An Interactive Relevance Feedback Interface for Evidence-Based Health Care

Abstract

We design, implement and evaluate EpistAid, an interactive relevance feedback system to support physicians towards a more efficient citation screening process for Evidence Based Health Care (EBHC). The system combines a relevance feedback algorithm with an interactive interface inspired by Tinder-like swipe interaction. To evaluate its efficiency and effectiveness in the citation screening process we conducted a user study with real users (senior medicine students) using a large EBHC dataset (Epistemonikos), with around 400,000 documents. We compared two relevance feedback algorithms, Rocchio and BM25-based. The combination of Rocchio relevance feedback with the document visualization yielded the best recall and F-1 scores, which are the most important metrics for EBHC document screening. In terms of cognitive demand and effort, BM25 relevance feedback without visualization was perceived as needing more physical and cognitive effort. EpistAid has the potential of improving the process for answering clinical questions by reducing the time needed to classify documents, as well as promoting user interaction. Our results can inform the development of intelligent user interfaces for screening research articles in the clinical domain and beyond.