Ranked (In)direct Citation Searching in Systematic Reviews: A methodological case study
Discuss this preprint
Start a discussion What are Sciety discussions?Listed in
This article is not in any list yet, why not save it to one of your lists.Abstract
Systematic Review (SR) is a prosperous study type in modern medicine and beyond. Many SR authors complement their primary database searches by supplementary techniques. Among these, citation-based techniques known as citation searching (CS) are widespread. Unranked Direct CS (UDCS) to identify directly cited and citing literature of seed references is currently most prevalent. Ranked (In)direct CS (RICS) additionally collects co-cited and co-citing literature combined with a ranking and cut-off procedure. However, RICS workflows remain non-standardized and tedious, and associated benefits unclear. This work aims to create a framework for the prospective international comparison of supplementary UDCS and RICS.
To prime RICS research, we developed the open-source Co*Citation Network application and assessed parallel supplementary UDCS and RICS retrospectively in three completed SRs and prospectively in one case study. Automated RICS collected and ranked cited, citing, co-cited, and co-citing literature of seed references from OpenAlex database and applied an empirical rank cut-off to approximate the volume of UDCS results. In RICS compared to UDCS, we consistently noted higher overlap with primary database search results. Title/abstract screening in the case study showed a precision (number needed to read) of 1.8% (57) for UDCS and 2.1% (48) for RICS results. After full text screening, two additional articles were included for review, one of which was identified by UDCS and RICS, and one exclusively by UDCS.
The present study indicates potential benefits of RICS for SR authors and will enable the formation of a research consortium to compare supplementary UDCS and RICS on larger scale.
Highlights
What is already known?
-
Citation searching is a widespread supplementary search technique in systematic reviews to complement the results of the primary literature search in bibliographic databases.
-
Compared to traditional citation searching (UDCS), Ranked (In)direct Citation Searching (RICS) which additionally incorporates co-cited and co-citing literature with a relevance ranking and cut-off procedure has been proposed as a potentially more effective approach, but has only been evaluated in proof-of-concept studies.
What is new?
-
The open-source Co*Citation Network application is introduced: the first tool to automate parallel UDCS and RICS on the basis of the OpenAlex citation index.
-
For the first time, parallel supplementary UDCS and RICS were applied and compared within a prospective systematic review, providing a concrete, replicable workflow.
-
The results indicate that RICS results may be more relevant than UDCS results.
Potential impact for RSM readers
-
The proposed supplementary UDCS and RICS workflow and ready-to-use collaboration infrastructure allow systematic review teams worldwide to contribute data to a large-scale comparative study.
