Semi-automating the manual literature search for systematic reviews increases efficiency

DOI: 10.1111/j.1471-1842.2009.00865.x Semi-automating the manual literature search for systematic reviews increases efficiency Andrea L. Chapman*, Laura C. Morgan & Gerald Gartlehner* *Department for Evidence-based Medicine and Clinical Epidemiology, Danube University Krems, Krems, Austria and Cecil G. Sheps Center for Health Services Research, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA Abstract Background: To minimise retrieval bias, manual literature searches are a key part of the search process of any systematic review. Considering the need to have accurate information, valid results of the manual literature search are essential to ensure scientific standards; likewise efficient approaches that minimise the amount of personnel time required to conduct a manual literature search are of great interest. Objective: The objective of this project was to determine the validity and efficiency of a new manual search method that utilises the SCOPUSÔ database. Methods: We used the traditional manual search approach as the gold standard to determine the validity and efficiency of the proposed SCOPUS method. Outcome measures included completeness of article detection and personnel time involved. Using both methods independently, we compared the results based on accuracy of the results, validity and time spent conducting the search, efficiency. Results: Regarding accuracy, the SCOPUS method identified the same studies as the traditional approach indicating its validity. In terms of efficiency, using SCOPUS led to a time saving of 62.5% compared with the traditional approach (3 h versus 8 h). Conclusions: The SCOPUS method can significantly improve the efficiency of manual searches and thus of systematic reviews. Key Messages Implications for Practice d The SCOPUS method can substantially reduce personnel time and thus the cost associated with conducting a systematic review. d Our findings are based on one medium-sized systematic review. Further method studies are needed to validate these findings and identify potential improvement to this approach. Implications for Policy d d High costs of trained personnel necessitate valid and efficient methods to conduct systematic literature searches. The SCOPUS method of conducting manual literature searches is as valid but more efficient than the traditional method. Background A systematic review is a summary of the medical literature that uses explicit methods to perform a Correspondence: Andrea Chapman, Department for Evidence-based Medicine and Clinical Epidemiology, Danube University Krems, Dr Karl-Dorrek-Straße 30, 3500 Krems, Austria. E-mail: andrea. chapman@donau-uni.ac.at thorough literature search and critical appraisal of individual studies and that uses appropriate statistical techniques to combine these valid studies. 1 A crucial part of any systematic review is the systematic literature search, a thorough, objective and reproducible search of a range of sources to identify as many relevant studies as possible. 2 Additionally, 22

Manual literature searches for systematic reviews, Andrea L. Chapman et al. 23 the systematic literature search is a major factor in distinguishing systematic reviews from traditional narrative reviews and helps to minimize bias. 2 The systematic literature search is often conducted in various sources and in several phases. In the initial phase, referred to as the primary literature search, systematic literature searches utilise bibliographic databases, such as MEDLINE, EMBASE, and The Cochrane Library. The result of the primary search is based on a search strategy that uses Boolean logic to combines key words and, when available, controlled vocabulary, such as MeSH Terms (Medical Subject Headings) in PUBMED. The basic procedure of the primary search is to develop a search strategy and apply this strategy when searching the various online databases, adjusting the search strategy to allow for syntactical and functional differences in the different online databases. Bibliographic software can be used to create a project database that contains all citations. The search results of the different searches can be downloaded directly into the project database. This method is practiced by institutions that conduct systematic reviews 3 and the basis of this methodological study. An additional phase of the search process is the manual literature search, also termed hand-searching or the hand search. The objective of the manual literature search was to identify studies that were missed by the primary search. Rather than beginning with key words or terms and developing a search strategy, the manual search begins with key sources: specific journals, conference proceedings or key articles. In contrast to the primary literature search, there seems to be no standard practice as to conducting the manual literature search. In the Cochrane Handbook, hand-searching is specified as a manual page-by-page examination of the entire contents of a journal issue or conference proceedings to identify all eligible reports of trials. 4 The Cochrane Handbook notes that reference lists of identified studies may be an additional source of studies. 5 This method is often referred to as snowballing, or pursuing references of references. 6 Our approach to the manual search, similar to snowballing, begins by identifying several key articles from the results of the final search, i.e. the results of the primary search and any other sources that were searched such as conference abstracts, evidence-based guidelines or dissertations. Key articles are relevant, high-quality studies or background articles. Each reference in the reference list of these key articles must be reviewed independently by two investigators. For titles that appear to be relevant for the topic of the systematic review, the abstracts of each reference must be reviewed, as in the final step of the primary search, in a dual-review process. (In the dual-review process two personnel read the abstracts independently of one another and decide if the citations that are likely to include studies or background information relevant to the systematic review.) Thus, the results of the manual search phase must also be assessed according to the inclusion and exclusion criteria that were defined at the inception of the systematic review. Methods research indicates that manual searches substantially increase the sensitivity of literature searches. 3,7 Sensitivity is the number of relevant reports identified divided by the total number of relevant reports in existence. 8 Additionally, manual searches are conducted in order to further minimise retrieval bias. We define retrieval bias as the failure to find relevant literature due to the retrieval capacity of a database. This could result from the technical design of the database or the classification of the literature by means of meta-data, such as deficient indexing of an article or inadequate key words. Retrieval bias differs from publication bias, which results from the selective publication of literature. 9 Consequently, manual searches are considered a vital part of any comprehensive systematic literature search and are explicitly recommended in methods guides for systematic reviews. 10 Nevertheless, manual searches require a considerable amount of personnel time of highly trained researchers who must review hundreds of references to determine the relevance of each citation. In our experience, only a fraction (5 10%) of the reviewed references usually proves to be new and not previously detected by electronic literature searches. As personnel time is the most costly factor in systematic reviews, this gain in sensitivity comes at a high cost. Consequently, finding more efficient approaches to conduct manual searches is an important and relevant issue for systematic reviewers. The usability of electronic databases of scientific literature has greatly improved over the past dec-

24 Manual literature searches for systematic reviews, Andrea L. Chapman et al. ades. SCOPUS, a relatively new literature database, was launched by Elsevier in 2004. SCOPUS is currently the largest database of citations and abstracts; it contains references from both research literature and web sources (http://www.info.scopus.com/detail/what/). To date, SCOPUS covers more than 15 000 peer-reviewed titles from more than 4000 international publishers, including 100% coverage of MEDLINE titles. A unique feature of SCOPUS is that it enables researchers to export entire reference lists of articles into bibliographic databases. On the basis of this tool, we are proposing a new, more efficient approach to conduct manual searches of literature during systematic reviews. Our study had two objectives. The first objective was to assess the yield of the manual search and the effort required to conduct the manual search. In this case we used the SCOPUS database to electronically gather relevant references from articles that had been selected for manual searches. We will refer to this method as the SCOPUS approach. The second objective was to assess these same factors in comparing the SCOPUS approach with a more conventional manual search approach, which we denote the traditional approach. Given that the traditional approach can be viewed as the current gold standard, the overarching objective of our study was to determine the validity and efficiency of the proposed SCOPUS approach. It is only possible to search SCOPUS using free text as SCOPUS does not have an indexing system such as those used in MEDLINE, EMBASE or The Cochrane Library. Consequently, SCOPUS is not useful for primary literature searches. Therefore, we have tested this method in conducting the manual literature search. Methods We tested the two approaches by drawing from a systematic review that compared efficacy and safety of constipation drugs used to treat chronic constipation and irritable bowel syndrome with predominant constipation. 11 This systematic review was funded by the Drug Effectiveness Review Project, a consortium that currently includes 14 states and one Canadian non-profit organisation, whose aim is to establish priorities for evaluation of drug classes through systematic reviews. 12 As this was a methodological project, we first discuss the design of the two manual searches of literature. We then describe how we evaluated the SCOPUS approach and the traditional approach. Designs of the manual searches The traditional approach used basic manual search strategies that the RTI-UNC EPC (Research Triangle Institute University of North Carolina Evidence-based Practice Center) follows in performing systematic reviews. To begin, authors typically select 15 20 landmark studies, systematic reviews or recent background articles that provided relevant information on the topic. Two researchers then independently review the reference list of each of these articles. They highlight references that appear to be relevant based on the title of the article. In the next step, supporting staff check the bibliographic project database to determine whether each selected reference had been detected during electronic literature searches. New citations are retrieved as abstracts or full text articles. Using this method, we selected 20 articles and followed the aforementioned procedure. In this project we used ENDNOTE Ò X as the bibliographic database. The SCOPUS approach began with the same 20 studies that had been selected as relevant resources for manual search in the traditional approach. The database manager then searched the SCOPUS database for each article and downloaded the reference lists into a bibliographic database (again ENDNOTE Ò X), denoted as the manual search database. In order to track these citations we tagged them in a custom field. To eliminate references that had already been detected by electronic literature searches, we imported the manual search database into a copy of the existing project database that contained all citations identified through electronic searches. In the final step, we eliminated duplicate citations. The remaining references from the manual search database could then be exported and dually reviewed by two researchers as described in the traditional approach. Figures 1 and 2 depict the two approaches graphically. Evaluation of the two approaches Two groups of researchers independently conducted either the SCOPUS approach or the traditional

Manual literature searches for systematic reviews, Andrea L. Chapman et al. 25 Select articles for manual search. Carry out dual review of reference list of each article for references of interest. Compare highlighted citations with the project database to identify new references. Import or manually enter new citations into the project database. Carry out dual review of new citations for references of interest. Figure 1 Traditional method of manual search Select articles for manual search. Search scopus for selected articles and download reference list of each into a manual search database. Remove duplicates in the manual search database. Merge manual search database with project database, discard duplicates, and print or export new references. Carry out dual review of new citations for references of interest. Figure 2 SCOPUS: approach to manual search approach. We applied two outcome measures to compare the two manual searches: completeness of article identification to measure validity and personnel time to measure effort. The traditional approach was considered to be the gold standard for evaluating the validity of the SCOPUS approach. Results Validity Both methods detected 89 references that met eligibility criteria of the systematic review and had been missed by electronic literature searches. Six references of interest were identified using the SCOPUS method but had been overlooked using the traditional method. However, none of these references ultimately met eligibility criteria of the systematic review. Effort The selected articles contained 817 references, all of which had to be reviewed during the traditional approach. The total time investment for the traditional method was about 8 h. Dual review of reference lists to identify citations of interest took a total of 3 h and 35 min. We invested an additional 2 h and 10 min to go through the highlighted citations from each reference list and determined which were already included in the database for the project. Finally, it took about 2 h to import or manually enter the highlighted references into the database. The total time investment for using the SCOPUS method was approximately 3 h. We spent 1 h and 35 min searching SCOPUS and downloading reference lists into an ENDNOTE Ò database. Overlapping references within reference lists were eliminated as each manual search reference list was imported to the database. Three hundred and sixty-nine references had to be reviewed by researchers during the SCOPUS approach. The dual review of this list of new references took 1 h. Overall, the SCOPUS approach eliminated 448 citations with a time savings of 62.5%. Discussion The SCOPUS approach utilises several features of bibliographic software to administer the results of the searches. In order to ensure that duplicates are not reviewed, bibliographic software can be used to remove citations that have already been reviewed during the abstract from the primary search. Additionally, bibliographic software allows for easy printing or exporting of the abstracts. By coding the

26 Manual literature searches for systematic reviews, Andrea L. Chapman et al. results of the manual search, it is possible to limit the print out or export file to new references only. Not only do these advantages make it highly desirable to link the manual search to a bibliographic database, using bibliographic software is a condition to the efficiency of the SCOPUS approach. This functionality is unique to SCOPUS: although other online tools also link articles to references, they do not allow exportation of the reference list to a bibliographic software program. One example is the Cited by feature in Google Scholar, which displays the references that are cited by the article in question. Another online tool that offers more developed features is Thomson Reuters Current Contents Connect Web of Science, which displays the references cited in the article in question. The references can then be linked to related records, which list common references of the original and secondary reference. Although it is possible to export results from Web of Science into ENDNOTE Ò, it is not possible to export the reference lists of articles. Results of these methods study indicate that our proposed approach to conduct manual searches for systematic reviews is as valid as the traditional approach but more efficient. Overall, without missing any studies, the innovative use of the SCOPUS database and a bibliographic program led to a time savings of 62.5%. As we used the new method for the first time, it is conceivable that with more experience, the time savings will even be greater. The new approach has multiple advantages. First, the SCOPUS method allows researchers to review one consolidated list of references. Reviewers do not have to consider already reviewed references from the primary search. The reference lists of all the articles selected for the manual search are consolidated in bibliographic software at the time of retrieving the reference lists. References that are found in more than one reference list can be removed electronically from the database. Thus, reviewers do not even see these duplicate references, whereas in the traditional approach, the reviewers potentially saw a reference again and again in the different reference list. Second, for new references that are deemed to be of interest, abstracts must be obtained for the first level of review. This process is streamlined using SCOPUS. The traditional method requires a reviewer to look up each citation of interest to find the abstract (generally on PUBMED or the INTERNA- TIONAL PHARMACEUTICAL ABSTRACTS database). In SCOPUS there are links to the abstracts of each reference in the list. Therefore, the reviewer must search only for the manual search records rather than identifying abstracts in other databases. This will lead to an additional timesaving that we had not considered in our study. Third, references that have been published as abstracts only (from conferences, etc.) are indexed in the reference list in SCOPUS. These citations do not have independent records in SCOPUS, but do have reference information that can be exported into a bibliographic program. Such references are currently not indexed in PUBMED. This requires personnel to spend additional time searching for them or manually entering them into the database. The time savings here are also significant and not included in our results. Our study also highlighted some limitations of the proposed method. SCOPUS requires a paid subscription while PUBMED is free of charge. Rates for access to SCOPUS vary based on the size of the institution acquiring the subscription but can be substantial. Additionally, SCOPUS currently does not include the reference lists of Cochrane Review. In our test, described in detail above, none of the key articles were Cochrane Reviews. Had the articles selected as key articles included a Cochrane Review, it would not have been possible to use SCOPUS to download the reference list. The reference list of Cochrane Reviews identified as key articles must be evaluated using the traditional method and thus increasing the time required. As Cochrane Reviews are often crucial sources of information, this is a major limitation of SCOPUS. Expanding SCOPUS to include the ability to download a reference list of Cochrane Reviews could further increase the efficiency of the new manual search approach. Disclosure statement This project received no financial support and the authors declare no conflict of interest. References 1 Bandolier. Glossary of diagnostic terms. Bandolier Office, April 2004. Accessible at: http://www.medicine.ox.ac.uk/

Manual literature searches for systematic reviews, Andrea L. Chapman et al. 27 bandolier/booth/diagnos/glossary.html (accessed 24 May 2009). 2 Higgins, J. P. T. & Green, S. (eds). Cochrane Handbook for Systematic Reviews of Interventions 5.0.1 [updated Section 6.1.1.2 Minimizing Bias. Accessible at: http:// www.cochrane-handbook.org. 3 Savoie, I., Helmer, D., Green, C. J. & Kazanjian, A. Beyond MEDLINE: reducing bias through extended systematic review search. International Journal of Technology Assessment in Health Care 2003, 19, 168 178. 4 Higgins, J. P. T. & Green, S. (eds). Cochrane Handbook for Systematic Reviews of Interventions 5.0.1 [updated Section 6.2.2.1 Handsearching. Accessible at: http://www. cochrane-handbook.org. 5 Higgins, J. P. T. & Green, S. (eds). Cochrane Handbook for Systematic Reviews of Interventions 5.0.1 [updated Section 6.2.2.5. Other reviews, guidelines and reference lists as sources of studies. Accessible at: http://www. cochrane-handbook.org. 6 Greenhalgh, T. & Peacock, R. Effectiveness and efficiency of search methods in systematic reviews of complex evidence: audit of primary sources. BMJ 2005, 331, 1064 1065. 7 Hopewell, S., Clarke, M., Lusher, A., Lefebvre, C. & Westby, M. A comparison of handsearching versus MEDLINE searching to identify reports of randomized controlled trials. Statistics in Medicine 2002, 21, 1625 1634. 8 Higgins, J. P. T. & Green, S. editors. Cochrane Handbook for Systematic Reviews of Interventions 5.0.1 [updated section 6.4.4 Sensitivity versus precision. Accessible at: http://www.cochrane-handbook.org. 9 Montori, V. M., Smieja, M. & Guyatt, G. H. Publication bias: a brief review for clinicians. Mayo Clinic Proceedings 2000, 75, 1284 1288. 10 AHRQ. US Preventive Services Task Force Procedure Manual. 2008 (Publication No. 08-05118-EF). 11 Gartlehner, G., Jonas, D. E., Morgan, L. C., Ringel, Y., Hansen, R. A., Bryant, C. M. & Carey, T. Drug Class Review on Constipation Drugs 2007. Accessible at: http:// www.ohsu.edu/ohsuedu/research/policycenter/derp/about/ final-products.cfm (accessed 7 July 2009). 12 Fox, D. M. Evidence of evidence-based health policy: the politics of systematic reviews in coverage decisions. Health Aff (Millwood) 2005, 24, 114 122. Received 18 February 2009; Accepted 20 August 2009