Handsearching still a valuable element of the systematic review

Abstract

Data sources

The Cochrane Methodology Register, Medline, Embase, AMED, Biosis, Cinahl, LISA, and Psycinfo were consulted along with researchers who may have carried out relevant studies.

Study selection

Studies were considered eligible if they compared searching by hand with searching one or more electronic databases to identify reports of randomised trials.

Data extraction and synthesis

the main outcome measure was the number of reports of randomised trials identified from searches made by hand compared with electronic searching. Data were extracted regarding the electronic database searched, the complexity of electronic search strategy used, the characteristics of the journal reports identified, and the type of trial report identified.

Results

Thirty-four studies were included. Handsearching identified between 92 and 100% of the total number of reports of randomised trials found in the various comparisons in this review. Searching Medline retrieved 55%, Embase 49% and Pyscinfo 67%. The retrieval rate of the electronic database varied depending on the complexity of the search. The Cochrane highly sensitive search strategy (HSSS) identified 80% of the total number of reports of randomised trials found; searches categorised as ‘complex’ (including the Cochrane HSSS) found 65% and ‘simple‘ searches found 42%. The retrieval rate for an electronic search was higher when the search was restricted to English-language journals, at 62% versus 39% for journals published in languages other than English. When the search was restricted to full reports of randomised trials, the retrieval rate for an electronic search improved: a complex search strategy retrieved 82% of the total number of such reports of randomised trials.

Conclusions

Hand searching is still valuable in identifying randomised trials for inclusion in systematic reviews of healthcare, particularly trials reported as abstracts or letters, those published in languages other than English, along with all reports published in journals not indexed in electronic databases. Where time and resources are limited, however, searching an electronic database using a complex search (or the Cochrane HSSS) will identify the majority of trials published as full reports in English language journals, provided, of course, that the relevant journals have been indexed in the database.

Commentary

Conducting a systematic review is a time-consuming business and the validity of the findings depends upon the nature of the underlying data and their interpretation. It therefore follows that identifying as unbiased and complete a set of relevant studies is a key step in the process. This methodological review was conducted to determine whether electronic approaches are as effective as searching by hand for identifying studies. A wide selection of databases was used as well as handsearching of journals to identify relevant studies: the number of trials identified was the main outcome measure.

in the 34 identified studies, the quality of the handsearching could only be judged appropriate in 50% of them because of the limited information supplied, whereas 29 out of the 34 electronic searches were considered appropriate. Although searching by hand produced higher yields (92–100%), retrieval rates from electronic databases varied according to the complexity of the search. The more complex searches, such as those employed by the Cochrane Collaboration, yielded 80% of trials. The electronic database results will of course vary with the topic under investigation and whether the relevant journals have been indexed in the databases searched. In addition, the literature included was related to the medical specialities: the effectiveness of electronic searches for dental topics may be different.

Although complex electronic searches using a range of databases may identify the majority of trials, limiting systematic review searches to just Medline with no handsearching (as is often the case in dental systemic reviews) is a problem. Glenny et al.¹ identified 65 reviews with only 12 attempting to identify all the relevant studies: 97% used Medline, 26% Embase, and 15% the Cochrane clinical trials database. As the authors of this review suggest, where time and resources are limited, searching electronic databases using a complex search (or the Cochrane HSSS) will identify the majority of trials, but the scale of the shortfall when only Medline is used with no handsearching has been illustrated by Bickley and Harrison.² in their analysis of the problem in the orthodontic literature, only 143 out of 304 trials were identified by using Medline alone.

The improvements in indexing trials on databases, the work of the Cochrane collaboration in identifying trials and retrospective re-indexing will continue to improve the hit-rate from electronic searches in the future. Searching journals by hand will continue to be a necessary element of the most thorough systematic reviews for the foreseeable future, however. In the meantime, the Cochrane Central Register of Controlled Trials (Central) is still likely to be the best single source of controlled trials and should be the first source searched by those carrying out systematic reviews of healthcare interventions.