Locating sex-specific evidence on clinical questions in MEDLINE: a search filter for use on OvidSP™

Research-based evidence is an important foundation for clinical decision making. In the past, women have often been underrepresented among participants in clinical research [1‐3]. Since the 1990s, however, health research funding organizations have taken initiatives to redress this bias, and researchers have begun to pay more attention to the equitable inclusion of men and women in clinical research and the analysis of the data according to sex [4‐11]. This has led to a new body of published research data on differences between and among men and women in the aetiology, diagnosis, treatment and prevention of diseases [12‐18]. These differences may be rooted in the biological or physiological "sex" characteristics that define men and women or in their socially constructed "gender" roles, behaviours or activities [3, 19]. For the sake of simplicity, in this paper we will use the term "sex-specific evidence" to refer to both sex and gender-related research-based data on differences between or among men and women.

One example of sex-specific evidence was reported by a study on malignant melanoma. The study found that men have a poorer life expectancy as compared to women, even when the melanoma is less thick [13]. Another example concerns the use of the AUDIT screening instrument for identifying alcohol problems. A systematic review revealed that the instrument was consistently less sensitive and more specific for women than for men when the recommended cut-off point of 8 was used. This led to the conclusion that for women a lower cut-off point (for instance 5) would be needed to identify these problems [18]. In order to provide optimal care to both men and women, it is essential that this new sex-specific evidence is easily accessible to anyone who is interested in evidence-based clinical practice (e.g., clinicians, guideline developers, health educators, researchers).

MEDLINE is one of the most widely consulted bibliographic databases for biomedical literature. In a previous project that investigated the uptake of sex-specific data in Dutch clinical guidelines [20], we discovered that finding sex-specific evidence in MEDLINE can be a difficult task. One problem is that searches for this type of evidence cannot be limited to a restricted number of journals, whereas sex-specific data are published in many different journals and not only in those focusing on gender medicine. MEDLINE offers a number of possibilities to facilitate searches for articles on sex-specific topics. Using the Check tags Male or Female is one possibility. However, these Check tags are assigned to an article if the concepts male or female are mentioned anywhere in the text. They will, therefore, retrieve all articles that mention either males (Check tag Male) or females (Check tag Female) and not only those providing information on differences among them. Another possibility is to use Medical Subject Headings, or MeSH, from the controlled vocabulary thesaurus. This thesaurus is composed by indexers of the National Library of Medicine (NLM) to describe the subject content of articles for MEDLINE. It contains several MeSH descriptors for sex-specific evidence (Table 1). Yet, it is not certain if these sex-specific MeSH capture all the relevant research reports. They may not be comprehensive enough, or they may be applied inconsistently by indexers. An example is the inconsistent indexing of studies evaluating sex as an effect modifier of the relationship under study [21, 22]. For these reasons, searching MEDLINE for articles that contain sex-specific evidence can be frustrating. On the one hand searches may yield many irrelevant research reports. On the other hand it is not certain that all relevant reports are included in the search results.

Search filters can be useful tools to facilitate detection of specific information in MEDLINE. As part of a follow-up project to improve attention to sex-specific factors in guideline development [23], two major Dutch guideline development organizations asked us to develop a search filter that could improve access to high quality evidence with respect to clinical questions, for both women and men. To date, little empirical work has been done in this area [24, 25]. One previously published filter contains some search terms for finding information on sex differences (see Table 2) [26]. We did not consider this filter suitable to our end, because it was developed to answer questions on women's health only. Moreover, the authors did not specify the criteria they had used for classifying the quality of the articles that had been identified by the filter. For that reason, we conducted a study that had the aim of developing and validating a search filter for locating relevant sex-specific evidence related to clinical questions in MEDLINE. This paper describes the results.

The filter was developed by researchers with ample experience in the retrieval of literature in biomedical databases and the development of search strategies.

Our goal was to develop a search filter that would facilitate the retrieval of high quality research describing sex-specific data on clinical questions related to conditions that can occur in both sexes. We excluded conditions that occur only in one sex, such as heavy menstrual bleeding, whereas literature on those conditions can be located simply by using disease-specific search terms (MeSH).

Jenkins has described a number of possible methodologies for filter development [27]. Given the absence of previous work on the selection of sex-specific search terms and the composition of a gold standard of studies for the evaluation and validation of those terms, we have followed Jenkins' recommendation to develop a so called "second generation filter". A characteristic of second generation filters is that search terms are subjectively derived and tested against a gold standard. A novelty of our approach was that we wanted to ensure that the papers that were selected for developing and validating the filter would also be considered as relevant by clinicians. To achieve this goal we composed two sets of published papers that had been critically reviewed for methodological soundness and clinical relevance by experts, prior to the study. The first set, that was used to develop the search filter, was selected from clinical guidelines that were developed by organizations that belong to the Appraisal of Guidelines for Research and Evaluation (AGREE) collaboration. These guideline organizations use the internationally developed AGREE instrument for testing the quality of the guidelines they produce [28]. Some of the AGREE quality criteria pertain to the selected evidence, which should be clearly evaluated for methodological soundness and clinical relevance. The second set of papers, that was used to validate the filter, was selected from a core set of leading clinical journals. The publication policy of these journals requires that published papers are carefully reviewed for level of clinical interest and methodological quality.

We found 662 articles reporting primary research on Alzheimer's disease and on asthma. One hundred and sixty-four of these papers were critically reviewed, as a first screening suggested that they might contain potentially relevant sex-specific information. This review found that 98 papers reported relevant sex-specific evidence on clinical questions. These 98 papers formed the reference set (see Table 5 for examples of sex-specific phrases found during the assessment). They were published in 24 different journals.

The SSS filter retrieved 125 of the 662 primary research papers (57 on Alzheimer's disease, 68 on asthma). Eighty-one of the retrieved papers contained relevant sex-specific information. Sixty-one of these papers could be identified by screening the title and abstract. For the other 20 articles the relevance became clear only after reading the body of the text. The recall of the filter was 83% and the precision 65% (Table 6). The recall for Alzheimer's disease was high (97%) but, as a trade-off, the precision was considerably lower (49%); for asthma these figures were respectively 77% and 78%.

A search using SS MeSH (the sex-specific MeSH as listed in Table 1 combined with the operator 'OR') captured 38 of the 662 primary research papers. The precision of this filter (79%) was high as compared to the SSS filter. Its recall, however, was low (31%) (Table 6).

The M&S filter was able to retrieve 118 papers with a recall of 74% and a precision of 62% (Table 6). The SSS filter had a slightly higher recall and precision. Upon closer inspection, the SSS filter was clearly more sensitive than the M&S filter for Alzheimer's disease, but for asthma the performance of both filters was in a comparable range. We did not execute a formal test to evaluate the differences in performance between the two filters.

In this project we developed a search filter to facilitate the detection of sex-specific research evidence relevant to clinical questions in MEDLINE. This filter has been developed for use on OvidSP™. It contains free text words for searching the title, abstract and MeSH of publications. Overall, the filter was able to locate 83% of all the publications reporting sex-specific data, with a precision rate of 65%. Thus, it was somewhat less successful in filtering out non-relevant information. This can be explained by the fact that there is always a trade-off between recall and precision: as a rule of thumb, high recall is accompanied by considerably lower precision and vice versa. A sensitive search filter is particularly useful for detecting information that is not readily available. As MEDLINE only contains a relatively small number of papers reporting sex-specific evidence on clinical questions, it is relevant to use a sensitive search filter, like ours, for detecting this information. Even though the filter also achieved a good precision rate, future studies may focus on potential ways to improve this.

We intended to identify search terms that were able to identify high quality sex-specific evidence on clinical questions. There is no gold standard for this type of evidence. We believe, however, that we have taken rigorous measures to make sure that the data sets that were used to develop and evaluate the filter contained high quality and relevant research evidence. To develop the filter we used research reports with sex-specific data that had been reviewed according to international standards for quality appraisal and considered relevant to clinical practice by guideline developers. To test the performance of the filter, we chose papers from a set of clinical journals that were selected by the National Library of Medicine on the basis of their quality and clinical relevance. Moreover, in the further selection of the reference studies we used rigorous and transparent quality criteria. In our opinion, the size of our gold standard (n = 98) was acceptable for establishing the filter's recall, given the fact that the minimum size of gold standards in filter development is a hundred [35].

A limitation of our study is the generalizability of the findings. The SSS filter was developed and tested on the basis of clinical studies with designs that are able to provide strong evidence for or against a causal effect, such as randomized controlled trials. It has been argued that such studies may not always provide sufficient information about the socio-demographic and sex-specific characteristics of the research population [37] and that relevant evidence may also be found in non-interventional designs such as observational or qualitative studies [38]. These study types were underrepresented in our validation set. It should also be noted that the selection process of the SSS filter's search terms included subjective choices. That is, the decision to search for a frequency of four and an adjacency of eight was based on previous experience with filter development. A different frequency (e.g. 3 or 5) or adjacency might have resulted in a different recall.

Further research is required in various directions. As a start, the filter should be tested against a gold standard that is larger in size than ours and on studies about other diseases. Furthermore, research is needed to investigate whether the SSS filter would perform differently in the retrieval of studies with non-experimental designs. Further work is also needed to create and validate a version of the SSS filter for other bibliographic databases and search interfaces.

Validation for the search interface PubMed would be a first important step, because it is widely used in the medical community. One obstacle is that some of the operators which are an important efficiency factor in the SSS filter, such as adjacency and frequency, are not available in the current PubMed interface (Jan 2009). It would be useful to investigate how the filter should be translated for use in PubMed.

Although the SSS filter was only validated against studies published in core clinical journals, it is probably also useful for searching other journals. This will also require further research. Two interesting sources of information that do not belong to the core set of clinical journals are the leading women's health journals Gender Medicine and the Journal of Women's Health (formerly also known as Journal of Women's Health and Gender-Based Medicine). We performed an extra search on Alzheimer's disease and on asthma in the 2007 and 2008 issues of these journals and found two publications reporting sex-specific evidence relevant to clinical practice [39, 40]. This suggests that it may be worthwhile to extend searches with the SSS filter to these leading women's health journals.

The SSS filter had a different aim than the previously developed filter by Montgomery and Sherif [26]. Yet, many of the search terms that were included in the two filters were similar. This may explain why their performance was rather similar.

Half of the papers of the development set and one-third of the papers of the reference set could have been identified through a search with sex-specific MeSH only. This indicates that it is not sufficient to rely on the index terms in MEDLINE (MeSH) to seek sex-specific research reports.

The SSS filter is able to identify articles that report sex-specific evidence by screening the title, the abstract and the MeSH that are assigned to them. It should be acknowledged, however, that many authors who report sex-specific data may not mention this in the title or the abstract of their article. For that reason the SSS filter may only be able to retrieve a subset of all available information on the topic. Indeed, this is a common restriction of all search filters.

We developed a search filter for the retrieval of high quality sex-specific clinical research data in MEDLINE. The recall is high and it has a good precision rate. Although the filter has been developed for guideline organizations, it has also potential relevance to a much wider spectrum of users of clinical evidence, e.g. researchers, medical educators and practitioners. The utility of the filter may vary for different disease categories, journals or research designs. This underlines the need for further evaluation.