The Systematic Review Data Repository (SRDR): descriptive characteristics of publicly available data and opportunities for research

On November 12, 2019, we ran custom MySQL queries on the SRDR database to gather descriptive statistics regarding SRDR users, projects, and website visits. A “user” is defined as a unique SRDR username and email address. A “project” is defined as a collaborative or individual enterprise among users within SRDR; a project may or may not be a systematic review. A user may have either a “public commentator account” or a “project contributor account.” The former type of account allows users only the ability to make comments on existing projects in SRDR. The latter type of account allows users to also create new projects and contribute modifications to existing SRDR projects in which they are participating. A “session” is defined as a unique visit from a user to the SRDR website; during a session, a user may open multiple projects and/or multiple pages within a project in SRDR. A “page” in SRDR is a webpage within the SRDR system.

For this analysis, we examined all projects with data made available publicly through SRDR as of November 12, 2019. We extracted the following information pertaining to each project:

Regarding registration in PROSPERO, if registration status was not provided on the “Published Projects” page of the SRDR website, we examined any available full report of the systematic review. We obtained full reports of AHRQ-funded systematic reviews from the AHRQ Website (https://​www.​ahrq.​gov/​research/​findings/​evidence-based-reports/​index.​html), Cochrane systematic reviews from the Cochrane Library (https://​www.​cochranelibrary.​com), and other systematic reviews from journal articles. If PROSPERO registration status was not provided in the full report, we searched the PROSPERO registry (https://​www.​crd.​york.​ac.​uk/​prospero/​#searchadvanced) using keywords from the title of the project. If these efforts did not yield a PROSPERO record for the systematic review, we considered it to be not registered in PROSPERO. Of note, PROSPERO does not allow registration of scoping reviews. Therefore, we did not examine PROSPERO registration status for technical briefs, evidence maps, and methods research.

We used SRDR to develop a standardized data extraction form that included 22 items (form available at https://​bit.​ly/​2VhR7DU; and data from this project are available publicly at https://​srdr.​ahrq.​gov/​projects/​published). Two investigators extracted and verified data (BTS and IJS). For each project, one investigator extracted all information and a second verified the data. All discrepancies were resolved through discussion.

Of note, when more than one project pertained to the same systematic review, we counted them as separate projects. Also, for each project, we extracted all information from SRDR (except for PROSPERO registration); we did not examine any available systematic review protocols, associated journal articles, or other external sources of information. As such, the information presented in this paper reflects the information in SRDR and not necessarily what might be reported elsewhere for a given systematic review.

We analyzed frequencies and percentages for categorical items and medians and interquartile ranges (IQRs) for continuous items. We compared categorical items descriptively. We performed Kruskal-Wallis tests for hypothesis testing of differences between medians of continuous items. We performed all analyses using Stata® version 16 (College Station, Texas, USA).

As of November 12, 2019, SRDR has had 2552 individual user accounts belonging to users from 80 countries. These include 1407 project contributor accounts from 53 countries; the remaining accounts are public commentator accounts. Countries with the most project contributor accounts are the USA, Canada, the UK, and Greece. From January 1 to November 12, 2019, the SRDR website had an average of 84 sessions per day and 70 visitors per day; each visitor visited approximately 4.6 pages per session. All told users have created 735 projects in SRDR, with a median of three collaborators per project (range 1 to 54). The total number of studies across all these 735 projects is 90,142.

We found that data are available publicly for 152 of the 735 projects in SRDR (21%). Considering full years since the 2012 launch of SRDR (i.e., 2013 to 2018), data from 25.3 projects per year, on average, have been made available publicly through SRDR (Fig. 1). The median time from project initiation in SRDR to public availability of the data through SRDR was 4 months (IQR 1 to 14). Note that this does not include the time spent on the systematic review steps before data extraction (e.g., question formulation, abstract screening) that have traditionally occurred outside SRDR.

The 152 published projects pertain to 148 distinct systematic reviews. One review was associated with four projects and another review was associated with two projects; in both instances, the multiple projects pertained to separate research questions of the same review. Three reviews each included one or more updated reviews, corresponding to seven SRDR projects. Because each update involved non-trivial changes to the research questions, we considered these seven projects as seven separate systematic reviews. In this paper, we consider projects as the unit of analysis. All told, the 152 projects include data from 15,621 studies (not accounting for overlap of some studies between projects).

Most of the published projects are in clinical fields (144/152 projects; 95%); the remainder are in public health (6/152 projects; 4%) or toxicology (2/152 projects; 1%) (Table 1). The primary focus of most projects is either interventions (therapeutic or preventive) (109/152 projects; 72%) or diagnosis (15/152 projects; 10%). The projects address a range of health areas. The most frequent health area addressed is mental and behavioral disorders (31/152 projects; 20%); the next most frequent is diseases of the eye and ocular adnexa (23/152 projects; 15%). For comparison, the most frequent health areas addressed in the 588 projects without publicly available data are diseases of the eye and ocular adnexa (85/588 projects; 15%); endocrine, nutritional, and metabolic diseases (50/588 projects; 9%); and neoplasms (37/588 projects; 6%). Appendix lists the specific topics addressed in all 152 projects with publicly available data, by health area.

Most projects are full systematic reviews (132/152 projects; 87%). Fewer than half of these (48/117 projects; 44%) were registered in PROSPERO. Among all 152 projects, most (91%) were funded by government sources and 95% have a US-based corresponding author. The data in almost two-thirds of all projects (94/152 projects; 62%) are structured, while the remainder are in flat files (58/152 projects; 38%).

About two-thirds of the projects (104/152; 68%) were funded by AHRQ. There are some notable differences between projects funded by AHRQ and those that were not (Table 1). Approximately one in five non-AHRQ-funded projects (21%) focus on exposure-outcome relationships, while none of the AHRQ-funded projects do. Almost 3 in 10 AHRQ-funded projects (29%) focus on mental and behavioral disorders, while only 2% of the non-AHRQ-funded projects do. Instead, almost half of the non-AHRQ-funded projects (46%) focus on diseases of the eye and ocular adnexa, largely due to the fact that Cochrane Eyes and Vision, based in London, UK, with a Satellite in Baltimore, Maryland, USA, has been encouraging its systematic review authors to use SRDR for data extraction. While all AHRQ-funded projects were government-funded, 3 in 10 non-AHRQ-funded projects (30%) were funded by non-government sources. Almost half of the non-AHRQ-funded projects (23/48; 48%) are Cochrane systematic reviews.

Overall, the 152 projects address a median of three research questions each (IQR 1–5) (Table 2). AHRQ-funded projects address a considerably higher number of research questions than non-AHRQ-funded projects (median 4 vs. 1; P = 0.0001). Overall, the 152 projects include a median of 70 studies each (IQR 20–130). AHRQ-funded projects include more than six times as many studies as non-AHRQ-funded projects (median 84 vs. 13.5; P = 0.0001).

There are some interesting differences between the findings of this investigation and others that have examined the characteristics of systematic reviews in health. For example, others have found that about half of systematic reviews have focused on interventions and that about 45% have been funded by government sources [11, 12]. In SRDR, 72% of publicly available reviews focused on interventions and 91% were funded by government sources. These higher proportions in SRDR are largely driven by the large proportion of reviews in it that have been funded by AHRQ (68%) or are Cochrane systematic reviews (15%). AHRQ-funded and Cochrane systematic reviews tend to focus more on interventions than do other systematic reviews. Similarly, another investigation found that the median number of included studies per systematic review was 15 [11], whereas the median number of studies per review in SRDR is 70. This difference is also largely driven by the predominance of AHRQ-funded reviews, which tend to be broader in scope and address more research questions than non-AHRQ-funded systematic reviews. Indeed, in SRDR, the median number of research questions per AHRQ-funded review is approximately eight times that of non-AHRQ-funded reviews.

Our finding that 44% of eligible systematic reviews were registered in PROSPERO is a considerable improvement from the 4% that was reported in a random sample of 300 systematic reviews published in 2014 [11]. However, since AHRQ and Cochrane strongly encourage PROSPERO registration, we would expect reviews in SRDR to be more likely to have registered protocols than other systematic reviews. While the higher percentage is a good sign, we urge all systematic review teams to register their systematic reviews prospectively in PROSPERO. Prospective registration offers many benefits, such as promoting transparency, reducing the potential for bias, and reducing the potential for redundancy [13, 14]. Additionally, in light of the growing numbers of abridged types of evidence syntheses, such as evidence maps/scoping reviews [15], living systematic reviews [16], and rapid reviews/technical briefs [17], we agree with Page et al. [12] that the PROSPERO registry should expand its eligibility criteria to include these other types of reviews.

To our knowledge, SRDR is one of a kind. It serves as a free, online, data management platform for collaboration among members of a systematic review team [8]. SRDR also offers free, open access to data about primary studies that have been extracted for systematic and other reviews on a range of topics. In this way, SRDR helps advance the open-access movement in science. We agree, however, with current guidance that those re-using shared data should cite the original data source (i.e., the systematic review and the SRDR platform from which the data were obtained) [5, 18]. To facilitate such citing, we provide each publicly available project in SRDR with an associated digital object identifier (DOI) for easy and persistent online identification.

The past decade has witnessed an almost threefold surge in the number of systematic reviews [11]. While some of these systematic reviews have been demonstrated to be redundant [19], conducting new systematic reviews on topics related to existing systematic reviews is often required. For example, an update to an existing systematic review may be needed if it is out of date and/or new studies are available. Other common scenarios that lead to a new systematic review being needed are when the eligibility criteria of an existing related systematic review were too narrow, a new type of intervention or comparator has emerged, or a broader search is needed [20]. In each of these and other related scenarios, a considerable amount of time and resources can be saved by using previously extracted data, where relevant. SRDR can help fulfill this need [8].

Efforts have been underway to begin strides towards a future in which data extraction for systematic reviews might be accurately and efficiently conducted using automation technologies [21‐23]. In that context, while archival of data would still serve the purpose of transparency, re-use of data might not be of much added value because technology would be able to conduct data extraction inexpensively. While such a “revolution in automation of systematic reviews” is on the horizon, we are not there yet [24]. Until we arrive at such a future, one in which the systematic review and broader research communities are comfortable with the accuracy of automated data extraction, re-use of data extracted by humans can help reduce redundancy and costs.

There appears to be support for sharing systematic review data. A 2014 survey found that 83% of systematic reviewers affiliated with the Cochrane Individual Participant Data (IPD) Review Group supported it [25]. However, we acknowledge that there might be barriers to completely relying on previously extracted data, especially when the previous systematic review team is different from the team undertaking the new systematic review. There may be concerns about whether the set of items extracted from studies in the previous review is adequate and whether the data extracted are accurate. While SRDR does not guarantee the adequacy and accuracy of extracted data, these aspects can be assessed by systematic review teams, such as through examination of data from a random sample of the studies. Alternatively, systematic review teams may choose to use the previously extracted data as the initial extraction in an approach similar to single (de novo) data extraction and verification.

SRDR can serve as a valuable platform for conducting methodologic research. Examples of such research that has already been conducted using SRDR are the Data Abstraction Assistant (DAA) Trial (a randomized controlled trial that compared different data extraction approaches [26‐28]) and the current study and the six other methodologic projects described in this paper [29‐34]. SRDR also can serve as a source of data for meta-research (i.e., methodologic and other types of research on research [4]). For example, researchers might analyze the populations, interventions, comparators, outcomes, funding sources, and/or other factors across systematic reviews, either within or across health areas.

Accessing and downloading publicly available data from SRDR is relatively straightforward through the SRDR Published Projects Website (available at https://​srdr.​ahrq.​gov/​projects/​published). As the SRDR management team, we are happy to help and/or partner with researchers to do so (see author contact information).

While the SRDR management team encourages and helps systematic review teams to make their data public and manages the website, we do not monitor the accuracy or completeness of the data. Inaccuracies in publicly available data in SRDR occur due to errors in data extraction from reports of primary studies [35]. It is not uncommon for such errors to be corrected in the final versions of reported data (e.g., journal publications), but not in SRDR. Because SRDR is not yet a data analysis platform, systematic review teams might not be vigilant about retrospectively updating data in SRDR to fix errors that might have been detected during data cleaning or analysis after exporting the data outside SRDR to statistical software or other applications. Second, SRDR is an evolving platform that is striving to improve how best to archive systematic review data for easy re-use. For example, while SRDR offers structured data extraction forms and structured data entry, these features are not always fully taken advantage of; more than one-third of the projects in SRDR have simply uploaded data as flat files. We recently developed an improved mechanism through which systematic reviews can import data from flat files into forms in SRDR so that the data can be shared in a structured format [36]. Third, it should be acknowledged that data have been made available for only a quarter of all projects in SRDR. While we, as the SRDR management team, do not require teams of systematic reviewers to make their data available, we encourage them to do so as quickly as possible. We believe that the median time from SRDR project initiation (i.e., data extraction phase) to public availability of data of 4 months is satisfactory. While we do not track the direction, magnitude, or statistical significance of results of systematic reviews in SRDR, a recent study by other investigators has demonstrated that statistical significance of results is not associated with the duration from protocol registration in PROSPERO to journal publication (for non-Cochrane systematic reviews) [37].

It should be noted that, since 2015, AHRQ has required EPCs to make each of their AHRQ-funded review’s data available publicly through SRDR upon completion of the review. For systematic reviewers working on non-AHRQ-funded projects, there is less of an incentive to make data available publicly. We strongly agree with Wolfenden and colleagues that platforms, such as SRDR, that make systematic review data available publicly can help maximize returns on the significant investments that are made in the systematic review enterprise [5]. We urge more systematic review teams to make their data available publicly.

The characteristics of the systematic reviews reported in this study are not intended to be representative of all systematic reviews. As discussed, most of the systematic reviews are AHRQ-funded or Cochrane reviews, leading to a predominance of reviews addressing clinical fields, evaluating interventions, and including higher numbers of studies than has been demonstrated in cross-sections of systematic reviews in other investigations.