Online dissemination of Cochrane reviews on digital health technologies: a cross-sectional study

Digital technologies have revolutionised healthcare and public health by offering unprecedented opportunities for improved medical interventions and diagnostics as well as monitoring of health [1, 2]. Digital technologies, such as smartphones and wearables, can assist healthcare decisions, including more accurate and timely diagnoses [3], and contribute to health promotion and disease prevention by empowering their users to take an active role in managing their health via access to personalised health information and self-monitoring [4]. Despite the growing accessibility of digital technologies, a persistent digital divide exists that limits the equitable access to innovations due to socioeconomic factors, geographical disparities, and varying levels of technological literacy [5]. Furthermore, digital technologies require systematic evaluation to investigate if they contribute to any health benefits in their users [6].

Scientific evidence required to evaluate health outcomes of digital technologies can be synthesised using systematic reviews. Systematic reviews play a crucial role as a valuable source of evidence for stakeholders, including policy makers, clinicians, researchers, and the general population. Among systematic reviews, Cochrane reviews employ high methodological standards [7] and are therefore especially valuable for stakeholders. Considering the rapid development of digital health, it is important to assess how Cochrane reviews addressed this topic so far, in terms of digital technology types, target populations, and health outcomes. Furthermore, it is also of interest to investigate how these reviews are disseminated online to improve their impact by potentially reaching any relevant stakeholders [8, 9].

Dissemination is defined as a proactive method of distributing scientific evidence to a specific audience through selected channels and planned strategies [10]. While peer-reviewed publications and conference presentations are effective strategies for disseminating research to academic audiences, other strategies, such as plain language summaries (PLS), policy briefs, reports, blogs, and communication via social media, may be better suited to reach policy makers or the interested public [11‐13].

As there are a multitude of channels which can be used to disseminate research, different measures can be used to assess impact of academic publications. For example, the number of citations of an academic publication is one method for estimating its academic impact. Different measures are needed to objectively assess the nonacademic impact of academic publications due to multiple and heterogeneous channels that could be used for dissemination. As such, the nonacademic impact of academic publications could be assessed by examining citations in policy documents, reports, guidelines, and mentions on social media. Altmetric data provide one method of tracing mentions of academic publications in nonacademic online channels [14] and thus could be used to objectively quantify the nonacademic impact of such publications.

The aim of this study was to investigate the online dissemination of Cochrane reviews on digital health technologies. This study addressed the following research questions:

This study is a meta-research (i.e. research on research) study. We used Cochrane reviews as the unit of analysis and a cross-sectional design to collect and analyse data from such reviews (i.e. data on review content and review dissemination). A protocol for this study was prospectively registered at the Open Science Framework [15]. The study adheres to the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) guideline [16]. The STROBE checklist is reported in Additional file 1. There were no changes to the original protocol.

The data source for this study was the Cochrane Database of Systematic Reviews [17]. This database listed 9033 records on 10 May 2023, including reviews with systematic methodology (e.g. systematic, scoping, or rapid reviews) and review protocols on any topics within healthcare and public health.

This study included Cochrane reviews with any methodology (e.g. systematic reviews) published up to 10 May 2023. The eligibility criteria for this study were defined based on the PICO (Population, Intervention, Comparison, Outcome) framework. The inclusion criteria were as follows: (1) any human population (P), (2) intervention or concept supported by any digital technology (I), (3) any or no comparison (C), and (4) any health outcome (O). The exclusion criteria were as follows: (1) nonhuman populations, (2) no focus on digital technologies, (3) no health outcome, and (4) other Cochrane publications (e.g. review protocols due to lack of data or original reviews if review update was published to reduce bias due to duplication).

Data collection process involved the electronic search for Cochrane reviews, selection of eligible reviews, and data extraction from the included reviews.

A list of variables (data items) for data extraction was developed by the team (Table 1).

Data items 1 to 5 (Table 1) were extracted as quantitative information (e.g. publication year) or qualitative, verbatim statements (e.g. description of digital technologies according to review authors) from the full text of the individual reviews. The qualitative data were subsequently processed into quantitative categories based on meaningful themes that inductively emerged from the data. For example, we assigned digital technologies into categories ‘mobile technologies’ or ‘nonmobile technologies’ based on the digital devices described in reviews. During data processing, we also considered the heterogeneous terminology used by review authors. For example, ‘mobile phone’ was used as a category to describe any portable telephone, including mobile phones, mobile telephones, videophones, smartphones, or cell phones. To reduce any biases in data processing, two researchers (M. K. and K. K. D. S.) discussed and agreed on all categories, one researcher (M. K.) processed all data, and another researcher (K. K. D. S.) checked all processed data. Any discrepancies were discussed, and the final agreement was reached by consensus between both researchers.

Data items 6 to 8 (Table 1) were extracted as quantitative information from the Cochrane website and the Altmetric data available open-access online. The Cochrane website lists various details of each Cochrane review, including a review abstract and a link to a full-text document, a plain language summary (PLS) in English that is a mandatory part of any Cochrane review, any translations of the PLS, and other review information (e.g. the number of clinical guidelines that cited the review and Altmetric data for each review). The Altmetric data are used to compute the Altmetric (attention) score that measures the attention towards academic publications by tracing their mentions in nonacademic online channels [14]. Therefore, it provides an objective method to quantify and explore online dissemination channels of academic publications. The Altmetric score is a weighted count of online mentions. It is computed as the sum of the quantity of online mentions (i.e. the number of mentions) weighted by the quality of online mentions (i.e. the source of mentions, such as Wikipedia) [18]. As of June 2023, the Altmetric score was computed based on the following 17 online channels with different weights [19] that were considered in this study:

To prevent score inflation, the computation algorithm for the Altmetric score considers channel types and their weights as well as other factors, such as duplicate posts in the same channels [19]. In general, only the first mention in a specific online channel counts towards the Altmetric score (e.g. if a Cochrane review is mentioned in multiple articles in the same news channel, then only the first mention contributes to the Altmetric score for that Cochrane review) [19, 20].

The academic online channels, including Mendeley readers and Dimension and Web of Science citations, were also traced by Altmetric data in June 2023, but not included in the computation of the Altmetric score [19] and, thus, not considered in this study.

To reduce any bias due to updates in Altmetric data that occur daily at midnight West European time, Altmetric data for all reviews were extracted on a single day (29 June 2023) during business hours in Germany. One researcher (M. K.) made a screenshot of Altmetric data for each Cochrane review and collated all screenshots into one document. A team assistant manually entered all Altmetric data into the data extraction sheet in Excel. To reduce bias in data entry, one researcher (M. K.) checked the Altmetric data in 10% of reviews. Since no errors were detected, no further checks were performed.

The data were synthesised using descriptive statistics (e.g. absolute and relative frequencies or measures of central tendency, including mean and standard deviation) in Excel and IBM-SPSS24. Depending on data availability and type, we planned to investigate if dissemination (i.e. Altmetric scores) is associated with any review characteristics (e.g. publication year) or evidence quality characteristics (e.g. certainty of evidence ratings) using a univariate linear or binary logistic regression analysis in IBM-SPSS24.

This study includes data from 100 Cochrane reviews that met the eligibility criteria out of 170 records identified in the search of Cochrane Database of Systematic Reviews (Fig. 1). A list of 100 included reviews and excluded reviews with individual reasons for exclusion is reported in Additional file 3. All data extracted from the 100 reviews are reported in Additional files 4 and 5, and data synthesis is reported in Additional file 6.

In agreement with the Cochrane guidelines, all reviews had a text-based plain language summary (PLS) that was available in 3–14 languages, including English. Most common translation languages were Spanish (n = 100), French (n = 88), and Arabic (n = 72). According to review information on the Cochrane website, 60 reviews were cited in 1–18 clinical guidelines.

According to the Altmetric data, the included Cochrane reviews were disseminated (i.e. mentioned online at least once) in 9/17 online channels that are used to compute the Altmetric score (Fig. 3). These included predominantly social media (X/Twitter and Facebook), while less than 50% of reviews were disseminated via blogs, policy sources, news, Wikipedia, Google + , YouTube, or patents. The Cochrane reviews were not mentioned in 8/17 online channels, including (post-publication) peer reviews, Weibo, Syllabi, F1000, LinkedIn, Reddit, Q&A, and Pinterest.

The included Cochrane reviews were mentioned 4661 times in 9/17 online channels (Fig. 4). Most traced mentions were found in the social media (X/Twitter and Facebook) and news outlets. Each Cochrane review was mentioned 1–271 times (50 reviews were mentioned 1–33 times, and further 50 were mentioned 34–271 times).

The Altmetric score for all reviews ranged between 1 and 553 (mean = 48, standard deviation = 67; 1–28 in 50 reviews and 29–553 in further 50 reviews). Most reviews (n = 97) had Altmetric scores between 1 and 150, and n = 3 reviews had Altmetric scores of more than 200 (i.e. 207 [21], 237 [22], and 553 [23]). These n = 3 reviews addressed topics relevant for the COVID-19 pandemic, including interventions to improve vaccination rates [23], digital technologies for contact tracing during epidemics [22], and interventions to reduce social isolation and loneliness among the elderly [21].

The online attention towards most reviews was high. Altmetric scores in 51 reviews were rated in up to the top 25% of all research outputs traced by Altmetric data (i.e. Altmetric scores of 1–6 in 10 reviews were rated below the top 25%, and Altmetric scores of 7–29 in 41 reviews were rated in the top 25%). Altmetric scores of 30–553 (in 49 reviews) were rated in the top 5% of all research outputs traced by Altmetric data.

To investigate if dissemination (i.e. Altmetric scores) is associated with any review characteristics, we performed a binary logistic regression analysis with one dependent and five independent variables (Table 2).

Dissemination (i.e. higher Altmetric scores) was associated with bibliographic review characteristics (i.e. older review publication year and PLS available in more languages). Dissemination was not associated with evidence quality (i.e. certainty of evidence, studies in review, or meta-analysis performed in review; Table 2). The same pattern of results was obtained in a sensitivity analysis when the three reviews with the highest Altmetric scores were removed from the binary logistic regression analysis (Additional file 6).

In addition to this academic publication, we planned to disseminate the results of this study via a conference poster at a scientific meeting [24] and PLS in English and in German (Additional file 7).

This cross-sectional study based on 100 Cochrane reviews on digital health technologies shows that such reviews (1) addressed different types of digital technologies focusing mainly on disease treatment or management in adult clinical populations and (2) were widely disseminated via nonacademic online channels. The online attention towards 90 reviews was high based on Altmetric scores in the top 25% or 5% of all research outputs traced by the Altmetric data. Dissemination was associated with bibliographic review characteristics (earlier publication year and more PLS translations), but not with evidence quality.

Our study shows that Cochrane reviews on digital health technologies were predominantly disseminated via the social media (X/Twitter or Facebook). Thus, Cochrane reviews might be used as important source of health information for users of such media. Interestingly, dissemination was not associated with evidence quality but rather with relevance of review topic (i.e. the highest Altmetric scores were recorded for review related to COVID-19 topics) and accessibility of review (i.e. more years since publication and with PLS in more languages). Similar trends are also observed in citations of scientific articles that reflect the scientific impact and relevance of article topics but less so research quality [25]. Previous studies using Altmetric data suggest that higher visibility in social media is associated with other characteristics of scientific articles not considered in this study, including high journal impact factors, published open access, and having informative titles [26‐28]. Various measures could be used to improve dissemination of Cochrane reviews based on their evidence quality rather than their mere presence online in various languages. For example, improvements in science communication may help academic authors to clearly communicate their scientific findings [9]. This may enhance the lay understanding of PLS for nonacademic stakeholders and thus facilitate their decision whether or not to disseminate a specific content. In particular, the evidence limitations, such as very low and low certainty of evidence, need to be adequately explained in the PLS. Furthermore, although not investigated in this study, more focus on enhancing health literacy and digital health literacy is needed to improve the understanding of scientific content for any stakeholders or population groups [29].

Poor understanding of scientific content and restrictions in access to scientific evidence may contribute to the delay in dissemination of findings from newer reviews and consequently translating research evidence into clinical practice (the so-called research to practice gap) [30]. Such time lags in the translation process are prevalent in diverse fields of healthcare, and it can take up to several years between publication and being implemented or mentioned in clinical guidelines [31, 32]. Cochrane attempted to reduce the research to practice gap for COVID-19 research by publishing relevant reviews in a timely fashion (e.g. as rapid reviews) and by establishing a register of COVID-19 publications so that they could be located online faster [33]. These measures together with the global interest in COVID-19 may have contributed to the finding that COVID-19 reviews had the highest Altmetric scores in this study indicating that they were disseminated online via various channels despite their young age (recent publication date) and potentially reaching the relevant stakeholders. Future research may examine in more detail the dissemination approaches used by Cochrane for their COVID-19 reviews to find out if similar approaches could also improve the online attention towards other Cochrane or non-Cochrane reviews.

According to Altmetric data, we show that there is high online attention to Cochrane reviews on digital health technologies. This result is in line with a high academic impact of Cochrane reviews. Specifically, Cochrane reviews in public health were cited on average 240% more than other papers in this field [8]. In contrast to academic impact, Altmetric data can be used as a proxy of online interest in academic publications. In the current study, X/Twitter was the platform where the included Cochrane reviews were mentioned most by far. However, such data have various limitations because it is unclear who does the dissemination (i.e. review authors or anyone with the internet access), what are the motivations to disseminate the scientific content, and how the receivers of online mentions interact with and use the information. Furthermore, the Altmetric score only counts the interactions on different online media platforms, but it cannot distinguish between positive and critical attention [34‐36]. Altmetric data may also be more prone to manipulation than traditional bibliometrics [35]. For example, Facebook mentions can be purchased [37], and it is unclear if mentions on social media platforms only passively exist or are actually read [35]. However, the strength of the Altmetric data is that the attention metrics for an academic publication are immediately traced and available within hours of publication [35, 36]. In contrast, scientific citation metrics, such as citation counts or journal impact factors, are typically available several years after publication [35, 36].

Altmetric score relies on a selection of online channels, some of which (e.g. Google +) are no longer traced. Potentially, there is a need to capture other online channels for dissemination, such as TikTok videos or Instagram clips. Online attention could also be increased when authors of academic publications use more knowledge translation strategies that is in turn associated with higher impact of such publications on end users via health policy and practice [38]. Especially, social media dissemination channels could be used to reach out to knowledge users [11, 12]. More visibility and potentially higher Altmetric scores could result from using hashtags for social media posts that enhance searching and finding of content and including a unique identifier of the study (i.e. a link to the Cochrane review and not only its abstract) in social media posts or blogs in the main text [39].

Based on the content of the Cochrane reviews included in this study, we provide some recommendations for future research on digital health technologies.

There were several limitations in this study. First, we did not investigate who disseminated the Cochrane reviews and why. Although review authors can disseminate their own Cochrane reviews, any planned dissemination is rarely described in the text of such reviews [9]. Dissemination via channels traced by Altmetric data can be done by anyone with Internet access. Future studies could investigate the motivations for disseminating the Cochrane reviews. Second, it is unclear in what specific (online) news outlets the Cochrane reviews were disseminated. Unlike listing the names of various online media channels (e.g. Facebook or Wikipedia), Altmetric data do not specify which (online) news outlets are traced despite that news mentions contribute the highest weight to the Altmetric score. Third, the associations between dissemination (i.e. Altmetric scores) and review characteristics were only weak, possibly because most Cochrane reviews had relatively high Altmetric scores (in the top 25% or 5% of all research outputs traced by Altmetric data). Future studies could investigate such associations in non-Cochrane reviews with more variable Altmetric scores and compare Altmetric data between Cochrane and non-Cochrane reviews. Fourth, the lack of association between dissemination and certainty of evidence rating could be due to a high heterogeneity of outcomes and digital technologies in the included Cochrane reviews. Certainty of evidence ratings in reviews (Cochrane and non-Cochrane) with the same outcome and the same digital technology could be used in future analyses. Fifth, despite a large sample size (100 reviews), the results of this study might not be generalisable to non-Cochrane reviews, Cochrane reviews in other fields than digital health technologies, and other methods of assessing dissemination than Altmetric scores.