Facilitating evidence uptake: development and user testing of a systematic review summary format to inform public health decision-making in German-speaking countries

To inform the development process, we searched the literature for existing tools for summarising the findings of systematic reviews in the health field. An inventory of the tools we identified and their characteristics is provided in Additional file 1: Table S1; we did not find a tool specifically developed for public health interventions. Within the research team we decided on guiding principles and core elements to be included in the summary format (Fig. 1).

We performed user tests guided by the ‘think-aloud’ method described in the ‘User Test Package’ of Cochrane Norway [25]. In user tests applying the think-aloud method, representatives of the target audience are asked to continuously verbalise their thoughts as they use a prototype of a product, while researchers observe them and listen to their comments [26]. The aim is to obtain a better understanding of the user experience, to observe the users’ problems with the product, to collect their suggestions and to improve the product on the basis of the structured observations and participant feedback. It has been applied to a variety of Cochrane products [27‐29] and other knowledge translation formats [30, 31].

We set out to recruit at least six participants in each of the three countries (18 in total), representing different institutions and levels of decision-making. For the purposes of this project, we defined public health decision-makers as those responsible for or involved in formulating or financing public health-relevant policies, programmes and regulatory measures. In each country, members of the research team identified relevant institutions representing public health decision-makers at national and sub-national levels (i.e. Bavaria for Germany, German-speaking part of Switzerland, lower Austria for Austria). Within these institutions, we identified departments relevant to our work and with whom we either had existing ties or with whom we were planning to establish such ties in the future. We recruited participants to cover a wide range of decision-making experiences and contexts, including heads of department (strategic decision-making) as well as members of staff with responsibility for specific programmes and content matter expertise (programmatic decision-making). We sent an invitation letter to identified individuals within relevant institutions informing them about the aims and process of the study. In case of no response, reminders were sent after 1–2 weeks or the invited individuals were contacted by phone.

User testing sessions were conducted face-to-face with individual participants and facilitated by one researcher at each site (LKB, MM, CK). Time and place were agreed upon with each participant individually; user testing sessions took place between June and August 2016, usually at the participant’s workplace. Participants were asked whether their contribution could be recorded and were assured anonymity. Every interviewee agreed to these terms and provided written informed consent. At the start of the interview, the participant was given a printed copy of the prototype summary to read on their own. Using a semi-structured interview guide, the interviewer then led participants through each part of the document, encouraging them to keep up a running monologue. The interview guide (Additional file 3) was based on the framework for user experience by Morville [32], which was adapted by Rosenbaum [33]. The eight facets that we chose to cover in our interview guide were credibility, usability, understandability, usefulness, desirability, identification, value and findability. An additional ninth facet, accessibility, assessed where the participant would expect to find the summary, and was included to help us develop a dissemination strategy and place future systematic review summaries appropriately.

The sessions were audio-taped and the interviewer took notes on the participant’s monologue and on their observations of the user experience. Finally, the interviewers asked the participants additional questions, e.g. about barriers to the use of systematic reviews in their everyday working life and their familiarity with systematic reviews.

At each of the three study sites, one researcher (LKB, MM, CK) excerpted relevant passages from the respective audio recordings. All data were de-identified before analysis. In a first round, each researcher paraphrased the relevant excerpt, entered it into Microsoft Excel, and manually coded it against the nine pre-determined facets, related it to the section of the product concerned (e.g. SoF table, key messages, GRADE, background), and categorised its quality (general comment; low, medium or serious problem; explicitly positive statement). A second researcher (LKB, MM, CK) checked the initial codings; disagreements were resolved within the team to ensure a consistent coding process. The coded excerpts were analysed for themes and issues that commonly applied across countries. The results of this analysis informed the content and design changes to the summary format.

The findings were summarised according to the section of the summary concerned and the quality of the comment. Within the team, we discussed how the participants’ comments and problems could be addressed; one researcher (JMS) then developed suggestions based on these discussions. Where several options were available, a communication specialist was consulted. The format was thus revised in an iterative process based on several rounds of discussion and revision within the group of researchers. We based the decision on how to handle the participants’ comments primarily on the feasibility of integrating them within the summary format, the pertinence of the comment and the recommendation of the communication specialist.

We conducted 18 interviews, six at each participating study site. Decision-makers were affiliated with different national and regional public health institutions, predominantly government agencies, and represented different levels of seniority ranging from research associate to head of department. Participants had been working in their current position between 6 months and 15 years. When asked whether they believed that the use of research evidence was valued in their position, 16 answered affirmatively, but four of them did not believe that it was common practise. All but one participant said they knew what a systematic review was, 12 said they had read one, and five had read selected chapters or a summary of a systematic review.

In the analysis, we could not discriminate the facet ‘value’ from other facets and found that participants’ responses to that question were already covered in previous responses. We therefore decided not to analyse this facet separately and reported related aspects under the best-fitting facet.

Overall, the summary was well-received and the participants indicated interest in receiving information about systematic reviews presented in this manner. The following sections present findings across the three countries for each of the eight facets.

At the beginning of the interview, the participants were asked how they rated the document’s credibility based on their first impression. Most considered the summary to be trustworthy. A large majority attributed this to the Cochrane logo placed on top of the first page.

“If it says Cochrane, I know, I can trust it.”

Another reason mentioned frequently was the provision of numbers and the inclusion of the grading of the certainty of the evidence.

“I see RCTs, I see concrete numbers, specifications regarding percentages, participants – it does give the impression that it was carefully researched and it seems that it is a reliable source, that I would trust.”

Furthermore, participants assumed credibility of the summary because it referred to the conflict of interest statements of the included primary studies and followed a clear and transparent design. Nevertheless, participants missed information that can be expected in a document from a credible source, such as a publication date and the names of the authors of the summary.

Many aspects of the layout and structure (e.g. headings and sub-headings, boxes) received praise. The box ‘Relevance for decision-makers’ and key messages were considered to be the most important sections of the summary.

Many participants described the information density as overwhelming and found that it hindered easy use of the document and quick extraction of relevant information. Feedback regarding the SoF table was particularly controversial. On the one hand, participants praised its structure and noted that they generally liked tables to obtain an overview. On the other hand, the actual use of the SoF tables presented a challenge, where several participants said that they preferred text and would like to receive more explanations. Another factor that limited the summary’s usability was the time needed to read and actually understand it.

“If I want to understand the text in-depth, then I would need 10–15 minutes. That’s long. That’s a lot of time. To read the key messages, which frequently show up in practice, 1–1.5 minutes is the most. If I can’t find what I am looking for, I put it aside.”

The participants suggested a range of specific changes to formatting and structure that would make the document easier to use, e.g. more prominent sub-headings and placing the box ‘Relevance for public health practice’ together with the key messages on the first page.

Most participants found the text-based parts of the summary understandable and the wording appropriate for decision-makers, public health practitioners and lay people.

However, understanding the SoF tables and GRADE ratings caused considerable problems. In the SoF tables, the most important source of confusion were statistical terms such as standard deviation, confidence interval, relative effect, illustrative comparative risk, outcome, and the differences between studies and independent comparisons. Participants appreciated the interpretation aid and said it helped them understand the tables, but additionally suggested the inclusion of the definitions of statistical terms used.

“This is quite a lot of information in such limited space, particularly in the [SoF] tables. I have the impression that they [the tables] are geared towards people with a strong background in statistics. For example: ‘standard deviations higher’. And it is also somehow assumed that one is familiar with the evidence rating. [...]. I am not sure that this statistical language is ideal for decision-makers.”

The explanations provided about GRADE and what the different ratings implied did not help participants interpret the results, often because they were not familiar with the term ‘effect estimate’. It was also mentioned that there was a discrepancy between the graphical representation of the ratings and the labels used to describe them, e.g. three out of four points to indicate ‘moderate’ quality. They also pointed out that ‘moderate’ could be interpreted very differently from its intended meaning and therefore recommended only stating the symbols.

“From our own studies, we know that people interpret these labels very differently. For me, [‘moderate’] would mean ‘in the middle’. This is not consistent with three crosses out of four. Therefore, symbols would be clearer to me.”

The participants also wanted information about what GRADE actually implies and how authors arrived at the respective rating.

Most participants stated that a summary was useful because it would allow them and their colleagues to obtain a quick overview of the respective topic. Moreover, the language used was considered to be appropriate for both experts and lay people; as the summary was provided by a credible source (Cochrane) it could be presented to external partners such as the Ministry of Health.

“Yes, very helpful. We have a high workload and we need to get the general idea fast. In the department we mostly have lawyers, who need the results presented in a condensed manner, without too many details.”

The most cited reasons for limitations in the summary’s usefulness were that the certainty of the evidence (‘moderate’ or lower) did not allow clear conclusions to be drawn and that it did not provide concrete recommendations adapted to the local situation.

“I would expect to find a recommendation for action, for implementation here.”

Accordingly, participants suggested giving practical examples of how the results could be implemented. They also said that it would help them if the summary already included contextual information about the legal and political conditions in their respective country or region.

Participants much appreciated a summary in German that was easy to understand. The elements carrying a lot of information condensed into simple messages, e.g. key messages, the graphical GRADE ratings and the box on ‘Relevance for decision-makers’, were found particularly appealing.

“What I thought was good is the box with the key messages. I would, of course, dive into it right away.”

Many participants disliked the high density of information, particularly in the SoF tables. Graphical illustrations and more explanations in text form were two examples named to overcome these problems.

Participants liked the visual emphasis of important information, either by placing such information in a box or by having key terms in bold letters.

“What I always like is that the key messages, like an abstract, including the results can be easily found right at the beginning, I find that nice because they are also visually highlighted.”

They also said that some aspects of the formatting made them miss important information, e.g. they overlooked the sub-headings because the font colour was too faint and the surrounding elements very dominant. One participant was confused because the results on alcohol-specific interventions were not found, although this was explicitly described in the title and background as one aspect of the review question.

Participants’ opinions diverged as to whether the summary was for people in positions like their own. Some welcomed the initiative and said that they were looking forward to receiving more such summaries. Others did not think of themselves as the right audience for this format because they lacked the necessary statistical knowledge or needed more comprehensive information.

“The question is who should use this [summary]. […] If it is for research associates who, at least in theory, have some statistical training, then it is fine exactly the way it is.”

“If I were better at statistical things, but that is just my personal background […] My background is specifically qualitative …”

On a similar note, several participants said that the depth of information was too much for political decision-makers. Title, key messages and a link to more detailed results would be more suitable for this audience.

When asked where they would expect to find systematic review summaries, most participants named those web-based sources where they would usually look for scientific information namely Google, Google Scholar, PubMed, the Cochrane Library, and the homepages of academic and government institutions. When asked how these summaries could be best disseminated, most of them mentioned newsletters and mailing lists and explained that they did not want to have to actively look for them. One participant, however, pointed out that a summary may easily get lost in the daily flood of emails. The need for advertising, for example, at suitable conferences, was emphasised.

When asked what stopped them from using evidence from systematic reviews, half of the participants named time constraints. The second most frequent reason was somehow related, i.e. that reviews are usually published in English, which takes longer to read and is more difficult to understand than one’s native language.

“We all speak English to some degree. But it does make a difference whether I have a review in English in front of me and for only a quick reading think ‘Hmm, not for now.’ But if I have this [summary] in German with key messages and [a section] ‘Relevance for decision-makers’ I think ‘Oh, yes, that does look quite interesting, I will have a quick look.’ That definitely helps.”

Other reasons mentioned were lacking access to publications and that reading scientific studies did not fall within their area of responsibility but within that of their members of staff.

We made five substantive changes on the basis of user test results:

With support from a health communication specialist, we made a number of additional minor changes to the structure and layout of the summary format (e.g. colour shadings). We also added the names of the authors of the summary on the last page of the summary format. The revised example summary for the Cochrane review on ‘Portion, package or tableware size for changing selection and consumption of food, alcohol and tobacco’ was translated into English and is available online (Additional file 4). Table 1 provides a comparison of the structure of the prototype and revised summary format.

We developed a German language summary format for systematic reviews of public health interventions in an effort to support the knowledge transfer to public health decision-makers in Germany, Austria and Switzerland. Through user tests, we explored our target group’s experience with this summary. Overall, the summary format was well-received and participants indicated their interest in it; they particularly appreciated the key messages, the section ‘Relevance for public health practice’ and to have information readily available in their own language. However, the user tests also revealed a range of problems. Many of these, for example, participants’ difficulties with understanding statistical terms and their wish for a less information-dense structure, were resolved through our revision of the summary format. Others, however, are more difficult to address, for example, the expressed need for recommendations and contextual information, and are discussed in more detail below.

Participants originated from different professional backgrounds and positions, and hence expressed diverging needs regarding the length of the summaries and related depth of information, ranging from a one-pager to be read in 2 minutes to a more detailed summary of eight pages or more. Sometimes, even individual participants gave conflicting feedback, at one point indicating that the document should be shorter to cater to their limited time frame, at a later point suggesting that more information should be added. Decision-makers’ preference for short texts and simple messages [4, 5, 14, 31], the seemingly contradictory wish for both brevity and detail [30], and the preference for an accessible structure [34] have also been reported by others. The revised summary format should cater to different types of decision-makers. Most likely strategic decision-makers with very limited time will only scan through the first page whereas programmatic decision-makers with more subject-specific knowledge and a scientific interest will use the full summary.

Participants generally appreciated the inclusion of a SoF table and stated that the provision of GRADE ratings conveyed credibility to the information; however, the problems in relation to the interpretation of GRADE ratings are well-known [31, 35]. Because previous user tests [29, 31] had demonstrated difficulties with understanding SoF tables and because a previous study [36] found that users want to have an explanatory comment, we included an interpretation aid in the prototype summary format and expanded this in the revised summary format. Our user tests corroborated the need for detailed explanations of technical terms found in former studies. There might be a learning effect where participants expect to find the SoF tables and GRADE ratings easier to understand upon repeated exposure [29].

Another challenging observation was the participants’ wish for information outside the scope of a systematic review. Many decision-makers deplored the absence of detailed recommendations and guidance for the implementation of findings, an observation also made in other user testing studies on knowledge translation products [30, 31]. Despite acknowledging the importance of these factors, the time and person-power needed to compile such information will make it difficult to produce locally adapted systematic review summaries in a standardised manner and on an ongoing basis.

A strength of our study lies in the participation of a broad range of public health decision-makers from distinct institutions and areas of competence in Germany, Austria and Switzerland. It reflects the diverse needs of strategic as well as more programmatic decision-makers at national and regional levels. In employing user testing, we pursued a non-quantitative approach to capturing the users’ experience with the systematic review summary. We only included six participants per country and may not have captured the full range of experiences with the product in relation to characteristics of sex, age, sociocultural or professional backgrounds.

We conducted our user testing with the summary of a single recent Cochrane review [24]. It is therefore possible that reactions to the summary format by some participants were specific to the content of this systematic review (e.g. not finding results for alcohol-specific interventions due to the non-availability of eligible studies). In a similar manner, reviews of other public health interventions may present with additional challenges. Further user testing could also serve to explore to what extent it might be useful to split larger reviews into several summaries (e.g. by splitting the systematic review on portion, package or tableware sizes for changing selection and consumption into three shorter summaries focusing on food, alcohol and tobacco, respectively).

The think-aloud method employed in this study has been successfully applied in several previous studies with similar aims [27, 29‐31], and we found it very suitable to this study’s needs. However, user testing creates an artificial situation and thus cannot fully reflect the real-life situation it is intended to shed light on. By conducting the user tests in the participants’ work places, where they would usually access and read a systematic review summary, we have attempted to attenuate this problem.

In addition, participants were likely aware that the interviewers had been involved in the preparation of the summaries, and this may have biased their responses. By continuing to encourage participants to think aloud, to describe their problems with the summary and to make suggestions for improvement, we sought to minimise any social desirability bias. One further strength lies in our analytical rigour. Three researchers independently coded and then cross-checked their colleagues’ work, thereby reducing subjectivity of interpretation.

There is one significant limitation with respect to our overall approach, namely that Cochrane systematic reviews focus on the validity of research findings, and – in line with a ‘knowledge shapes policy model’ [37], where evidence affects decision-making through either short-term direct use or longer term thought enlightenment [38] – efforts to increase the uptake of research findings are primarily concerned with better dissemination and communication of evidence. However, uptake of research findings also requires that evidence obtained through a global systematic review be examined with respect to its applicability in a local decision-making context. A systematic review summary format can help with this translation from generic findings to specific circumstances, but the summary format by itself is not sufficient to enable evidence uptake and use.