Languages for different health information readers: multitrait-multimethod content analysis of Cochrane systematic reviews textual summary formats

Health literacy is defined as the “constellation of skills, including the ability to perform basic reading and numerical tasks required to function in the health care environment” [1]. It has been recognized for a long time as an important factor both for understanding health information and prediction of health status [1, 2]. Health information should be easily accessible to people with low levels of health literacy [3], which means that written information should be easily readable and written in a plain language [4], preferably a sixth-grade reading level in the USA [5], which translates to the primary education of 11 to 12 year-olds. A number of studies have shown that materials containing information about health are often presented above patients’ readability level [5‐8]; also, patients and the general population have low health literacy [9], with resultant deficits in decision-making [10]. The most common readability tool for health information is the SMOG index (“Simple Measure Of Gobbledygook”). It emerged as the most appropriate among 12 different readability formula tested for their characteristics and predictive validity for public and patient material produced by the National Cancer Institute in the US [8]. Studies into other language characteristics also point to the importance of subjective expressions of attitudes, sentiments and feelings, as well as linguistic fluency for processing and interpreting the facts presented in a textual information [11, 12].

While there are many efforts to translate health information of different kinds to a form suitable for patients and the general public [4‐8], particular challenge for achieving appropriate health literacy are systematic reviews of health interventions, because they summarize evidence from individual studies to help doctors and patients make informed choices about health treatments. An example of the effort to make customized health information is Cochrane, an international professional community dedicated to producing systematic reviews of health interventions [13]. Cochrane has made a massive effort to develop separate summary presentation formats for different users [13]. The basic form of summarizing the results of a Cochrane systematic review is the Scientific abstract. It is written by review authors and aims at health professionals and researchers. Healthcare practitioners and professionals are also targeted with Cochrane Clinical Answers, which are “readable, digestible, clinically focused” presentation formats, produced by Cochrane [14]. Press releases are summary formats about Cochrane systematic reviews provided for media professionals; these are written by Cochrane. Finally, Plain language summaries are formats written for the lay public – consumers, patients and their families, and are translated into several languages. Plain languages summaries are written by review authors, but they often do not follow Cochrane writing standards [15], and are thus diverse in style, words usage, and possibly in literacy requirements.

The aim of our study was to compare the linguistic characteristics of different textual formats about the same health information. We used the availability of different summary formats for individual Cochrane systematic reviews to assess the linguistic characteristics of these formats. We compared their readability and used three different natural language processing tools to assess features that reflect different psychological and emotional processes embedded in different textual summary formats.

For 162 unique Press releases at the Cochrane web-site, we could identify 35 Cochrane Clinical Answers, 158 Scientific abstracts, 156 English Plain language summaries and their translations into French (n = 101), German (n = 41) and Croatian (n = 156) (Fig. 1).

Our study showed low overall reading ease of textual formats that summarize the results of Cochrane systematic reviews. Although Plain language summary formats were significantly easier to read than other formats, the required literacy level was still high: on average, over 10 to 15 years of education needed for easy reading of a plain language summary across different languages, which is much higher than the recommended US 6th grade reading level [1, 2, 9].

These results have to be interpreted with the limitations inherent to all readability studies, such as appropriateness for health information [26], score variability for different languages, as was the case for the Croatian language in our study [17], and imprecise education level thresholds for brief texts, such as used in our study [27]. The reasons for differences in SMOG index for different language translations of the same information are not clear, as there is little literature on the use of the SMOG formula to compare different languages. The original SMOG formula, developed for the English language, was used for studying health-related textual information in German [28]. Further, the study that compared SMOG scores for the same text unrelated to health in English, Spanish and French found that the scores for the English were lower (i.e. the text was more readable) than in French and Spanish [29]. Our study showed the same finding for textual health information (lower SMOG score for French than for English Plain language summary), but much lower scores (i.e. better readability) for German and Croatian translations. A further limitation is the use of Cochrane as a single source of health information summaries. However, the collection of different Cochrane formats for individual systematic reviews was a suitable model for the comparison of language characteristics in different summary formats presenting the same information to different audiences. The comparison showed significant statistical differences between the formats, but no actual “clinical” difference [1, 2, 9]. We were also limited by the small sample of Cochrane press releases which were the starting point for the collection of matching text summary formats. Finally, readability can be affected by visual formatting of the text or structured presentation [30, 31], which was not tested in our study.

The low readability of different textual summaries of systematic reviews has important implications for the efforts in translating textual health evidence to different audiences. On the one hand, lower readability necessarily leads to less accessible health information, especially for patients as a non-specialist audience. On the other hand, complex scientific literature is becoming increasingly more difficult to read [32], which may decrease the accuracy of information understanding, and thus affect health decisions guided by that information [33]. This means that there should be a balance between reading ease and accurate information about health. Cochrane has been testing different interventions to this aim, including PLEACS standards (Plain Language Expectations for Authors of Cochrane Summaries) for writing plain language summaries [15]. However, the authors of Cochrane systematic reviews, who write both the Scientific abstract and the Plain language summary formats, often do not follow these standards [15]. Other formats, such as infographics, may also not be better for understanding the results of a Cochrane systematic review in comparison to a well-written plain language summary [34].

The availability of new technologies for natural (human) language processing provided us with the unique opportunity to identify and quantify affective states and subjective information in different textual systematic review summary formats. Subjective experience of people when they process a piece of information influences whether they will consider the information as truthful, whether they will like it, and whether they will have confidence in the information [11]. Linguistic characteristics of the text contribute to the way people react to health information [35]. Sentiment analysis of physician-patient communication showed that patients adhered more to the advice of physicians who used fewer words related to negative emotion and fewer singular first-person pronouns [36]. Clinical advertisements by cancer centers in US consumer magazines and TV networks were shown to use emotional appeals to evoke hope and fear, but not providing sufficient information about objective information about risks vs. benefits, and costs [37].

We used three different natural language analysis tools and assessed different language traits to compare textual formats about health evidence targeting different audiences. Scientific abstracts had not only the longest format but also fewer emotional words. Scientific abstracts are intended for professionals and emotional “coldness” is to be expected as it presents information needed for physicians and other experts to gain greater insight into the methodological aspect of research required for critical assessment [38]. In contrast to this, Press releases were written more engagingly, had more clout (high expertise and confidence), as well as an overall more positive sentiment, more positive and engaging personality traits and were more engaging than either Scientific abstracts or Plain language summaries. Plain language summaries also had the highest “Tentative” tone, which may be related to difficulties in explaining complex scientific findings, which are by nature contradictory and in some way uncertain, and tentative [39].

As evidenced by data from all three language analysis tools, the style of writing in Press releases followed the standards of the health journalistic profession to “get attention, arouse interest and stir emotions” [38]. On the other hand, the language style and emotional content of the Plain language summaries was more similar to that of the Scientific abstracts, which can be expected from texts written by the same person – authors of the systematic review. This left the Plain language summaries in a metacognitive “no man’s land”: they lacked the cognitive fluency to facilitate judgment about the presented information but also lacked the details for a full critical interpretation of the methods, findings and limitations of the research. We also found apparently contradictory finding by the two different language analysis tools, like lower analytical tone for the Scientific abstracts, as measured by Watson Tone Analyzer (Fig. 3) but higher analytical thinking according to LIWC analyzer (Fig. 5). This apparent difference may be related to the different dictionaries used by the two methods, but it is not possible to precisely identify its source as the dictionaries for the two tools are not openly accessible.

It is difficult to assess the “clinical” significance of the statistically significant differences observed for language characteristics in our study, as there are, to the best of our knowledge, no similar linguistic comparison studies of different textual formats for the same health information. Future research should test hypotheses that can be generated from our study, such as how linguistic characteristics of the text affect the ability of a reader to understand and interpret the presented information.

General health literacy research ranges between two extremes: the proposition that news stories should read like research articles in order to accurately express scientific findings [40], and the proposition that news stories should involve a more human, emotional element into the text in order to improve understanding of health information [41, 42]. Our study showed that the Press releases about Cochrane systematic reviews followed the latter approach of more conversational language and paring of the scientific details, although they were aimed at supposedly highly health-literate consumers – health journalists [38]. Plain language summaries had a mix of formats both for high and low health-literate readers, which may not be an efficient method of health evidence translation to the patients and lay public in general [1, 2, 9].

Our study found that the language of professionally written press releases was generally more emotionally and cognitively engaging than that of scientific abstracts and plain language summaries, which are written by the authors of systematic reviews. Future research using novel language processing tools may help bring together the needs and wishes of different health information users and provide more evidence on the text characteristics that have impact on the cognitive processing of the users, as well as on the importance of presenting sufficient details about scientific validity and interpretation of health evidence.