Introduction

People are confronted every day by claims about the effects of treatments. Many, if not most, of these claims are unsupported by evidence [1][2], meaning they can put patients at risk of harm[3]. Where claims seek to sell a treatment, patients may spend their own money on treatments of no known benefit, or may seek inappropriate treatments, which can waste public resources on a large scale [4]. Additionally, misleading claims can exacerbate peoples’ natural tendency to overestimate the benefits of treatments and to underestimate their potential risks[5]. This might lead patients to seek inadequately evaluated treatments with unrecognised adverse effects[6][7]or avoid treatments likely to help them [8].

Similarly, health professionals may treat patients using methods that have been insufficiently evaluated [9] or act based on comparisons which have not assessed treatments using patient-valued outcomes. Professionals may also exploit treatment claims to fulfil their own, conflicted, interests [10]. Finally, as treatment claims often contradict each other and cause confusion, patients may simply become disillusioned with, and lose respect for, the relevance of research evidence [11].

To address these problems, people need to be able to think critically about the treatment claims they come across [12]. They should be able to assess the strength of the supporting evidence underlying a treatment claim, the conflicts of interest among those making the claims, and the relevance of the research evidence to their healthcare needs [13][14]. These processes are part of a wider range of skills people should use to make well-reasoned decisions on a day-to-day basis.

Increased general knowledge about how to evaluate treatment claims would mean people could more confidently use information about health claims to promote their own health [15][16][17][18][19][20]. It would mean that health professionals, together with their patients, could make more balanced decisions [21]. It would also enable greater patient involvement in shared-decision making [22][23], with fewer resources wasted and fewer avoidable harms.

Despite its importance, the promotion of critical thinking—the use of logic and evidence to assess the strength of the arguments that underlie claims—is not widespread. Furthermore, the effects of learning-resources that aim to do this have very rarely been formally evaluated. This lack of assessed learning-resources disadvantages people who wish to think for themselves about treatment decisions. To address this, it would be valuable to collate and make accessible an inventory of relevant, and where possible formally evaluated, existing learning-resources to help people who are responsible for teaching others how to assess treatment claims.

There have been efforts since 2011 to create an inventory of learning-resources for this purpose [24][25] (Fig 1). In October 2015, in Vienna, the Informed Healthcare Choices (IHC) project group and the James Lind Initiative (JLI) co-convened an international, multidisciplinary workshop for people interested in helping others to make sense of treatment claims. One of the presentations at the workshop was used to gauge support for a further attempt to develop and maintain an inventory of learning-resources, as well as to inform the development and evaluation of new resources. After written expressions of interest were submitted, it was agreed that the JLI, which is funded by the National Institute for Health Research (NIHR), would oversee the formation of such an inventory. Thus, we have created the Critical thinking and Appraisal Resource Library (CARL).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Timeline demonstrating the stages that lead to the development of CARL.

(A) Inventory by L Kendall Krause created. (B) The James Lind Initiative begins adding resources to Testing Treatments interactive. (C) European Communication on Research Awareness Needs (ECRAN) inventory created. (D) The Informed Healthcare Choices Project (IHC) formed, Members begin to identify useful resources. (E) Discussions between IHC and JLI lead to expressions of support for a new library, JLI agrees to coordinate development until 2019.

https://doi.org/10.1371/journal.pone.0178666.g001

In summary, our objectives were to develop the Critical thinking and Appraisal Resource Library for those who are responsible for encouraging critical thinking about treatment claims. These include: teachers of schoolchildren, undergraduates and graduates, for example those teaching evidence-based medicine, or those communicating treatment claims to the public. In selecting resources, we draw attention to those resources that have been formally evaluated with pre-defined criteria, for instance by the resource creator or by independent research groups.

Methods

Results

The website ‘Testing Treatments interactive’ currently provides access to over 300 open-access resources in CARL and more are being added following the same search and screening processes described earlier. We expect this number to grow with periodic further searches and through CARL users proposing additional resources using an online suggestion form.

The IHC Key Concepts that currently have the highest average number of resources are those grouped under ‘Comparisons: are they fair and reliable?’. This contains IHC Key Concepts relevant to randomised trials and systematic reviews, amongst other topics. The Key Concept group with the lowest average number of resources per Key Concept is ‘Choices: are the findings relevant?’, which contains IHC Key Concepts that address how users may apply evidence and treatment claims in their own decisions. The average number of resources per Key Concept may change as further searches are conducted, but it may also highlight which areas are lacking, and therefore inform development of additional resources by interested parties.

Of the resources that have been formally evaluated [31][32][33][34][35][36][37][38][39][40][41][42][43][45][46] the Key Concept group to which most are coded is ‘Comparisons: are the results fair and reliable?’ and this reflects that many of the resources currently available are used for teaching evidence-based medicine. The group that has the fewest formally evaluated resources is ‘Choices: are the findings relevant?’. This is likely to be because there are fewer Key Concepts in this group. Work is ongoing to expand the number of Key Concepts in this section. The most common formats of resources are Text and Audio, mostly derived from Testing Treatments [3]. The least common format of resources is currently Lessons. (Fig 2).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Graph to show the proportion of resources that are of each format in the CARL database.

Vertical axis is percentage of total resources of the specified target users. Horizontal represents the Key Concept group in question. (Blue) Text. (Red) Audio. (Grey) Videos. (Yellow) Websites. (Purple) Cartoons. (Green) Lessons.

https://doi.org/10.1371/journal.pone.0178666.g002

Specific IHC Key Concepts with relatively few resources include:

• Increasing the amount of a treatment does not necessarily increase the benefits of a treatment and may cause harm
• Beliefs about how treatments work are not reliable predictors of the actual effects of treatments
• Treatment decisions should take account of both beneficial and harmful effects
• Average differences between treatments can be misleading
• The treatments evaluated in fair comparisons may not be relevant or applicable
• Results for a selected group of people within fair comparisons can be misleading

Additionally, relatively few resources for teachers of children and young people have been identified, as well as few for ‘advisors’, ‘communicators’ and ‘researchers’.

The largest number of learning-resources is for ‘Learners’, divided into: ‘General Learners’ i.e. those who wish to educate themselves about the Key Concepts, and ‘Higher Learners’, including undergraduate and postgraduate health profession students (see Fig 3). Many of the learning-resources included in CARL had no specified Target User Group when originally identified in searches. These have been coded to specific user groups after discussion. Mostly, where resource target user groups were not specified, the resources were coded as for ‘General Learners’. This is because they may be relevant to one or more of the ‘Intermediaries’ listed above, or to anyone who wants to teach themselves how to use Key Concepts to assess the validity of treatment claims.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Graph to show the proportion of resources that are targeted at each Target User group in the CARL database by each of the Key Concept groups.

Vertical axis is percentage of resources in a group. Horizontal is the Key Concept group in question. (Pink) Teachers of school age children. (Orange) Teachers of students in further education. (Light blue) Independent learners with general interests. (Yellow) Higher level learners, such as those in further education or with higher level knowledge. (Red) Advisors, e.g. policymakers. (Green) Communicators, e.g. journalists. (Purple) Researchers interested in developing educational resources.

https://doi.org/10.1371/journal.pone.0178666.g003

Discussion

The Critical thinking and Appraisal Resource Library—CARL—has been created to help those who teach people how to evaluate claims about the effects of treatments. This report describes how CARL has been developed. We hope CARL will become an increasingly improved and indispensable directory for teachers, communicators, advisors, researchers and learners.

Systematic and online searches for additional learning-resources relating to each Key Concept will be conducted periodically. It is anticipated that additional resources will be recommended to us by users of the site and by those who share our interests.

With the initial phase of resource identification and coding complete, future emphasis will be on encouraging formal evaluation of the effects of resources on understanding and ability to apply Key Concepts, by independent groups who express interest in the project.

Some examples of formally evaluated resources include the booklet ‘Know Your Chances’ [42], which was compared objectively with a decision aid in two randomised controlled trials, in populations of different socioeconomic backgrounds [34]. Both trials found that people who had read ‘Know Your Chances’ were better able to understand risk than those who had used the decision aid.

Another formally evaluated resource is ‘Thinking, Doing, Talking Science’ [41], which promotes an alternative method of teaching science to primary school children, with an emphasis on greater cognitive challenge, interactivity and scientific reasoning [40]. This approach was evaluated in a randomised controlled trial that demonstrated an average increase in pupil progress in science, compared with control children, equivalent to two extra terms of teaching. It also showed that progress was relatively greater in students from poorer socioeconomic backgrounds. This study did not measure pupils’ understanding or ability to apply Key Concepts, but rather how well they did in national exams. It demonstrated that, in these terms, scientific reasoning improved.

A challenge in formally evaluating the impact of learning-resources is how to define and measure their effects. The IHC Project has created the Claim Evaluation Tools (Austvoll-Dahlgren et al. Submitted), a collection of multiple choice questions that test knowledge of the Key Concepts (www.testingtreatments.org/create-test-claim-evaluation-tools-database). These have been validated in Ugandan primary school children [44] for which information is publicly available through www.testingtreatments.org, and will potentially help interested researchers to conduct formal evaluations of resources.

In a randomised trial involving 120 schools, the Claim Evaluation Tools have been used to assess the effects of IHC primary school learning-resources (including a comic book text) on primary school children in Uganda [45]. In a linked trial, IHC resources were. used to evaluate the effects on parents of primary school children who have listened to an IHC podcast [46].

Helping children to think critically about treatment claims is important, particularly at an age when they may be more open to thinking critically, and have the time to learn the skills needed. Acquiring these skills as young children could provide a foundation upon which to strengthen their ability to make reasoned decisions. Indeed, the Key Concepts upon which the content of CARL is organised apply not only to claims about the effects of treatments, but also to other arguments and assertions—economic, social and political.

To summarise, the James Lind Initiative has coordinated the development of an open-access Critical thinking and Appraisal Resource Library (CARL), to help teachers and learners to increase general knowledge of Key Concepts relevant to assessing claims about the effects of treatment. These resources are currently available at www.testingtreatments.org.

The Fair Comparisons Network

Expressions of interest in CARL have been encouraging and have led to the development of a ‘Fair Comparisons Network’. This an informal list of individuals who share a common interest in promoting critical thinking and critical appraisal skills. The network is open for other members with similar interests to join and to help identify resources for inclusion in CARL. We hope that, as the Fair Comparisons Network expands, people who share an interest in a specific target user group or resource format will work together to increase the proportion of that are formally evaluated learning-resources. People who are not already members of the Fair Comparisons Network can join it by emailing Patricia Atkinson at patkinson@jameslind.net. Emails should include Name, Contact Information, and areas of interest.

Supporting information