What is the effectiveness of printed educational materials on primary care physician knowledge, behaviour, and patient outcomes: a systematic review and meta-analyses

Printed educational materials (PEMs) are a simple, relatively inexpensive knowledge translation (KT) intervention for the dissemination of clinical information (such as clinical practice guidelines, journal articles, or evidence-based PDF or email summaries), aimed at improving the provision of care. A recent Cochrane review found that PEMs may have a small (0.02–0.13 standardized mean difference) beneficial effect on health professional practice outcomes [1]. However, despite continued publication of randomized controlled trials (RCTs) utilizing this intervention, we still know little about which behaviours can be influenced by PEMs, within which settings, and how to optimize the effect of these interventions for various health professionals.

Primary care physicians (PCPs) are required to have a vast and comprehensive knowledge base to treat different patient groups and diseases. On average, they have been observed to have 3.2 questions for every 10 patients they see [2] but these questions often go unanswered. PEMs are a potential strategy for meeting these needs. Non-interactive PEMs are easy to implement and scale across various primary care clinics. Reviews of the literature have found that printed resources (including books) remain a common source of information for physicians [3‐5], with one systematic review finding that 50–80 % of physicians used printed materials for information [3]. However, if there is no demonstrated effectiveness of these interventions on knowledge, behaviour, or patient outcomes when targeted at PCPs, they should not be implemented as behaviour change techniques. To our knowledge, this is the first review to examine the effect of PEMs on PCPs.

Though interactive computer-based KT interventions such as those integrated within electronic health records have been shown to be effective in changing behaviour and are increasingly more popular than non-interactive paper-based interventions, they are expensive and require technological infrastructure and training, obstacles to implementation given limited budgets and overworked clinicians [6]. Surveys show that only 64 % of Canadian PCPs [7] and 41.5 % of American physicians [8] use electronic medical records, limiting the reach of complex interventions that are integrated into electronic records and possibly unintentionally leaving PCPs out of these interventions. With many different software vendors being used across practices (for example, there are 14 certified electronic medical record products to date in Canada alone [9]), creating a one-size-fits-all solution is challenging. As such, PEMs, a non-interactive and low-tech intervention, will likely continue to be used to disseminate new evidence and important clinical information or as a part of multi-component KT interventions.

The objective of this review was to examine what effect PEMs have on PCP knowledge, behaviour, and patient outcomes, in comparison to no intervention or to other single- or multi-component educational interventions. This review contributes to existing literature by examining the effect of interventions specifically designed for PCPs. The primary care setting is considerably different from other health care settings, and PCPs are likely to experience barriers unique to their setting and their scope of practice. Physicians are the population of interest to limit participant heterogeneity as we anticipated that differences in training and role among diverse primary care clinicians may influence behaviour change. PEMs for PCPs may have different content and may target different behaviour than PEMs for other professionals. More importantly, PCPs may respond differently than other clinicians to PEMs, and we anticipate PEMs have a different effect size when targeting behaviour change in different providers. We also examined the quality of reporting of PEM interventions in included studies.

A systematic review protocol was written for this review and registered with PROSPERO, the international prospective register of systematic reviews (registration no. CRD42013004356). We based the methods for this review on those described in the Cochrane Handbook for Systematic Reviews of Interventions and the Cochrane Effective Practice and Organisation of Care Group (EPOC) [10]. It is reported using the PRISMA Statement for Reporting Systematic Reviews [11].

We included 40 studies and conducted eight meta-analyses with data from 26 studies. The reported quality of evidence ranged from low to high and many studies were unclear on important methodological factors such as allocation sequence generation (15 studies unclear), baseline outcome measurement (15 studies unclear), and baseline characteristic measurement (nine studies unclear). No significant effect was seen across the eight meta-analyses. However, six of the meta-analyses had three or fewer studies included.

Statistical heterogeneity was high for both physician cognition meta-analyses, possibly due to different ways of measuring knowledge outcomes, the different topic areas, duration of the intervention, risk of bias, or the type of PEM intervention. Clinical heterogeneity was visible across studies in all meta-analyses as the topic of the interventions varied greatly. Behaviour change of PCPs may vary by clinical area addressed by the intervention. Though heterogeneity in the population was limited by the review’s focus on PCPs, it was difficult to determine how the population varied across studies due to limited reporting on participant age, time since graduation, and academic affiliation. Methodological heterogeneity was also a concern as study durations varied widely and many factors could not be assessed such as how the intervention was delivered, the source of the evidence, setting of the participant’s clinical practice, and participant demographic details.

The narrative synthesis examined 71 study-level outcomes across three outcome categories. When compared to usual care, PEMs resulted in significant improvement in outcomes for one of four clinical patient outcomes, 13 physician behaviour outcomes, and no physician cognition outcomes. Though it is difficult to draw conclusions from the narrative syntheses of these outcomes, it is evident that a large majority of the measured outcomes did not statistically improve as a result of PEMs. These results differ from the review by Giguère and colleagues [1], which included all health professionals, and found a median improvement in SMD for categorical practice outcomes of 0.02 and a median improvement in SMD for continuous professional outcomes of 0.13 when compared to usual care. Only 11 of our 40 studies overlapped with the Giguère review. Our results suggest that the effect found in the review by Giguère and colleagues may not be applicable to PCPs and that the effect was a result of interventions targeted at specialist physicians. Most of the studies included in the Cochrane review (42 of 45) included physicians, though the clinical speciality was not clear in many of the included studies and no subgroup analyses were conducted by type of health professional. The difference in effect size may be a result of differences in PEM content between PCP and other clinician PEMs, in the evidence needs of PCPs, in the practice context, or in the behaviours targeted by the PEM. The methods used to synthesize data in the Cochrane review included the use of more than one outcome per study and may have resulted in an inflated effect size. The authors of the Cochrane review also used a different approach when more than one measure was provided in a study by abstracting the median measure whereas we used a random selection process. Furthermore, our review builds on the existing literature as it examines the intervention’s effect on knowledge, an outcome that was excluded from the Cochrane review.

Though it is assumed that the cost of these interventions is small relative to interactive educational tools or active evidence-dissemination strategies, the costs need to be systematically assessed. Three of the studies included in this review provided an economic analysis, but two of the three studies provided costs for a more active intervention and not for the control PEM. Before widely disseminating any behaviour change intervention, information on cost effectiveness of the intervention is critical.

This systematic review has found that PEMs do not improve patient, PCP behaviour, or PCP knowledge outcomes. A previous Cochrane systematic review of 45 studies (14 RCTs and 31 ITSs) noted that similar, passive dissemination strategies have small to moderate effects (1). However, the Cochrane review included all health care professionals, excluded knowledge outcomes, and combined more than one outcome per study (possibly inflating the result) in its meta-analyses.

These findings are of particular importance to those interested in knowledge translation (guideline developers, quality agencies, government agencies, pharmaceutical companies, etc.) and in developing, disseminating, and evaluating PEM interventions. Individuals and groups interested in developing PEMs should note that in their current state, there is no evidence that PEMs are effective at improving PCP outcomes. However, it is difficult to ascertain whether the PEMs tested have been optimized and whether adequate reach of the intervention has been achieved. There are too few details in published studies on how these materials were developed (use of theory, use of evidence-based design, involvement of the end-user) and too limited descriptions of the materials. The results also have implications for research funding bodies. Continuing to fund studies on PEMs with little description of the intervention, inadequate power, or a lack of optimization of the intervention is a poor use of resources. Researchers interested in conducting future studies of PEMs for primary care should invest resources in better design of the tools and should provide detailed descriptions of the intervention to determine if PEM optimization can result in improved outcomes. Trials should be sufficiently powered to detect the small effect of these interventions. Head to head studies of different designs of PEMs may help us understand how certain design elements may contribute to effectiveness, though these studies should continue to include usual care comparisons. Otherwise, we should accept the role of PEMs as simple tools for the dissemination of information or as a part of other more interactive interventions, rather than as tools to influence knowledge and behaviour of PCPs.