Placebo response and effect in randomized clinical trials: meta-research with focus on contextual effects

Only trials included in the latest update of the Cochrane review “Placebo interventions for all clinical conditions” by Hróbjartsson and Gøtzsche [14] were considered eligible, and therefore, no new literature search was performed [15, 16].

Only randomized trials having an experimental intervention group, a placebo comparator group, and a no-treatment (control) group were considered eligible. As in the original meta-analysis [14, 17], participants were patients with any somatic or psychiatric disease or symptoms. Besides the exclusion criteria described in detail in the Cochrane review [14], trials without an intervention group and trials written in languages other than English were excluded.

A data-extraction form was developed for data collection. One reviewer (S.H.H.) extracted data and selected the outcome of interest based on the description in the Cochrane review [14]. A second reviewer (R.C) was consulted when necessary, and doubts were discussed to consensus. The extracted data included the year of publication, study design, number of participants randomized and analyzed, baseline characteristics (average age, proportion of women, diagnosis, chronic or non-chronic condition), time of outcome measurement, type of placebo (pharmacological, physical, or psychological), experimental and no-treatment intervention and primary outcome.

Further, the type of outcome was categorized in (i) patient-reported outcomes (e.g., pain); (ii) observable patient-reported outcomes (e.g., vomiting); (iii) observer-reported outcomes dependent on cooperation of the patient (e.g., forced expiratory volume); (iv) observer-reported outcomes not dependent on patient cooperation (e.g., blood pressure); and (v) laboratory data (e.g., hemoglobin concentration). Other extracted information included dropout rate; blinding of participants, providers, and observers; allocation concealment; settings (i.e., single-center or multicenter); and information given to participants (i.e., whether participants were informed that the trial involved a placebo intervention). End-of-treatment data were preferred over follow-up data to reduce bias due to patients leaving the primary trial and effects consequentially diminishing. Change from baseline was preferred, but if only final values were available, these were used. For crossover trials, data were extracted from the first treatment period only, to avoid any carry-over effect. If that was not possible, summary data were used (naively) as if they had been derived from a parallel-group trial.

Risk of bias of the included studies was assessed and compared to those from the original Cochrane review [14]. This approach enabled us to rate the risk of bias in domains from the Cochrane Risk of Bias tool [18] as High, Low, or Unclear risk. In addition, a simple risk-of-bias assessment proposed in the Cochrane review [14] was used, where trials with low risk of bias were defined as fulfilling the three following criteria: (i) adequate concealment of allocation, (ii) dropout rate no more than 15%, and (iii) inclusion of at least 50 patients.

A Restricted Maximum Likelihood (REML) mixed-effects model was applied to combine the log_e(PCEs) across trials. To evaluate the degree of heterogeneity, the I² index was estimated describing the percentage of total variation attributable to heterogeneity [21].

As outlined in the protocol, a number of pre-specified stratified analyses were performed: (i) time of outcome measurement, (ii) type of intervention, (iii) type of outcome, (iv) blinding of participants and treatment providers, (v) blinding of outcome assessor, (vi) allocation concealment, (vii) risk of bias, (viii) information to participants, and (ix) trial settings. Furthermore, the patient’s condition (i.e., chronic or non-chronic) was assessed at trial-level (based on the trial’s eligibility criteria).

Meta-regression analysis (REML models) was conducted involving covariates at trial-level to investigate whether individual covariates could explain heterogeneity (i.e., reduce variability) of the PCE among studies. These variables were (i) patient age, (ii) proportion of females, (iii) sample size, and (iv) year of publication. A meta-regression analysis was also performed to investigate the association between PCE and the corresponding placebo effect (i.e., SMD), as reported in the Cochrane review [14]. Small-study bias was examined using a funnel plot and Egger’s test [22]. Furthermore, based on the various risk-of-bias assessments for each trial, pre-specified exploratory sensitivity analyses were conducted in order to assess whether possible biases could affect the estimates. Trial characteristics, such as overall risk of bias, sample size [23], and trial settings (single-center or multicenter trial) [24], were further used in the pre-specified analyses to evaluate possible bias across studies. P values were obtained as part of the meta-regression analysis; i.e., based on F tests if more than two groups were compared and t tests if only two groups were compared. All statistical tests were performed using STATA/IC 15.1 (Stata Corp LLC, TX, USA).

All 202 trials included in the main analysis of the Cochrane review by Hróbjartsson and Gøtzsche [14] were screened for inclusion (Fig. 2). Five trials [25‐29] were not in English. The remaining 197 trials were read in full text, of which 7 trials [30‐36] did not meet the inclusion criteria. Relevant outcome data were not accessible in 4 trials [37‐40], leaving 186 trials for inclusion in the meta-analysis.

The included trials were conducted in 23 countries and published between 1961 and 2008, with the majority published after 1994. There were 11 crossover trials [41‐51], of which 10 (total of 250 patients) had no data available from the first period only, and thus were handled as parallel trials by using summary data (ignoring the design). There were 54 trials (total of 5,160 patients) having more than three arms. In 18 of those [52‐69], one or more additional arms were disregarded (a total of 796 patients) in order to calculate the PCE for groups that had comparable interventions. In the remaining 36 trials with more than three arms, all arms were relevant and therefore included which left a total number of 16,655 patients for the meta-analysis.

Trial characteristics are summarized in Table 1. The mean age of participants ranged from 0 (infants) to 83.8 years, with an average mean age of 39.9 years (age was not reported in 11 trials). The percentage of females ranged from 0 to 100, with mean percentage of 58.6 (female/male proportions was not reported in 23 trials). Patients included in the trials had a broad spectrum of conditions, such as depression, hypertension, obesity, or headaches. Pain was the most common outcome measure (reported in 53 trials). Other frequent measures were for anxiety, medication use, smoking cessation, and nausea. Interventions were categorized into three types: pharmacological (e.g., medication or supplements given orally or via injection/inhalation), physical (e.g., acupuncture or transcutaneous electrical nerve stimulation), and psychological (e.g., cognitive therapy or hypnosis). In 22 trials, patients were not informed that the study involved placebo. There were 97 single-center trials and 33 multicenter trials (unclear in 56 studies). Explicit trial characteristics are presented in Additional file 3.

Methodological characteristics were assessed by Hróbjartsson and Gøtzsche [14] and are listed in detail in the original Cochrane review. In summary, adequate sequence generation was reported in 39 trials (21%); 28 trials (15%) reported adequate concealment of allocation; 59 trials (32%) were judged to have adequate blinding of patients and providers; and 81 trials (44%) to have adequate blinding of outcome assessor. Overall, 81 trials (44%) had a sample size of at least 50 patients, and 80 trials (43%) had a dropout rate of 15% or lower. Following the simple risk-of-bias assessment proposed in the Cochrane review, 16 trials [70‐85] (9%) fulfilled all three criteria and were therefore judged as having low risk of bias among the included trials.

As illustrated in Additional file 4, there was a considerable degree of heterogeneity across PCE’s from various trials (I²=93.6). The pooled PCE across all 186 trials corresponded to 54% of the observed effect (0.54, 95%CI 0.46 to 0.64). There was statistically significantly higher PCE in trials with pharmacological and physical interventions, compared to psychological, and PCE was non-significantly higher for patient-involved outcomes (patient-reported outcomes and observer-reported outcomes dependent on patient cooperation). Trials where patients were not informed that the study involved placebo had non-significantly higher PCE. Trials with binary outcomes had statistically non-significant, but potentially higher PCE’s compared to trials with continuous outcomes. None of these factors, however, explained much of the observed heterogeneity. Publication year and sample size did not have any effect on heterogeneity either, and no association was found with PCE (see Additional file 5). There was a significant association between PCE and mean age and percentage of females, where PCE decreased with higher mean age of participants (slope = 0.986; 95%CI 0.976 to 0.995) and increased with higher percentage of females (slope = 1.006; 95%CI 1.000 to 1.012). However, only the proportion of females in the trial populations seemed to explain some of the between-study variance (reduction in τ² = 14.1%, I² = 92.5); see Table 2.

Neither patient’s condition/diagnosis (e.g., fibromyalgia, cancer, smoking; increase in τ² = 4.5%, I² = 93.8) nor outcome domain (e.g., pain, nausea, smoking cessation; increase in τ² = 0.7%, I² = 93.5) was an important factor in reducing the between-study variance. However, whether the condition was chronic or not reduced between-study variance slightly (slope = 0.69; 95%CI 0.49 to 0.96; reduction in τ² = 3.0%); chronic conditions had a significantly lower PCE.

As illustrated in Fig. 3, there was an association between PCE and the corresponding placebo effect, estimated as SMD and reported in the Cochrane review [14], where PCE increased with an increasing placebo effect (slope = 1.55; 95%CI 1.07 to 2.24).

The risk of small-study bias across trials was assessed using funnel plot, followed by Egger’s test [22]. The funnel plot was asymmetrical (Fig. 4), which Egger’s test confirmed (p < 0.001). Small studies tended to show smaller PCE. Furthermore, the funnel plot showed a vertical line of points (ln[PCE] = − 4.6) that indicated the trials where either the intervention group or the placebo group showed no improvement or got worse, requiring adjustment in the form of the described imputation of a small treatment effect. For sensitivity, when comparing the fixed-effect estimate for the PCE to the random-effects estimate, the PCE increased from 0.54 (95%CI 0.46 to 0.64) to 0.82 (95%CI 0.80 to 0.83). This indicates a discrepancy between random and fixed-effect models. However, a subsequent visual inspection of the funnel plot did not indicate an important small study bias.

When conducting the pre-specified sensitivity analyses according to the risk-of-bias assessment, the blinding of the outcome assessor was the most important factor, being associated with a significantly higher PCE. Furthermore, the PCE was significantly higher in trials with concealed allocation, and there was a non-significant increased PCE in trials with an overall low risk of bias. Neither trial setting (single-center or multicenter) nor sample size had an important effect on PCE or explained much of the reported heterogeneity, with blinding of outcome assessor apparently being the most important factor (reduction in τ² = 5.8%).

What influenced the heterogeneity the most were the 18 trials [41, 46, 48, 86‐100] (including 890 patients) where either the intervention or placebo group showed no improvement or got worse; these trials required imputing a very small treatment effect in order to calculate PCE so they could be included in the meta-analysis (reduction in τ² = 80.6%, I² = 84.4). Due to this imputation technique, a sensitivity analysis was performed where the aforementioned 18 trials were excluded.

A meta-analysis of the remaining 168 studies (total of 15,765 patients) resulted in PCE of 0.72 (95%CI 0.67 to 0.79), τ² = 0.2080, and a slightly lower heterogeneity (I² = 85.7). Many of the trends apparent in the main analysis (e.g., higher PCE in trials with concealed allocation) diminished or disappeared (see Additional file 6). Ten of the 18 excluded trials had psychological interventions [87‐90, 92, 93, 95, 97, 99, 100], which led to a radical shift in PCE for the remaining trials with psychological interventions. This shift in PCE, in turn, led to a non-significantly higher PCE in trials with physical interventions compared to pharmacological and psychological interventions. The PCE was still higher in trials with concealed allocation, blinded outcome assessor, and overall low risk of bias, but only the blinding of outcome assessor resulted in significantly higher PCE (slope = 1.22; 95%CI 1.03 to 1.44). The significantly higher PCE in trials of non-chronic conditions diminished and was no longer significant. None of these factors were important in explaining the between-study variance. Neither sample size, publication year, whether trials had binary or continuous outcomes, mean age of participants, nor percentage of females seemed to have any effect on heterogeneity, and no association was found with PCE in the sensitivity analyses.

For the sensitivity analysis of the association between PCE and placebo effect, an additional five trials with binary outcomes were excluded because in these trials, no improvement was seen in the no-treatment group. In order to be eligible for inclusion in the synthesis, these trials needed adjustment (0.5 added to each cell of the 2 × 2 table) prior to calculating the OR, and afterwards SMD. The association found earlier between PCE and placebo effect diminished and was non-significant after the exclusion of all 23 trials that needed adjustment. The association between PCE and placebo effect (SMD) for the sensitivity analysis is illustrated in Fig. 5.

A new funnel plot and Egger’s test was conducted for the remaining 163 trials in the sensitivity analysis, which also resulted in an asymmetrical funnel plot (Additional file 7; Egger´s test, p < 0.001), suggesting that small studies report smaller PCE.

This study shows, based on 186 trials, that the majority (average PCE: 54%) of the overall treatment effect of diverse interventions across conditions was attributable to contextual effects. Several contextual factors and trial characteristics were found to impact the observed variation in PCE. The factors that increased the PCE, and thus are potentially valuable, were adequate allocation concealment, blinded outcome assessor, lower mean age of participants, higher proportion of females, larger placebo effect, and trials of non-chronic conditions. As anticipated the PCE was higher in trials with low risk of bias and using patient-reported outcomes. The proportion of females was the only pre-specified covariate that significantly reduced the between-study variance; other factors had only minimal impact on the between-study variance.

The sensitivity analysis showed insignificant variation of PCE for the most part; only blinding of outcome assessor remained significant. No factors were found to significantly reduce the between-study variance, and no association was found between PCE and mean age, percentage of females, or placebo effects, respectively.

Our findings support those reported in earlier PCE study by Zou et al. [12], who also found larger PCE in studies with concealed allocation. Neither study could explain, however, whether this is due to greater contextual effects or smaller experimental (intervention group) effects. Also, our findings support those found by Whiteside et al. [19], who reported an increase in PCE with higher proportions of females. There are, however, two crucial differences between this study and previous PCE studies. First, the two previous studies examined a single condition, (osteoarthritis [12] and fibromyalgia [19]), where the primary outcome was pain. Second, both studies excluded trials where one or more groups did not improve or even worsened. Due to these exclusions, it might be more appropriate to compare the findings from the sensitivity analysis in this study to the previous PCE studies. The overall PCEs reported in previous studies are similar to the findings from the present sensitivity analysis (75% in osteoarthritis and 60% in fibromyalgia compared to 72% in the present study). Zou et al. [12] reported the lowest PCE for treatments delivered with oral medications and higher PCE for treatments delivered via physical means and needles/injections. In the present sensitivity analysis, there was also insignificantly higher PCE in physical interventions compared to pharmacological. The categories in this study, however, cover a broader range of interventions than the categories used by Zou et al. [12]; nevertheless, our study generally corroborates earlier findings. We found that there was a direct association between PCE and the placebo effect reported in the Cochrane review [14]. This association, however, was not apparent in the sensitivity analysis, meaning that the PCE was constant even though the placebo effect diminished.

There are several similarities between the findings in the Cochrane review [14] and the findings in this study. The Cochrane review reported a larger placebo effect in trials with patient-reported outcomes, in trials where patients were falsely informed that no placebo was involved in the study, and in trials with concealed allocation. Similar results were seen in the sensitivity analysis in the present study, where trials with patient-reported outcomes suggested a higher PCE compared to outcomes not dependent on patient involvement. Trials with concealed allocation had significantly higher PCE in the main analysis but not in the sensitivity analysis. These similarities between PCE and placebo effect reported in the Cochrane review indicate that the difference between the improvements seen in the intervention and placebo groups is less prominent in studies with a large placebo effect.

A major limitation of this study was the inclusion of studies with high risk of bias, mainly due to lack of sequence generation and allocation concealment in 79% and 85% of trials, respectively. Only 16 trials (9%) out of 186 fulfilled all three predefined criteria for low risk of bias. An updated literature search may have added more studies at low risk of bias, as the majority of the included trials did not describe factors such as sequence generation and allocation concealment adequately. Since there were trends for higher PCE in trials with a low risk of bias this could have affected the outcomes. Furthermore, the meta-analytic methods used did not allow for patient-and clinical-related variables changes over time. Many changes in the patient populations could be experienced in 47 years.

This study correlates with previous studies of bias, reporting that bias due to unblinded assessor or inadequate allocation concealment tends to overestimate treatment effect [101‐103]. Furthermore, a small-study bias was detected, whereby small-sample studies report smaller PCE. Based on the presence of small-study bias, and that studies with a lower risk of bias tended to have higher PCE, the PCE reported in this study could have been underestimated.

As described above, there was a considerably high degree of heterogeneity, which may have affected our study’s outcomes. Although subgroup analyses were carried out, the heterogeneity remained high, and no factors were able to explain the heterogeneity, except for the 18 trials that needed adjustment prior to analysis. Furthermore, not all the important contextual factors were included in the model. This could lead to uncontrolled confounding in the meta-analytic study, leading to biased effect estimates. Thus, the conclusions claimed in this article could be incorrect. A more in-depth sensitivity analysis would have been helpful to assess the magnitude of these biases. However, many contextual factors that are known to be important, such as the patient-provider interaction and patient’s expectations, were not considered because these factors are rarely reported in RCTs.

In contrast to earlier PCE studies, we decided to also include trials where either the intervention or placebo group showed no improvement or even worsened (i.e., where the PCE became negative). That decision might have been both a strength and a limitation. Earlier PCE studies excluded such trials because the measure of PCE does not allow negative values when the ratio is log-transformed, and worsening in a group could indicate side/nocebo effect, which is not the focus of interest for the PCE [12]. However, we viewed the exclusion of studies that did not fit to the pre-specified hypothesis as a source of bias; we therefore decided to add a miniscule effect to the groups that had not improved, in order to be able to perform the necessary calculations. This was the case for 17 trials with continuous outcomes and one trial with binary outcomes. Afterwards, it was clear these 18 trials had great impact on the results, and it was questionable whether this kind of adjustment was optimal for retaining the trials in our study.