How evidence-based is an 'evidence-based parenting program'? A PRISMA systematic review and meta-analysis of Triple P

There are a large number of published evaluations of Triple-P parenting interventions and we were able to identify 33 English-language studies which measured a child-based outcome and which compared Triple P interventions with a comparison condition. Most of the studies involved families who responded to media advertisements. These families clearly have children whose parents are finding difficulties with their behavior but may well not be typical of such families in the population. They are more likely to be motivated and literate and are sufficiently confident to present for treatment as volunteers. These characteristics would be likely to lead to high levels of compliance with treatment and better than average treatment response. Only five studies [5, 6, 21, 33, 34] did not rely upon self-referral by potential participants. All the studies involved only children over two years old. There are many forms of the Triple-P program [2], and new versions emerge regularly, but for simplicity we have not distinguished between the levels of intervention. Nevertheless, most of the studies reported on the effectiveness of small group-based Triple-P interventions, usually level 4 or 5, and synthesis of results from other Triple P levels would be limited by low numbers.

Most of the studies were relatively small and the great majority used a waiting-list control design in which the participants on the waiting list were offered active treatment immediately after the post-intervention data collection. It is, therefore, not possible to draw conclusions about the longer term effectiveness of Triple P relative to a comparison condition. Before- and after- data from the intervention group were usually presented more prominently (and frequently) than between-group comparisons, and this method of reporting often obscured interpretation of group effects and tended to increase the impression of a positive effect from the intervention. Only two trials used an active comparison group, and neither of these showed any advantage for Triple P in terms of child-based outcomes. Trials with waiting list controls or usual care provide intervention estimates which reflect the combined specific and non-specific effects that will accrue in practice, and are more likely to show between-group differences than trials with active controls [35].

A range of child-based outcome measures was used but the most commonly reported was the ECBI, completed by parents. In the majority of cases (31/33) the main informant was the mother, and these data were synthesized in our main meta-analysis. Despite some differences in methodological approach, the ES obtained in our meta-analysis of maternally-reported ECBI scores (0.61) is impressive and is broadly in line with that reported by other authors [9, 11, 12].

There is some evidence of publication bias from our analyses of the published work and there is additional evidence that results of a number of evaluations of Triple-P have not been published [30, 36]. Further evaluation of publication bias is not possible given the uniform lack of Triple-P trial registration. Thus, despite the apparent consistency of more recently published work, there is still the possibility that these studies represent a particularly favorable picture of Triple-P. The International Committee of Medical Journal Editors (ICMJE) agreed in 2005 that only registered trials would be considered for publication. Allowing for publication time lags, about one third of the studies in our meta-analysis pre-dated the guidance and two thirds could potentially have benefitted from adopting these recommendations.

There are considerable limitations in relying on maternal report data alone: 17 papers gave outcomes reported by informants other than mothers. Five of these studies showed a relative benefit for Triple P on one or more outcome measures compared with no treatment. Meta-analytic synthesis of paternally-reported data identified a pooled ES of 0.42 but there was significant heterogeneity and the overall effect size was not significantly greater than zero. Multiple outcomes were used in five of the six papers reporting significantly positive results: primary outcomes were not pre-specified (in common with all the reported trials), and corrections were not made for multiple comparisons. Paternal reports are often difficult to assess because of missing data which may not be missing at random. The incorporation of independent direct observations of parent and child behavior into trial design provides important confirmatory information [37, 38], and seven of the Triple P trials (see Table 4) included data from independent observers. Two of these seven studies reported benefit in one or more observational subscale.

It is possible that the discrepancy between maternal and paternal (or independent) reports of child behavior may be accounted for by the fact that maternal mental state improved significantly with most Triple-P interventions [10] and this may have led to a more positive maternal evaluation of the child's behavior, reflecting more optimistic states of mind. Fathers are less likely to attend sessions and independent observers are unconnected with the intervention. Related attribution effects have been reported in relation to Triple-P [39]. It is also possible that mothers are more accurate than fathers in reporting their children's behavior difficulties. One paper [13] reported a planned subgroup analysis for lone parent families - and reported no benefit from the triple P intervention.

All of the papers considered here, with one exception [26] were authored or influenced by Triple-P affiliated personnel. This is commonly observed in the early stages of development of non-pharmaceutical interventions but readers should interpret findings accordingly, particularly when authors may gain financially from the intervention under study. Although authors of Triple P interventions receive royalty payments from sales of training and materials [27], only one of the articles we obtained declared any conflict of interest. Conflict of interest may be of particular importance in interpreting studies (such as many of those reported here) in which subgroup analyses are reported [40]. Outcome reporting bias [41] may also be an important consideration in this respect, with possible significance for the interpretation of meta-analyses [42].

Claims that whole-population parenting programs have significant impact on public health are particularly important, because these may have led to substantial commitment of public funds. We were unable to find any convincing evidence of benefit from the Triple P program in the three whole-population studies eligible for inclusion in the present review.

The studies to date give proof-of-concept that group-based Triple-P may be effective in the short term according to maternal report of child behavior but, given the high risk of bias (or unknown risk of bias when reporting is poor), they do not support the view that Triple P provides other benefits to children.

The lack of convincing evidence of benefit from whole-population interventions is in line with previous work in which no significant improvement in child-based outcomes resulted from a public health parenting program [43] and with a more recent large-scale independent evaluation of Triple P in Zurich which demonstrated no impact on child behavior [44]. Along with findings from a previous systematic review [12], the results of our meta-regression support the view that some benefit might be achieved if interventions were focussed on the families of children with more severe problems. A recent Cochrane review of parent training interventions for children with established conduct problems and those at high risk of conduct disorder [45] provides robust evidence of the effectiveness of such targeted programs. It is, therefore, likely that an effective case-finding approach combined with offers of interventions to families with identified problems may be more effective than the 'public-health' approach.

Only one of the studies [26] included in our review had no apparent conflict of interest. We are aware of two further independent evaluations of Triple P which were ineligible for inclusion in our review - one published in German [46] and one recent large scale trial [44]. Both produced negative results. These findings mirror the frequently observed failure of independent replication of positive results from a range of developer-led studies. Theoretical models developed by Eisner [47], describing the mechanisms by which conflict of interest can lead to research bias, may help to explain this phenomenon.

Given the substantial cost implications, health care providers and policymakers would be well advised to apply the same standard of evidence when purchasing behavioral interventions as they do to the purchase of pharmaceutical agents or medical devices. Compulsory clinical trial registration and full and open declaration of conflicts of interest would address many of the deficits noted in our review. Pending the implementation of such mechanisms, unproven interventions should only be carried out in the context of a robust independent evaluation.

Developers and evaluators of psychological interventions should be encouraged to adhere to the guidelines dealing with good publication practice for communicating company sponsored medical research (GPP2 [48]). Journal editors and reviewers should be encouraged to adhere to CONSORT guidelines for both text and abstracts [49]: we believe that authors who choose journals that do not adhere to these guidelines, and editors who choose not to adopt them, do the field, as well as their own work, a disservice. There should be a clear expectation that instrument subscales should be reported in full and covered even-handedly in article abstracts. Care providers and policy makers should assess the generalizability of findings from socially advantaged and volunteer samples to their own situation. There is a need for registered, large, multicenter trials, with prospectively defined, long-term outcomes and active comparison groups rather than further evaluations using waiting list control groups. Rigorous systematic reviews of parenting interventions (for example [45]) attest to the importance of including data from independent observers in trials and in reviews, both in order to reduce risk of bias and to provide more convincing data on the effects of parenting interventions. Whole-population data on child behavior, reported by multiple informants, and linked to provision of parenting interventions, may be an alternative approach to the evaluation of public health parenting programs.

The study was unfunded, apart from a contribution made by the Gillberg Neuropsychiatry Centre to pay for statistical analysis. The funder played no part in the study design, analysis or interpretation of the data except insofar as one of the authors (CG) is employed within the Gillberg Neuropsychiatry Centre at the Sahlgrenska Academy, University of Gothenburg.