The relative efficacy of bona fide psychotherapies for post-traumatic stress disorder: a meta-analytical evaluation of randomized controlled trials

Eligibility criteria for studies to be included in this meta-analysis were (see [23]): (1) a randomized and controlled study design (RCT), investigating the relative efficacy of (2) at least two bona fide psychotherapies; (3) therapies needed to be conducted in two or more sessions; (4) participants were adults that were (5) diagnosed with PTSD according to the valid edition of the DSM at the time of the respective study (DSM-III or DSM-IV); (6) PTSD symptom severity was assessed with self-report or clinician rating. Studies that provided insufficient information to compute effect sizes were excluded as were component, dismantling, and parameter studies. Moreover, we also excluded studies that contained mere additional analyses on previously published data, ensuring that primary data entered analysis only once.

Target group, treatment format and study quality did not serve as eligibility criteria in the present study. In order to examine the possible effect of various patient and treatment characteristics as well as the influence of study quality, moderator analyses were conducted (see the section on Additional Analyses, below). Taylor and Harvey [32] examined the effects of psychotherapy specifically for people who have been sexually assaulted and Ehring et al. [14] for adult survivors of childhood abuse. Sloan et al. [19] and Bisson et al. [12] examined, amongst others, group therapies. However, other meta-analyses did not limit themselves to specific target groups or treatment formats [13, 16‐18, 20, 34].

To qualify as an ‘active treatment’, the bona fide definition [23, 29] required that treatments had to be delivered by a trained therapist who adapted the treatment to patients on the basis of a therapeutic relationship (i.e., no delivery of a non-modifiable standard protocol, e.g., progressive muscle relaxation); treatments also needed to be conducted personally and face-to-face (i.e., no online treatments or treatments conducted with, e.g., audio material). Moreover, at least two of the following four criteria had to be fulfilled with regards to their descriptions in the studies: (a) a citation to an established school or approach to psychotherapy; (b) a description of the therapy that contained a reference to a psychological process (e.g., operant conditioning); (c) a reference to a treatment manual that was used to guide the delivery of the treatment; (d) the identification of active ingredients of the treatment and citations for these ingredients.

Primary studies were identified in two ways. We obtained the 15 studies investigated by Benish et al. [23] and performed a literature search in the databases MEDLINE, Cinahl Health, Psychology and Behavioral Sciences Collection, PsycINFO, PsycArticles, PubMed, Social Sciences Fulltext, and Web of Knowledge, using the keywords PTSD, posttraumatic, post-traumatic, post traumatic in combination with psychotherapy and treatment, and also in combination with relative efficacy, outcome study and comparative efficacy. We refined the search results by restricting it to ‘adults’, ‘humans’ and ‘randomized controlled trials’ in some databases (see Fig. 1). In addition, we searched reference sections of previous meta-analyses of PTSD treatment, controlled studies of psychotherapy outcomes for PTSD, and literature reviews of PTSD treatment [31, 35, 36].

Moreover, we meticulously re-assessed the bona fide status of all treatments included in previous meta-analyses and of the treatments of the 27 studies included in the review of Cloitre [31]. This review was specifically mentioned by Ehlers et al. [30] in their criticism with regards to a biased study sample of Benish et al. [23].

Prior to study selection, descriptions of 112 treatments classified as bona fide in previous meta-analyses on PTSD [23], alcohol abuse [24], and depression [26] as well as descriptions of treatments not classified as bona fide in these study samples were screened to ensure that we applied the bona fide criteria correctly. Moreover, we closely adhered to the descriptions and examples provided by [29].

The second author extracted the data and coded the studies. Cases where coding appeared unclear were resolved by discussion and consensus between both authors. Moreover, coding of included studies was compared afterwards with coding provided in other meta-analyses for the same studies, where available [13, 16, 17, 32, 35]; in these cases, no differences in coding were observed. Wherever possible, intention-to-treat (ITT) analyses were used in meta-analytical computations.

Coded and extracted data items pertained to (a) study characteristics: publication year, type of publication, country, type of analysis (ITT or completer only); (b) patient characteristics: total number, number per treatment arm, percentage of women, mean age, type of trauma, mean time since trauma, PTSD symptom severity prior to treatment, percentage of comorbid disorders (depression, anxiety, substance use disorder), site of patient recruitment (community, clinical, community and clinical mixed, other), use of psychotropic drugs; (c) treatment characteristics: type of treatment, treatment format (individual vs. group), treatment length, total number of sessions, session length; and (d) all relevant data with regards to the primary and secondary outcomes, clinically significant change, and dropout (see the section on Summary Measures, below).

Risk of bias was evaluated on the basis of the seven criteria of the Practice Guidelines from the International Society for Traumatic Stress Studies [37], complemented by standards of high-quality studies proposed by Cuijpers et al. [38]. Study quality was evaluated with respect to whether (1) randomization was conducted adequately and independently from researchers and therapists (by using, e.g., computer software); (2) assessment was blinded; (3) studies reported reasons for dropout or analyses on differences between dropouts and completers; (4) therapists were specifically trained in the provided treatment; (5) adherence to treatment manuals or protocols was ensured with adequate measures or checked empirically; (6) ITT-analyses were conducted and adequately reported; (7) a treatment manual or protocol existed. Items were coded 0 = no, 1 = yes, and summed, resulting in an omnibus measure of study quality that ranged from 0 to 7. Additionally, we also investigated criteria (1), (2), and (3) separately. These criteria agree closely with the Jadad et al. [39] criteria¹ that are widely-used in medical research and that were already also used in psychotherapy research [16, 22]; with regards to blinding, we considered only blinded assessment; double-blind trials, as proposed by Jadad et al. [39] are adequate in medical research, but are generally not feasible in psychotherapy research.

First, between-group differences in primary and secondary outcomes (see below) were meta-analysed at post-treatment and, where available, at follow-up. Benish et al. [23] restricted their analyses to post-treatment differences. All direct measures of PTSD symptom severity (e.g., Clinician Administered PTSD Scale [CAPS], Impact of Events Scale [IES], PTSD Symptom Scale Self-Report [PSS-SR]) were considered primary outcomes. This was in contrast to Benish et al. [23] who considered measures of anxiety (e.g., State-Trait Anxiety Inventory [STAI]) primary outcomes as well. Anxiety is one of the key symptoms of PTSD and anxiety disorders are among the most frequent comorbid disorders of PTSD [37]. However, measures of anxiety do not address the full range of PTSD symptoms. Hence, they were not considered primary outcomes in the present study. On the other hand, Benish et al. [23] computed a second effect size, aggregating all post-treatment outcome measures that were reported in individual studies. As these included primary outcomes as well, there was a substantial overlap of this second effect measure with the primary outcome measure. Instead of following such an approach, we considered all measures of symptom severity of comorbid disorders (i.e., anxiety, depression, substance use disorder), trauma-related symptoms (e.g., trauma-related guilt), general symptom distress, social functioning, and quality of life as secondary outcomes, thus having a true second measure for a more differentiated analysis, see [6, 9, 10]. Following [12], we also distinguished primary outcomes with regards to self-reports and clinician ratings. While the meta-analysis of Van Etten and Taylor [20] did not find any differences in efficacy with regards to assessment, others did [32, 40]. A complete list of primary and secondary outcome measures is displayed in Appendix (Tables 5 and 6).

Second, additional outcomes pertained to treatment differences in percentages of clinically significant changes and dropouts at post-treatment and, where available, at follow-up. Where available, recovery rates were used to assess clinically significant change. Otherwise, rates of participants improved were utilized. As in the analysis of primary and secondary outcomes, data of ITT analyses were used, where available. Dropout rates were calculated either on patients dropping out of treatment or on the basis of ITT participant numbers (i.e., after randomization).

Based on descriptions and references given in the respective studies, treatments were categorized into specific types that were in line with previous meta-analyses [13, 16, 17] and a review [31]. Treatment categorization in these studies followed a bottom-up strategy, sorting and clustering treatments according to their provided descriptions consistently into groups. The present meta-analysis also adhered to this bottom-up treatment categorization scheme, adhering as closely as possible to treatment descriptions in studies for the categorization of treatments. Similar to previous meta-analyses of PTSD comparing several treatment categories, categories resulting from this process were sometimes narrower and sometimes broader, depending on whether therapies followed a specified treatment manual (e.g., eye movement desensitization and reprocessing [EMDR] [41]), or were combined because of a similar theoretical approach (e.g., exposure therapies). In order to maximize statistical power, we were careful not to narrow categories down too much, as this would have yielded a large number of categories and only few studies per category. To compensate for this broadening of categories, we conducted also meta-analyses for distinct subgroups (see below) and controlled for outliers.

Types of treatment included in the analyses were: Exposure therapies (EX), prolonged exposure (PE [42]), and both combined (EXPE), EMDR [41], cognitive behavioural therapies (CBT), and present-centered therapies (PCT). PCT included bona fide psychotherapies that were mostly linked to elements of Roger’s client-centered psychotherapy [43] and encompassed supportive, present-centered, problemsolving, and psychoeducative elements. Treatments coded as CBT included some therapies that utilized, but did not primarily rely on, elements of other therapies (e.g., exposure [44, 45]), and treatments such as Seeking Safety [46] or Image Rehearsal [47].

All treatments were coded whether they were trauma focused (TF) or not (NTF). TF treatments were defined as focusing on the memory for the traumatic event and/or on trauma-related appraisals [12]. Najavits and Hien [48] recommended to name non-trauma focused therapies for what they are providing and not just to qualify them for what they are not providing. Nevertheless, we adhered to TF/NTF distinction as it is frequently used in this field of research. The TF/NTF distinction served to strictly compare two categories on the same level of abstraction and to address the question of whether trauma focused treatments were more efficacious than non-trauma focused ones. Again, treatment descriptions in studies were utilized as closely as possible to code whether treatments were trauma focused or not. Within the CBT category we also examined the efficacy specifically of trauma focused treatments that included elements of exposure (TFCBT + EX).

For primary and secondary outcomes, Hedges’ g was used as effect size measure, similar to previous meta-analyses [16‐18, 32], but in contrast to Benish et al. [23] who used Cohen’s d. Hedges’ g is widely recommended in the meta-analytical literature to control for the overestimation of effects with Cohen’s d in small samples and its use is also recommended in the Practice Guidelines from the International Society for Traumatic Stress Studies [37]. In cases where studies reported more than one outcome (primary or secondary), effect sizes were obtained for each outcome and then averaged within outcomes; dependency between outcomes was accounted for by assuming a correlation of .50 in the computation of the variance of the averaged effect size as in Benish et al. [23]. In cases where both full-scale and subscale scores of a measure were reported, full-scale scores were used. In cases where only subscale, but not full-scale, scores were reported, published coefficients on the intercorrelation of subscales was used instead of assuming a uniform correlation of .50 in computing the averaged effect size variance (e.g., for the IES–revised form, coefficients were taken from [49], p. 1491). For additional outcomes, percentages of clinically significant changes and dropouts, risk ratios (RR) were computed and used in analysis.

For all analyses, random-effects models were utilized, assuming that treatment differences were not represented by a single population effect. With regards to analyses themselves, two independent approaches were utilized: (1) conventional meta-analysis of direct comparisons, contrasting specified types of treatment against all other available treatments, restricted to types with five or more comparisons, see [38], contrasting also trauma focused versus non-trauma focused treatments, and contrasting further all exposure-based treatments (EXPE) against all other treatments. The conventional analyses comprised comparisons regarding primary and secondary outcomes, clinical significance, and dropout as well as analyses of publication bias, sensitivity analyses, and moderator analyses; (2) the Wampold homogeneity test (WHT) with regards to primary and secondary outcomes at post-treatment [4, 50].

For conventional meta-analyses, the software metafor [51] in R was used. Heterogeneity was tested with the Q test and assessed with the I ² statistic that describes the ratio of true heterogeneity to the total observed dispersion with a range of 0 % to 100 %, reporting also 95 % confidence intervals. Heterogeneity was assumed small for I ² = 25 %, moderate for I ² = 50 %, and high for I ² = 75 % [52].

With regards to the interpretation of standardized mean differences, i.e., Hedge’s g, we adhered to benchmarks of Cohen [53] interpreting g = 0.20 as small, .50 as medium, and 0.80 as large effects. For the interpretation of whether effects are likely clinically meaningful, we adhered to Ferguson [54] and the NICE Guideline Development Group [12] who proposed thresholds of g = .41 and g = .50, respectively. With regards to the interpretation of risk ratios pertaining to recovery/improvement rates and dropout, we adhered to thresholds devised by the NICE Guideline Development Group [12], regarding risk ratios ≤ 0.80 or ≥ 1.25 as clinically meaningful. Where 95 % confidence intervals of risk ratios included these thresholds, results were interpreted as ‘limited evidence’ of differential efficacy. Significance was set to p < .05.

The omnibus hypothesis, whether there were any differences in treatment efficacy overall, regardless of treatment categorization, was investigated with the WHT. The WHT requires in a first step the random assignment of a positive (+) or negative sign (−), in equal proportion [4], to the effect sizes of the head-to-head comparisons. The aggregated effect of the randomly aligned treatment differences will approach zero if k, the number of comparisons, is large, but this is not of direct interest in the WHT. In the second, crucial, step, the homogeneity of the obtained effect size distribution is investigated via the Q test. If the null hypothesis of no overall treatment differences is not true, then there should be a disproportionate number of observations in the tails of the distribution [4]. Therefore, if homogeneity has to be rejected, the null hypothesis of no overall treatment differences must also be rejected. This null hypothesis has been tested with the WHT in a number of previous analyses [4, 23‐25] and has never been rejected.

However, two aspects may compromise the validity of the WHT: (1) the Q test is known for its low test power when k is small [52]; in the meta-analysis of Benish et al. [23] k was 17, heterogeneity may thus have been overlooked in previous research; (2) the WHT relies on only one run of random sign assignment. As stated above, random assignment need not result in an aggregated effect size of null; the probability of this event approaches unity only for k → ∞. Calculation of the Q statistic depends on the aggregated effect size. Random fluctuations of the aggregated effect size thus affect also the Q test of the WHT. Performing the WHT only once for a given dataset may therefore capitalize on random fluctuations whose possible effect is greater for small k. In conclusion, the Q test of the WHT cannot be considered an exact test, and Q values, as well as I ² values derived from this Q statistic, cannot be considered exact statistics of effect size heterogeneity.

In the present study, we sought to overcome this problem by applying the WHT repeatedly on 30 independent runs of random sign assignments, which can be thought of as a Monte Carlo permutation test. Monte Carlo permutation tests are akin and asymptotically equivalent to (exact) permutation tests (i.e., Fisher’s exact test) which obtain the distribution of a test statistic under the null hypothesis by calculating all possible values of this statistic by rearranging (i.e., permuting) the observed data in all possible ways. P values are obtained by computing the proportion of test statistic values that are greater than the test statistic of the observed data. Monte Carlo permutation tests sample only a number of permutations and do not perform all possible permutations. In the present case and with regards to the WHT, no clearly defined test statistic could be computed for the observed data: as explained above, the Q statistic of the WHT is itself not exact, but intrinsically fuzzy and subject to random fluctuations. However, the principles of Monte Carlo permutation tests could be applied to sample the distribution of Q values of the WHT and to obtain the number of significant Q tests for a given data set. Instead of relying on the p value of a single WHT, we counted the number of p values that were less than .05 among 30 replications of the WHT. Assuming the null hypothesis of no overall treatment differences is true, we expected 1.5 significant results (5 % of 30) with a nominal α of .05. To determine whether the null hypothesis of the WHT had to be rejected, the obtained number of significant results among the 30 replications was compared to this expected value, using a binomial test. This test had a power of more than 95 % to reject the null hypothesis of the WHT of no overall treatment differences, if the true number of significant Q values was at least 7.5 (i.e., 25 % of Q tests were significant).

Publication bias in meta-analysis has to be investigated in order to check whether studies in the analysis are representative of the overall population of completed studies. A systematic bias could occur either because of one-sided search-strategies or because small studies or studies without significant results were not published on a regular basis. Publication bias was investigated with the trim-and-fill procedure [55] with regards to all meta-analyses of direct comparisons, using metafor in R.

In addition to the 15 studies included in Benish et al. [23], we retrieved seven further studies that fulfilled the inclusion criteria, resulting in a total sample of 22 studies [42, 44‐47, 57‐73] (as the re-assessment of the study selection is an important topic of interest, see Additional file 1 for full references of the excluded studies). In sum, 161 abstracts were thoroughly screened and 46 studies assessed in full-text for eligibility, see Fig. 1.

Studies were excluded because they (reference numbers pertain to the reference list in Additional file 1: (a) compared treatments that were not bona fide [1‐5]²; (b) were component or dismantling studies [6‐12]³; (c) reported preliminary analyses of studies that were not included in the present meta-analysis themselves [13]; (d) reported additional analyses on data of studies already included in the present meta-analysis [14‐17]; (e) did not examine adults and/or did not ascertain a PTSD diagnosis according to DSM-III or DSM-IV [18‐20]; (f) included treatments that were delivered in a standard protocol that could not be adapted to individual patients [21‐23]; (g) examined only two participants in one treatment arm [24]. This study [24] was excluded on statistical grounds as estimates of population variance, on which Hedges’ g depend, are undefined when n < 3.

From the studies included in the review of Cloitre [31], only 3 studies remained which fully met the inclusion criteria and therefore were also included in the present meta-analysis [46, 65, 71]. Deviating from Benish et al. [23], the Schnurr et al. [71] study was included. This study investigated a PCT about which Wampold et al. [29] argued it was not bona fide because it bore little resemblance to PCT commonly applied in practice. However, according to the information provided in the study itself, the treatment fulfilled enough criteria (b, c, and d; see the section on Eligibility, above) to be considered bona fide. Moreover, Benish et al. [23] included another study of the same authors [70] that contained basically the same treatment, only delivered in a group format. On these premises, the Schnurr et al. [71] study was therefore included.

Basic characteristics of the 22 included studies, quality ratings, and individual effect sizes of primary and secondary outcomes are displayed in Table 1. Overall, 1694 patients participated in the studies. Studies were published between 1989 and 2010 and 18 (82 %) were conducted in English-speaking countries, 11 of which in the United States. Patients’ mean age ranged from 21.5 to 59.4 years, seven studies contained only women (total n = 664), three studies only men (total n = 460), in the remaining 12 studies (total n = 570) on average 56.0 % of patients were female. Twelve studies included patients with a specific trauma history or otherwise specific patient characteristics (total n = 1175): Childhood sexual abuse and adult rape victims (five studies, total n = 305), war veterans (four studies, total n = 744), fugitives (two studies, n = 51), comorbid substance use disorder (one study, n = 75). Fourteen studies provided information on time since trauma, which ranged from 1.2 to 22.9 years, with a mean of 7.1 years. With regards to PTSD symptom severity prior to treatment, eleven studies included severely traumatized patients (total n = 902; CAPS: 60–79, IES > 43), five studies included extremely traumatized patients (total n = 556; CAPS > 79). The remaining six studies did not report on severity or used instruments that provide no cut-offs with regards to severity (i.e., PDS, PSS-I, PSS-SR, PCL).

Details regarding the categorization of treatments in the present meta-analysis and in previous meta-analyses are reported in Appendix (Table 7). The psychodynamic therapy investigated in Brom et al. [57] was neither categorized with regards to type of treatment, as it did not fit with any type, nor included in direct head-to-heads comparisons of TF and NTF treatments as it remained unclear whether or not it was trauma focused [29]; it served only as an available comparator in meta-analyses of direct comparisons of specific treatment types. The hypnotherapy investigated in Brom et al. [57] was categorized as EX, as the description in the study explicitly stated that “The emphasis of the hypnotherapists in our study was on behavioral therapy. The goal was to bring the patient in contact with the reality of the traumatic event and to bring about a decrease in the conditioned responses triggered by the event” (p. 607) and that hypnosis was used to facilitate this goal (meta-analyses [12, 13] coded this treatment as ‘other’ treatment). The Hien et al. [46] study investigated the efficacy of Seeking Safety, a manualized CBT that addresses both PTSD and substance use disorder, comparing it with another CBT (relapse prevention) that addresses only substance use disorder. Both treatments were coded as CBT, but in the CBT related meta-analyses only Seeking Safety was pooled with the other CBT treatments for PTSD (and contrasted with relapse prevention), as relapse prevention did not directly address the PTSD symptomatology.

Twenty-four head-to-head comparisons were reported in the 22 studies; one study compared three treatments. There were 15 comparisons involving CBT (of these, 10 were TFCBT + EX), seven involving EX, seven involving PE, six involving EMDR, and six involving PCT. Therapies were conducted in 12 sessions that lasted 90 min on average. Only four (8.9 %) treatments were carried out in a group format. Follow-up assessments were reported in all (100 %) studies with regards to at least one of the defined outcomes; time of follow-up ranged from 1 to 15 months after end of treatment, 4.7 months on average. Nine (40.9 %) studies reported a second follow-up, ranging 6 to 24 months after end of treatment, 10.0 months on average.

Ratings of study quality are presented in Table 1. Most studies reported reasons for dropout or reported analyses of differences between completers and dropouts, ensured adherence to treatment manuals or protocols, and provided a treatment manual or protocol (18 [81.8 %] studies each). Fifteen (68.2 %) studies each reported blinded assessment and whether therapists were specifically trained in the provided treatment. Only eight (36.4 %) studies reported adequate randomization, whereas only seven (31.8 %) intention-to-treat analyses. Jadad ratings ranged between 0 and 4 points, with 2.2 points on average.

In head-to-head comparisons between trauma focused treatments and non-trauma focused treatments, this meta-analysis revealed a significant difference in efficacy. Trauma focused treatments were slightly more efficacious with regards to PTSD symptom severity at post-treatment (g = 0.14), at the first (g = 0.17) and the second follow-up (g = 0.23). This result corresponds with results of other meta-analyses [12, 14], but this time based strictly on head-to-head comparisons of bona fide therapies only.

The results of this meta-analysis with regards to the specific types of therapy corroborate at a first glance that bona fide psychotherapies of PTSD appear mostly similar with regards to their efficacy [23]. There was no strong indication of cognitive behavioural therapy, exposure therapies, prolonged exposure, EMDR, or of trauma focused cognitive behavioural therapy to be more efficacious than any of the other available therapies. However, sensitivity analyses revealed that some of these null findings could be traced to low quality studies. Excluding these studies, we obtained evidence of an advantage of prolonged exposure and exposure therapies (g = 0.19) over other treatments considering PTSD symptom severity at post-treatment, and limited evidence of a higher efficacy of prolonged exposure (RR = 1.26) regarding recovery rates at post-treatment, cf. [16]. PCTs appeared somewhat less efficacious than other treatments with regards to PTSD symptom severity at post-treatment (g = −0.20; adjusting for publication bias) and at follow-up (g = −0.17), mirroring recent findings with regards to a somewhat diminished efficacy of bona fide supportive therapies also in the treatment of depression [27]. However, PCTs were equally effective with regards to secondary outcomes as their available comparison treatments. No significant and clinically meaningful differences with regards to dropout were observed. All differential treatment effects were at most only small in size [53] and far below proposed thresholds for clinically meaningful differences [12, 54].

Application of the omnibus test of overall effect size heterogeneity (see Benish et al. [23]) indicated that the null hypothesis of no treatment differences had to be rejected in the full study sample; effect size heterogeneity was larger than expected by chance. Similarly, this null hypothesis had to be rejected in the study sample of Benish et al. [23]. Our analysis was based on an improved application of this analytic method, using a Monte Carlo permutation approach and basing our conclusions on 30 independent runs of the Wampold homogeneity test, and differed also by utilizing a stricter definition regarding primary outcomes. Despite these differences, the obtained results cast doubt on the results of the Benish et al. [23] meta-analysis, but also on other meta-analyses that used this method [4, 24, 25, 33]. We thus caution on the inconsiderate application of the omnibus test of overall effect size heterogeneity and recommend a Monte Carlo permutation approach, as used in the present study, also in future applications.

Substantial effect size heterogeneity was also apparent in conventional meta-analyses of specific treatment types and sensitivity analyses revealed that it could be traced to a number of influential studies. The Schnurr et al. [71] study was identified as influential, being subject to controversy before [29, 30], regarding the bona fide status of the PCT investigated in one arm of this study. Yet, we also identified a number of other, apparently less controversial, studies. Some of these [45, 59, 70] had a higher study quality and often larger patient samples than remaining studies, indicating that their exclusion may not be considered warranted. Studies of Ironson et al. [62] and Resick et al. [68] were of comparably lower quality, suggesting that their exclusion may be considered warranted.

Excluding influential studies, exposure therapies and prolonged exposure appeared more efficacious than other treatments with regards to PTSD symptom severity and recovery rates, whereas PCTs appeared equally efficacious as other treatments with regards to PTSD symptom severity. As the latter finding depended on the exclusion of large and high-quality studies [70, 71], we deem them likely biased. Moreover, systematic effects of study quality were also observed in comparisons involving cognitive behavioural therapy; differences at post-treatment and follow-up were larger in studies with lower quality. Lastly, we obtained evidence of publication bias, that, when adjusted for, revealed as its most significant result that PCTs were robustly less efficacious than other treatments with regards to PTSD symptom severity.

Study quality appeared thus influential with regards to aggregated tests of differential efficacy of bona fide psychotherapies of PTSD. However, based on the available evidence, obtained results appeared fickle and need to be interpreted with caution. Even though another seven studies over-and-above the 15 studies investigated by Benish et al. [23] were identified and included in the present meta-analysis, as much as nine (41 %) of all included studies examined less than 20 patients within one or both treatment arms, and another five (23 %) less than 30 patients; study quality appeared, overall, only mediocre. In order to be able to draw firmer conclusions, more high-quality studies using ITT analyses are still needed in this field of research.

As the current debate on the relative efficacy of PTSD treatments focused a lot on the topic of whether individual studies should be included or excluded in meta-analysis [29, 30], the sensitivity analyses reported here show that this factor could actually be decisive in terms of the overall meta-analytical result regarding equal or different efficacy of compared treatments. Concerning the consequences with regards to the dodo bird verdict and the question of what works in psychotherapy, we thus recommend a refinement and clarification of the bona fide definition which should ideally result in a clear decision rule with regards to the inclusion and exclusion of especially supportive and present-centered therapies in analysis. Otherwise, the selection of study samples may overly depend on the theoretical position of the study authors, or meta-analyses from authors with different theoretical positions might not be adequately comparable.

Using additional terms in the literature search, like the names of specific psychotherapies (e.g., “CBT”, “exposure therapy” or “EMDR”), might have provided additional studies that were overlooked using the present search terms.

Treatment categorizations adhered to the descriptions provided in the studies themselves, which may have introduced spurious heterogeneity with regards to categorization and may have impacted on our results. Moreover, there were too few studies for most of the specified treatment types to systematically test effect moderation with moderator analysis. Thus, effect size heterogeneity that could not be traced to influential studies could mostly not be examined with regards to proposed moderator variables. This affected specifically meta-analyses on EMDR, where heterogeneity was substantial and could not be traced to single influential studies.

The assessment of eligibiliy criteria, in particular the bona fide status of treatments, is still a major point of discussion. Hence, independent codings of a second rater, in addition to comparing codings of the present study with codings provided in other meta-analyses, might have benefitted the reliability of our decisions to include or exclude studies in the present meta-analysis.

Studies were very variable with regards to time of follow-up. Bradley et al. [13] suggested limiting comparisons at follow-up to assessments that took place six or more months following end of treatment. In the present study, all available follow-ups were used for analysis, regardless of exact time of follow-up, which may have influenced results. Specifically with regards to longer follow-up periods results could be confounded, because patients might have sought additional, external, treatment. This needs to be controlled both in primary studies and meta-analytical investigations. However, not enough information was available in primary studies to investigate this topic in the present meta-analysis.

Distinguishing different types of recovery and improvement might accentuate differences in efficacy, particularly because improvement is defined by study authors in different ways [13, 74]. As only nine studies reported recovery rates at post-treatment, combining recovery and improvement rates allowed the assessment of clinical significance, but may have influenced results.

Effects of researcher allegiance were neither examined nor controlled for in the present study. Researcher allegiance may affect differences in treatment efficacy and was reported to explain 12 % of variation in outcome regarding various trauma focused treatments [33]. However, allegiance failed to explain differences of efficacy in a number of other meta-analyses [22, 27, 75], and it is not clear whether allegiance ratings reliably reflect true differences in allegiance [76] or how well meta-analytical reviews may adjust for it.

Considering public health issues, a variety of topics not considered here may still have to be evaluated with regards to the comparative efficacy of treatments [48]. Cost of treatments, the level of education and the intensity of training of therapists, the preferences of patients and the applicability of treatments for a wide range of patients may influence what works best for patients as well as what is affordable to implement in public health care. For example, present-centered therapies could be easier to deliver at a lower cost to more varied patients with a less skilled workforce, and these advantages might compensate for a lower efficacy with regards to health outcome. More research is needed here.

More research on traumatized populations with severe comorbidities, like psychosis (mostly an exclusion criteria in PTSD studies), as well as on patients with additional and multiple life burdens and comorbidities, such as homelessness, unemployment, suicidality or with a criminal justice involvement, is also needed [48]. Information on the efficacy of PTSD treatments in such highly vulnerable populations is currently lacking.