How big is the effect of spinal manipulation on the pressure pain threshold and for how long does it last? – secondary analysis of data from a systematic review

The effect of a treatment is, in fact, the difference between the differences in outcome in these two interventions, i.e. (i) the follow-up measurement of the treatment group minus the baseline measurement is estimated and also (ii) the follow-up measurement of the placebo group minus the baseline measurement. Thereafter, (iii) the difference between these two differences is subjected to a test for statistical significance. However, if the baseline measurements in the two groups is more or less identical, then there is no need to take the baseline measurements into account, and it would be sufficient to test only if the difference between the outcome measurements is statistically significant.

However, the statistical significance (i.e p-value) is not an indication of the clinical significance. The clinical significance can be judged by comparing the estimates of the treatment and placebo groups and by calculating, for example, the numbers needed to treat, in clinical studies. In experimental studies, the clinical significance would often be assessed by calculating what is called an ‘effect size’.

To show how big an effect of a particular treatment might be, one could proceed to calculate the ‘difference of differences’, as explained above. However, it would be difficult to compare studies that used different methods and units. Therefore, an index is often used, such as the Cohen’s d or Hedge’s g coefficients.

To relate the effect size to the concept of clinical significance, Cohen created a scale that can be used for either the d or the g coefficients. According to this scale, an effect size of 0.2 represents an overlap between the compared populations in their distributions of means of about 85%, which Aron [1] suggests could be compared to a difference of height between 15 and 16 years old girls, thus considered ‘small’. An effect size of 0.5 represents an overlap of about 67%, which, in the same way, could be interpreted as a difference of height between 14 and 18 years old girls, thus considered ‘medium’. An effect size of 0.8 (or above) is an overlap of 53% (or less) and represents a difference of height between 13 and 18 years old girls, thus considered ‘large’. As another example, Cohen suggests that the difference in IQ between holders of a PhD degree and a ‘typical college freshman’ is comparable to an effect size of 0.8 [2].

The Cohen’s d coefficient is commonly calculated by the subtraction of the mean of the experimental group from the mean of the control group, and the division of this result by the standard deviation. However, this equation should be used under specific conditions, which are described extensively in the literature albeit without a clear consensus. Therefore, the effect size can be calculated in different ways and unless it is clearly described, there is room for errors and confusion. (Please, see Additional file 1 for more details).

This work consists of a secondary analysis of data from our previous systematic literature review, using data from eight randomized controlled trials that reported the regional effect of spinal manipulation on PPT in asymptomatic subjects compared to a sham procedure [8].

The search strategy and extraction of data for the original review have been extensively reported (Additional file 3). The flow chart for the screening process presented in the previous review has been included in this report for information. (Please, see Additional file 4).

For the present review, a descriptive and a quality checklist were created (Tables 1-2) to fit our new objectives. The quality score for the research method in general as reported in the previous article was included in the descriptive checklist for information (Table 1).

The present quality checklist (Table 2) is based on the various recommendations from the creator of the original effect size index [9] and its coeval authors [10], supported by more recent texts on the same subject ([2]; [11]). This consists of information on:

- whether the between-group effect size was provided [11],

- whether the formula for calculation was provided [10] or if, at least, an exact reference was provided (document and page, not just the name of a textbook), the reporting of the number of study participants, the exact mean values and standard deviations necessary to calculate the effect size [12], and the reporting of 95% confidence intervals (95% CI) [9]. If all this information was available, it would be possible to calculate the effect size.

Where effect size had not been calculated, it was our intention to do so, with the formulae provided in the Additional file 1 and verifying all calculations by a blinded third person. The information on the effect size at each time of measurement (provided by the authors or calculated by us) was collected in a table (Table 3) and illustrated in Fig. 1. We chose to report, when needed, the data concerning what happened on the “dominant side” [13], as we cared only for the regional effect.

Effect sizes were calculated using Eqs. 1 to 7 (as described in Additional file 1). The effect sizes and their 95% CI were calculated with Microsoft® Excel, version 16.17 (180909). Statistical significance of the effect size was defined as when the 95% CI does not include ‘zero’ [14]. The effect size was defined as small (d < 0.5), medium (0.5 < d < 0.8) or large (d > 0.8) [9].

These studies have been extensively described in our previous review [8], with a general description available in Additional file 2. Briefly, the eight studies included in our new analyses provided information on the PPT at different follow-up times; four studies immediately after, one study one minute after, five studies five minutes after, two studies ten minutes after, one study fifteen and thirty minutes after the interventions. The quality of the eight studies was established in the previous review to range from ‘medium’ to ‘good’ (Table 1).

In the present review, no additional quality score for the effect size was given, as no definitive consensus can be found on this subject. However, we noted that no study reported a between-group effect size. Instead, they reported ‘effect sizes’ of intra-group differences, i.e. in fact, the ‘outcome sizes’. Further, two studies failed to report any effect size at all (Table 1). In addition, no study reported the exact formula they used, and only one provided a ‘precise’ reference (Table 2). No additional descriptive information on the ‘effect size’ was given (95% CI or SD (d)). In sum, the reported ‘effect sizes’ were not real effect sizes, not transparent and possibly not comparable.

Fortunately, all studies had exploitable information, with reported numbers of study participants in each group, exact mean values and exact standard deviations, and only one study provided values only in a figure, which made it possible to retrieve information although it, for this reason, lacked precision. Thus, all authors had provided sufficient information to make it possible for us to calculate their between-group effect sizes (Table 3).

In this additional analysis of data from a previous systematic review, we were confused by the reported effect sizes. A systematic approach revealed that no study reported between-group size differences, instead using the within-group differences, when reported at all. Further, none provided details on how this ‘effect size’ had been calculated. Therefore, we used information provided in the reviewed articles to produce our own estimates.

According to our own calculations obtained from data available in the eight reviewed studies, the estimated effect size of spinal manipulation on the PPT in asymptomatic subjects is ‘medium’ immediately after the intervention (T0), ‘mainly large’ five minutes after (T + 5) and ‘mainly medium’ ten minutes after the intervention (T + 10). No certain estimation of the effect size can be reported beyond T + 10, but it may be small after 30 min.

Using the examples provided in the introduction [1, 2] to explain the clinical importance of the different effect sizes, the ‘medium’ effect size immediately after SM would thus correspond to a difference in height between 14 and 18 years old girls. The ‘large’ effect size five minutes after SM would correspond to the difference in IQ between holders of a PhD degree and a ‘typical college freshman’. The ‘medium’ effect size ten minutes after the intervention would, again, correspond to a difference in height between 14 and 18 years old girls.

The effect of SM on the PPT in asymptomatic subjects is therefore reported to be a reasonably large but probably short-lasting phenomenon. Whether these changes can also be ‘appreciated’ by study subjects, in such a way that they can differentiate between a small, medium and large effect size, is not known. Nevertheless, it serves as a comparator with other interventions in the same domain. For example, it could be used to compare the effect over time or effects of different types of interventions.

A description of the studies is found in the Table 1. Our quality checklist was established according to the various recommendations in the literature, including those provided by the creator of the Cohen’s d coefficient. There is no definitive consensus on the calculation of the effect size nor on how to assess its quality, so we did not judge the quality of work in the reviewed articles but used our systematic approach to obtain a general understanding of the various effect size values (Table 2). This can be done online to obtain the effect size with its SD (d) and 95% CI, with A Practical Meta-Analysis Effect Size Calculator [22]. A blinded third researcher verified the calculations. Obviously, this approach assumes that the groups that are being compared are fairly similar at base-line. We did not investigate if this was the case.

Depending on the type of data (continuous or categorial), treatment effect can be reported in other ways than with Cohen’s d. Relative Risk, Odds Ratio, Number Needed to Treat (NNT) and Area Under the Curve are other possibilities [23].

This additional analysis of data from our previous review on the effect of spinal manipulation on the pressure pain threshold in asymptomatic subjects revealed that all reviewed studies that reported an effect size used the within-group rather than the between-group differences. However, the within-group effect size is a purely descriptive outcome, interesting perhaps to understand the full picture of an effect, but it should never be provided alone. Therefore, the between-group calculations need to be calculated properly and in a transparent manner, to ensure that they are correct and comparable to other reports. We provide some information on how to do this in our Additional file 1, and we also provide references for these calculations.