Strength in numbers? The fragility index of studies from the Scandinavian knee ligament registries

A large number of studies related to anterior cruciate ligament (ACL) reconstruction have been published from the Scandinavian knee ligament registries over the past decade [1, 2]. Many of these studies have aimed to determine predictors and risk factors for an additional ACL reconstruction, i.e. a revision or a contralateral ACL reconstruction, or used the presence of postoperative laxity as a measurement of primary ACL reconstruction failure [2]. A P value of less than 0.05 or a 95% confidence interval (CI) excluding the defined null value have been used as the threshold of significance when drawing conclusions on statistically significant predictors. Meeting these criteria implies that the null hypothesis, stating that there would be no difference in outcome depending on the investigated predictor, has been rejected, meaning that the predictor is likely to have a true effect on the outcome after ACL reconstruction.

The concept of a P value was first described by Sir Ronald Fisher and aids in the interpretation of a given result [3]. Although Fisher never did set a threshold for significance, a P value of less than 0.05 shows that a result that is similar to or more extreme than that observed would be found in fewer than 5% of repeated tests, on condition that the null hypothesis was true. It is therefore commonly accepted that a level of significance of 5% is sufficient to conclude that the observed result has not occurred by chance. Nonetheless, the P value says nothing about the robustness of an analysis and the interpretation of P values is many times misunderstood by researchers [4‐6]. The fragility index (FI) was developed to evaluate the robustness of significant findings in randomised controlled trials (RCTs). More specifically, the FI describes the minimum number of patients in the group with the fewest events that would need to change from a non-event to an event to change the result from significant to non-significant [7]. Although the FI has not previously been applied to registry studies, it should follow the same principle. For example, there are studies from the Scandinavian knee ligament registries reporting that the use of hamstring tendon (HT) autograft significantly increases the risk of ACL revision compared with the use of patellar tendon (PT) autograft [8‐10]. The FI for these studies would describe how many patients in the PT group would need to change from not undergoing an ACL revision to undergoing one to change the analysis to non-significant. The FI is thus a measurement of the number of events (e.g., ACL revisions) on which the statistical significance depends. In other words; the lower the FI, the more fragile the result.

Recently, the FI was evaluated for 48 RCTs in sports medicine and arthroscopic surgery [11]. Worryingly, the median FI of the included studies was two [11], meaning that drawing conclusions in current clinical trials of sports medicine is in fact based on the outcome of a very limited number of patients. One of the main methodological strengths of the Scandinavian registries is the prospective data collection from a large population. In fact, the registries together comprise data from over 70,000 primary ACL reconstructions [12]. Large study samples increase the robustness of a statistical analysis, however, the FI of the statistically significant findings presented from the Scandinavian registries has not been evaluated. This is important knowledge since it allows for a more precise interpretation of the results and promotes an evidence-based approach in the clinical care of patients. The purpose of this systematic survey was to evaluate the applicability of the FI to registry studies by determining the FI of all analyses from the Scandinavian registries related to any of the following dichotomised outcomes; ACL revision, contralateral ACL reconstruction and the presence of postoperative residual knee laxity.

This most important finding of this study was that the FI varied substantially across dichotomous analyses from the Scandinavian knee ligament registries. Although almost one third of the analyses had an FI of zero, the analyses related to age generally had the most robust FI, with a mean FI of 178.5 (range 1–1089). In fact, the majority of the analyses had a higher FI than what previously has been reported from RCTs related to orthopaedic surgery [11, 37]. However, the variable FI underlines that there are difficulties in the interpretation of robustness in analyses from these registry studies.

The FI has previously been applied exclusively to RCTs. A median FI of 2 (IQR 1–3) was reported when assessed in 40 RCTs related to orthopaedic spine surgery [37]. Similarly, a median FI of 2 (IQR 1–2.8) was found in 48 RCTs related to arthroscopy and sports medicine surgery [11]. It was concluded that the statistical significance in current orthopaedic RCTs is fragile and that relatively small sample sizes and few outcome events are contributory factors [37]. The large study samples provided by registry studies could theoretically increase the robustness of significant findings. On the other hand, it is not known whether the use of the FI is feasible for registry studies, as there are some fundamental discrepancies in the study design compared with RCTs. A well-designed RCT is thought to exclude confounding factors by assuming an equal distribution of both measured and unmeasured factors due to randomisation and blinding. Registry studies are instead susceptible to confounders and bias, which is commonly dealt with by statistical adjustments. The FI is calculated independently of whether or not the tested P value originates from an adjusted analysis, which is important, as most analyses included in this study were adjusted. Moreover, despite the fact that the FI has previously been applied to time-to-event outcomes in RCTs [7, 11], time-to-event outcomes are more common in registry studies. Time-to-event is not considered when calculating the FI, which means that the FI becomes a rougher measurement for these analyses.

It is obvious that the FI had greater variability in registry studies compared with previous studies of RCTs [11, 37]. The variability of the FI questions the feasibility of using this metric on registry studies, especially since a strong contributor to the variable FI probably is the heterogeneous data analyses. For example, there is no consensus on how to stratify age groups in the Scandinavian knee ligament registry studies. This aggravates a comparison of the FI across studies, since the FI in addition to describe the actual robustness also will be affected by group size and age difference between groups. Moreover, almost one third of the analyses had an FI of zero, which is difficult to interpret for analyses which originally used statistics other than Fisher’s exact test and found significance. Does an FI of zero indicate fragility or is it the result of applying a statistical test that was not deemed to be the most appropriate test in the original study? Interestingly, Walsh et al. [7] applied the FI to 399 trials published in high-impact journals and found that 70% of the trials with an FI of zero originally were analysed using time-to-event analysis [7]. This indicates that time-to-event analyses are particularly susceptible to an FI of zero, which might explain the large proportion of analyses with an FI of zero in this study. Not surprisingly, the mean and median FI were considerably higher when the analyses with an FI of zero were excluded in the subanalysis. Further research is needed to determine the most appropriate methodology for FI calculation in studies using time-to-event analysis. Until then, it could be argued that time-to-event analyses with an FI of zero should be excluded to not severely skew the overall FI.

To draw conclusions regarding the feasibility of using the FI on registry studies, the FI perhaps needs to be assessed in a larger number of registry studies, which could strengthen the data and narrow the range of the FI. That could potentially also enable a determination of the most robust predictors for ACL failure. In this study, most predictors included analyses from only one or two studies. The inclusion of few analyses per predictor makes the FI analysis sensitive to outliers and makes the process of determining the most robust predictor vulnerable. One should however bear in mind that only significant analyses are considered for the FI calculation and few included analyses for a predictor could mean that the reported significance is an exception among several non-significant findings. Patient sex could be used to exemplify this, where only one analysis was included with an FI of 35. Although the FI for patient sex is difficult to interpret based on a single analysis, it should be remembered that seven other studies using ACL revision as the outcome and three other studies using contralateral ACL reconstruction as the outcome found non-significant results when comparing patient sex in the Scandinavian registries [2]. The interpretation of the FI must therefore also be set in the context of the cumulative evidence.

Not all predictors were, however, limited by few analyses, and the feasibility of using the FI is strengthened when determining the FI for the two predictors with most analyses (age with 19 analyses and graft fixation with 12 analyses), as the result reflects previous literature on the subject. The literature is unanimous when it comes to young age as a risk factor for an additional ACL reconstruction [38‐42]. In agreement with this, the FI for age was by far the highest, which indicates that the FI calculation is able to provide a reliable estimate of robustness. This is further emphasised by the higher FI in analyses with an increasing age difference between the compared groups. With regard to graft fixation, six of 12 analyses had an FI of 0 and the mean FI for all the analyses was 37.4. For this reason, graft fixation does not predict an additional ACL reconstruction with the same certainty as age, which is supported by the contradictory literature regarding the impact of graft fixation [43‐45].

In the light of the limitations associated with using the FI on registry studies, it is our opinion that the FI could be regarded as a rough measurement of robustness for registry studies and that it could be used to compare confidence in the results across analyses with a similar statistical methodology. There is reason to believe that the FI could be a valuable method for registry studies, especially for those using statistical methods that are perfectly compatible with the FI, such as Chi-square or Fisher’s exact test. The large amount of data comprised in registries should intuitively increase the robustness, but it is important to find a metric to quantify this objectively in registry studies. Although there might be outcomes in registry studies that are more or less appropriate for the use of FI, researchers should be encouraged to calculate and report the FI whenever possible. The strength in numbers of registry studies does not compensate for other limitations, such as confounders, bias and an inability to provide causality. Care must be taken not to overestimate the effect of a higher FI in registry studies compared with RCTs, as RCTs still remain the gold standard to determine the efficacy of an intervention. This study is also limited by the fact that analyses from seven studies needed to be excluded, as data on study arm size and the number of events were not reported. Future studies should preferably report these numbers. It should also be mentioned that the outcome of additional ACL reconstruction may underestimate the true rate of failed ACL reconstructions. This becomes especially relevant to consider in studies with small FIs, since the robustness of significance in these circumstances is likely to be even more vulnerable to “hidden” failures not proceeding to an ACL revision.

There was large variability in the FI in analyses from the Scandinavian knee ligament registries and almost one third of the analyses had an FI of zero. The FI is a rough measurement of robustness when applied to registry studies, however, future studies are needed to determine the most appropriate metric for robustness in registry studies. The use of the FI can provide clinicians with a deeper understanding of significant study results and promotes an evidence-based approach in the clinical care of patients.