Clinical research on postoperative trauma care: has the position of observational studies changed?

In evidence-based medicine, the value of observational studies, such as cohort and case–control studies, has generally been regarded as relatively insignificant [1]. Randomized controlled trials (RCTs) have long been considered the gold standard [2]. In the strive for optimal evidence-based medicine and subsequent patient care, the utility of conducting Randomized Controlled Trials (RCT) in trauma surgery has been questioned over the past years [3]. In addition, observational studies are published frequently, while the number of surgical RCTs from North America has decreased over the last decade [4]. This raises questions how the quality of studies evolved over the last years. In addition, as a result of evolving clinical experience and meeting patient demands, there is more emphasis on patient-reported outcome measurements. Patients are more involved in the decision-making process concerning their treatment, which should ultimately lead to shared decision making.

In trauma surgery, ankle fractures are among the most common fractures and the indication for surgery has not changed in the past decades [5]. Furthermore, the postoperative care regimes of ankle fractures are studied for over 30 years and recommendations have shifted only slightly in the last decades, as we demonstrated recently. As a consequence, the circumstances in which studies were performed did not change substantially in this field of medicine. Therefore, treatment of ankle fractures can be used as an example to investigate the evolution in study methodology. The goal of this study was to evaluate the changes in time in the design, quality and outcome measures of studies investigating the postoperative care of ankle fractures.

A total of 25 studies were included, 6 of which from the 80s, 11 from the 90s and 8 from the twenty-first century (Table 1). From all studies, 12 (48 %) were RCTs.

For postoperative care of ankle fractures, a change in the study design and presented outcome variables was observed over the last three decades. This study demonstrated an increase in the published number of cohort series on the treatment of ankle fractures, with increasing number of patients included. According to the Cochrane tool, the reported quality of RCTs has improved in the last three decades whereas the reported quality of observational studies has remained unchanged. However, when quality was evaluated with the MINORS criteria, equal improvement was observed for both RCTs and observational studies. Therefore, evaluation of quality is under influence of the quality assessment tool used. In addition, a gradual shift from physician measured to patient-reported outcome variables was observed.

There is a growing debate on the need of RCTs to evaluate the effectiveness of surgical procedures [35]. Results of this study indicate a relative decrease in the number of published RCTs over the past three decades in this field of research. Literature shows that one in five surgical RCTs is discontinued early, and one in three completed trials remains unpublished [35]. For (trauma) surgical research, such a trend might be explained by several factors.

First, including patients in a surgical RCT has become more and more challenging [36]. Both patient and surgeon have treatment preferences and surgeons may be better skilled in one of the study procedures. Ideally, randomisation distributes any unknown factors, thereby eliminating unforeseen confounders. However, patients may refuse or prefer the “new” technique and, therefore, do not wish to receive random treatment [37]. In addition, blinding of both patient and surgeon is frequently impossible and/or undesired by both patient and surgeon, which also neutralizes one of the advantages of an RCT over a cohort series.

Second, in the last decade the bureaucratic burden of ethical committee approval and monitoring has increased substantially, especially for conducting RCTs. This is accompanied by an increased financial demand. These procedures were developed to increase patient consent and safety, but sometimes might seem undue. For research projects in a field with limited financial resources, it is therefore increasingly difficult to meet the intensified quality demands [36]. Only a limited amount of funding went to surgical research in spite of the relatively large contribution of surgery to effective treatment [38]. These practical obstacles might have caused a substantial decrease in the quantity of RCTs in surgical research compared to other fields of medicine [4, 39].

However, as the goal of research remains to improve the quality of care for patients with an ankle fracture, alternatives are sought. This might reflect the subsequent increase in the number of published cohort series. An alternative explanation might be that due to the eagerness to produce papers in the highly competitive scientific world, where the number of publications counts heavily, there may be a bias towards writing more relatively inexpensive articles that describe cohort and retrospective studies. However, this was not demonstrated by the absolute number of published papers over the last three decades in this study, nor by a decrease in quality.

Reported quality of RCTs improved over time. This might be explained by several quality assessment tools that became available for RCTs in the 90s. The introduction of these tools provided guidance for constructing well-designed studies, with more standardised reporting and thereby improved reported research quality. Reporting of studies might have improved by reporting standard such as the CONSORT [2]. In the current study, we used the Cochrane risk of bias assessment tool to assess the quality of the studies included. This tool was primarily designed for RCTs, but is also recommended for quality assessment of observational studies [40, 41]. In addition, we used the MINORS criteria [8]. This tool is primarily designed for non-randomized studies. A combination of both tools can provide the overall quality of studies. Due to the design of the Cochrane risk of bias assessment tool, the design of a study is decisive for the results on a domain of the tool (e.g., a randomised trial has the potential to score positive on all domains contrary to observational studies). Therefore, observational studies will inevitably have a higher risk on bias in the Cochrane risk of bias assessment tool. In addition, we used the MINORS criteria [8]. This tool is especially designed for non-randomized studies. Other tools for observational studies were designed more recently, which are not frequently used [42]. It should always be kept in mind that evaluation of quality is under influence of the quality assessment tool used. Similarly to RCTs, the MINORS criteria might provide guidance for constructing well-designed observational studies and further improve observational research quality. Perhaps both RCTs and cohort series are needed to provide a complete image of a disease or treatment modality. RCTs are designed to compare specific items head to head, while large cohort series are more capable of exposing rare complications or the effect of a whole package [43].

A convincing finding of the present study is the increased use of patient-relevant outcomes. The next step would perhaps consist of consensus recommendations to define standard outcome measures for future ankle fracture studies. If all studies applied the same set of outcome measures at the same timepoints, this would make it much easier to compare and to use study results. In rheumatology, such an initiative (called OMERACT) was highly successful [44].

In the 80s, primary outcomes were focused on radiological results and objective variables. The attending physician or investigator could measure these objective variables. In the twentieth century, outcomes increasingly focused on the implications for the patient and were reported as primary end-point of several studies. In the last decade, over half of the included studies mentioned return to work as outcome measure which is, in our opinion, a legitimate patient-reported end-point.

As ankle fractures are represented in the top five of most prevalent fractures and have an important socio-economic impact on both patient and society, the observations made in this study might be applicable to broader areas of (trauma) surgery research [45]. However, results of this study might not be applicable to other non-surgical specialties. This study can be seen as a case report and should be tested in further extent and on a larger scale in trauma surgical studies.

Results of this study indicate a relative decrease in the number of published RCTs, which might be a reflection of the practical difficulty to conduct RCTs in trauma surgery due to surgeon, patient and system-based factors. Furthermore, the desire for more patients orientated outcome variables is reflected in the changes seen in the last decades.