Level of clinical evidence presented at the open and closed American Shoulder and Elbow Surgeons annual meeting over 10 years (2005–2014)

The ASES annual scientific meetings are important venues that focus on the presentation of high quality and evidence-based outcome measures for the most important procedures addressing injuries or disease of the shoulder and elbow.

Overall, the presentations over the past ten years at ASES meetings predominantly used lower quality study designs, with 68% of all presentations graded as either level III or IV evidence. The most common presentations were level IV studies, which encompassed 41% of paper presentations from the open meeting and 46% of paper presentations from the closed meeting between 2005 and 2014. Level IV studies are mostly case series, which describe outcomes from a single cohort of patients without comparison to a control group. The lack of a control group affords level IV studies with inherent sources of bias including: measurement bias, selection bias, and bias from confounding factors that are unaccounted for. As such, the attendees of these meetings should avoid using the outcomes generated from such studies as the sole basis of treatment decisions. While research of lower methodological quality may not always have direct clinical relevance, these studies are important sources of information that can be used to generate hypotheses to direct the focus of future studies with improved study designs [6].

Between 2005 and 2014, there was a significant decrease in the presentation of level IV evidence in both the open and closed annual meetings combined and in the open meeting alone. These changes could indicate an increased understanding and use of new procedures concerning the shoulder and elbow. Due to the aging population, degenerative shoulder problems such as osteoarthritis and rotator cuff disease are becoming increasingly common [7]. The use of shoulder arthroplasty has increased substantially since the United States Food and Drug Administration approved the use of reverse total arthroplasty in 2003 [8]. This procedure has been a very important contribution for treatment of conditions such as rotator cuff insufficiency, arthropathy and some fractures [7]. The use of elbow arthroscopy was initially described as unfeasible in 1931, however newer technology and a better understanding of the anatomy of the elbow has since allowed an increase in the prevalence of arthroscopic elbow surgery [4]. As data is compiled from early research evaluating these new procedures, the production of research with improved study designs becomes progressively more feasible. Furthermore, many Orthopaedic journals including the Journal of Bone and Joint Surgery, American Journal of Sports Medicine, and JSES have required and included the LOE as part of the publication of research manuscripts. Not only do published LOE allow clinicians to understand the context with which to interpret a study, but it also provides Orthopaedic surgeons a greater awareness of the importance of studies with higher methodological quality or LOE [9]. Most Orthopaedic journals, including the JSES, have demonstrated increased production of publications with level I and II evidence since the introduction of LOE in these journals [10]. Therefore, we could attribute the improvement in the methodological quality of research at ASES meetings to an increased quantity of research submitted in addition to the influence of EBM, facilitated partially by the inclusion of LOE in most Orthopedic journals at the beginning of our study period.

Although the statistical analysis did not indicate a significant change in the proportion of level I studies between 2005 and 2014, the relatively small sample size of level I evidence may have contributed to the negative finding. The percentage of level I evidence increased from 15% before 2010 to 20% after 2010. Of the 17% of studies that were classified as level I, 57% were therapeutic indicating that 10% of all studies were RCTs. The percentage of level I evidence at ASES meetings between 2005 and 2014 represents a much higher number than that presented at the AAOS (7% in 2010) meeting and even greater than the percentage of level I and II presentations combined at the Pediatric Orthopaedic Society of North America (POSNA) meetings (14% over 2001, 2002, 2007 and 2008) [11, 12]. The proportion of level I research is especially impressive considering the recognized difficulty in performing a randomized study with surgical interventions. Some key challenges inherent to surgical trials include patients’ reluctance to randomization as well as the difficulty blinding surgeons to the interventions [13]. These challenges limit the number of RCTs conducted, but also impact the quality of those produced as Bhandari et al. reported that the majority of randomized trials published in The Journal of Bone and Joint Surgery did not report the use of key methodological features such as proper allocation concealment, exclusion of patients, and blinded assessment of outcomes [14]. RCTs studying elbow pathology in particular, are not of the highest quality and have not shown an improvement in their quality in recent years [15]. On the other hand, McCormick et al. have reported that, according to the Jadad score, RCTs studying rotator cuff pathology are of high quality [14, 16]. Although there are difficulties conducting RCTs in surgery, Farrokhyar et al. have reported that RCTs for novel surgical interventions are important and can be produced if the feasibility is properly assessed initially and close attention is addressed to specific methodological details [12].

Overall, statistical analyses indicate a significant improvement in the LOE of presentations at the open and closed annual ASES meetings from 2005 to 2014. The average LOE during the period between 2005 and 2009 was 3.07 and improved to 2.75 in the period between 2010 and 2014. Voleti and colleagues evaluated the poster and paper presentations at the AAOS meetings and demonstrated significant improvements in the average LOE between 2001 (3.46) and 2010 (2.88) [11].

The program committee for ASES meetings diligently screen submitted abstracts and include only presentations of the highest quality, however, the quality of abstracts that are being submitted is often the limiting factor to such selection. Therefore, the changes identified in the present study are likely indicative of an improvement in the methodological quality of the research being submitted for presentation at these meetings. One method that can be used to educate the audience as well as the submitting authors on the importance of high methodological quality is having authors include the LOE of their study accompanying the abstract. However, Shmidt et al. studied presentations for which authors provided a LOE at AAOS meetings and have reported that authors tend to rate their own studies with a higher LOE than an independent reviewer [17]. Therefore, if LOEs are to be included with meeting presentations, it may be beneficial to have these verified by an independent reviewer.

A strength of this study is that it is the first, to the best knowledge of the authors, to evaluate the methodological quality of the research presented at the ASES annual scientific meetings. There was high agreement amongst the reviewers in terms of the evaluation of LOEs, indicating that the evidence was categorized consistently. Limitations of this study include the availability of abstracts (no abstracts from the years 2007, 2010, 2011 and 2013) and the limited methodological information available based on the 300 word abstract alone. Missing data may have affected the statistical tests assessing the methodological quality over time.