An Analysis of References Used for the Orthopaedic In-Training Examination: What are Their Levels of Evidence and Journal Impact Factors?

Although the references recommended for the Orthopaedic In-Training Examination (OITE) have been evaluated in certain subspecialty domains, suggested reference level of evidence (LOE), impact factor, and citation age have not been evaluated comprehensively to our knowledge.

We present an analysis of all references cited in the OITE recommended readings for each test question including the duration of time between their initial publication and their use in the OITE, which we defined as citation age, LOE, and the impact factor of the journals referenced.

We evaluated all references for the 2010 to 2012 OITE administrations (three examinations; 825 questions total). Publication characteristics, including citation age, were noted. The LOE for each journal article and the impact factor of each journal were determined; differences in LOE and impact factor were compared between test sections. A total of 1817 references were cited in the 825 questions we evaluated; this denominator was used in all calculations that follow.

The recommended reading references included 1337 journal article references (74%), 469 text references (26%), and 11 multimedia sources (0.6%; eg, websites, instructional DVDs). The three most commonly recommended journals were general orthopaedic journals, The Journal of Bone and Joint Surgery (American Volume), Journal of American Academy of Orthopaedic Surgeons, and Clinical Orthopaedics and Related Research ^®. The majority (72.2%) of the cited journal references were published within 10 years of the test date, with a mean ± SD citation age of 8.3 ± 7.4 years. The majority of the cited journal articles were Levels IV and V evidence (mean, 4.16 ± 1.1). The Spine section had higher LOE (3.74; p < 0.001), although the practical relevance of such a difference is questionable, as all but two sections’ LOE rounded to Level IV evidence. The Spine and Basic Science sections were published in journals with a larger mean impact factor (Basic Science, 7.16 ± 12.67; Spine, 5.73 ± 12.08; p < 0.001).

Our data show that the majority of the recommended readings for the OITE stem from higher impact general orthopaedic and major subspecialty journals. Furthermore the observed mean LOE of the recommended readings shows a preponderance of Levels IV and V research. These data may suggest that test-takers may find benefit in the review of high-level general orthopaedic journals, and review articles in particular while preparing for the OITE, although further study is necessary to determine optimal test preparation strategies. Finally, our study provides a baseline analysis of the study designs of OITE recommended references, and may provide insight for educators designing resident educational curricula.