Skip to main content
  • Research article
  • Open access
  • Published:

Associations between resident physicians’ publications and clinical performance during residency training

Abstract

Background

Both research and clinical medicine requires similar attributes of efficiency, diligence and effective teamwork. Furthermore, residents must succeed at scholarship and patient care to be competitive for fellowship training. It is unknown whether research productivity among residents is related to broad measures of clinical achievement. Our goal was to examine associations between the quantity of internal medicine residents’ publications and validated measures of their knowledge, skills and multi-source evaluations of performance.

Methods

This was a longitudinal study of 308 residents graduating from Mayo Clinic from 2006 to 2012. We identified peer-reviewed articles in Ovid MEDLINE between July of each resident’s match year and the end of their graduation. Outcomes included American Board of Internal Medicine (ABIM) certification examination scores, mini clinical examination (mini-CEX) scores, and validated assessments of clinical performance by resident-peers, faculty and non-physicians. Performance assessments were averaged to form an overall score ranging from 1 to 5. Associations between quantity of resident publications – and ABIM, mini-CEX and performance assessment scores – were determined using multivariate linear regression.

Results

The residents published 642 papers, of which 443 (69.0 %) were research papers, 198 (30.8 %) were case reports, and 380 (59.2 %) were first-authored. On adjusted analysis, multi-source clinical performance evaluations were significantly associated (beta; 99 % CI; p-value) with the numbers of research articles (0.012; 0.001–0.024; 0.007), and overall publications (0.012; 0.002–0.022; 0.002).

Conclusions

To our knowledge, this is the first study to demonstrate that scholarly productivity based on journal publication is associated with clinical performance during residency training. Our findings suggest that residents who invest substantial efforts in research are not compromised in their abilities to learn medicine and care for patients.

Peer Review reports

Background

To become reflective and competent doctors, resident physicians must be trained to interpret the literature, apply evidence to patient care and demonstrate competence in research methods [1]. Therefore, the Accreditation Council for Graduate Medical Education (ACGME) requires that residents participate in scholarly activities [2].

Dedicated research time during medical school and residency is associated with increased publication by learners [3]. The Lincoln Hospital found that that 21 % of residents who participated in a research elective published, compared to only 5 % of residents who did not have the elective [4]. Likewise, a Mayo Clinic study demonstrated that internal medicine residents, all of whom engage in research rotations, publish at significantly higher rates than similar residents at other institutions [5].

Students and residents who receive dedicated research time and publish are more likely to select careers in academic medicine and ultimately achieve higher academic rank [3, 512]. For example, Johns Hopkins medical students with research experience were nearly three times more likely to pursue an academic medical career compared to those without such experience [9]. Similar findings have been demonstrated among resident physicians in the fields of urology [8], general surgery [6], neurosurgery [11], pediatrics [13], and emergency medicine [10].

Although research experience during medical school and residency appears to be correlated with future careers in academia, few studies have addressed potential associations between scholarly productivity and performance during residency, and research has failed to demonstrate a connection between publishing during medical school and subsequent performance during residency training [1416]. For instance, researchers at one academic medical center found that the quantity and quality of publications during medical school had no apparent impact on future resident performance in terms of supervisory ratings and standardized test scores [17].

It remains unknown whether research productivity among resident physicians is related to broad measures of their clinical performance during residency training. Substantial time investment is required to be successful at research [1822], which suggests that residents who excel in scholarship might suffer in their abilities to care for patients [2325]. However, both research and clinical medicine require similar traits like efficiency, diligence, and effective teamwork, and residents must be successful at both scholarship and patient care to compete for fellowship training. Consequently, we hypothesized that there would be an association between residents’ publications, knowledge acquisition, and clinical accomplishment. Our goal was to utilize a longitudinal study design to examine potential associations between the quantity of internal medicine residents’ peer-reviewed publications and diverse measures of their clinical performance using validated measures of knowledge (ABIM certification examination scores), skills (mini-clinical examination scores) and multi-source evaluations of clinical performance.

Methods

Resident research curriculum

The Mayo Clinic internal medicine resident research curriculum extends across three years of training and includes the following topics: project planning, protocol design, basic principles of biostatistics and epidemiology, human subjects’ protection, and scientific writing. Educational resources are also available through the Mayo School of Graduate Medical Education, Mayo Center for Clinical and Translational Science, and biannual scientific writing workshops. Each year, residents may engage in month-long research electives, which require completion of a standardized application form, a mentor’s letter of support, and final approval by the associate program director for resident scholarship. Following their electives, residents receive evaluation and feedback from their mentors.

Independent measure: resident publications

To quantify resident scholarship in terms of amount (counts), authorship role (first author, yes or no), and category (original research and review articles versus case report), we identified peer-reviewed articles indexed in Ovid MEDLINE between July of each resident’s match year and the end of their graduation calendar year. Articles were found by searching author last name, first initial, and middle initial when available. The resulting references were then examined manually for full-name matches. For references without full first names or where multiple authors were listed under the same name, we checked the Electronic Residency Application Service record. We excluded non-journal articles and any publication dated before July 1 of the resident’s match year. For each resident, we recorded the total number of peer-reviewed articles, case reports, and first-authored publications.

Validity of outcome measures and co-variables

The measures used in this study are supported by validity evidence. Research has demonstrated predictive validity of the ABIM certification examination [26]. Several studies have established the validity of in-training examination (ITE) scores, including correlation with resident conference attendance and self-directed reading, and no correlation with empathy [2729]. Validity and reliability of the mini clinical evaluation exercise (mini-CEX) has been verified by previous studies at the Mayo Clinic and elsewhere [3036]. We use a traditional version of the mini-CEX; however, the items are on 5-point, as opposed to 9-point, scales. Clinical performance assessments of Mayo Clinic internal medicine residents are completed by peers, senior medical residents, faculty, and non-physician professionals. Validity of these assessments is based on elements from previously published instruments, input from experts with experience in scale design, factor analysis showing multiple dimensions, and excellent internal consistency reliability [37].

Study design and data analysis

This was a longitudinal study of 308 post-graduate-year three (PGY-3) residents graduating from Mayo Clinic Rochester from 2006 to 2012. Associations of resident publications – with the outcomes of ABIM average percent correct, mini-CEX and clinical performance assessment scores – were determined using multivariate linear regression models. Scores for the clinical performance assessment items were averaged across year-three within assessor group to form an overall score ranging from 1 to 5. Co-variables adjusted for included sex (male; female), medical school (U.S. versus international medical graduate), track (categorical; clinician-investigator), PhD (yes; no), age, and percent correct on ITE examination. The threshold for statistical significance was set conservatively at 0.01 to account for multiple comparisons. This study of 308 residents provided 90 % power to detect a medium Cohen’s f2 effect size of 0.25 for adjusted associations between the number of publications and each outcome variable. Statistical analyses were performed using SAS version 9.3 (SAS Institute Inc., Cary, NC). The Mayo Clinic Institutional Review Board deemed this study exempt under 45 CFR 46.101, on the basis of existing data that were recorded in such a manner that the subjects cannot be identified.

Results

Of the 308 residents, 197 (64.0 %) were male, 259 (84.1 %) were U.S. medical graduates, 279 (90.6 %) were categorical, and 11 (3.6 %) had PhD degrees. Their average age was 29.7 years. The residents published 642 papers, of which 443 (69.0 %) were research papers, 198 (30.8 %) were case reports, and 380 (59.2 %) were first-authored (Table 1).

Table 1 Description of the Resident Cohort (N = 308)

Upon adjusted analysis, multi-source evaluations of clinical performance were significantly associated (beta; 99 % CI; p-value) with the number of overall publications (0.012; 0.002–0.022; 0.002), and research articles (0.012; 0.001–0.024; 0.007 [Table 2]). In other words, each additional publication was associated with a 0.012-point increase in the expected mean PGY-3 multisource evaluation score after adjusting for all covariates. There were no statistically significant associations between quantity of publications and ABIM certification examination or mini-CEX scores.

Table 2 Associations between resident publications and measures of professionalism, clinical performance and medical knowledge

Discussion

To our knowledge, this is the first study to explore relationships between residents’ scholarly activity in terms of publications and broad measures of their clinical performance during residency training. We found that residents’ overall numbers of publications in peer-reviewed journals were associated with validated, multi-source assessments of their clinical performance. These findings suggest that a potential overlap exists between the skills required for research and medical practice. Additionally, these findings indicate that residents who invest time in research may not be compromised in their abilities to care for patients and contribute to healthcare teams.

Although there was a positive association between resident scholarship and clinical evaluation scores, it is noteworthy that there were no negative associations between resident scholarship and any aspect of clinical performance. In other words, our findings would suggest that research does not detract from clinical training, which is the main goal of residency training [25, 38, 39]. These findings support a study of residents who, despite logging fewer clinical cases than their colleagues who did not do research, performed better on oral examinations at graduation [23]. Additionally, these findings underscore a previously-untested assumption that providing education in research during residency training likely improves residents’ skills across all of the ACGME core competencies [40].

Potential explanations for the association between resident scholarship and clinical performance include external and internal forms of motivation, as well as traits that are adaptive to both research and clinical practice [41]. As for external motivation, it is recognized that selection criteria for competitive fellowship training programs and academic appointments includes a strong emphasis on peer-reviewed publications [40, 42, 43]. Examples of internally motivating factors are intellectual curiosity and a personal drive for excellence that would translate to all aspects of medicine, including scholarship and patient care. Similarly, it has been observed that achievement in research and clinical practice require the abilities of time management, efficiency, diligence and effective teamwork [44]. It is possible that residents who are widely recognized for their research achievements enjoy a “halo effect,” which could render them attractive across diverse aspects of their work [45, 46]. Lastly, it has been shown that residents who participate in research have a higher satisfaction with residency training; this may enhance their clinical accomplishments and push them towards careers in academic medicine [47, 48].

This study expands upon previous research regarding scholarship among medical students and residents. One study did not identify a correlation between publications during medical school and subsequent in-training exam scores or global faculty-of-resident assessments during residency training [17], although this study may have been limited by low publication rates. Another study revealed that residents with research experience are more likely to pursue academic careers [9]. One study demonstrated a correlation between research experience and clinical achievement during residency [23]. However, the variables in that study were research elective and performance on an oral examination. Our study involved measured numbers of peer-reviewed publications by internal medicine residents, along with validated assessments of clinical performance, and the universally-important outcome measures of mini-CEX and ABIM certification scores.

This study has several limitations. It was conducted at a single institution, which may limit generalization of findings to other settings. Specifically, aspects of our residency research program that might be distinctive and are recognized as valuable for success include a dedicated research director, robust and longitudinal research curriculum, statistical resources, faculty with established research careers, and a strong culture of mentoring support [5, 39, 4951]. Similarly, the assessment of clinical performance that was utilized in this study is unique to the Mayo Clinic; nonetheless, this assessment reflects the same ACGME competencies that are utilized by other institutions [2], and validity of the instruments’ scores include relations to other meaningful variables such as resident physician professionalism and well-being [52, 53].

Conclusions

This study showed that numbers of journal publications by residents were positively associated with their clinical performance assessment scores during residency training. These findings suggest that research does not interfere with residents’ abilities to care for patients and work in healthcare teams. Potential explanations for the study findings are that certain traits – such as discipline, critical thinking ability, organization, and ability to work in teams – are adaptive to excellence in both research and clinical medicine. To be selected for fellowship programs, residents must demonstrate both scholarly and clinical success. Therefore, additional research should determine whether there is a relationship between publication during residency and future accomplishments such as selection into competitive fellowship programs.

Abbreviations

ABIM:

American board of internal medicine

ACGME:

Accreditation council for graduate medical education

ITE:

In-training examination

mini-CEX:

Mini clinical evaluation exercise

PGY-3:

Post-graduate year 3

References

  1. Schon DA. The Reflective Practitioner. 1st ed. New York: Basic Books, Inc.; 1983.

    Google Scholar 

  2. Common Program Requirements Effective July 1, 2013. Accreditation Council for Graduate Medical Education. Available at: https://www.acgme.org/acgmeweb/Portals/0/PFAssets/ProgramRequirements/CPRs2013.pdf. Accessed December 8, 2013.

  3. Dyrbye LN, Davidson LW, Cook DA. Publications and presentations resulting from required research by students at Mayo Medical School, 1976–2003. Acad Med. 2008;83(6):604–10.

    Article  Google Scholar 

  4. Kanna B, Deng C, Erickson SN, Valerio JA, Dimitrov V, Soni A. The research rotation: competency-based structured and novel approach to research training of internal medicine residents. BMC Med Educ. 2006;6:52–60.

    Article  Google Scholar 

  5. West CP, Halvorsen AJ, McDonald FS. Scholarship during residency training: a controlled comparison study. Am J Med. 2011;124(10):983–7.

    Article  Google Scholar 

  6. Bhattacharya SD, Williams JB, De la Fuente SG, Kuo PC, Seigler HF. Does protected research time during general surgery training contribute to graduate' career choice? Am J Surg. 2011;77(7):907–10.

    Google Scholar 

  7. Holmes JF, Sokolove PE, Panacek EA. Ten-year experience with an emergency medicine resident research project requirement. Acad Emerg Med. 2006;13(5):575–9.

    Article  Google Scholar 

  8. Yang G, Zaid UB, Erickson BA, Blaschko SD, Carroll PR, Breyer BN. Urology resident publication output and its relationship to future academic achievement. J Urol February. 2011;185(2):642–6.

    Article  Google Scholar 

  9. Brancati FL, Mead LA, Levine DM, Martin D, Marolis S, Klag MJ. Early Predictors of career acheivment in academic medicine. JAMA. 1992;267(10):1372–6.

    Article  Google Scholar 

  10. Burkhardt J, Kowalenko T, Meurer W. Academic career selection in American emergency medicine residents. Acad Emerg Med. 2011;18 suppl 2:S48–53.

    Article  Google Scholar 

  11. McClelland III S. Pre-residency peer-reviewed publications are associated with neurosurgery resident choice of academic compared to private practice careers. J Clin Neurosci. 2010;17(3):287–9.

    Article  Google Scholar 

  12. Rezek I, McDonald RJ, Kallmes DF. Pre-residency publication rate strongly predicts future academic radiology potential. Acad Radiol. 2012;19(5):632–4.

    Article  Google Scholar 

  13. Ledley FD, Lovejoy Jr FH. Factors influencing the interests, career paths, and research activities of recent graduates from an academic pediatric residency program. Pediatrics. 1993;92(3):436–41.

    Google Scholar 

  14. Stohl HE, Hueppchen NA, Bienstock JL. Can medical school performance predict residency performance? Resident selection and predictors of successful performance in obstetrics and gynecology. J Grad Med Educ. 2010;2(3):322–6.

    Article  Google Scholar 

  15. Dirschl DR, Dahners LE, Adams GL, Crouch JH, Wilson FC. Correlating selection criteria with subsequent performance as residents. Clin Orthop. 2002;399:265–71.

    Article  Google Scholar 

  16. Carmichael KD, Westmoreland JB, Thomas JA, Patterson RM. Relation of residency selection factors to subsequent orthopaedic in-training examination performance. South Med J. 2005;98(5):528–32.

    Article  Google Scholar 

  17. Cavalcanti RB, Detsky AS. Publishing history does not correlate with clinical performance among internal medicine residents. Med Educ. 2010;44(5):468–74.

    Article  Google Scholar 

  18. Bland CJ, Schmitz CC. Characteristics of the succesful researcher and implications for faculty development. J Med Educ. 1986;61(1):22–31.

    Google Scholar 

  19. Goldman L. Blueprint for a research career in general internal medicine. J Gen Intern Med. 1991;6(4):341–4.

    Article  Google Scholar 

  20. Bauchner H. Mentoring clinical researchers. Arch Dis Child. 2002;87(1):82–4.

    Article  Google Scholar 

  21. Toto RD. How to launch a successful career in clinical research: tips on making the most of available resources. J Investig Med. 2008;56(6):843–6.

    Google Scholar 

  22. Rosier RN. Institutional barriers to the orthopaedic clinician-scientist. Clin Orthop. 2006;449:159–64.

    Article  Google Scholar 

  23. Schott NJ, Emergick TD, Metro DG, Sakai T. The cost of resident scholarly activity and its effect on resident clinical experience. Anesth Analg. 2013;117(5):1211–6.

    Article  Google Scholar 

  24. Robbins L, Bostrom M, Marx R, Roberts TS, Sculco TP. Restructing the orthopedic resident research curriculum to increase scholarly activity. J Grad Med Educ. 2013;5(4):646–51.

    Article  Google Scholar 

  25. Kroenke K. Conducting research as a busy clinician-teacher or trainee. Starting blocks, hurdles, and finish lines. J Gen Intern Med. 1996;11(6):360–5.

    Article  Google Scholar 

  26. Ramsey PG, Caroline JD, Inui TS, Larson EB, LoGerfo JP, Wenrich MD. Predictive validity of certification by the american board of internal medicine. Ann Intern Med. 1989;110(9):719–26.

    Article  Google Scholar 

  27. Garibaldi RA, Trontell MC, Waxman H, et al. The in-training examination in internal medicine. Ann Intern Med. 1994;121(2):117–23.

    Article  Google Scholar 

  28. Garibaldi RA, Subhiyah R, Moore ME, Waxman H. The In-training examination in internal medicine: an analysis of resident performance over time. Ann Intern Med. 2002;137(6):505–10.

    Article  Google Scholar 

  29. West CP, Huntington JL, Huschka MM, et al. A prospective study of the relationship between medical knowledge and professionalism among internal medicine residents. Acad Med. 2007;82(6):587–92.

    Article  Google Scholar 

  30. Cook DA, Beckman TJ. Does scale length matter? A comparison of nine-versus five-point rating scales for the mini-CEX. Adv Health Sci Educ Theory Pract. 2009;14(5):655–64.

    Article  Google Scholar 

  31. Cook DA, Beckman TJ. Reflections on experimental research in medical education. Adv Health Sci Educ Theory Pract. 2010;15(3):455–64.

    Article  Google Scholar 

  32. Norcini JJ, Blank LL, Duffy FD, Fortna GS. The mini-CEX: a method for assessing clinical skills. Ann Intern Med. 2003;138(6):476–81.

    Article  Google Scholar 

  33. Kogan JR, Bellini LM, Shea JA. Feasibility, reliability, and validity of the mini-clinical evaluation exercise (mCEX) in a medicine core clerkship. Acad Med. 2003;78(10 suppl):S33–35.

    Article  Google Scholar 

  34. Hatala R, Ainslie M, Kassen BO, Mackie I, Roberts JM. Assessing the mini-Clinical Evaluation Exercise in comparison to a national specialty examination. Med Educ. 2006;40(10):950–6.

    Article  Google Scholar 

  35. Holmboe ES, Huot S, Chung J, Norcini J, Hawkins RE. Construct validity of the miniclinical evaluation exercise (miniCEX). Acad Med. 2003;78(8):826–30.

    Article  Google Scholar 

  36. Durning SJ, Cation LJ, Markert RJ, Pangaro LN. Assessing the reliability and validity of the mini-clinical evaluation exercise for internal medicine residency training. Acad Med. 2002;77(9):900–4.

    Article  Google Scholar 

  37. Beckman TJ, Mandrekar JN, Engstler GJ, Ficalora RD. Determining reliability of clinical assessment scores in real time. Teach Learn Med. 2009;21(3):188–94.

    Article  Google Scholar 

  38. Applegate WB, Williams ME. Career Development in academic medicine. Am J Med. 1990;88(3):263–7.

    Article  Google Scholar 

  39. Rothberg MB. Overcoming the obstacles to research during residency: what does it take? JAMA. 2012;308(21):2191–2.

    Article  Google Scholar 

  40. Potti A, Mariani P, Saeed M, Smego Jr RA. Residents as researchers: expectations, requirements, and productivity. Am J Med. 2003;115(6):510–4.

    Article  Google Scholar 

  41. Ryan RM, Deci EL. Intrinsic and Extrinsic Motivations: Classic Definitions and New Directions. Contemp Educ Psychol. 2000;25(1):54–67.

    Article  Google Scholar 

  42. Souba WW, Tanabe KK, Gadd MA, Smith BL, Bushman MS. Attitudes and opinions toward surgical research. A survey of surgical residents and their chairpersons. Ann Surg. 1996;223(4):377–83.

    Article  Google Scholar 

  43. Khan K, Levstik M. Ranking in Canadian gastroenterology residency match: what do residents and program directors want? Can J Gastroenterol. 2010;24(6):369–72.

    Google Scholar 

  44. Calhoun KH, Bailey BJ. Thriving in Academic Medicine. Otolaryngol Head and Neck Surg. 1997;117(6):575–9.

    Article  Google Scholar 

  45. Iramaneerat C, Yudkowsky R. Rater errors in a clinical skills assessment of medical students. Eval Health Prof. 2007;30(3):266–83.

    Article  Google Scholar 

  46. McLaughlin K, Vitale G, Coderre S, Violato C, Wright B. Clekship evaluation-- what are we measuring? Med Teach. 2009;31:2.

    Article  Google Scholar 

  47. Takahashi O, Ohde S, Jacobs JL, Tokuda Y, Omata F, Fukui T. Residents' experience of scholarly activities is assocaited with higher satisfaction with residency training. J Gen Intern Med. 2009;24(6):716–20.

    Article  Google Scholar 

  48. Zahtz G, Vambutas A, Hussey HM, Rosen L. Resident research experience and career path association: a national survey of recent otolaryngology graduates. Otolaryngol Head and Neck Surg. 2014;151(1):46–51.

    Article  Google Scholar 

  49. Levine RB, Hebert RS, Wright SM. Resident research and scholarly activity in internal medicine residency training programs. J Gen Intern Med. 2005;20(2):155–9.

    Article  Google Scholar 

  50. Alquire PC, Anderson WA, Albrecht RR, Poland GA. Resident research in internal medicine training programs. Ann Intern Med. 1996;124(3):321–8.

    Article  Google Scholar 

  51. Schultz HJ. Reserach during internal medicine residency training: meeting the challenge of the Residency Review Committee. Ann Intern Med. 1996;124(3):340–2.

    Article  Google Scholar 

  52. Beckman TJ, Reed DA, Shanafelt TD, West CP. Resident Physician well-being and assessments of their knowledge and clinical performance. J Gen Intern Med. 2012;2012(27):3.

    Google Scholar 

  53. Cullen MW, Reed DA, Halvorsen AJ, Wittich CM, Baumann Kreuziger LM, Keddis MT, et al. Selection criteria for internal medicine residency applicants and professionalism ratings during internship. The Mayo Clin Proc. 2011;33(1):197–202.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Thomas J. Beckman.

Additional information

Competing interests

None of the authors have any competing interests.

Authors’ contributions

LS, AW, CW, DR, AH and TB contributed to conception and design. LS, CW, AH and GE contributed to data acquisition. All authors contributed to literature review, interpretation of data, and drafting and revision of the article. All authors approved the final manuscript for publication. All authors are in agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Seaburg, L.A., Wang, A.T., West, C.P. et al. Associations between resident physicians’ publications and clinical performance during residency training. BMC Med Educ 16, 22 (2016). https://doi.org/10.1186/s12909-016-0543-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12909-016-0543-2

Keywords