Skip to main content
Erschienen in:

Open Access 01.08.2017 | Short Communication

An evaluation system for postgraduate pediatric residency programs: report of a 3-year experience

verfasst von: Liviana Da Dalt, Pasquale Anselmi, Sara Furlan, Silvia Carraro, Eugenio Baraldi, Egidio Robusto, Giorgio Perilongo

Erschienen in: European Journal of Pediatrics | Ausgabe 9/2017

Abstract

The way a postgraduate medical training program is organized and the capacity of faculty members to function as tutors and to organize effective professional experiences are among the elements that affect the quality of training. An evaluation system designed to target these elements has been implemented within the framework of the Pediatric Residency Program of the University of Padua (Italy). The aim of this report is to describe some aspects of the experience gained in the first 3 years of implementation of the system (2013–2015). Data were collected using four validated questionnaires: the “Resident Assessment Questionnaire”, the “Tutor-Assessment Questionnaire”, the “Rotation-Assessment Questionnaire”, and the “Resident Affairs Committee-Assessment Questionnaire”. The response rate was 72% for the “Resident Assessment Questionnaires”; 78% for the “Tutor-/Rotation-Assessment Questionnaires” and 84% for the “Resident Affair Committee-Assessment Questionnaires”. The scores collected were validated by psychometric tests.
Conclusion: The high rates of completed questionnaires returned and the psychometric validation of the results collected indicate that the evaluation system reported herein can be effectively implemented. Efforts should be made to refine this system and, more importantly, to document its impact in improving the Pediatric Residency Program.
What is known:
• The elements that influence the quality of postgraduate training programs and the knowledge, performance, and competences of residents must be regularly assessed.
• Comprehensive evaluation systems for postgraduate residency programs are not universally implemented also because quite often common guidelines and rules, well-equipped infrastructures, and financial resources are missing.
What is new:
We show the feasibility of implementing an evaluation system that targets some of the key elements of a postgraduate medical training program in Italy, a European country in which the regulations governing training programs and, notably, the evaluation of residents are still being developed.
Hinweise
Communicated by Mario Bianchetti
Abkürzungen
AQ
Assessment Questionnaire
Resident Affair Committee-AQ
Resident Affair Committee-Assessment Questionnaire
Resident-AQ
Resident-Assessment Questionnaire
Rotation-AQ
Rotation-Assessment Questionnaire
Tutor-AQ
Tutor-Assessment Questionnaire

Introduction

Postgraduate medical training programs should be continuously monitored to guarantee the quality of the education curriculum [4]. The way the program is run and the capacity of the faculty members to function as tutors and to organize effective professional experiences are elements which, among others, can affect the learning experience of residents and, ultimately, the quality of doctors licensed to practice medicine. The Pediatric Residency Program of the University of Padua (Italy) has adopted an evaluation system specifically designed to target these elements. This report describes the experience gained over the first 3 years (2013–2015).

Materials and methods

The Pediatric Residency Program of the University of Padua is a Ministerial accredited and ISO-9001 certified 5-year program that offers a Diploma of Specialization in Pediatrics [1]. An average of 90 residents attends the program every year. Approximately, 80% of the learning activities takes place in the clinical setting under faculty supervision. During their first 3 years, residents rotate through an average of 12 Pediatric Divisions of the Department of Woman’s and Child’s Health of the University Hospital of Padua. These rotations range from 3 to 6 months. During the last 2 years, residents select elective rotations involving at the most two divisions, each lasting from 9 to 15 months. Each Division Chief is responsible for organizing the rotation, for being tutor for residents and for evaluating them. The Resident Affairs Committee is the body in charge of running the program.
To monitor the program, the Resident Affairs Committee of the Pediatric Residency Program has developed an evaluation system according to which Division Chiefs (tutors) are requested to evaluate residents, and residents are requested to evaluate the quality of the training provided by each Division Chief, the professional experience acquired during the various rotations, and the functioning of the Resident Affairs Committee.
This evaluation system is based on four validated questionnaires: the “Residents Assessment Questionnaire (AQ)”, the “Tutor-AQ”, the “Rotation-AQ” and the “Resident Affairs Committee-AQ” [2, 3] that consist of various items evaluated on a four-point rating scale (from 1 “poor” to 4 “excellent”). The “Resident-AQs” are completed by the Division Chiefs, who, as mentioned above, are also the “tutors” that the residents evaluate by completing the “Tutor-AQ” questionnaire. The “Resident-, Tutor- and Rotation-AQs should be completed within two weeks from the end of each rotation, while the “Resident Affairs Committee-AQs” should be completed and returned at the end of each academic year. The “Tutor-AQ” is anonymous. The results of these questionnaires, except “Resident Affairs Committee-AQs”, are communicated by the Resident Affair Committee to the residents and tutors during individual private meetings. Once a year, the average scores obtained during a 12-month period by each tutor and rotation are presented to the whole faculty for discussion. Only residents of the first 4 years undergo regular evaluation (“formative assessment”) because the results of the “Resident-AQs” are meant to provide information to adjust their training program, if necessary. Fifth-year residents receive only a single final evaluation (“summative evaluation”) regarding their one year-long conclusive elective. Consequently, they are not required to complete either the “Tutor-AQ” or the “Rotation-AQ”.
An internet-based system has been developed to handle the entire evaluation process. The web-based system provides the Resident Affairs Committee with a dashboard command to monitor the implementation and the correct use of the evaluation system. On the whole, questionnaires can be completed in less than 10 min. Tutorials to instruct new residents and faculty members on how to complete the questionnaires are organized annually. The psychometric properties of the questionnaires, although previously validated, are regularly monitored to verify (a) the internal consistency of all the questionnaires, namely, to verify whether differences among the scores are true and not due to measurement errors [5]; (b) the strength of the items in defining a unidimensional measure [7]; and (c) the level of agreement among respondents, namely, that the tutors substantially agree in their evaluations of the residents, and that the residents agree in their evaluations of tutors, rotations and Resident Affairs Committee [6, 8]. At the end of the first 3 years of implementation of the evaluation system, students and faculty were asked to provide feedbacks about the system in terms of (a) purposes, (b) comprehensiveness, and (c) user-friendliness. These items are evaluated on a four-point rating scale, from 1 “poor” to 4 “excellent”.

Results

The results of the questionnaires

The number of questionnaires expected and collected in the study period is reported in Table 1.
Table 1
Number of questionnaires collected in the years 2013, 2014, and 2015
Questionnaire
Year
N. residents
Number of evaluations
Expected
Provided within 15 days
Provided
Resident-AQ
2013
71
227
95 (42%)
153 (67%)
2014
71
237
143 (60%)
165 (69%))
2015
54
158
104 (65%)
129 (82%)
Total
  
622
342 (55%)
447 (72%)
Tutor-and Rotation-AQ
2013
71
227
163 (73%)
185 (81%)
2014
71
237
139 (59%)
170 (80%)
2015
54
158
101 (64%)
130 (82%)
Total
  
622
403 (65%)
485 (78%)
RAC-AQ
2013
86
86
68 (79%)
71 (83%)
2014
89
89
58 (65%)
75 (84%)
2015
73
73
60 (82%)
63 (86%)
Total
  
248
186 (75%)
209 (84%)
AQ Assessment Questionnaire, RAC Resident Affair Committee
Resident-AQ
Tutors returned 447 “Resident-AQs”, which represented 72% of the total ones which were supposed to be returned; 55% were returned within the 15-days requested. The number of “Resident-AQs” received per resident in a year varied from one (for fourth year residents) to four (for the ones attending the first 3 years).
Tutor-AQ and Rotation-AQ
Four-hundred and eighty-five “Tutor-AQs” and “Rotation-AQs” were returned (i.e., 78% of those expected) of which 65% were returned with the 15-day deadline.
Resident Affairs Committee-AQ
Two-hundred and nine “Residents Affairs Committee-AQs” were returned (i.e., 84% of those expected; 75% of them were returned within 15 days. Ninety-nine residents evaluated the Resident Affairs Committee from one to three times.
Tutors tended to be generous in assessing the performance of residents (Fig. 1a), while residents in judging tutors, the professional value of the rotations and the Resident Affairs Committee used more often the entire scale of scores (Fig. 1b–d).
Figure 2 shows an example of the reports of the combined “Tutor-AQs” and “Rotation-AQs” that were presented during the yearly general faculty meeting: the average scores assigned to each tutor (Division Chief) and to each rotation in a year are shown.
The psychometric properties of the questionnaires (as outlined in the “Methods”) were invariably confirmed.

Feedback of students and tutors regarding the evaluation system

Feedbacks on the system were received from 76% of residents and 58% of tutors. Interestingly, 80% of the tutors but only 60% of the residents felt that the purpose of the evaluation system was clear, that the meaning of the different items was easy to understand and that the entire system would help to improve the performance of residents and the quality of training. Most residents and tutors thought that the internet-based system was easy to use.

Discussion

The relatively high rates of questionnaires completed and returned in time indicate that the evaluation system adopted within the framework of the University of Padua’s Pediatric Residency Program was successfully implemented. In fact, the analysis of the validity of the various responses provided, as assessed by conventional psychometric tests, and the fact that most tutors and residents returned the questionnaires within the time set, indicate that the task of completing the questionnaires was taken seriously by all.
The aim of the evaluation system was to provide residents, tutors and the Resident Affairs Committee with feedback regarding their performance, based on objective data, and thus to provide a base on which to continuously improve the program. We do not yet have objective data of the impact of this evaluation system on the quality of the Pediatric Residency Program. However, the presentation, during the yearly faculty meetings, of the scores obtained by each tutor and rotation has stimulated efforts to improve. Indeed, the scores assigned to tutors and rotations improved over the years. Furthermore, the fact that all the Division Chiefs were interested in having private discussions on the scores received is a sign that they took seriously the judgments received. The same applies to the residents, who were eager to receive, privately, feedback on their performance. Fortunately, no conflicts seemed to have emerged between residents and tutors and no one was offended by negative judgments. We believe that this evaluation system set a positive circle, considering the improvement of many scores of “Tutor-AQs” and “Rotation-AQs” documented over the years and the impression we had had that everybody took quite seriously the judgments received.
We acknowledge some intrinsic limitations of the system, namely, the fact that the “Tutor-AQ” was meant to target only the performance of the Division Chief, thus neglecting the contribution of all the other staff members. More should be done in order to refine it and, more importantly, to document its impact in improving the program.
Finally, we are aware that comprehensive evaluation systems require common guidelines and rules, well-equipped infrastructures and financial resources. However, we believe that good organization, strong motivation, and a culture of educational commitment can result in an excellent residency program, even with limited financial resources. This is the ultimate reason why we thought appropriate to share this limited experience.

Acknowledgements

This study has been made possible thanks to a grant awarded by the University of Padua (Italy) for inter-departmental research projects. We thank Jean Ann Gilder (Scientific Communication Srl., Naples, Italy) for writing assistance.

Compliance with ethical standards

This study was funded by grant awarded by the University of Padua, Italy for shared research projects among Departments.

Conflict of interest

None of the Authors has received research grants from Private Companies nor has received, from them, a speaker honorarium nor owns their stocks. The authors declare that they have no conflict of interest.

Ethical approval

All procedures performed were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards. Due to the nature of the research present, however, no informed consent was required from the individual participating in the study.
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Unsere Produktempfehlungen

e.Med Interdisziplinär

Kombi-Abonnement

Für Ihren Erfolg in Klinik und Praxis - Die beste Hilfe in Ihrem Arbeitsalltag

Mit e.Med Interdisziplinär erhalten Sie Zugang zu allen CME-Fortbildungen und Fachzeitschriften auf SpringerMedizin.de.

© Springer Medizin

e.Med Pädiatrie

Kombi-Abonnement

Mit e.Med Pädiatrie erhalten Sie Zugang zu CME-Fortbildungen des Fachgebietes Pädiatrie, den Premium-Inhalten der pädiatrischen Fachzeitschriften, inklusive einer gedruckten Pädiatrie-Zeitschrift Ihrer Wahl.

© Springer Medizin

Literatur
1.
Zurück zum Zitat Da Dalt L, Callegaro S, Mazzi A, Scipioni A, Lago P, Chiozza ML, Zacchello F, Perilongo G (2010) A model for quality assurance and quality improvement for post-graduate medical education in. Europe–Med Teach 32(2):e57–e64CrossRef Da Dalt L, Callegaro S, Mazzi A, Scipioni A, Lago P, Chiozza ML, Zacchello F, Perilongo G (2010) A model for quality assurance and quality improvement for post-graduate medical education in. Europe–Med Teach 32(2):e57–e64CrossRef
2.
Zurück zum Zitat Da Dalt L, Anselmi P, Bressan S, Carraro S, Baraldi E, Robusto E, Perilongo G (2013) A short questionnaire to assess pediatric resident’s competencies: the validation process. Ital J Pediatr 39:41CrossRefPubMedPubMedCentral Da Dalt L, Anselmi P, Bressan S, Carraro S, Baraldi E, Robusto E, Perilongo G (2013) A short questionnaire to assess pediatric resident’s competencies: the validation process. Ital J Pediatr 39:41CrossRefPubMedPubMedCentral
3.
Zurück zum Zitat Da Dalt L, Anselmi P, Furlan S, Carraro S, Baraldi E, Robusto E, Perilongo G (2015) Validating a set of tools designed to assess the perceived quality of training of pediatric residency programs. Ital J Pediatr 41:2CrossRefPubMedPubMedCentral Da Dalt L, Anselmi P, Furlan S, Carraro S, Baraldi E, Robusto E, Perilongo G (2015) Validating a set of tools designed to assess the perceived quality of training of pediatric residency programs. Ital J Pediatr 41:2CrossRefPubMedPubMedCentral
4.
Zurück zum Zitat Epstein RM (2007) Assessment in medical education. N Engl J Med 356(4):3 87–3 96CrossRef Epstein RM (2007) Assessment in medical education. N Engl J Med 356(4):3 87–3 96CrossRef
5.
Zurück zum Zitat Linacre JM (1989) Many-facet Rasch measurement. MESA Press, Chicago Linacre JM (1989) Many-facet Rasch measurement. MESA Press, Chicago
6.
Zurück zum Zitat Smith EV (2001) Evidence for the reliability of measures and validity of measure interpretation: a Rasch measurement perspective. J Appl Meas 39:281–311 Smith EV (2001) Evidence for the reliability of measures and validity of measure interpretation: a Rasch measurement perspective. J Appl Meas 39:281–311
7.
Zurück zum Zitat Wright BD, Linacre JM (1994) Reasonable mean-square fit values. Rasch Meas Trans 8:370 Wright BD, Linacre JM (1994) Reasonable mean-square fit values. Rasch Meas Trans 8:370
8.
Zurück zum Zitat Wright B, Stone M (1999) Measurement essentials, 2nd edn. Wide Range Inc, Wilminton Wright B, Stone M (1999) Measurement essentials, 2nd edn. Wide Range Inc, Wilminton
Metadaten
Titel
An evaluation system for postgraduate pediatric residency programs: report of a 3-year experience
verfasst von
Liviana Da Dalt
Pasquale Anselmi
Sara Furlan
Silvia Carraro
Eugenio Baraldi
Egidio Robusto
Giorgio Perilongo
Publikationsdatum
01.08.2017
Verlag
Springer Berlin Heidelberg
Erschienen in
European Journal of Pediatrics / Ausgabe 9/2017
Print ISSN: 0340-6199
Elektronische ISSN: 1432-1076
DOI
https://doi.org/10.1007/s00431-017-2967-z

Neu im Fachgebiet Pädiatrie

Podcast

Magendarminfekte: Dos and Don'ts bei Kleinkindern

Die Leitlinie zur akuten infektiösen Gastroenteritis wurde 2024 umfassend überarbeitet. Cola und Saft zur oralen Hydratation und einige Antidiarrhoika sollten vermieden werden. Von einem standardmäßigen Erregerpanel wird abgeraten. Die Hintergründe zu diesen und weiteren Empfehlungen erläutert Leitlinienkoordinator Prof. Carsten Posovszky.

Ab sofort gelten die neuen Verordnungsausnahmen für Lipidsenker

Freie Fahrt für Lipidsenker? Das nicht, doch mit niedrigerem Schwellenwert fürs Infarktrisiko und neuen Indikationen hat der G-BA die Verordnungs-Handbremse ein gutes Stück weit gelockert.

ADHS-Kranke verlieren sieben Lebensjahre

  • 11.02.2025
  • ADHS
  • Nachrichten

Ein ADHS verkürzt die Lebenserwartung um rund sieben Jahre, bei Frauen sind es sogar fast neun Jahre. Ein Großteil der erhöhten Sterblichkeit lässt sich auf modifizierbare Risikofaktoren wie Alkohol, Rauchen und psychische Begleiterkrankungen zurückführen.

Abdominale CT bei Kindern: 40% mit Zufallsbefunden

Wird bei Kindern mit stumpfem Trauma eine CT des Bauchraums veranlasst, sind in rund 40% der Fälle Auffälligkeiten zu sehen, die nichts mit dem Trauma zu tun haben. Die allerwenigsten davon sind klinisch relevant.

Update Pädiatrie

Bestellen Sie unseren Fach-Newsletter und bleiben Sie gut informiert.