nach oben

Canadian Journal of Anesthesia/Journal canadien d'anesthésie

Erschienen in:

01.02.2019 | Reports of Original Investigations

The reliability of a portfolio of workplace-based assessments in anesthesia training

verfasst von: Damian J. Castanelli, MBBS, Joyce M. W. Moonen-van Loon, PhD, Brian Jolly, PhD, Jennifer M. Weller, MD

Erschienen in: Canadian Journal of Anesthesia/Journal canadien d'anesthésie | Ausgabe 2/2019

Einloggen, um Zugang zu erhalten

Abstract

Purpose

Competency-based anesthesia training programs require robust assessment of trainee performance and commonly combine different types of workplace-based assessment (WBA) covering multiple facets of practice. This study measured the reliability of WBAs in a large existing database and explored how they could be combined to optimize reliability for assessment decisions.

Methods

We used generalizability theory to measure the composite reliability of four different types of WBAs used by the Australian and New Zealand College of Anaesthetists: mini-Clinical Evaluation Exercise (mini-CEX), direct observation of procedural skills (DOPS), case-based discussion (CbD), and multi-source feedback (MSF). We then modified the number and weighting of WBA combinations to optimize reliability with fewer assessments.

Results

We analyzed 67,405 assessments from 1,837 trainees and 4,145 assessors. We assumed acceptable reliability for interim (intermediate stakes) and final (high stakes) decisions of 0.7 and 0.8, respectively. Depending on the combination of WBA types, 12 assessments allowed the 0.7 threshold to be reached where one assessment of any type has the same weighting, while 20 were required for reliability to reach 0.8. If the weighting of the assessments is optimized, acceptable reliability for interim and final decisions is possible with nine (e.g., two DOPS, three CbD, two mini-CEX, two MSF) and 15 (e.g., two DOPS, eight CbD, three mini-CEX, two MSF) assessments respectively.

Conclusions

Reliability is an important factor to consider when designing assessments, and measuring composite reliability can allow the selection of a WBA portfolio with adequate reliability to provide evidence for defensible decisions on trainee progression.

Australian and New Zealand College of Anaesthetists. Anaesthesia training program curriculum. Melbourne: Australian and New Zealand College of Anaesthetists, 2012. Available from URL: http://www.anzca.edu.au/documents/anaesthesia-training-program-curriculum.pdf (accessed October 2018).

Van Gessel E, Mellin-Olsen J, Ostergaard HT. Niemi-Murola L; Education and Training Standing Committee, European Board of Anaesthesiology, Reanimation and Intensive Care. Postgraduate training in anaesthesiology, pain and intensive care: the new European competence-based guidelines. Eur J Anaesthesiol 2012; 29: 165-8.CrossRefPubMed

Stodel EJ, Wyand A, Crooks S, Moffett S, Chiu M, Hudson CC. Designing and implementing a competency-based training program for anesthesiology residents at the University of Ottawa. Anesthesiol Res Pract 2015; 2015: 713038.PubMedPubMedCentral

van der Vleuten CP, Schuwirth LW, Driessen EW, et al. A model for programmatic assessment fit for purpose. Med Teach 2012; 34: 205-14.CrossRefPubMed

Bok HG, Teunissen PW, Favier RP, et al. Programmatic assessment of competency-based workplace learning: when theory meets practice. BMC Med Educ 2013; 13: 123.CrossRefPubMedPubMedCentral

Norcini J, Burch V. Workplace-based assessment as an educational tool: AMEE Guide No. 31. Med Teach 2007; 29: 855-71.CrossRef

Weller JM, Castanelli DJ, Chen Y, Jolly B. Making robust assessments of specialist trainees’ workplace performance. Br J Anaesth 2017; 118: 207-14.CrossRefPubMed

Weller JM, Jolly B, Misur MP, et al. Mini-clinical evaluation exercise in anaesthesia training. Br J Anaesth 2009; 102: 633-41.CrossRefPubMed

Weller JM, Misur M, Nicolson S, et al. Can I leave the theatre? A key to more reliable workplace-based assessment. Br J Anaesth 2014; 112: 1083-91.CrossRefPubMed

10.

Watson MJ, Wong DM, Kluger R, et al. Psychometric evaluation of a direct observation of procedural skills assessment tool for ultrasound-guided regional anaesthesia. Anaesthesia 2014; 69: 604-12.CrossRefPubMed

11.

Moonen-van Loon JM, Overeem K, Donkers HH, van der Vleuten CP, Driessen EW. Composite reliability of a workplace-based assessment toolbox for postgraduate medical education. Adv Health Sci Educ Theory Pract 2013; 18: 1087-102.CrossRefPubMed

12.

Wragg A, Wade W, Fuller G, Cowan G, Mills P. Assessing the performance of specialist registrars. Clin Med (Lond) 2003; 3: 131-4.CrossRef

13.

Downing SM. Reliability: on the reproducibility of assessment data. Med Educ 2004; 38: 1006-12.CrossRefPubMed

14.

Crossley J, Russell J, Jolly B, et al. ‘I’m pickin’ up good regressions’: the governance of generalisability analyses. Med Educ 2007; 41: 926-34.CrossRefPubMed

15.

Moonen-van Loon JM, Overeem K, Govaerts MJ, Verhoeven BH, van der Vleuten CP, Driessen EW. The reliability of multisource feedback in competency-based assessment programs: the effects of multiple occasions and assessor groups. Acad Med 2015; 90: 1093-9.CrossRefPubMed

16.

Crossley J, Jolly B. Making sense of work-based assessment: ask the right questions, in the right way, about the right things, of the right people. Med Educ 2012; 46: 28-37.CrossRefPubMed

17.

Pelgrim EA, Kramer AW, Mokkink HG, van den Elsen L, Grol RP, van der Vleuten CP. In-training assessment using direct observation of single-patient encounters: a literature review. Adv Health Sci Educ Theory Pract 2011; 16: 131-42.CrossRefPubMed

18.

Lockyer J, Violato C, Fidler H. A multi source feedback program for anesthesiologists. Can J Anesth 2006; 53: 33-9.CrossRefPubMed

19.

Brennan RL. Generalizability Theory and classical test theory. J Appl Meas Educ 2010; 24: 1-21.CrossRef

20.

van der Vleuten CP, Schuwirth LW, Scheele F, Driessen EW, Hodges B. The assessment of professional competence: building blocks for theory development. Best Pract Res Clin Obstet Gynaecol 2010; 24: 703-19.CrossRefPubMed

21.

Castanelli DJ, Jowsey T, Chen Y, Weller JM. Perceptions of purpose, value, and process of the mini-Clinical Evaluation Exercise in anesthesia training. Can J Anesth 2016; 63: 1345-56.CrossRefPubMed

22.

Driessen EW, van Tartwijk J, Govaerts M, Teunissen P, van der Vleuten CP. The use of programmatic assessment in the clinical workplace: a Maastricht case report. Med Teach 2012; 34: 226-31.CrossRefPubMed

23.

Van Tartwijk J, Driessen EW. Portfolios for assessment and learning: AMEE Guide no. 45. Med Teach 2009; 31: 790-801.CrossRefPubMed

24.

Ten Cate O, Chen HC, Hoff RG, Peters H, Bok H, van der Schaaf M. Curriculum development for the workplace using Entrustable Professional Activities (EPAs): AMEE Guide No. 99. Med Teach 2015; 37: 983-1002.CrossRefPubMed

25.

Wisman-Zwarter N, van der Schaaf M, Ten Cate O, Jonker G, van Klei WA, Hoff RG. Transforming the learning outcomes of anaesthesiology training into entrustable professional activities: a Delphi study. Eur J Anaesthesiol 2016; 33: 559-67.CrossRefPubMed

26.

Jonker G, Hoff RG, Ten Cate OT. A case for competency-based anaesthesiology training with entrustable professional activities: an agenda for development and research. Eur J Anaesthesiol 2015; 32: 71-6.CrossRefPubMed

27.

Eva KW, Macala C. Multiple mini-interview test characteristics: ‘tis better to ask candidates to recall than to imagine. Med Educ 2014; 48: 604-13.CrossRefPubMed

Titel: The reliability of a portfolio of workplace-based assessments in anesthesia training
verfasst von: Damian J. Castanelli, MBBS
Joyce M. W. Moonen-van Loon, PhD
Brian Jolly, PhD
Jennifer M. Weller, MD
Publikationsdatum: 01.02.2019
Verlag: Springer International Publishing
Erschienen in: Canadian Journal of Anesthesia/Journal canadien d'anesthésie / Ausgabe 2/2019
Print ISSN: 0832-610X
Elektronische ISSN: 1496-8975
DOI: https://doi.org/10.1007/s12630-018-1251-7

Neu im Fachgebiet AINS

Mehr Frauen im OP – weniger postoperative Komplikationen

21.05.2024 Allgemeine Chirurgie Nachrichten

Ein Frauenanteil von mindestens einem Drittel im ärztlichen Op.-Team war in einer großen retrospektiven Studie aus Kanada mit einer signifikanten Reduktion der postoperativen Morbidität assoziiert.

Delir bei kritisch Kranken – Antipsychotika versus Placebo

16.05.2024 Delir Nachrichten

Um die Langzeitfolgen eines Delirs bei kritisch Kranken zu mildern, wird vielerorts auf eine Akuttherapie mit Antipsychotika gesetzt. Eine US-amerikanische Forschungsgruppe äußert jetzt erhebliche Vorbehalte gegen dieses Vorgehen. Denn es gibt neue Daten zum Langzeiteffekt von Haloperidol bzw. Ziprasidon versus Placebo.

Eingreifen von Umstehenden rettet vor Erstickungstod

15.05.2024 Fremdkörperaspiration Nachrichten

Wer sich an einem Essensrest verschluckt und um Luft ringt, benötigt vor allem rasche Hilfe. Dass Umstehende nur in jedem zweiten Erstickungsnotfall bereit waren, diese zu leisten, ist das ernüchternde Ergebnis einer Beobachtungsstudie aus Japan. Doch es gibt auch eine gute Nachricht.

Darf man die Behandlung eines Neonazis ablehnen?

08.05.2024 Gesellschaft Nachrichten

In einer Leseranfrage in der Zeitschrift Journal of the American Academy of Dermatology möchte ein anonymer Dermatologe bzw. eine anonyme Dermatologin wissen, ob er oder sie einen Patienten behandeln muss, der eine rassistische Tätowierung trägt.

Update AINS

Bestellen Sie unseren Fach-Newsletter und bleiben Sie gut informiert.

hier abonnieren

Die Highlights vom Kongress des American College of Cardiology 2024

Springer Medizin

The reliability of a portfolio of workplace-based assessments in anesthesia training