Skip to main content
Erschienen in: Perspectives on Medical Education 2/2018

Open Access 09.03.2018 | Eye-Opener

Progress test utopia

verfasst von: Cees van der Vleuten, Adrian Freeman, Carlos Fernando Collares

Erschienen in: Perspectives on Medical Education | Ausgabe 2/2018

download
DOWNLOAD
print
DRUCKEN
insite
SUCHEN

Abstract

This paper discusses the advantages of progress testing. A utopia is described where medical schools would work together to develop and administer progress testing. This would lead to a significant reduction of cost, an increase in the quality of measurement and phenomenal feedback to learner and school. Progress testing would also provide more freedom and resources for more creative in-school assessment. It would be an educationally attractive alternative for the creation of cognitive licensing exams. A utopia is always far away in the future, but by formulating a vision for that future we may engage in discussions on how to get there.

Progress testing

Innovations in education move slowly. An innovative way of assessing knowledge through progress testing was introduced in the late 1970 s [1]. A progress test is a comprehensive written test using preferably scenario-based multiple choice items that is periodically administered to all the students in a curriculum. Naturally with every test administration a new test is developed or new items are retrieved and reviewed from an item bank. Progress testing has many advantages [2, 3], but we restrict ourselves to two. First, progress testing fits with modern constructivist education because it does not reinforce test-directed learning [4] and promotes long-term knowledge and learning methods [5]. Gone are the days when you could train doctors using a finite syllabus. Over a 5-year course the expansion of medical knowledge will be exponential. Repeated testing enhances learning and helps to create the scaffolds for future learning (https://​youtu.​be/​l_​x-5haIVRs).
The second advantage is the information that it provides to learners and schools. The learner receives not only profile scores on every test, but also on their longitudinal knowledge growth [6]. Schools can see in which areas the curriculum is effective and which not. When multiple schools participate in progress testing comparative information can be obtained which is even more useful for curriculum evaluation [7]. All depending on the number of tests per year, the number of data points available in a learner’s full curriculum is impressive and provides a wealth of information that no other assessment approach could deliver. There are high correlations between progress tests, allowing early detection of lower performing learners and providing opportunities for early remediation. Despite these advantages, it has taken a long time for schools to adopt this testing approach. In recent years, we see that schools have successfully introduced progress testing [8], also outside medicine [9]. We know of many more schools that have not published about their implementation. In this paper we would like to take a more radical position by using progress testing to significantly advance assessment in medical education. We realize it is utopia, but we would nevertheless make the argument to fuel the discussion about where we should go.

Progress test utopia

In our utopia schools would work together to collaboratively develop and administer progress testing. It is a waste of resources for schools to reinvent the wheel and start an expensive process of item production and test administration. Progress testing is just that, using tests to measure progress (https://​www.​youtube.​com/​watch?​v=​Mc_​-fVmEf1M). That means that individual schools can still have unique curricula and different learning methods. The tests are just the tools to see if the student is making progress within those unique curricula.
In the Netherlands six out of the eight medical schools have combined forces and collaboratively develop and administer progress tests. It has led to an enormous reduction of cost [10], while at the same time it has the benefits as sketched above for the approximately 10,000 learners involved and for the schools themselves. At the same time, it has silenced any discussion on national exams. Naturally, the Netherlands is but a tiny country with few medical schools. Could we not do the same internationally with English speaking schools? Imagine the synergy that this would provide? Let us sketch a few examples of predictable positive developments.
Progress testing with all of its educational benefits would be a low-cost assessment strategy. Schools would have to contribute yearly with a few but high quality items to a central bank. Some cost would be associated with the central administrative and psychometric expertise that is required, but when more schools buy in this would be a marginal cost. Schools would also need to agree on a blueprint for the test. The European Board of Medical Assessors offers progress testing in an online adaptive form (http://​www.​ebma.​eu). This allows a further reduction of the length of the test by about 50%, while the reliability is significantly higher (above 0.90). This is particularly advantageous for students in junior years, a period known to have lower reliabilities in its traditional, paper-based form [2]. With an existing large item bank, schools could flexibly administer progress tests whenever they wish and as frequently as they wish. This would probably deliver the cheapest form of assessment per unit of measurement information in the ability of a test to differentiate learners’ cognitive competence.
Many countries have national licensing exams, but also many countries have no licensing exams. Some countries are preparing or contemplating national exams. Big bang final exams may be useful as a licensing strategy to protect the public [11, 12], but they are not so useful as an educational strategy that would allow both students and schools to improve their performance longitudinally [13]. Moreover, licensing exams are costly. With a comprehensive progress testing approach the need for cognitive national licensing exams evaporates. Written tests predict other written tests and progress tests have shown to accurately predict performance on licencing tests [14]. It is a complete win-win. Schools know how they perform relative to each other, while learners profit from all the feedback. And all of this at a very reasonable cost. With all the knowledge on progress testing, embarking on a traditional licensing exam with all the potential side effects [15] seems almost an outdated strategy.
Finally, an educational synergy should be sketched that might have a phenomenal impact on assessment in medical education. Once a system of progress testing is in place in a school’s assessment strategy, the big question is what it replaces. In some schools the progress test is the only form of knowledge testing [8]. Imagine the resource implications of such a strategy! Other assessment strategies would be possible as well. With a periodic progress test in place, one would have a very precise view of a learner’s knowledge. There is not much point repeating such assessments again locally. It allows assessment in the school to be more creative. With the massive move towards competency-based learning, there is a great need for more authentic forms of assessment and assessing behaviours. It is these assessments that a school might focus on. With the resources being saved, more expenditure is available for these authentic forms of assessment. This would have a massive impact and would drive education in a very desirable direction.

Conclusion

Utopia is a sketch of the ideal. Given that educational changes move slowly, we are aware we are far away from that ideal. Nevertheless, we might start small and team up with a few medical schools willing to pursue the vision we have shared here: implementing and optimizing a cost-effective assessment tool capable of fostering excellence in learning. It would already be fantastic if some of the current schools using progress testing were to collaborate and reach some of the above synergies. All the psychometric and technological tools necessary for our utopia to become a reality are already available. Together we could make a big difference in assessment in medical education. Let’s start tomorrow.
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://​creativecommons.​org/​licenses/​by/​4.​0/​), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
download
DOWNLOAD
print
DRUCKEN
Literatur
1.
Zurück zum Zitat Van der Vleuten C, Verwijnen G, Wijnen W. Fifteen years of experience with progress testing in a problem-based learning curriculum. Med Teach. 1996;18:103–9.CrossRef Van der Vleuten C, Verwijnen G, Wijnen W. Fifteen years of experience with progress testing in a problem-based learning curriculum. Med Teach. 1996;18:103–9.CrossRef
2.
Zurück zum Zitat Wrigley W, Van der Vleuten CP, Freeman A, Muijtjens A. A systemic framework for the progress test: strengths, constraints and issues: AMEE Guide No. 71. Med Teach. 2012;34:683–97.CrossRef Wrigley W, Van der Vleuten CP, Freeman A, Muijtjens A. A systemic framework for the progress test: strengths, constraints and issues: AMEE Guide No. 71. Med Teach. 2012;34:683–97.CrossRef
3.
Zurück zum Zitat Schuwirth LW, van der Vleuten CP. The use of progress testing. Perspect Med Educ. 2012;1:24–30.CrossRef Schuwirth LW, van der Vleuten CP. The use of progress testing. Perspect Med Educ. 2012;1:24–30.CrossRef
4.
Zurück zum Zitat Pugh D, Regehr G. Taking the sting out of assessment: is there a role for progress testing? Med Educ. 2016;50:721–9.CrossRef Pugh D, Regehr G. Taking the sting out of assessment: is there a role for progress testing? Med Educ. 2016;50:721–9.CrossRef
5.
Zurück zum Zitat Heeneman S, Schut S, Donkers J, Van der Vleuten C, Muijtjens A. Embedding of the progress test in an assessment program designed accoring to the principles of programmatic assessment. Med Teach. 2017;39:44–52.CrossRef Heeneman S, Schut S, Donkers J, Van der Vleuten C, Muijtjens A. Embedding of the progress test in an assessment program designed accoring to the principles of programmatic assessment. Med Teach. 2017;39:44–52.CrossRef
6.
Zurück zum Zitat Muijtjens AM, Timmermans I, Donkers J, et al. Flexible electronic feedback using the virtues of progress testing. Med Teach. 2010;32:491–5.CrossRef Muijtjens AM, Timmermans I, Donkers J, et al. Flexible electronic feedback using the virtues of progress testing. Med Teach. 2010;32:491–5.CrossRef
7.
Zurück zum Zitat van der Vleuten CP, Schuwirth LW, Muijtjens AM, Thoben AJ, Cohen-Schotanus J, van Boven CP. Cross institutional collaboration in assessment: a case on progress testing. Med Teach. 2004;26:719–25.CrossRef van der Vleuten CP, Schuwirth LW, Muijtjens AM, Thoben AJ, Cohen-Schotanus J, van Boven CP. Cross institutional collaboration in assessment: a case on progress testing. Med Teach. 2004;26:719–25.CrossRef
8.
Zurück zum Zitat Coombes L, Ricketts C, Freeman A, Stratford J. Beyond assessment: feedback for individuals and institutions based on the progress test. Med Teach. 2010;32:486–90.CrossRef Coombes L, Ricketts C, Freeman A, Stratford J. Beyond assessment: feedback for individuals and institutions based on the progress test. Med Teach. 2010;32:486–90.CrossRef
9.
Zurück zum Zitat Bennett J, Freeman A, Coombes L, Kay L, Rickett SC. Adaptation of medical progress testing to a dental setting. Med Teach. 2010;32:500–2.CrossRef Bennett J, Freeman A, Coombes L, Kay L, Rickett SC. Adaptation of medical progress testing to a dental setting. Med Teach. 2010;32:500–2.CrossRef
10.
Zurück zum Zitat van der Vleuten CP, Heeneman S. On the issue of costs in programmatic assessment. Perspect Med Educ. 2016;5:303–7.CrossRef van der Vleuten CP, Heeneman S. On the issue of costs in programmatic assessment. Perspect Med Educ. 2016;5:303–7.CrossRef
11.
Zurück zum Zitat Archer J, Lynn N, Coombes L, et al. The impact of large scale licensing examinations in highly developed countries: a systematic review. BMC Med Educ. 2016;16:212.CrossRef Archer J, Lynn N, Coombes L, et al. The impact of large scale licensing examinations in highly developed countries: a systematic review. BMC Med Educ. 2016;16:212.CrossRef
12.
Zurück zum Zitat Melnick DE. Licensing examinations in North America: is external audit valuable? Med Teach. 2009;31:212–4.CrossRef Melnick DE. Licensing examinations in North America: is external audit valuable? Med Teach. 2009;31:212–4.CrossRef
13.
Zurück zum Zitat Van der Vleuten CPM. Validity of final examinations in undergraduate medical training. Br Med J. 2000;321:1217–9.CrossRef Van der Vleuten CPM. Validity of final examinations in undergraduate medical training. Br Med J. 2000;321:1217–9.CrossRef
14.
Zurück zum Zitat Norman G, Neville A, Blake JM, Mueller B. Assessment steers learning down the right road: impact of progress testing on licensing examination performance. Med Teach. 2010;32:496–9.CrossRef Norman G, Neville A, Blake JM, Mueller B. Assessment steers learning down the right road: impact of progress testing on licensing examination performance. Med Teach. 2010;32:496–9.CrossRef
15.
Zurück zum Zitat Harden RM. Five myths and the case against a European or national licensing examination. Med Teach. 2009;31:217–20.CrossRef Harden RM. Five myths and the case against a European or national licensing examination. Med Teach. 2009;31:217–20.CrossRef
Metadaten
Titel
Progress test utopia
verfasst von
Cees van der Vleuten
Adrian Freeman
Carlos Fernando Collares
Publikationsdatum
09.03.2018
Verlag
Bohn Stafleu van Loghum
Erschienen in
Perspectives on Medical Education / Ausgabe 2/2018
Print ISSN: 2212-2761
Elektronische ISSN: 2212-277X
DOI
https://doi.org/10.1007/s40037-018-0413-1

Weitere Artikel der Ausgabe 2/2018

Perspectives on Medical Education 2/2018 Zur Ausgabe