Tablet versus paper marking in assessment: feedback matters

The transition from paper-based to computerized recording of candidate performance brings challenges and risks, but also anticipated and unanticipated benefits. In this work, we found that the adoption of new technology led to the provision of improved quantity and also quality of comments left by OSCE examiners, which are used for the provision of student feedback. The electronic nature of the comments may shorten the time interval before feedback can be provided to students, and may also facilitate the future provision of feedback to all (rather than just some) candidates.

There was a reduction in the number of occasions where examiners left only non-specific praise for candidates using the iPad devices. While positive comments allow candidates to know something was done correctly, there are risks in using this type of language. Ambiguous comments, such as ‘good effort,’ do not inform the recipients which aspects of the performance were of value [11]. In a study of helpful and unhelpful feedback techniques, Hewson and Little [12] describe an incident deemed unhelpful due to the non-specific praise delivered, establishing the importance of ensuring feedback is based on observations, including when praising someone. There was a difference in feedback provision between the formative and summative contexts when using iPads. The explanation for this is uncertain. Although a learning curve associated with the iPad could influence this, a more plausible explanation is that the nature of formative and summative assessments differs, and that examiners may purposefully provide more detailed feedback at formative encounters, and restrict their comments to weaker candidates at summative encounters.

The reduction in the overall number of comments left may be complex and influenced by several factors including examiner training, overall competence of the candidates and the time constraints of individual stations.

A key strength of the work is that the same test items were used for both paper- and tablet-marked OSCEs in separate academic years and cohorts. This allowed us to interrogate candidate/examiner data for nearly 4000 OSCE stations, and be confident that the introduction of the new technology did not weaken the provision of quality or quantity of examiner comments.

Although the OSCE station material was identical between assessments, the individual examiners were not, and the initial examiners using tablet devices might have been self-selecting. However, we purposefully used our standard recruitment measures for examiners for the iPad OSCEs, since we wished to ensure that the technology would work irrespective of the seniority and background of examiners. It is also possible that with the new technology, the stability and rise in the comments provided reflected not the design of the app, but bias due to early adoption of new technology [13], with examiners interacting more with the iPad than they might otherwise do with paper. However, tablet computers have an increasing footprint in healthcare service contexts, and as such the novelty factor is likely to be modest. We also acknowledge that other factors such as cohort effect, administration differences and slight curricular changes would inevitably make identical comparisons between the two cohorts impossible.

A further potential limitation of this study is that for both iPad-based OSCE mark schemes, the comments that have been analyzed are from a section entitled ‘Comments’. Whilst these are used for feedback purposes during student support interviews, these comments are not necessarily written by the examiners with the intent that they will be directly given to the students. Nevertheless, we have demonstrated the improvement in the quality of the comments for use as feedback, using an iPad compared with a paper-based system. Another next step could be to change the title on the iPad marking system to ‘Candidate feedback’ instead of ‘Comments’ and encourage the examiners at the pre-examination briefing to leave ‘Feedback for the candidates’ rather than emphasizing that ‘their comments will be helpful’ when reviewing the students’ OSCE mark sheets. Some examiners may assume that this is what the ‘Comments’ are for, but making this explicit may further enhance the feedback quantity and quality received.

Although several qualitative frameworks exist to analyze written feedback, we judged that none were suitable for the context of brief comments in an OSCE. The rating scale used to assess the quality of the free-text comments was developed and piloted in an iterative manner with independent testing of reliability to maximize its utility. The scale took into consideration several aspects of feedback that were desirable or conversely could be considered unhelpful. Further validation would be valuable.

A further strength of using tablet devices was that missing data (i.e. unmarked items on the checklist) never occurred, due to the design of the app. Although the number of marks missed during the paper-marked assessments was small, this probably underestimates the true incidence since examiners may be prompted (or remember) to complete unmarked items in advance of later administrative checking systems. Ultimately, less missing marks results in a higher quality of data.

An under-anticipated benefit of using tablet devices was the very considerable saving both in administrative staff time and cost of single-use specialist computer-readable paper sheets. While this has been described previously [3], the advantages were well in excess of our expectations. Raw results can be available for analysis within minutes following the final OSCE bell. Tablet devices are individually expensive, but are reusable and are deployed in multiple different assessment and other contexts in our institution.

The exams under scrutiny in this work were first year OSCEs and the stations were straightforward. Careful analysis of our planned further deployment of tablet marking in more advanced stations in senior years will be required, to determine if tablets will remain a suitable vehicle for capturing comments for use as feedback, which may be more complex than in early years curricula.

In conclusion, the use of tablet devices in OSCE assessments is associated with improved examiner comment for use as feedback compared with the traditional, paper-based data capture using computer read mark sheets.