Students as stakeholders in assessment: how students perceive the value of an assessment

In order for assessment to support future learning, students must engage with the results of assessment to identify areas for future learning. We adapt a validity framework to be student-relevant, and apply this framework to evaluate a newly implemented assessment. Students rated the exam well overall, commented on the educational value of the assessment, but were disappointed when they felt that the educational goals of assessment were not being met. This study suggests that students are able to engage critically with assessment, and lays a foundation for considering assessees as stakeholders beyond simplistic notions of acceptability.

Assessment is a key component in the education of healthcare professionals. It has several education roles [1‐5] such as providing direction and motivation for further learning [6‐8], supporting future learning [6, 8‐12], and identifying areas for improvement [6, 12]. For assessments to fulfil these roles defensibly, it must be of high quality. Several criteria exist for ‘good assessment’ including: validity, coherence, reproducibility-consistency, equivalence, feasibility, educational effect, catalytic effect, and acceptability [13, 14]. The relative importance of these criteria varies depending on the type of assessment (formative vs. summative), the stakes of the assessment (high vs. low), and the consideration for stakeholders’ needs. A stakeholder can be any individual who has an interest in the outcome of the assessment (e. g. Norcini & McKinley [15]), including the general public, faculty/teachers, health system/regulators, and examinees themselves. Exploring the perceived value of an assessment tool, including perceptions of acceptability, credibility, and fairness could be one means to include different stakeholders in the assessment process (e. g. Norcini et al. [14]; Van Der Vleuten [16]).

If we focus on the presumed educational roles of assessment, it is not unreasonable to consider students as relevant stakeholders in the context of assessment for learning [6, 8‐12, 17‐19]. Students must be able to receive, interpret, and infer meaning behind their assessment score in order for assessment to be used to provide direction and motivation for learning [6‐12] and to identify areas for improvement [6, 20]. Students report that formative assessment can enhance the learning process, provided that they perceive the assessment to be effective [17]. Trainees who perceive an assessment process as ‘unfair’ (i. e. ineffective) may be less likely to accept the feedback provided [21, 22], and more likely to dismiss important feedback when they consider the assessment process to be flawed [21‐23]. This active dismissal of assessment-generated feedback likely results in missed learning opportunities, which in turn undermines the educational role of assessment. Therefore, student perceptions of the value of an assessment can have downstream effects beyond whether students ‘like’ a particular assessment tool or approach [14, 17, 18].

We suggest that in order to facilitate the integration of assessment feedback to support learning, stakeholder perceptions should move away from simplistic notions of face-value acceptability (as whether or not a particular stakeholder group ‘likes it’) into broader notions of how stakeholders perceive the value of assessments. In order to investigate students’ perceptions of the value of a newly implemented assessment, the goals of this study were twofold: (1) to adapt a validity framework to be used to document students’ perceived value of assessment, contextualized according to an adaptation of Messick’s validity evidence [24]; (2) use the student-relevant validity framework to explore how students perceive the value of a newly implemented assessment.

The goal of this study was to explore whether students could be considered to be valuable stakeholders when conducting a validation of an assessment. Specifically, we conceptualized students as having a voice beyond a simplistic perception of acceptability—that students, as receivers of assessment scores, are responsible for interpreting those scores in the context of evaluating their own performance and as a tool to further their own learning. This consideration of students as stakeholders, and as score interpreters, underpinned the development of a questionnaire aimed at exploring students’ perceived value of an assessment using a student-relevant validity framework. We based the development of our framework on a modern validity theory [24]—and relying on previous work [27], we expanded definitions of the different categories of evidences of validity as defined by Cook et al. [27] to be relevant from an examinee perspective, and developed specific questionnaire items for students to rate when considering a newly implemented integrative assessment.

Participants’ critical comments suggest that the students were engaging with the assessment in a sophisticated manner—including having expectations of the role of this assessment and expressed displeasure when they felt those educational roles were not being filled. These comments suggest that students can engage in judging an examination beyond simple ‘acceptability’. For example, participants had clear expectations of what the assessment ‘should do’ in order to support their learning, and were disappointed when they perceived that these goals were not met. Participants expressed disappointment with an exam that was not (to their view) difficult enough to identify knowledge gaps, or used repeated content and therefore was not testing the application of knowledge that would be required in a clinical setting. This remarkable engagement with, and critical consideration for, the educational goal of assessment emphasizes the potential value of conceptualizing students as meaningful contributors to a validation study if the assessment is intended to have an educational impact. Furthermore, this critical student engagement also provides support for a core tenant of this work—that students are able to provide a well-considered opinion regarding the perceived value and validity of an assessment, given that assessment scores are interpreted in order to support future learning.

Participant ratings differed across categories of validity evidence and ratings were higher on Content, Response Process, Consequential, and Internal Structure while they were lower for Relationship to Other Variables. In our student-relevant conceptualization of validity, Relationship to other variables validity evidence addresses how well the participants felt the exam aligned with future clinical scenarios, other assessments, and future performance. Although the perceived validity of the assessment was favourable, the significantly lower scores for Relationship to Other Variables suggests that students did not necessarily feel that performance on this cumulative examination reflected their future ability to perform in clinical settings, nor related strongly to their performance on other forms of assessment. This was further supported by comments in the open-ended questionnaire items. It seems, at least for these cohorts and this exam, that the appropriateness of an exam includes consideration for how exam performance reflects what is expected of examinees in their future practice. In other words, the value of assessment for learning that will support future clinical practice, may prove to be one of the more dominant characteristics considered by students when judging the validity of an assessment and the educational value of the scores generated by that assessment. Further, if students do not feel that performance on an exam reflects performance in future clinical settings, it may be unlikely that assessment-generated feedback would be integrated to improve later performance. This potential dismissal of assessment-generated feedback was also reflected in the relationship between actual exam performance and perceptions of the assessment’s value. A significant correlation was found between exam score and subscore for Response Process and Relationship to Other Variables—indicating that those who performed more strongly on the exam felt that the exam was more likely to relate to later clinical performance, and more likely to reflect ‘true’ performance than those who received lower examination scores. These lower ratings generated by those who performed less well on the exam suggest that performance and perceptions of an examination are linked, which aligns with previous work [34‐36]. Therefore, participants may feel that failing this exam (receiving a low score as feedback) does not necessarily translate into poor clinical skills, knowledge, or judgement, and so may undervalue, or ignore, potential performance-improving feedback derived from this assessment. Further, participants with more experience with this type of examination (the Year 2 participants) did rate the examination significantly higher than those for whom it was their first experience (Year 1 participants).

We often discuss the evidence supporting the validity of an assessment and the roles we expect from assessment, however we may be too rarely attending to assessment as a component of a larger educational process—a process that exists in order to ‘gate-keep’ but also to shape and support learning [9]. Here, we have argued for, and piloted work to contextualize validity evidence within a student-relevant perspective. Assessment can support learning [10, 12] through steering and fostering the learning of an individual student by providing students with performance-based feedback [36‐38]. If we contextualize assessment within a feedback framework, it is not only the choice of tool that allows assessment to drive further learning, but the acceptance of the tool, and acceptance of the scores generated by the tool, by those being assessed. The opinion that an examinee holds in regards to the perceived fairness, credibility, or validity of an assessment tool will influence how likely a trainee is to accepting and reflecting on the feedback provided by the assessment to allow for growth [13]. Watling and colleagues demonstrated that the perception trainees have of their assessment process can alter the extent that feedback contributes to learning, even when the assessment tool is based on sound assessment measures [39, 40]. Similarly, when an evaluation process is perceived to be unsatisfactory by those being evaluated, its capacity to foster learning becomes questionable [41]. When a student perceives an assessment process as unfair, they may view an unfavourable outcome negatively and not accept or integrate the feedback provided [23, 34‐36], which may hamper future development.

This study has limitations. The first is that the current study had a small sample size; and a sample that may not be representative of the entire medical class. Web-based surveys tend to have a low response rate [42], particularly when no reminders are sent and as such a 20% response rate is within a reasonable range. Further, we do not wish to claim that the findings reported here are generalizable; instead, the purpose of this study was to explore considering students as interpreters of assessment data, therefore opening the possibility of applying validity frameworks (e. g. [24]) to understand student views of assessment quality and educational value. A further limitation is that this study had a single site; however, given the nature of the newly implemented examination (it only exists at a single site), a multi-institution study was not feasible, nor possible. Future work could examine the transferability of the student-relevant validity framework, adapted as needed to a particular assessment context in other settings. Finally, the work reported here was not intended as a full validation study, but rather as an exploration of the notion of students as score interpreters within the framework of assessment for learning. As such, we make no claims regarding the validity of the assessment used within this study, nor the ‘accuracy’ of generated scores. Instead, we simply suggest that evidence of validity can be adapted beyond exam administrators as score interpreters, and explore the notion that a validity framework may be applicable to students-as-score-interpreters in the context of the educational role of assessment.

In this study, we conceptualized students as valuable stakeholders in assessment. Specifically, we considered the roles of students as interpreters of assessment scores—students are the receivers and final ‘actors’ in the assessment process. The recognition of students as having agency within the assessment process may help to contextualize undesirable aspects of student behaviour [15]—cramming to pass the exam, selective studying, and dismissing negative feedback as invalid. The perception that an examination lacks credibility, or value, and the presence of undesirable behaviours on the part of an examinee, may undermine the institutional missions of assessment and decrease the positive impact of sound assessment. Work reported here is a shift from more classic approaches to validity where the ‘interpreters of scores’ are those who administer and monitor assessments, to acknowledging the potential impact of how students interpret and contextualize their assessment scores in order to benefit (or not) later learning. In order to maximize the educational benefits of assessment tools, we must be mindful of unintended negative consequences of assessments, including examinee perceptions of the validity and credibility of assessment.