A critical perspective on the modified personal interview

Scoring involves assigning scores to observations; for the MPI, this refers to interviewers assigning Likert ratings based upon observing candidates within a station [13‐18]. At this level, two assumptions require support: (1) evaluators consistently assign scores that appropriately reflect the candidates’ abilities, and (2) the tool appropriately measures constructs of interest.

For the first assumption, evaluators should be trained in using the assessment tool [13‐18]. MPI interviewers are trained through mandatory online sessions; optional in-person sessions offer MPI demonstrations that are recorded for future reference [5]. Since in-person sessions are voluntary, they are poorly attended [12]. This causes evaluators to miss the chance to engage in observational learning, which can cultivate essential nontechnical skills—like decision-making—for accurately rating candidates [19]. Moreover, observing errors in rating applicants and how to overcome them is valuable external feedback that evaluators can use to improve their rating abilities and recognize erroneous tendencies [20]. Without such insight, evaluators become more susceptible to halo error, which is when their fixation on a single construct leads them to rate all constructs similarly [12]. As a result, evaluators assign scores that do not accurately represent what they observed, jeopardizing the scoring inference. Supporting the presence of halo error are high correlations (r = 0.70–0.86) between the four Likert ratings given in a single station (‘within-station correlations’), as these suggest that evaluators’ overall impressions of one construct may influence scoring of all four [12]. Such correlations are present at all four stations.

For the second assumption, it is important to note that, in addition to pre-determined questions, evaluators can devise questions during interviews [5]. Since the presentation questions can greatly impact how they are answered, they must be constructed such that candidates understand what is being asked; this ensures that differences in scores are attributable to differences in ability. Although specific questions cannot be obtained and scrutinized, Hanson et al. (2016) quantified the degree to which questions influence scoring using a generalizability (G) study [5]. They found that only 0.011% of the variation in scoring—a negligible amount—is due to the questions themselves, regardless of whether they were pre-determined or impromptu [5].

Generalization involves making overall judgements about test performance based on assigned scores [13‐18]. In order for scores to reflect test performance, components chosen for a given test (e. g. evaluators, stations, questions, etc.) must be representative of all components that could have theoretically been chosen [13‐18]. As such, two assumptions require support: (1) assessment content appropriately and adequately samples relevant topics and constructs, and (2) the assessment is highly likely to assign similar scores on subsequent tests that use entirely new components [13‐18].

For the first assumption, it is important to ensure the assessment sufficiently covers constructs it aims to assess; one strategy for this is blueprinting, or mapping assessment content onto the constructs being assessed [21]. This seems to have been done for the MPI by adapting themes from the University of Toronto’s Leadership Education and Development program (LEAD)—a leadership program that also uses the MPI. LEAD used the following themes for blueprinting: (1) self-reflection and personal insight, (2) bandwidth and adaptability, (3) ability to work in teams, and (4) vision and expectations of [the program] [11]. Aside from exchanging ‘bandwidth and adaptability’ for ‘ethical decision-making’, these constructs directly translate to those assessed by the University of Toronto’s medical program. Since LEAD is comprised of medical students, using these same constructs to screen medical school applicants is appropriate. After reworking LEAD constructs for the medical school MPI, questions are developed for evaluators [12]. Although it seems that the medical school MPI was blueprinted, there is no explicit confirmation or description of this in the literature. This is problematic because if MPI content does not reflect the constructs it claims to assess, evaluators may inaccurately judge candidates’ performance or inadvertently assess different constructs.

For the second assumption, the MPI must have a high inter-interview reliability to show that it can discriminate consistently between candidates [5]. However, inter-interview reliability is just 0.56 for the MPI, which is below accepted standards (0.70–0.90) and supports only moderate confidence for distinguishing between candidates consistently. One potential cause for this is construct-irrelevant variance—systematic error in assessment data due to factors unrelated to constructs of interest [21, 22]. This means that factors other than the applicant’s competence in the assessed constructs influence their scores, compromising accurate judgement of their performance using these scores. The G study by Hanson et al. (2016) quantifies the influence of construct-irrelevant variance on the MPI [5]. Their results show that 52% of variation in scores is from a combined effect of the station and applicant; 18% is from a combined effect of the station, applicant, and items; and only 17% is from true differences between applicants [5]. These percentages show that scores assigned to candidates are significantly influenced by situational factors, suggesting context specificity [2, 6]. Further support for context specificity is within-station correlations between the four constructs (r = 0.70–0.86); scores for constructs within a station are highly related, despite being inherently different [12]. Additionally, low correlations (r = 0.06–0.27) between common constructs across stations (‘between-station correlations’) suggest that there are not enough MPI stations to sufficiently sample the assessed constructs and overcome context specificity; between-station correlations are high when constructs are aptly sampled [12]. Therefore, since situational influences are dynamic and change with repeated assessments, the MPI is unlikely to score similarly on subsequent administrations.

Extrapolation involves judging whether test performance reflects real-world performance [13‐18]. For the MPI, this means inferring that candidates will succeed academically and non-academically in medical school (i. e. real-world performance) based on success during the interview (i. e. test performance). At this level, the assumption requiring support is that the chosen comparator metrics represent medical school performance [13‐18].

To support this assumption, Hanson et al. (2016) examined the MPI’s predictive validity against performance on a first-year objective structured clinical examination (OSCE)—a scenario-based assessment—that tests physical examination and communication skills [5]. The authors used these scores as comparator metrics because this OSCE is encountered in medical school. The results show that MPI total scores are moderately correlated with physical examination skills (r = 0.24), communications skills (r = 0.33), and total OSCE scores (r = 0.30) [5]. These associations suggest that the MPI shows promise in selecting candidates who will perform well in first-year medical school, supporting the extrapolation inference. However, one shortcoming is that Medical College Admission Test (MCAT) scores and grade point averages (GPA) were not used as additional comparators, despite their ability to modestly, yet significantly, predict medical school and licensing examination performances [12, 23‐25]. Nevertheless, there is still some support for this assumption through predictive validity evidence from the OSCE metric.

Implication involves considering consequences of a decision on all stakeholders involved [13‐18]. For the MPI, this means considering consequences of medical admissions decisions on the candidates, medical program, medical profession, and society. The assumption requiring support is that the MPI will lead to more positive outcomes for these stakeholders.

To support this assumption, it is important to consider benefits for each stakeholder. For candidates, medical school acceptances provide positive emotional reactions; rejections would cause negative emotional reactions and loss of application fees. Furthermore, the MPI benefits candidates by increasing the number of applicants that the University of Toronto can interview, resulting in more interview invites being sent [5]. For the medical program, the MPI requires less interviewers than the panel interviews, reducing burden on the University’s resources [5]. Specifically, the MPI allowed just 160 evaluators to interview 600 candidates; the panel interview required 290 interviewers for 587 candidates [5]. Moreover, since MPI scores are related to first-year OSCE performance, accepted candidates are more likely to succeed academically, reflecting positively on the program. For the profession and society, reduced error rates in practice are strong evidence. However, no data show that those accepted through the MPI commit less errors than those accepted through other formats—a gap in the literature. Nonetheless, there is still some support for this assumption as the MPI benefits candidates and the medical program.