The UKCAT-12 study: educational attainment, aptitude test performance, demographic and socio-economic contextual factors as predictors of first year outcome in a cross-sectional collaborative study of 12 UK medical schools

The present analysis takes into account the aims which UKCAT set for itself [20, 34], as well as various previous studies of aptitude tests (and the criticisms of those studies). It therefore looks at:

◦ The predictive validity of A-levels and Scottish Highers for performance in the first year of medical school studies.

◦ The predictive validity of UKCAT for performance in the first year of medical school studies.

◦ The incremental validity of UKCAT over and above existing measures of educational attainment, both General Certificates of Secondary Education (GCSEs)/AS-levels/A-levels and Scottish Highers/Advanced Highers.

◦ The specific predictive ability, with and without taking educational attainment into account, of the four subscales of UKCAT.

◦ An assessment of whether ‘Theory ‘and ‘Skills’ measures at medical school are predicted differently by educational attainment and UKCAT aptitude measures.

◦ An assessment of whether the predictive validity of any of the measures is different in the 12 medical schools that have taken part in the study.

◦ The role of demographic and socio-economic factors in moderating any of the findings.

It should be noted that the present study is restricted to medical school entrants and, therefore, it cannot look more generally at how social and other factors relate to educational attainment and UKCAT performance in the entire set of medical school applicants (rather than entrants). The analysis also considers only simple predictor-outcome correlations, and makes no attempt to calculate construct validity, taking into account the unreliability of predictor and outcome measures, restriction of range due to selection, and the right-censorship of predictor variables such as A-level scores. All of that is considered in a separate paper, which carries out a meta-regression of construct validity not only in the UKCAT-12 study, but also in five other cohort studies [35].

Ethical permission for the study was provided by UCL. The medical schools providing data for the analysis did so on the basis of strict anonymity of the institutions themselves. We have also had no access to raw, non-anonymized data, and have had to accept the data as provided as being correct and accurate. Data were provided by the Health Informatics Centre (HIC) at the University of Dundee as anonymized, encrypted files, with a randomized identification code for applicants in each year, which allowed merging of various datasets. Data analysis was carried out by ICM and CD. SN and JD did know the identity of medical schools, but did not process the anonymized data.

Conventional statistical analyses used IBM SPSS 20 (International Business Machines Corporation, Statistical Package for the Social Sciences, Armonk, New York, USA), with missing values handled using the EM method in Missing Values Analysis. Multilevel modeling used MLwiN v 2.24. (Centre for Multilevel Modelling, University of Bristol, Bristol, UK).

Of 4,811 medical students on the four-point outcome scale (OutcomeFirstYear4pt), 4,056 (84.3%) passed all their first year examinations without re-sits, 565 (11.7%) progressed from the first year after re-sits, 94 (2.0%) were required to repeat the first year, and 96 (2.0%) left the medical school (proportions which are very similar to the 81%, 14%, 1% and 4% reported in a cohort of medical students entering in 1981 [45]. Altogether 109 students left medical school, in 55 cases for Academic Reasons, and in 49 for non-Academic reasons (3 after repeating the first year, and 10 after passing the first year exams). Figure 1a shows that the distribution of OverallMark is approximately normal, with some leftward skew. Distributions of TheoryMark and SkillsMark in Figure 1b,c are also approximately normally distributed, the correlation between them being 0.566 (Figure 1d).

Table 3 shows correlations of the 22 background variables, as well as the 6 conventional measures of GCE and SQA achievement, with zEducationalAttainment, zUKCATtotal score, and performance at medical school. It should be remembered that many of the background variables are themselves inter-correlated, and in the following analyses multivariate statistics are used to tease apart the relationships. Among this population of entrants to medical school, who are not, of course, representative of applicants, the 22 background variables together accounted for 3.9% of variance in educational attainment, and 14.0% of variance in UKCAT total score, although that difference may in part reflect selection on educational attainment and, hence, greater restriction of range.

Overall there was a highly significant correlation between prior educational attainment (zEducationalAttainment) and OverallMark (r = .362, n = 3432, P <.001), which was significantly stronger (z = 3.76, P <.001) for SQA qualifications (r = .464, n = 715, P <.001, 95% CI .406 to .522) than for GCE qualifications (r = .331, n = 2,717, P <.001, 95% CI .298 to .364), the relationships being shown in Figure 2. OverallMark was not as strongly correlated with the more conventional measures of three best A-levels (r = .185, n = 2,717, P <.001) and five best Scottish Highers (r = .121, n = 715, P = .001), primarily due to restriction of range and ceiling effects, although both correlated strongly with zEducationalAttainment (A-levels: r = .690, n = 3,432, P <.001; Highers; r = .328, n = 715, P <.001). Students with lower attainment at A-level did, though, perform less well, the regression model suggesting that students with BBB performed about 1.1 SDs below those with AAA, a substantial effect. As explained in the Technical Report [34], our analysis looked in detail at various aspects of measures of Educational Achievement. In particular, we note that for GCE examinations predicting OverallMark: i) AS-level results provided an incremental prediction over A-levels; ii) GCSEs provided an incremental prediction over A- and AS-levels; iii) grade at General Studies A-level provided an incremental prediction over (other) A-levels; iv) grades on all four core-sciences provided an incremental prediction over summed A-level grades, further exploration finding that a key predictor appears to be the minimum core science grade attained, low values predicting poorer performance at medical school; v) there was no evidence that grades at any of the four core sciences were particularly predictive of medical school performance, with simple correlations of OverallMark with grades in Biology, Chemistry, Math and Physics being .182, .143, .125 and .172 (all P <.001, n = 2,645, 2,739, 18,750 and 685). For SQA examinations in relation to OverallMark: i) ‘Highers Plus’ scoring provides incremental prediction over conventional Highers scoring; ii) Advanced Highers provides incremental prediction over Highers/HighersPlus; iii) None of the core sciences showed specific incremental prediction, either at Highers or HighersPlus; iv) Advanced Highers grades at Biology and Chemistry (but not Math and Physics), provided incremental prediction; v) As with A-levels, the minimum core science grade attained seems to have predictive value. Finally, because SQA qualifications had a higher predictive validity than GCE qualifications, we compared the predictive validity of the qualifications in Scottish medical schools (where entrants have either GCE or SQA qualifications) and other medical schools (where entrants have GCE qualifications). In Scottish medical schools, SQA results had higher correlations with outcome than did GCE results for students on the same course, whereas GCE predicted outcome equivalently in Scottish and non-Scottish schools. SQA results do have greater predictive power, perhaps because of the inclusion of Advanced Highers results. However, elsewhere we show that despite the higher correlation with outcome, the construct validity of SQA results is somewhat lower than that for GCE results [35].

Prior educational attainment correlates with a wide range of background variables (TR Table 5). An important question, though, concerns the extent to which background variables continue to predict outcome after educational attainment has been taken into account. OverallMark was regressed on the 22 background variables, with an alpha set at 0.001 to account for repeated testing. Four background measures were significant, in order of entry: Ethnic2, non-White students performing less well (beta = −.126, P <10^-14); being a mature student, students over the age of 21 performing better than non-mature students (beta = .057, P <.001); UKCATdayOfTakingPctileRank, students who took UKCAT late performing less well (beta = −.089, P <10^-7); and DFES.AvePointEntry, students from high-attaining secondary schools performing less well (beta =−.085, P <10^-7). Figure 3 explores DFES.AvePointEntry in more detail. Figure 3c shows that average points per exam entry are substantially lower in non-selective schools than selective schools. The average points are divided into four groups (boundaries 205, 230 and 250), with almost no selective secondary schools in the lowest group and almost no non-selective secondary schools in the highest group. Effects upon overall score were estimated with a regression model in which there were significant effects of three best A-level grades (beta = .205, P <.001) and secondary school-level average points (beta = −.085, P = .005), and an almost significant effect of selective secondary schooling (beta = −.056, P = .059), the fitted regression lines being shown in Figure 3a. There was no evidence of interactions. Actual mean scores are shown for candidates whose secondary schools were in the four groups of average points, and it can be seen, particularly for entrants with AAA grades in both non-selective and selective secondary schools that overall scores at medical school are lower in those from secondary schools with higher average point scores. From the regression lines it can be estimated that one grade at A-level (the difference between AAA and AAB) is equivalent to 85 points on the average point score, so that an entrant with ABB at A-level from a secondary school with an average score of 175 (at about the 1st percentile of the non-selective schools) performs similarly at medical school to a candidate with AAA at A-level from a secondary school with an average score of 265 (at the 99th percentile of the selective secondary schools).

The correlation between UKCAT scores and OverallMark was 0.148 (n = 4,811, P <0.001). Because UKCAT is often said to be particularly helpful in selecting mature entrants (where educational qualifications are often out of date or not applicable), we compared the predictive validity of UKCAT in mature and non-mature students (Figure 4). The correlation with OverallMark was higher in mature students (r = .252, N = 690, P <.001), than in non-mature students (r = .137, n = 4,076, P <.001). Mature students had somewhat more variable raw UKCAT scores (SD = 237.7 compared with 200.0), but regression showed that that was not the cause of the increased correlation with OverallMark. The incremental validity of UKCAT after taking educational attainment into account was assessed by regressing OverallMark firstly on zEducationalAttainment, and then on zUKCATtotal. zUKCATtotal significantly improved the prediction of OverallMark (t = 3.54, 3,429 df, P <.001), but the beta coefficient was only 0.057, whereas the beta coefficient for zEducationalAttainment after taking UKCAT into account was 0.351. In practice, many admissions tutors can only use three best A-levels and, therefore, we repeated the exercise with Alevel_TotalbestN, when the beta for UKCAT was .101 but for A-levels was .168. The previous analysis of OverallMark in relation to the background variables, after taking educational achievement into account, had found four background variables related to medical school performance. We repeated the regression analysis after inclusion of UKCAT as well as educational achievement. All of the four variables significant previously were again significant, suggesting that UKCAT performance cannot eliminate the effects of ethnicity, secondary schooling, day of taking UKCAT, or being a mature student in first year medical school performance. In addition, Sex was also a significant predictor of OverallMark, males performing less well, after taking UKCAT and other measures into account (beta = −.056, P <.001).

UKCAT has four subtests - Abstract Reasoning, Decision Analysis, Quantitative reasoning and Verbal Reasoning - which may correlate differently with educational attainment and with medical school performance, particularly perhaps between Theory and Skills measures. Table 2 shows correlations between the four UKCAT sub-scales, and it can be seen that they are significantly, but only moderately, correlated, suggesting that they are indeed measuring different cognitive processes. Each sub-scale correlates with the total UKCAT score (but it is, of course, a part of it). The subscales all correlate to much the same extent with educational attainment, except for verbal reasoning which has a rather lower correlation. All four sub-scales correlate with OverallMark at medical school, although verbal reasoning correlates significantly more highly than the other three sub-scales, a pattern which is clearer still for the marks from Theory exams, whereas all four sub-scales show low and similar correlations with SkillsMark. In the Technical Report [34], we describe further analyses showing that of the subtests, it is Verbal Reasoning, which particularly contributes unique variance to predicting medical school performance after Educational Achievement has been taken into account, both overall and for TheoryMark, and also for SkillsMark, when higher Verbal Reasoning predicted a lower SkillsMark. Verbal ability may predict better than other subtests due to verbal tests being less subject to practice and coaching effects [26].

Although medical school performance is a continuous measure, students eventually end up in one of four categories, OutcomeFirstYear4pt, with the lower categories having important consequences for the students and their careers. We, therefore, compared the four groups on UKCAT scores and the measures of prior educational attainment. Table 4 shows comparison between the groups using one-way analysis of variance. The overall pattern is that the students who perform less well tend to score lower on the various measures. An exception is that in several cases the group who had failed showed scores that were higher than those who are repeating the first year, as for instance, on measures of prior educational attainment and on several of the UKCAT scores. That may be because the group of failures is not homogenous. Numbers in the Fail group are relatively small, with 55 leaving for academic reasons and 49 for non-academic reasons (although reasons for leaving medical school are often complex and not readily classified [46]). A comparison is available in the Technical Report [34].

Medical schools differ [47], and it is possible that variables which predict outcome in one medical school will predict better or worse in other medical schools. The 12 medical schools in UKCAT-12 allow such possibilities to be assessed. The analyses will begin with a model of the importance of educational achievement, which will be described in some detail, and then a number of other factors will be considered as well.

A three-level model is fitted (see Figure 5), with individual students at the first level, who are nested within the 12 medical schools at the second level, which in turn at the third level are nested within either Scotland or elsewhere. Note that because zEducationalAttainment is not available for mature students, this analysis is restricted to non-mature students. The outcome variable is OverallMark, which is standardized within medical schools and cohorts (and hence overall effects typically have means close to zero). The main predictor is zEducationalAttainment. A dummy variable, at the student level, SQAorGCE indicates whether students took SQA or GCE qualifications (and it has already been suggested that SQA attainment predicts medical school outcome better than GCE attainment). OverallMark can be predicted at level-1 by zEducationalAttainment and SQAorGCE, and by their interaction. The slope of the regression of OverallMark on zEducationalAttainment and SQAorGCE, and their interaction can also show variance between medical schools and between Scottish and non-Scottish medical schools. Figure 5 shows the full fitted model, estimates being shown with their standard errors in parentheses, so that estimates are significantly different from zero with P <.05 on a two-tailed test if their value is at least twice their standard error (and these are indicated in Figure 5). zEducationalAttainment is a strong indicator of OverallMark, the interaction with SQA qualifications is also significant, predicting OverallMark more strongly than do GCE qualifications. No other terms are significant, which in particular means a) that educational achievement has the same predictive value in all 12 medical schools, irrespective of whether they are in Scotland or elsewhere, and b) the increased predictive effect of SQA qualifications is the same in all medical schools, in Scotland or elsewhere. The important conclusion is that although medical schools might have differed in the predictive value of educational attainment, perhaps because of differences in teaching methods, curriculum or whatever, there is no evidence that they do so.

The three-level model for zEducationalAttainment was fitted with the addition of sex, the contextual secondary school attainment measure DFES.AvePointEntry, and ethnicity (White), as well as the interactions of those measures with zEducationalAttainment. After taking zEducationalAttainment into account, male students underperformed (estimate = −.0699, SE .0309), non-White students underperformed (estimate = −.2504, SE = .0357), and students from secondary schools with a higher DFES.AvePointEntry, score (that is, higher-attaining secondary schools) performed less well overall (estimate = −.1112, SE = .0183). There was no evidence that any of the measures interacted with educational attainment and particularly important for interpreting these results is that there was no evidence, for any of the three measures, of variance between the 12 medical schools. In other words, males, for instance, underperformed to the same extent in all 12 medical schools, which is important for understanding and interpreting such effects.

Multilevel modeling of the prediction of UKCATtotal, as well as the subscores, was broadly similar to that for educational attainment, except that differences between medical schools in Scotland and elsewhere were not considered, so that the model had two levels. Age of students (<21, 21+) was included as a previous analysis suggested that UKCAT predicted better in mature students. The overall fitted model is shown in Figure 6. UKCAT significantly predicts outcome (estimate = .1295, SE .0174). Mature students also perform better than non-mature students (.3315, SE .0738), and there is a significant interaction between maturity and UKCAT, UKCAT predicting more strongly in mature students (estimate = .1208 SE .0548). There was no evidence that UKCAT, age or the interaction of UKCAT and age behaved differently in their predictive ability at any of the 12 medical schools.

Academic qualifications, typically A-levels and Scottish Highers, have long been the mainstay of medical student selection, the course being academically very demanding, and an ability to cope both with the cognitive load and the science content being seen as important. UKCAT-12 confirms the importance of prior educational attainment and performance on the course correlating with educational attainment. The role of A-levels and Highers has been questioned, in large part because there is little variation in educational attainment in students actually on the course. However, UKCAT-12’s large samples make clear that even small amounts of under-attainment (AAB, ABB or BBB compared with AAA at A-level) correlate with poorer performance in the first year of medical school. There is, therefore, clear evidence justifying not only the continued use of school achievement for selection but also supporting the development of validated approaches utilizing more of the available information (such as GCSE, AS-level and additional details, such as ‘Highers Plus’ and Advanced Highers). A key question for selection concerns widening access and two aspects merit consideration here. First, the analysis of construct validity, described in detail in a separate paper [35], allows the calculation of how well those with lower qualifications are likely to perform if admitted. Second, we can now make justifiable estimates of how contextual measures, such as personal and educational background might be ‘factored in’, in such a way that they optimize the potential of medical school entrants best.

The UKCAT-12 identified a number of aspects of educational achievement of practical importance.

Educational attainment is clearly a strong predictor of outcome, but it is currently limited by so many applicants getting top grades (a problem which might be partly mitigated by A* grades at A-level, although at present government policy is in the future to decouple AS- and A-level assessments). A partial solution is to consider also AS-levels and GCSE results, both of which have incremental value over A-levels (and a wider range of performance). AS-levels and GCSEs also have the practical advantage of being available at the time of selection, rather than merely being estimated grades for exams yet to be taken. Scottish Highers particularly have the problem of many candidates being at ceiling, but using the full range of marks available at Highers (A1, A2 and so on), as well as using Advanced Highers can result in much better prediction.

Our constructed measure of Prior Educational Achievement demonstrates that more can be achieved by the detailed assessment of secondary school-based performance measures. Though care would need to be taken to assess impact on widening access and this measure cannot be applied to graduate applicants.

In summary, even with the limitations of currently available measures of educational attainment, the results of this study suggest that it would be possible to obtain better prediction of medical school performance by drawing upon information currently neglected in selection: namely AS-level, GCSE, and the full range of marks provided by Scottish Highers, with the introduction of A* grades at A-level also perhaps being of help.

The replicability of these findings, and the reasons for them, can be left for future studies, but all show the potential importance of large-scale studies of selection both for forcing theoretical interpretation on the nature of medical education, and for providing a clear evidence-base for answering important practical questions about selection.

The UKCAT-12 data show that older students, female students and white students all perform better at medical school, with all effects being significant after taking educational attainment and UKCAT scores into account. The ethnic difference found is similar to that found in a meta-analysis [52]. The UKCAT-12 study particularly adds to previous studies because the multilevel modeling shows that the effects of maturity, sex and ethnicity are equivalent at all 12 medical schools, making it unlikely that the differentials found in performance are the result of institutionally-specific effects.

Students educated at selective schools performed less well at medical school than those educated in non-selective schools. Contextual measures were also available for secondary schools in England, and students also performed less well who came from secondary schools with a higher value-added score at key stage 5, or at which students gained more points at A-level (with two different ways of calculating those points). Multiple regression showed that the effect was not due to selective secondary schooling as such, but principally related to the average A-level attainment level of pupils at a secondary school, with the apparently paradoxical finding that the higher the achievement of pupils overall, the less well pupils from that secondary school did at medical school. The result is not novel, and a similar finding had previously been reported by the Higher Education Statistics Agency (HESA) [3, 53]. The mechanism of the finding is not completely clear, but one possibility is that higher achieving secondary schools achieve their results in part by contributing extra ‘polish’ to a student’s work, polish which no longer generalizes or transfers when they then move to a university where such support is not present. Alternatively, it may be that academic aspirations are higher in those from less academic environments, due to the ‘Big Fish Little Pond Effect’ [54]. That the effect found by HESA is now found in medical students suggests that there is a strong argument for using the contextual measure of average A-level attainment at a secondary school in making admission decisions.

UKCAT-12 also considered a range of other socio-economic measures, both directly of the socio-economic classification of a student themself, based on self-description of parental occupation using the self-completion scale of the Office for National Statistics [43], and indirectly, as contextual measures of deprivation on a range of scales, from the English Deprivation Indices [44]. The findings are straightforward; as has been reported before [45], socio-economic background seems to bear little or no relationship to medical school performance once educational attainment is taken into account. Both in the UK [55] and the US [56], for university admissions in general, evidence suggests that the barrier for college entry is lower application rates, rather than universities discriminating against those from lower socio-economic status (SES) backgrounds. The implication is that socio-economic contextual factors may not be of utility in selection, although they could well be of use in encouraging higher rates of application from disadvantaged groups.

Performance at UKCAT does correlate with first-year performance at medical school. The correlation is small but significant for secondary school leavers and is larger for mature entrants. The incremental validity of UKCAT after taking the current educational attainment used for selection into account is small but significant and as such provides sufficient added value for UKCAT to be an adjunct to current selection processes. However, the data analysis also demonstrates the potential of the improved predictive validity of using fuller information on educational attainment. Measures of educational attainment probably predict university outcome better because they provide evidence simultaneously in three domains: intellectual ability (typically fluid intelligence), substantive subject knowledge about science (crystallized intelligence), and a combination of motivation, appropriate study skills and personality [57]. UKCAT aims to assess fluid intelligence and other attributes thought to be important in decision making, but it specifically attempts not to measure science knowledge. The most likely explanation of the relatively lower predictive validity of the UKCAT aptitude tests is that they do not assess domain knowledge (and, hence, indirectly personality, motivation and so on), although personality and motivation are included in more recent versions of UKCAT, they are not included here.

UKCAT predicts outcome significantly better in mature than non-mature students. Mature students often have unusual combinations of academic qualifications, sometimes taken a while ago, and UKCAT is potentially a useful tool for assessing such applicants. UKCAT also predicts differently in males and females, the predictive validity being less in males than females, reflecting in part the fact that males do less well at medical school but do somewhat better overall on UKCAT.

Options for improving the predictive validity of the cognitive tests in UKCAT are several-fold. Reliability of UKCAT could be somewhat improved by increasing the test length, although the relatively low construct validity, described elsewhere [35], suggests that the benefit would not be large. This would also incur increased costs and additional candidate inconvenience. The UKCAT Verbal Reasoning sub-test has the highest correlations with outcome and review of content and style of this sub-test is underway, but without making the test unbalanced, and there is little theoretical sense in, say, downgrading quantitative abilities for a technical, science-based course. Other cognitive subtests might be added, but the ubiquity of ‘g’ probably means that there are no obvious domains of general mental ability which are not currently covered. The performance of Section 2 of BMAT [23] and the science knowledge tests of MCAT [32] suggest that tests which include substantive science knowledge have higher predictive validity than ‘pure’ aptitude tests, which essentially measure only fluid intelligence, and do not assess acquired knowledge or the motivation and personality necessary to acquire it. However, UKCAT originally set out specifically not to include such measures [20] as they are currently available through traditional measures of educational attainment. Non-cognitive measures, primarily of aspects of personality have been piloted by UKCAT, and might contribute additional variance [29]. UKCAT is also currently piloting a situational judgment test, which might also assess separate constructs from those presently assessed.

The emergence of aptitude testing for medical selection resulted from weaknesses in the existing information available to selectors and these persist. Differentiating in a transparent and fair way between the many applicants who apply for each place in a medical school is challenging and the predictive validity and small incremental validity provide some justification for use of UKCAT. Certainly UKCAT, which is now an integral component of many medical school’s systems, seems in many ways more justifiable than the use of UCAS personal statements, which have been less well studied, are open to criticism for difficulty in scoring consistently, and are subject to a range of influences, including social opportunity, and have not been shown to predict success in medical school [17, 58, 59]. There is also evidence that utilizing UKCAT has a positive impact on widening access [19]. With on-going refinements and, in particular, the introduction of non-cognitive tools, such as situational judgment tests, the existing assessment may be able to improve further.

The primary impetus for the UKCAT-12 study was to evaluate UKCAT in the context of medical student selection. The data analysis confirms that UKCAT scores have operational utility when used alongside measures of educational attainment. The UKCAT-12 study, we believe, particularly indicates the value of ‘big data’ in evaluating medical education and lessons learned in the UK are probably generalizable elsewhere. A noteworthy feature is the use not only of data collected on students themselves but also integration of other large-scale databases, such as the Department for Education data on secondary school performance in England, and the English Deprivation Indices. For the first time it is possible to consider the construct validity of selection tools for medicine in the UK and these, using the UCKAT-12 study, as well as five other cohort studies, are published separately [35]. Two future developments would be particularly welcome. Though considerable data are now being amassed, it does not cover all worthwhile outcomes, or even all UKCAT medical schools, let alone UK medical schools. Extending studies such as this to include more medical schools with follow-up into later undergraduate years, as well as post-graduate (as has been carried out for other cohort studies [48]), is vital.

In addition, as cohorts of students graduate and go into practice, other databases will become available for addressing questions of predictive validity both for postgraduate assessments and also outcome measures in clinical practice itself, thus enabling a range of large-scale longitudinal studies. Such databases, studies and analyses would enable much faster progress towards the target of the Royal Commission on Medical Education, the Todd Report of 1968, which it called, “an objective evaluation of student selection” [60].