Reassuring evidence on competency based selection

Patterson et al provide reassuring evidence on the predictive validity of competency based selection.1 A fair, valid, and feasible selection method for entry to training for general practice is vital to maximise the effectiveness of training and to minimise failure in summative assessment, currently costing about £40 000 ($76 564; €57 958) per doctor given an extension of training.

The paper answers two important questions: do methods at an assessment centre predict performance in identified competencies and do candidates selected by the three part competency based process perform better than candidates selected by other methods?

The answer to both is a qualified “yes.” Higher performers at the assessment centre scored more highly than lower performers at three months into training. Candidates selected by the competency based process scored more highly than those selected by other methods. In each case statistical significance was reached in global scores and in two competencies.

Improvement in problem solving reached statistical significance in both comparisons, professional integrity in neither. The small numbers acknowledged by the authors may be responsible for the failure to reach statistical significance in some competencies. It is possible that some of the competencies are currently better assessed than others.

The important question is whether using the six assessment centre competencies is an appropriate yardstick against which to measure performance of registrars and so judge the selection process. Educators will be interested in the ability of this selection method to predict future performance in other outcomes such as Membership of the Royal College of General Practitioners and Summative Assessment examinations, and performance in career posts in general practice.

The competencies are based on research by Patterson et al, who used focus groups of general practitioners and patients discussing critical incidents (good and bad) and also observed behaviour in consultations of general practitioners identified as good by patients and colleagues. This gives the competency based methods a unique and compelling quality and makes it difficult to interpret direct comparisons to other measures—even the highly respected examination for membership of the Royal College of General Practitioners, whose content validity depends on blueprinting against General Medical Council and World Organisation of National Colleges, Academies and Academic Institutions of General Practice and Family Medicine documents, informed by the opinion of a panel of examiners.

The challenge is to improve coherence in vocational training by identifying which competencies are appropriate at selection, which should be tackled in training, and which should be tackled at exit assessment, and to continue to develop methods in all three areas. The process has begun in the work of the National Summative Assessment Board and the Royal College of General Practitioners in developing a new assessment.

Patterson et al have made an invaluable contribution both in their original work and in the continuing development and monitoring of the selection process.1