Development and preliminary validation of a leadership competency instrument for existing and emerging allied health professional leaders

A cross-sectional survey design was used in this study. Data was collected from July 2013 until February 2014 at the Allied Health Division of the Singapore General Hospital. In total, 106 AHPs (34 men and 72 women) of different professional grades (49 principal-grade AHPs, 41 senior-grade AHPs, and 16 junior-grade AHPs) completed both the AHEAD and LPI instruments. These 106 AHPs are come from Dietetics, Medical Social Services, Occupational Therapy, Physiotherapy, Pharmacy, Podiatry and Speech Therapy departments. Department heads selected promising AHPs with the potential for growth and development, as well as existing leaders in their respective department. This study was exempted from review by the SingHealth Centralised Institutional Review Board.

To ensure that AHEAD items are applicable to all within the division, we first conducted item-generation steps through a literature review. This was followed by item-reduction using pretesting and input from discussion groups. The Leadership Practices Inventory [4] served as an intellectual and structural model during the development of AHEAD.

The aim of this phase was to examine the factorial, convergent, and discriminative validity of the AHEAD instrument. To this end, each participant completed both the AHEAD and the LPI instruments through self-evaluation, followed by a facilitated discussion with their supervisor to reach an agreed score. In addition, participants were provided a resource guide which contained descriptors and suggested developmental resources. The subsequent scores were then sent to the team for collation.(Additional file 2)

In our study, factorial validity was examined by assessing the extent to which the items under “skills” and “values” conformed to their respective theorized constructs [9]. Cross-sectional convergent validity was examined by assessing the extent to which AHEAD was associated with the LPI (described later). Discriminative validity – the ability of an instrument to discriminate between known groups, was examined by assessing the extent to which AHEAD discriminated AHPs of different professional grades – namely, in ascending order, junior, senior, principal, and senior principal grades.

We used descriptive statistics to characterize the study sample: we used means with SDs and medians with IQRs for continuous variables and frequencies with percentages for categorical variables. To examine the factorial validity of AHEAD, we used exploratory factor analysis (EFA). Given that no prior analyses have been performed on AHEAD, EFA is an appropriate first-step technique to explore the nature and number of factors that account for the covariation between the items [12]. We performed EFA on the polychoric correlation matrix of the ordinal-scaled AHEAD items, and we used ordinary least squares extraction with oblique (oblimin) rotation. Of note, an oblique rotation was used because we expected “Values” and “Skills” to be correlated. To determine the number of factors to be extracted, we relied on 3 principles: (i) inspection of the scree plot, (ii) interpretability of the factors, and (iii) results from Horn’s parallel analysis which uses the “mean eigenvalue criterion” [13‐15]. Finally, we examined the internal consistency of each factor using Cronbach’s alpha [9, 16].

To examine convergent validity, we computed Spearman correlation between AHEAD and LPI. To examine the discriminative validity of AHEAD and LPI in differentiating AHPs of different professional grades – an ordinal outcome with 4 categories – we computed the rank-biserial correlations which described the correlations of AHP grades with AHEAD and LPI. To statistically compare discriminative validity of AHEAD and LPI, we used separate proportional odds models and regressed AHP grades on AHEAD and LPI [17]. From each ordinal model, we computed the concordance index (c-index), where a value of 1 represents perfect discrimination and 0.5 represents chance discrimination. Next, we used a bootstrapping procedure (1000 repetitions) to calculate the difference in c-indices between AHEAD and LPI models. Specifically, by subtracting the c-index of the LPI from that of AHEAD, positive values indicated that the discriminative validity index of AHEAD was superior to those of LPI.

All statistical analyses were done with R software, version 3.0.1, using the psych [18] and rms [19] packages.

In our study, moderate correlations were demonstrated between AHEAD and LPI scores (Table 3). To the extent that the LPI is a well-established and validated measure of leadership competency [10], these correlation results suggest that AHEAD was assessing a similar underlying construct as the LPI, thereby providing evidence for its convergent validity. More importantly, AHEAD outperformed LPI in terms of discriminative validity (Fig. 1).

AHEAD scores may be more discriminative because given that the instrument was developed by AHPs, its conceptual focus may be more relevant to the AHP compared with the LPI. Furthermore, having to justify AHEAD ratings by citing examples may require participants to give a more accurate and honest view of themselves. In contrast, the LPI may be less favourable as a discriminative measure because the comparatively simpler way of rating oneself with LPI may influence respondents to rate themselves more hastily which may have introduced some discrepancy in results. This is especially so for junior AHPs, who may have rated themselves more highly than they would have if they had to produce supporting evidence. Finally, the AHEAD scores may be more discriminative because individual items descriptors were provided in the assessment form whereas only domain descriptors were available for the LPI. Conceivably, the detailed AHEAD item descriptors may bring respondents and their supervisors to a common level of understanding, which in turn, would have enhanced the predictive accuracy of the AHEAD scores.

Our study has limitations. First, our sample size was modest but this limitation may be countered by the few number of factors that we examined, the relatively large number of items per factor, and the moderately-high factor loadings [32]. Thus, based on the simulation results by Mundfrom et al. [32], we believe our sample size was at least adequate. Second, our focus on AHP leaders may limit study generalizability to this subset of AHPs. However, by limiting our study to AHPs who had the potential to lead and excel, we have provided a more robust test of convergent and discriminative validity of the AHEAD instrument. Third, we studied only AHPs so it remains uncertain if our findings are applicable to other healthcare professionals such as nurses, physicians, and hospital administrators. This limitation must, however, be balanced against the fact that the AHEAD instrument is one of the few leadership competency instruments developed in and for healthcare professionals. Accordingly, future studies should evaluate whether AHEAD could be used (or modified for use) in other healthcare disciplines or settings, preferably adopting a confirmatory factor analysis approach. Fourth, this cross-sectional study focused on convergent and discriminative validity of AHEAD; hence, it would be informative to evaluate, in a longitudinal fashion, the predictive validity of AHEAD scores with future important leadership outcomes – for example, leadership self-efficacy and follower outcomes and satisfaction [33].