A rank subdivision of equivalent score for enhancing neuropsychological test norms

Neuropsychological assessment of cognitive functioning is a key part of clinical care: diagnosis and treatment planning, treatment evaluation and research [1], prediction of long-term outcomes [2], the individual ability to perform activities of daily living [3], and effects of neurosurgery [4]. Based on clinical history, observations, initial medical history, and behavioural aspects, it is possible to choose neuropsychological tests to employ for each patient. Neuropsychological tests must be psychometrically sound to adequately identify a deficit, accompanied by well-defined standard procedures and accurate normative data [5]. The quantitative evaluation of individuals’ neuropsychological functioning via psychometric tests requires the raw scores to be scored based on the performance of a representative population. In this way, we can compare the performance of a single individual against the performance of a population with similar demographic characteristics (age, education, and gender).

To standardize scores, normative values must be derived from a sample of healthy individuals [6]. Many standardization methods have been proposed to establish normative values, and many of them rely on a non-parametric approach [6]. Neuropsychological data may be affected by ceiling/floor effects and high inter-individual variability. Therefore they do not generally conform to distributional assumptions [7, 8]. For this reason, non-parametric approaches should be preferred when drawing norms. The most well-known non-parametric approach is the Equivalent Score (ES) method [9]. The ES method represents regression-based approaches to standardize neuropsychological tests by non-parametrically drawing cutoffs and controlling for inferential errors [10]. The overall procedure to extract the ESs for a test comprises several steps. The first one entails examining an undivided, demographically composite sample, and calculating the contribution of the demographic variables through multiple regression. Then the original scores have to be adjusted by adding/subtracting that contribution. This permits obtaining test norms based on a relatively small number of participants, in the order of hundreds, compared to the thousands required for covering samples of any age, education, and gender group.

The second step requires deciding with controlled risk whether an adjusted score passes a specific threshold that can be determined using the non-parametric tolerance limits (npTL). Two limits have to be set from the sampled adjusted scores distribution. The outer tolerance limit (OTL) is the cutoff that guarantees (with 95% probability) that no more than 5% of the reference population score is actually below it. The inner tolerance limit (ITL) is the cutoff that guarantees (again with 95% probability) that no less than 5% of the reference population score below it. Therefore, a diagnosis of pathology is rather safe if the score is below the OTL, and a diagnosis of normality is also rather safe if the score is above the ITL. The area between the two limits represents the uncertainty of the classification accuracy.

Finally, starting from the npTLs, the ESs can be determined. The ESs represents an ordinal five-point scale that maps percentile ranks of adjusted scores of a test [9]. As the nature of ESs is non-parametric, they do not depend on the test scores dispersion. As such, ESs possess the properties of the ordinal scale. The ESs permit drawing clinical judgements of the score obtained from a test: ESs equal to 0 and 1 meaning defective and borderline, respectively; ES equals to 2 meaning low-end normal and ESs equal to 3 and 4 meaning normal. The OTL represents the limit of the ES equal to 0; the median value of the test score distribution corresponds to the limit of the ES equal to 4. These are the two fixed points in the ES computation. The ESs that lie between ES0 and 4 can be identified by subdividing the range of adjusted scores between the OTL and the median.

To determine the intermediate limits — ES from 1 to 3 — practitioners have mostly relied on a parametric model (i.e. a z-score-based approach) because of the “possible normality of the underlying ability” [7‐9]. However, relying on this approach may be problematic. Indeed, determining the intermediate ESs using a parametric approach creates a methodological inconsistency with the non-parametric approach used to determine the two fixed ESs: As explained, the ES0 and ES4 are derived using the OTL and the median of the distribution, both of which are determined using distribution ranks. Additionally, and most importantly, even though we agree that test scores could be expected to be normally shaped in the population, they are usually not normally distributed in normative samples. Certainly, partialling out the effects of demographic characteristics through regression analysis permits obtaining scores (i.e. adjusted scores) that do not have nonlinearity issues. However, adjusted scores might still be non-normally distributed [11, 12]. For example, some tests show a ceiling effect regardless of the demographic variables [11, 13]. Furthermore, the rank-based subdivision was the rule for identifying test cutoffs when TLs and ES methods were unavailable. In a review of normative neuropsychological studies, the authors divided the percentile distribution (from the 5th to the 50th) into three equal parts using percentile intervals [12].

In this paper, we propose that the calculation of the intermediate ESs should be based on a non-parametric rank subdivision of the adjusted scores distribution instead of a parametric distribution model (i.e. z-score). The advantage of the proposed approach is that partitioning the score distribution into three equal parts is performed regardless of the shape of the distribution, like those from a test’s error score. In this way, the three intermediate ESs have the same density, and consequently, the rank classification is more accurate. Furthermore, this method of rank subdivision permits defining each ES scale step as what it is: a rank scale. This paper aims to provide evidence of the good performance of the non-parametric approach to establishing intermediate ESs. To do this, we performed three simulation studies in which different score distributions were considered (i.e. non-parametric vs parametric). To further explore the rank-based approach, we also compared it with another simple approach that could be used to define the intermediate ESs which is based on the direct subdivision of the dependent variable into three equal groups from the OTL to the median. In the case of parametric distribution, this approach should correspond to the z-score subdivision. However, compared to the z-score approach, it is much simpler to implement. Finally, a guide for calculating the TLs and ESs and the R script to implement them is provided.

Three simulations were performed. In all the simulations, OTL and ITL were assessed using the non-parametric approach (i.e. npTL), while the intermediate ESs were computed through the z-score approach, the rank subdivision approach, and the subdivision approach based on the value of the dependent variable. The normality of the distributions was checked using the Shapiro–Wilk test.

In the first simulation (i.e. simulation A), we created a sample of 300 not-normally distributed scores. In this sample, the scores represent a test in which the lower the score, the better the performance, such as the execution time of a performance test [14] or a reaction time test [11]. Thus, the npTLs are one-sided in the right part of the distribution.

Like simulation A, simulation B was conducted on a sample of 300 not-normally distributed scores. However, in this simulation, the scores represent a test in which the higher score represents better performance, such as a memory recall test [15‐17]. As a consequence, the npTLs are one-sided in the left part of the distribution.

Compared to the other simulations, simulation C was conducted on a sample of 1000 normally distributed scores, where the npTLs are one-sided in the left part of the distribution.

While the calculation of the ESs using a z-score-based approach is well described in the R script provided in [7], the R script for calculating ESs using the rank subdivision approach is available at https://​osf.​io/​v28x6/​. This includes both commands for reporting the ES values and the observation number of the ESs. The interested users can easily adapt the R script by simply assigning the adjusted score of the target test to the variable “x” (see the R script). Once they adapt and run the script, the rank of the target observations, the ITL, the OTL, and the 4 cutoffs between the ES0 to ES4 is computed and reported. In addition to reporting the rank positions, ES points are accompanied by corresponding values for the tests. Statistical analyses and figures were performed with R statistical environment [18]. The calculation of TLs was performed using the tolerance package [19].

In this paper, we aim to refine the method of calculating the ESs of neuropsychological tests. We provided the rationale for using the rank subdivision of ES score to enhance neuropsychological test norms. The provided simulation examples, with different sample sizes and distributions, showed that the rank subdivision was able (by definition) to resist skewed data. The other two approaches (i.e. the z-score and dependent variable score) give similar results when the adjusted score is normally distributed but different when the data are non-normally distributed.

The use of rank subdivision was motivated by its definition “Equivalent scores are based on the ranks of the adjusted scores; their nature is basically non-parametric, and does not depend on the test scores dispersion” (Capitani & Laiacona, 2017, p. 1224) [9]. This is in contrast with the use of the z-score approach used to define intermediate ESs. Different distributions lead to different results. To the best of our knowledge, the most appropriate solution remains the rank ES: for its simplicity, and because it closely follows the definition of ES, which is an “ordinal five-point scale that maps percentile ranks of adjusted scores of a test” (Capitani & Laiacona, 2017, p. 1223) [9]. The second advantage of rank-based ES is the resulting direct rank subdivision based on population density. There are a number of old and new pieces of evidence that the z-score subdivision does not produce an equal density subdivision in intermediate ES and consequently does not follow perfectly the ranking definition provided above [21‐23].

However, rank-based ES scores have some clinical implications, both positive and negative. Given that our proposed approach provides a more accurate subdivision of the ES1-ES3 values based on sample density, it has the main advantage of enhancing the accuracy of diagnosis. As can be seen from Fig. 1, the z-score approach yielded a narrower ES1 width than the rank approach. Compared to the z-based approach, it will be easier to find more patients that fall into the ES1 and fewer in the ES2 with the rank-based approach. We recognize that classification based on these two approaches could change. Nevertheless, the clinical implications of this difference are minimal since the two most important points for the diagnosis remain unchanged: the ES0 and the ES4.

According to Capitani and Laiacona [20], the ES calculation (using the z-score method) also produces the same interval on the z-axis. Therefore, a simple and similar method for defining ES1-ES3 based on the direct subdivision of the dependent variable was tested. The results show that the two procedures overlapped perfectly when the data had a normal distribution. When the data are non-parametric, the similarity between these two methods depends only on the shape of the distribution. In simulation A, an approximate result was obtained, but the results were different in simulation B. For those unfamiliar with statistics, the cutoff definition based on the direct subdivision of the dependent variable is the simplest method. Furthermore, to avoid the calculation of rank ES cutoffs, we provide a simple table that reports all rank observations points.

One of the advantages of using ES is its intrinsic ease of interpretation and the fact that it is used in a wide range of neuropsychological tests. Using this approach, a full neuropsychological evaluation may be scored with a single scoring system and test results may be compared easily to identify specific deficits. However, the ES method presents some limitations. The few steps of ES have the disadvantage of including an extensive range of accurate scores. Consequently, it is difficult to appreciate small differences in the case of the evaluation of rehabilitation. Secondly, we are aware that the more critical point for defining a pathological score is the OTL and the correspondent ES0. The intermediate ESs are aleatory, even more so with the procedure listed in this study. The two approaches for calculating ESs (i.e. z-score and rank) will need to be detailed in future normative data studies.

Whether a test has a double scoring system: percentiles and ES, there is the possibility of choosing one over the other according to the situation at hand. For example, a clinician could use ES for reporting a clinical evaluation of a patient, but the same clinician could use percentile for rehabilitation evaluation [6]. In the case of bilateral scoring, i.e. lateralized scores, such as those used for spatial neglect evaluation, ES scoring could not be applied. In these cases, the 95% TL with 95% CI remain valid (OTL and ITL), but they need to be calculated on two sides, obtaining only two cutoffs (i.e. left and right) without using ES scoring [11, 23‐25].

In summary, subdividing intermediate ESs using the here so-called rank-subdivision of the distribution of the scores can improve the scoring of neuropsychological tests. Thus, future research aiming at scoring normative data should consider the best procedures at the very least.