Development of a new Rasch-based scoring algorithm for the National Eye Institute Visual Functioning Questionnaire to improve its interpretability

The 25-item National Eye Institute Visual Functioning Questionnaire (NEI VFQ­25) is a patient-reported outcome (PRO) instrument originally developed for use in patients with age-related macular degeneration (AMD), cataracts, diabetic neuropathy and glaucoma [1, 2]. It has been widely used in clinical trials in neovascular AMD [3, 4], diabetic macular edema (DME) [5, 6], macular edema due to retinal vein occlusion (RVO) [7] and choroidal neovascularization (CNV) secondary to pathologic myopia (PM) [8].

When using PROs in clinical studies, it is critical that the instrument selected provides a valid measurement of the concept of interest in the specific context of use [9‐11]; this has become especially relevant in recent years because the use of data from PRO instruments, such as the NEI VFQ-25, in decisions about healthcare resource allocation is increasing [12‐14]. The NEI VFQ-25 has undergone psychometric evaluation in patients with varying ocular conditions and the general population [15‐20]. However, important limitations have been identified which may affect the interpretation of clinical trial results based on the NEI VFQ-25: for example concerns with reliability and validity [16, 17, 19, 20], as well as the dimensional structure of the NEI VFQ-25 validity [16, 17, 19, 20]. Furthermore, a more complete understanding of the content, clinical validity, and interpretability of the NEI VFQ­25 is likely to be critical to regulatory acceptance of PRO-based labelling claims for new drugs and devices [7, 21].

Classical test theory is associated with four key challenges: first, the analysis is framed in ordered counts, not interval-level measurement; second, findings are both sample and scale dependent; third, missing data cannot be handled easily; and fourth, the standard error of measurement around individual patients’ scores is assumed to be a constant value regardless of the person’s location on the range of a scale [22‐24]. Modern psychometric methods, such as Rasch Measurement Theory (RMT), provide a more robust approach with which to examine issues such as validity and interpretability compared with traditional psychometric methods [22‐24]. Rasch analysis has previously been used to “re-engineer” the NEI VFQ-25 scale to comprise two valid and unidimensional subscales, namely visual functioning and emotional well-being [16, 17, 19, 20].

This study uses a large and well-described patient population to extend previously published research proposals for a two-scale structure to further our understanding of how the NEI VFQ-25 can: 1) capture the patient perspective and include clinically relevant and meaningful domains; through 2) exploiting the benefits of Rasch Measurement Theory, and in particular item maps and threshold plots to improve interpretability; and ultimately 3) provide a scoring algorithm that can ensure an equivalent frame of reference across different clinical settings for patients with retinal diseases.

Our psychometric evaluation of the NEI VFQ-­25, which supports previous research [17‐19], suggests that the instrument can be improved as a measure of vision-related QoL. Importantly, by using RMT analysis, our findings provided a direct evidence-base upon which to propose a modified scoring system (NEI VFQ-­28-­R), which subsequently demonstrated improved psychometric performance. Furthermore, compared with the original NEI VFQ-25, the two-domain structure of the NEI VFQ-­28-­R measures activity limitation and socio-emotional impact over a wider range of visual functioning than the original 11-sub-domains of the NEI VFQ-25. Our analyses identified the same item misfit and threshold disorder as previous Rasch analyses. This suggests that previously reported limitations of the NEI VFQ-25 were not sample- or analysis-dependent, and warranted further recommendations to improve the validity of the instrument.

The Rasch-based scoring of the NEI VFQ-­28­-R places items and participants on the same linear scale of vision-related QoL. The location of participants on the scale indicates the impact of vision problems on their QoL, while the location of items indicates the perceived difficulty of activities for participants. This provides a better understanding of the measurement scale and how it relates to the range of visual functioning in the study population at an individual or group-level, than that provided by the original NEI VFQ-25 scoring conventions. In this paper, we defined interpretability in the context of exploiting the clinical hierarchy of the item ordering in the Rasch item map (and ultimately subsequent threshold plots) to define and describe the meaning of total sub-scale scores. With the items now on a continuous scale which matches the sample ability, score changes can be interpreted as specific functioning or well-being lost or gained. A comparison of scores can then be linked to specific ability of the patients.

The modified instrument, therefore, enables the identification of specific activities likely to be affected in patients with a known level of visual functioning as their vision improves or deteriorates. For example, on average, a patient with a high level of visual functioning experiencing a reduction in score as a result of progressive visual impairment will probably experience an impact on their ability to drive at night. Further deterioration in visual functioning may impact the patient’s ability to participate in hobbies that require them to see well up-close and may increase their need for help from others. Similarly, a patient with poor visual functioning experiencing improvements in vision as a result of treatment may become better able to go out to see movies, plays or sports events, and is likely to have a reduced need for help from others. This type of information is potentially of great value to clinicians in describing probable impacts on vision-related activities and socio-emotional functioning, and in guiding patient expectations regarding disease progression or treatment benefits.

It is important to highlight that while the psychometric performance and clinical interpretability of the NEI VFQ-­28­-R was improved compared with the NEI VFQ­25, scale-­to-­sample targeting indicated that a ceiling effect was still present. As such, the standard error associated with person estimates is lowest at the less impacted end of the continuum for both the NEIVFQ-28-R and the NEI VFQ-25 (around 0.2 logits; further information available from authors), respectively. This suggests the items associated with the lowest random error, and therefore most potential precision focus include core daily functioning (e.g., difficulty participating, shaving/styling, going down stairs), perceiving the environment (e.g., difficulty recognizing faces, peripheral vision, reading mail/bills, seeing television, reading street signs, finding objects), and burden (e.g., need help, reliance on others, needing to stay at home). However, the persistent ceiling effect means that the NEI VFQ-­28­-R may be unable to discriminate between participants with the highest levels of visual functioning.

Analysis of scale-­to-­sample targeting for the NEI VFQ­-28­-R among participant subgroups revealed that targeting to the scale was substantially better for participants with poorer visual acuity in the better-seeing eye (Early Treatment Diabetic Retinopathy Study [ETDRS] letter score, ≤ 58; approximate Snellen equivalent, 20/80 or worse) than for those with better visual acuity. This limitation may be addressed by adding items to the higher end of the visual functioning scale, but is an important consideration for comparisons of clinical trials in which the baseline visual acuity of the patient populations differs. Change from baseline assessments may be misleading, as a change from a ceiling score may not be feasible, regardless of the associated clinical benefit. Once again, importantly, by using RMT analysis, our findings provided a direct evidence-base upon which to attempt to identify items most relevant to patients with higher visual functioning. It is important to highlight that the item maps presented in this paper are the mapped item locations, not item thresholds. The threshold locations are more spread than the item locations (item location is a mean of item thresholds), and so ultimately it would be important to take these mapped item locations into consideration when interpreting the total scores from the two proposed sub-scales.

Our findings demonstrate that it is important to assess the psychometric properties of patient-reported outcome measures in each population to ensure they are reliable and valid for each specific population. This can be thought of as a quality control or calibration process (similar to calibrating scales to measure weight or a sphygmomanometer to measure blood pressure) whereby the measurement tool is checked for validity before the results are analysed so as to ensure accurate and precise measurement to reduce systematic bias [17‐19].

Many studies have utilized Rasch analysis to optimize the psychometric properties of questionnaires. For example, the Impact of Vision Impairment questionnaire (IVI) was developed using classical test theory methods and originally comprised 32 items with five subscales [17‐19]. Thorough re-examination using Rasch analysis demonstrated that the IVI’s most optimal structure was 28 items in three subscales, and a recent study has used Rasch techniques to shorten the scale further into 15 items in two subscales. Consequently, it is not uncommon for scales to be modified after undergoing additional validation in specific population samples; in fact, this serves to improve measurement precision and increase robustness of subsequent parametric testing using the questionnaire scores.

Our reengineering of the NEI VFQ does not have implications for other work which has used the NEI VFQ-25 to develop a utility measure from the NEI VFQ-25 items [18, 34, 35], as questionnaires and utility instruments are quite separate instruments with separate purposes, development processes and analysis requirements. We recommend to administer the NEI VFQ-25 items in full (including additional questions, and without modifications to the scale) to patients. This consistency in administration will allow improved comparisons of the measure to other studies, and for use in other purposes such as the VFQ-UI or other utility measures derived from these items. Additionally, our findings may inform future studies using the NEI VFQ-25 about the importance of assessing its psychometric properties in each population sample and by giving an a priori indication of its likely dimensional structure.

Finally, our study has two main limitations. First, it is a retrospective analysis of existing clinical trial data including patients diagnosed with retinal disease. Additional prospective evaluation will be required to establish the performance of the NEI VFQ­-28-­R in this patient group and those diagnosed with a cataract or other conditions associated with impaired vision, to establish the replicability and generalizability of our findings. Second, the two-domain structure (Activity Limitation and Socio-emotional Functioning) was proposed based on previous studies [18, 36]. In addition, the item hierarchies are empirically produced. However, this scoring structure proposes just one way that the items could be scored. The structure will require further consideration, qualitative research and clinical anchoring. In relation to this, it is important to flag that unidimensionality [42] is an important element of any Rasch analysis. However, dimensionality is a complex idea [43], made further complicated by the original NEI VFQ-25 was not developed with modern test theory principles in mind. [2] Thus, for this exploratory psychometric analysis [44], took recourse to the conceptual framework of the original authors [2] (which suggests for a single score) and the subsequent research supporting the two sub-scales structure [17].

In summary, for patients with retinal diseases, the proposed NEI VFQ-­28­-R, which has Rasch-based scoring and a two-domain structure, provides improved psychometric performance and clinical interpretability relative to the original version. This Rasch-based approach provides an opportunity to move beyond working with raw scores to using instruments in a way that could facilitate item-level interpretation. Combined with the grouping of items into two clinically meaningful domains, the Rasch-based scoring in this revised instrument may allow identification of the probable impact of visual impairment on patients’ activity and socio-emotional functioning, helping to guide patient expectations.