Psychometric properties of Patient Reported Outcome Measures (PROMs) in patients diagnosed with Acute Respiratory Distress Syndrome (ARDS)

Acute Respiratory Distress Syndrome (ARDS) is a severe life threatening condition, which develops if the lungs become severely inflamed due to an infection or injury. Although there is a low incidence (approximately 78–280 cases per million population), ARDS is associated with a high mortality rate of 40 % or greater [1‐9]. It is estimated that due to long intensive care unit and hospital stays, the cost of every saved life from ARDS is approximately £43,000 (2010 prices) [10, 11]. Patients who survive ARDS tend to have a high number of comorbidities and a poor health-related quality of life (HRQoL) with 35 % unable to return to work 24 months after hospital discharge [12, 13]. Health care costs tend to increase after surviving ARDS due to the need for hospital readmission and inpatient rehabilitation [14].

Patient reported outcomes (PROs) such as symptoms and health utilities can be measured through self-reported questionnaires of health status or HRQoL, which are completed by patients at different time points and are otherwise known as patient reported outcome measures (PROMs). PROMS can be used to compare patients’ self-reported health status or HRQoL at two separate points in time, allowing analyses of the change in health status or HRQoL with respect to an intervention [15, 16]. The inclusion or use of poorly designed or inadequately targeted PROMs in a study that has not considered their psychometric properties can have adverse consequences. These include an additional burden to the patient, an increase in study costs and ethical concerns surrounding patients having to complete measures that are incapable of capturing the patient’s perspective [17]. This could also lead to missing data, unreliable information and biased results. These consequences should therefore be avoided with further research and evidence of the psychometric properties of PROMs in particular populations.

The OSCAR (Oscillation in ARDS) study was conducted to assess the effectiveness and cost-effectiveness of High Frequency Oscillatory Ventilation (HFOV) against conventional artificial ventilation for adults with ARDS. The OSCAR study included the EQ-5D-3 L, SF-12 and the St Georges Respiratory Questionnaire (SGQ) in patients diagnosed with moderate to severe ARDS.

There is limited evidence regarding the psychometric properties of PROMs used in critical care and no evidence in patients with ARDS [18‐21]. Menn et al [18] found that in patients with severe Chronic Obstructive Pulmonary Disorder (COPD) hospitalised for exacerbations, the EQ-5D-3 L appeared to be a suitable measure of HRQoL, whereas the SF-12 appeared to be less suitable for a self-assessment due to the high proportion of missing values. Additionally, the psychometric properties of the SGRQ were satisfactory in this population group, although there was a recognition that no utility values (preference based outcomes) could be derived using this PROM [18]. Therefore, considering that there has been limited information on the properties of alternative PROMS in critical care, and no previous assessments in the context of ARDS, the objective of this study was to assess the psychometric properties of the EQ-5D-3 L, the SF-12 and its preference based derivative the SF-6D, and the SGRQ, in patients diagnosed with ARDS.

A total of 795 patients were randomised within the OSCAR study of whom 168 were complete study responders, meaning all three source PROMs (EQ-5D-3 L, SF-12 & SGRQ) were completed and returned at both 6 months and 12 months post randomisation. The baseline characteristics of the study population are shown in Table 1. There was no difference between patients who responded at both 6 and 12 months, incomplete responders and patients who did not complete any questionnaires at either 6 or 12 months follow up in terms of sex or ARDS type. However, there was a significant difference in average age between all three groups (P < 0.001). Incomplete responders tended to be younger than both patients who responded to questionnaires at both follow up points and also patients who did not complete any questionnaires at either follow up point (P < 0.05). There was also a difference in average weight between patients who responded at both follow ups and patients who did not complete any questionnaires at either follow up point (P < 0.001).

The aim of this study was to compare and assess the psychometric properties of the EQ-5D-3 L, SF-12 and its preference based derivative the SF-6D, and the SGRQ, in patients with moderate to severe ARDS. This study aimed to provide evidence for the use of generic and condition specific PROMs in future clinical trials and trial based economic evaluations associated with critical care and specifically ARDS. The results of the study showed significant variation between properties. Response rates were varied with the EQ-5D generating the highest response rates and the SGRQ generating the lowest response rates. Cronbach’s alpha scores showed that all PROMs were deemed acceptable to the study population. Results also showed that there was no statistically significant difference between known groups with minimal effect sizes. All utility and summary scores correlated statistically with the exception of the SF-12 PCS and MCS scores which did not correlate. When assessing the empirical validity of the EQ-5D-3 L compared to the SF-6D, results showed that the SF-6D was an efficient and empirically valid alternative to the EQ-5D-3 L. Lastly, the EQ-5D-3 L and SGRQ were more responsive compared to the SF-12 and SF-6D with the EQ-5D-3 L generating greater effect sizes than the SGRQ.

When comparing and assessing the psychometric properties of PROMs, there are specific difficulties that need to be addressed to provide transparency of the analysis. For instance, HRQoL can be measured in different ways: the EQ-5D-3 L and SF-12 are generic PROMs and measure general health whereas the SGRQ is a condition specific PROM that measures HRQoL in relation to respiratory concerns. Each of these PROMs also measures HRQoL over different time periods: the EQ-5D-3 L asks individuals about their health state “today,” whilst the SF-12 and the SGRQ measures utilised in the OSCAR study ask individuals about their health during or over the past 4 weeks.

High response rates were seen for all measures due, in part, to the strategy adopted to maximise patient responses. A letter was sent out to survivors 2 weeks before the follow up questionnaires were due at 6 and 12 months, as a reminder to the patient that they would be receiving a questionnaire related to their health from the trial group in the next few weeks. When the follow up questionnaire was sent to patients, a freepost envelope was also sent to maximise response rates. The PROMs within the follow up questionnaires were ordered as follows: the SGRQ, the EQ-5D-3 L and then the SF-12. This may partly explain our response rates where the highest response rate was observed for the SGRQ symptom score. Hence, a large symptom score response rate may be due to the patient having to first answer the first part of the SGRQ (Questions 1–8), which results in a symptom score. The second part of the SGRQ (Questions 9–16) results in activity and impact scores which had lower response rates. Therefore, due to the volume of the SGRQ (50 items), participant fatigue and potential repeatability of dimensions relating to respiratory health may have led to lower response rates for the latter half of the SGRQ.

The EQ-5D-3 L was found to have an acceptable reliability and internal consistency. The SF-6D and SF-12 showed greater reliability and internal consistency than the EQ-5D-3 L. The SGRQ had high Cronbach’s alpha statistics that exceeded 0.95. Scores higher than 0.95 are not necessarily desirable, as this indicates that the items may be entirely redundant [45]. Hence, there is a possibility of item reduction or creating a derivative within the SGRQ that could potentially aid in increasing response rates, although this has to be counter-balanced against the broader goals of the measure.

Construct validity was tested using known group’s comparisons. With no outcomes measures being collected at baseline, the known groups comparison was a partial analysis to ascertain construct validity. For the known groups based on APACHE II scores and the PaO₂/FiO₂ ratio, it was found that there was no difference between known groups and all effect sizes were small. This highlights the need for research to be conducted on measuring and valuing health for the unconscious health state in ARDS patients so that baseline assessments can also be conducted.

There was no correlation between SF-12 PCS and MCS scores which would be expected as they are delineated across different dimensions, which should provide divergent constructs that are not overlapping. There was strong correlation between the EQ-5D-3 L and the SF-6D. All four SGRQ summary scores correlated highly with each other, further highlighting the overlap between its dimensions and constructs. This could be due to the SGRQ being a condition specific measure but also due to the greater amount of items used within the SGRQ. The majority of Spearman Rank correlations were statistically significant at the 95 % confidence level, which shows that there was overlapping constructs between measures. There was particularly strong correlation between the pain dimension in the EQ-5D-3 L and bodily pain sub domain in the SF-12. There was also a strong correlation between the anxiety dimension in the EQ-5D-3 L and the mental health sub-domain in the SF-12. Hence, both results provide evidence for convergent validity. Furthermore, EQ-5D self care and EQ-5D usual activities did not correlate with the anxiety/depression dimension of the EQ-5D-3 L or the mental health sub-domain of the SF-12 which shows some evidence for discriminant validity between dimensions or domains. All in all, correlation at both the utility/summary score level and dimension specific level generated values which revealed expected overlapping and non-overlapping constructs. Hence, the PROMs displayed convergent and discriminant validity.

Empirical validity was tested using relative efficiency statistics to see whether the EQ-5D-3 L and SF-6D generated utility or summary scores that reflected hypothesised differences in external indicators of health status for current general health and total respiratory health. For both the external indicators of current general health and total respiratory health, the SF-6D was found to be more efficient than the EQ-5D and therefore considered to be an empirically valid alternative multi attribute utility measure to the EQ-5D. This shows that the SF-6D is capable of discriminating between external indicators of health status in keeping with the results of other studies [39].

As no baseline outcome measures were collected, it was difficult to comprehensively test internal responsiveness and external responsiveness and hence basic floor and ceiling effects were used to assess responsiveness. The EQ-5D-3 L generated ceiling effects where respondents chose the highest response on ordinal scales that cannot be improved. In order to address this issue, the EQ-5D-5 L has been created where there are five response levels within each dimension, which should decrease ceiling effects [46]. The SF-12 and SF-6D had an advantage in responsiveness due to lack of or absence of floor or ceiling effects. The SGRQ had large ceiling effects, which shows that it has a limitation in its ability to register health changes.

External responsiveness was also analysed where a reference measure, Question 1 of the SF-12 that focuses on current general health, was used to assess whether the changes registered by a measure over time resemble those expected based on an external measure of health. This reference measure was chosen as it is not used in the calculation of any utility or summary scores. We found that the EQ-5D-3 L and SGRQ were more responsive than the SF-12 and SF-6D. As a small sample has been analysed not all mean comparisons are statistically significant or could be performed. Additionally, there may be a response shift bias, which indicates that a patient’s values for health changed over the course of time in the SF-6D, which is further driven by the SF-12 PCS Score.

Many of the differences between the EQ-5D-3 L, SF-6D and SGRQ were expected considering differences in their descriptive system, scoring function, valuation and range of utility and scoring systems. Furthermore, poor health states are valued more highly (in utility terms) in the SF-6D compared to the EQ-5D-3 L [25, 47, 48]. Additionally, the SF-6D is generally better able to detect smaller changes in health compared to the EQ-5D-3 L [48]. There is also evidence which suggests that the time trade off method results in greater values for mild or moderate health states and lower values for severe health states, which may partly explain our findings [47].

The limitations of this study include having a small sample of responders which limited some analyses and decreasing the generalizability of our results. This study was also disadvantaged as no data regarding HRQoL was collected at baseline. Instead, patients were assumed to have an EQ-5D-3 L utility score of -0.402 (representing an unconscious health state) in the separate trial-based economic evaluation [49]. Unconsciousness is not a defined health state by the SF-12 or the SGRQ and hence there were no pre-defined SF-12 or SGRQ QoL or HRQoL values for the baseline unconscious health state. In the current EQ-5D-3 L value set, the unconscious state has been assigned a utility value of -0.402 [49]. This value suggests that the general public, on average, considers unconsciousness to be “worse than dead (<0)” but better than being conscious and experiencing problems on all dimensions (-0.543). Future research must clearly take into account methodological issues surrounding measuring and valuing the unconscious health state in critical care. Issues include whether being unconscious in this setting represents one health state or a number of health states, if an individual has any feelings or emotions when unconscious, whether people can value unconsciousness without knowledge of the preceding and subsequent health states and whether being asleep is equivalent to being unconscious for a short time.

Lastly, in order to determine the robustness of the EQ-5D-3 L and SF-6D utility scores in this population, advocacy for mapping research is encouraged. Mapping exercises could be based on using the SGRQ summary scores from OSCAR or similar trials and mapping onto the EQ-5D-3 L, EQ-5D-5 L and SF-6D in order to assess if there is any difference in the original utility scores derived from the OSCAR trial and the estimated utility scores derived from the mapping exercises.

This study highlights the complications that can arise when trying to assess the psychometrics of PROMs in intensive care contexts and hence advocates for researchers and policy makers to notice this gap in evidence and follow through with building evidence surrounding utility values for the unconscious health state. In summary, it was considered that generic instruments were suitable to measure HRQoL in this population and showed good properties for most criteria whereas more consideration has to be given to the role of condition specific instruments in this context. The selection of a preferred PROM for evaluative studies within the ARDS context should ultimately depend on the relative importance placed on individual psychometric properties and the importance placed on generation of health utilities for economic evaluation purposes.