A systematic review of the measurement properties of the European Organisation for Research and Treatment of Cancer In-patient Satisfaction with Care Questionnaire, the EORTC IN-PATSAT32

The literature search was part of a larger systematic review (Prospero ID 42017057237 [14]) investigating the validity of 39 different PROMs measuring quality of life of cancer survivors included in an eHealth application called “Oncokompas” (Amsterdam, the Netherlands) [15‐18]. The databases Embase, Medline, PsycINFO, and Web of Science were searched for publications that investigated measurement properties of these 39 PROMs including the EORTC IN-PATSAT32. The search terms were the PROM’s name, combined with search terms for cancer, and a precise filter for measurement properties [19]. The first search was performed in July 2016. The full search terms can be found in Appendix A. An additional search (up to July 2017) was performed using the same search terms, and a subsequent manual search in Google Scholar and Pubmed for missing records, to search for recent studies.

Studies were included that reported original data on cancer patients, and on at least one of the following measurement properties of the IN-PATSAT32 as defined by the COSMIN taxonomy [10‐13]: internal consistency, reliability, measurement error, structural validity, hypothesis testing (for construct validity), criterion validity, cross-cultural validity, and responsiveness. Validation studies on other PROMs, which also reported original data on the IN-PATSAT32, were included. Studies that were only available as abstracts or conference proceedings were excluded, as well as non-English publications. Titles and abstracts, and the selected full-texts were reviewed by two independent raters (KN and FJ). Disagreements were discussed until verbal agreement on consensus.

Two independent researchers (KN and FJ) extracted information from eligible papers on each of the measurement properties defined by the COSMIN taxonomy [10]. Relevant data included the type of measurement property, its outcome, and information on methodology. Disagreements were discussed until verbal agreement on consensus.

Data synthesis consisted of three steps. First, the quality of the methodology of the included studies was rated using the 4-point scoring system of the COSMIN checklist [10‐12]. Methodological aspects regarding design requirements and preferred statistical methods, specific to each measurement property under consideration, were rated as either “poor”, “fair”, “good”, or “excellent”. The methodological quality was operationalized per measurement property per study as the lowest score they received on any of the methodological aspects. The final ratings can be found in Appendix B.

Second, criteria for good measurement properties were applied to the results of the included studies, following the COSMIN guidelines for systematic reviews of PROMs [13]. Each measurement property in each individual study was rated as sufficient (+), insufficient (−), or indeterminate (?), according to predefined criteria. For indeterminate ratings, the methodological rating was non-applicable. All of these ratings were qualitatively summarized to determine the overall rating of the measurement property. If all studies indicated a sufficient, insufficient, or indeterminate rating for a specific measurement property, the overall rating of this measurement property was accordingly. If there were inconsistencies between studies, explanations were explored (e.g., differences in methodological quality). If explanations were found, they were discussed until consensus was reached and taken into account during interpretation. If no explanations were found, the overall rating would be inconsistent (±).

Third, we used the modified GRADE approach [13] to rate the quality of the evidence available for the measurement properties of the IN-PATSAT32. This approach takes into account (i) methodological quality, (ii) directness of evidence, (iii) inconsistency of results, and (iv) precision of evidence. The overall quality of evidence was rated as high, moderate, low, or very low. Measurement properties that were rated as indeterminate in the previous step did not receive a rating as there was no evidence to rate. All ratings (methodological quality, measurement property rating, and GRADE rating) were rated by one researcher (KN), whose ratings were checked by a second independent researcher (AH). Discrepancies in ratings were discussed until verbal agreement on consensus.

The initial search identified 980 abstracts of which 10 were relevant to the IN-PATSAT32 (Fig. 1). Three abstracts and one full-text were excluded for not providing unique information on a measurement property. One study not captured by the search, but known to the authors, was added before data extraction. The search update up to July 2017 identified three more abstracts of which one was excluded for not providing unique information on a measurement property. No full texts were excluded from this search update. In total, nine studies were included in this review (see Supplementary Table 1). These nine studies reported on the structural validity (six studies), internal consistency (five studies), reliability (two studies), and hypothesis testing (six studies) of the IN-PATSAT32, but lacked information on measurement error, criterion validity, responsiveness, and cross-cultural validity. We were able to calculate measurement error for two studies.

Six studies reported on structural validity. Methodological quality of these studies was rated as “good” [20], “fair” [21], or “poor” [22‐25] (Table 1). The poor ratings were due to using Multitrait Item Scaling (MIS) instead of confirmatory or exploratory factor analysis (CFA/EFA). The fair score was due to lack of information about the handling of missing values. Results of the MIS analyses were consistent across studies, as well as with the original validation study. However, MIS is an indirect way of testing structural validity. Therefore, no conclusions can be drawn on basis of these studies. Two articles [20, 21] presented results of principal component analyses (PCA). Hjörleifsdóttir [20], of “good” quality, extracted four components with an eigenvalue > 1, with a balanced distribution of explained variance. Pishkuhi [21], of “fair” quality, extracted five components with an eigenvalue > 1, and one of those components explained most of the variance. The factor structures found in these two studies were inconsistent with the 11 subscale (and three single-item scales) model as reported in the initial study [8], leading to an insufficient rating.

Five studies reported on internal consistency of the IN-PATSAT32, and their methodological quality was rated as “good” [20], “fair” [21], or “poor” [22‐25]. The main reason for the poor ratings was that the unidimensionality of the scales was not tested appropriately. The values for Cronbach’s alpha of five studies [21‐25] are presented in Supplementary Table 2. One other study [20] presented Cronbach’s alpha values for scales they had established: nurse satisfaction (α = .95), doctor satisfaction (α = .93), information satisfaction (α = .91), and service satisfaction (α = .67). However, as these scales do not represent the subscales recommended for this questionnaire [8], this study is not included in Supplementary Table 2, nor further taken into account. All but one subscale (hospital access) showed Cronbach’s alpha values that would qualify for a sufficient rating. However, as none of the studies provided any evidence of unidimensionality for the subscales, Cronbach’s alpha cannot be properly interpreted [26]. The inconsistentency of Cronbach’s alpha coefficients across studies is noteworthy for the subscale hospital access (α = .36–.86).

Two studies [21, 25] reported on test-retest reliability (see Supplementary Table 3). Methodological qualities were rated as “fair” due to lack of information about the handling of missing values [21, 25], not reporting the type of correlation coefficient [21], and a short time interval (30 min) [25]. One study [21] showed high test-retest correlations (r > .85), leading to a sufficient rating on test-retest reliability. However, as the type of correlation coefficient was not reported, it is unclear whether these values represent appropriate estimates of test-retest reliability [27, 28]. The other study [25] showed acceptable test-retest correlations (ICC > .70), except for doctors’ availability (ICC = .64) and general comfort (ICC = .67), leading to a sufficient rating.

While none of the studies presented results regarding measurement error, the standard error of measurement (SEM) and smallest detectable change (SDC) could be calculated for the two studies reporting test-retest reliability [21, 25]. Methodological quality was “good”, due to the need to calculate measurement error indirectly (Table 2). Since no minimal important change (MIC) was reported, a criterion for good measurement error could not be applied. While there is no evidence for or against good measurement error, the SDC could be compared to the maximum range of the subscales. The SDC represents the minimum change score over time of which we can be certain does not represent measurement error. Most SDC scores were between 20 and 30, representing 20–30% on the 100-point scale. There were a few notable outliers: Doctor availability (29.17–46.40), waiting time (25.05–44.70), and hospital access (29.39–34.48).

The synthesized ratings of the measurement properties can be found in Table 5. Internal consistency was rated indeterminate as no tests of unidimensionality were reported. Measurement error was rated indeterminate as no MIC was reported and could not be calculated with the available data. Structural validity was rated insufficient with evidence of low quality. Test-retest reliability and construct validity (hypothesis testing) were judged to be sufficient, both with evidence of moderate quality. The indeterminate findings [23, 24] for construct validity were not taken into account in this synthesis, as they did not provide evidence for or against construct validity. Studies of “poor” quality were outweighed by studies with better quality. One study of “good” quality provided insufficient evidence on convergent validity for construct validity [29], while three studies of “fair” quality provided sufficient evidence on known-group comparison and convergent validity for construct validity [21, 22, 32].