Measuring quality of life in Duchenne muscular dystrophy: a systematic review of the content and structural validity of commonly used instruments

This review contains two searches. The first search (Search A) was designed to identify PROMs used to measure QoL in DMD in peer-reviewed publications. The second search (Search B) was used to identify literature reporting on the measurement properties of these PROMs in DMD. Search B also included the recommended practice of searching for the development papers of PROMs to enable a full COSMIN assessment of their content validity [12, 13, 15]. Full copies of the searches are contained in Additional file 1, for reproducibility.

Data extraction was undertaken by two reviewers using a pre-prepared data extraction sheet, with consensus on any ambiguities reached through discussion. The data extraction sheet was informed by the tools developed by COSMIN on reporting guidance [16], and included study characteristics (authors, year, journal, country, language, study type); details of the PROM used (name, mode of administration, recall period, total N subdomains, subdomain names, total and subdomain N items, total and subdomain response levels, total and subdomain score ranges); DMD sample characteristics if applicable (N, age, percentage ambulatory, total and subdomain PROM score, total and subdomain observed ranges); details of PROM development if applicable (construct definition, target population, original language, intended context of use, patient involvement); details of content validity results if applicable (summarised results, e.g. findings from a cognitive debriefing exercise); and details of structural validity results if applicable (analytic model, summarised results, e.g. fit statistics, tests of model assumptions for IRT/Rasch).

The methodological quality of the PROM development papers, and studies on content and structural validity were assessed (at the study level) using up-to-date COSMIN standards via the new COSMIN risk of bias checklist [24]. A total rating for relevance, comprehensiveness, and comprehensibility (content validity aspects) of a PROM is determined separately, alongside a total rating for the methodological quality of a structural validity study [24]. When rating the methodological quality of the studies, each COSMIN standard (or item) is ranked on a 4-point scale: “very good”, “adequate”, “doubtful”, and “inadequate”. Total ratings are determined using the lowest rating for any item for that study (i.e. worst score counts) [25]. Studies were initially rated independently by two reviewers, and, in the case of divergence, consensus was reached in a subsequent face-to-face meeting. This information on risk of bias is used to inform quality of evidence (see section 2.2).

In order to synthesise and assess evidence on content validity, two reviewers independently rated the results of PROM development studies, content validity studies, and the content of the PROM itself on 10 COSMIN criteria [13], agreed upon by international consensus [12]. These criteria included: whether the included items were relevant for (I) the construct of interest, (II) the population of interest, and (III) the context of use of interest; whether the (IV) response options and (V) recall period were appropriate; whether (VI) all key concepts were included; whether (VII) the PROM instructions and (VIII) PROM items and response options were understood by the population of interest as intended; whether (IX) the PROM items were appropriately worded; and whether (X) the response options matched the question. Ratings for each source of evidence were made separately, using COSMIN guidance [13] (p.54) and could either be positive (+), negative (−), or indeterminate (?). Reviewers’ ratings were made based on the judgement of the researchers, who have experience in PROM design and work with people with DMD, including direct qualitative research [26, 27]. When reviewers considered whether the items were relevant or comprehensive for the construct of interest, they were compared against the CMQM [9]. Accordingly, a PROM would be sufficiently comprehensive (+) if it included items covering physical, psychological, and social aspects of QoL. When judging the appropriateness of the recall period, reviewers considered any defined recall period of up to 4 weeks as appropriate (+), as children aged 8 years and above can recall up to this length of time with sufficient accuracy [28]. When rating the appropriateness of response options, bearing in mind the target sample (i.e. a child or adult PROM), reviewers took into account the numerical range, how the response options were visually displayed, and the perceived cognitive complexity of the options (including wordiness, degree of variation throughout the questionnaire, and the use of reversed ordering).

Following the above assessment, an overall (qualitatively synthesised) judgment on the relevance, comprehensiveness, and comprehensibility of each PROM was made, which could be sufficient (+), insufficient (−), or inconsistent (±), using COSMIN guidance [13] (p.58). For example, if all sources of evidence were rated positive (+) for relevance, then the overall rating for the PROM would be sufficient (+). As recommended by COSMIN [13], more weight was given to content validity studies, then development studies, then ratings of the PROM by reviewers. Ratings were compared and combined across the two reviewers by consensus. As per COSMIN guidance [13], only available evidence was taken into account when assessing content validity, so, for example, if there were no content validity studies in DMD available for that PROM, assessment was made based on the ratings of any PROM development studies and the ratings of reviewers. The fact that the PROM had no content validation studies in DMD is then reflected in a lower quality of evidence rating (see below). An example content validity rating spreadsheet for the KIDSCREEN-52, including the rules for synthesising the individual ratings is included in Additional file 2.

Evidence on structural validity was assessed against the updated COSMIN criteria for good measurement properties, using the same rating scale as above [16]. Specifically, a positive (+) rating would be given for a CFA featuring a CFI, TLI or comparable measure > 0.95 OR RMESA < 0.06 OR SRMR < 0.08. For an IRT/Rasch model, a positive (+) rating would be given for no violation of unidimensionally (e.g. assessed with the fit statistics above) AND no violation of local independence (e.g. residual correlations among items after controlling for the dominant factor < 0.20) AND no violation of monotonicity (e.g. evidenced graphically or item scalability > 0.30) AND adequate model fit (e.g. χ² < 0.01, infit/outfit mean squares ≥0.5 and ≤ 1.5 OR Z-standardized values > − 2 and < 2. A negative (−) rating would be given if these criteria were not met in the data and an indeterminate (?) rating would be given if model fit was not reported.

Finally, the quality of the evidence was graded using a modified GRADE approach [29], as either “high”, “moderate”, “low”, or “very low”. The GRADE approach takes into account the risk of bias of studies (or study quality); (in) consistency across studies; imprecision (based on sample sizes); and indirectness (of evidence) [16]. The evidence is assumed to be high, then is downgraded by 1–3 points based on the degree of risk of bias (including quality and absence of content validity studies), 1–2 points based on inconsistency, and 1–2 points based on indirectness. Further details on how to apply all of the above criteria are provided elsewhere in comprehensive manuals, which were followed when conducting this review [13, 16]. The quality of this systematic review itself was appraised against a recently developed COSMIN checklist to assess the quality of systematic reviews of health-related PROMs [22].

After removing duplicates, 1733 records were identified through database searching for Search A. Of these, 1521 were excluded at the title/abstract review stage, leaving 212 papers for full-text review. Of these 212 papers, 84 were excluded as they were not full-text published research articles; 25 did not meet the required sample criteria of at least 75% of the sample being boys or men with DMD; 21 were judged not to be assessing QoL; 16 were not published in English; and finally 11 papers did not feature a multi-item PROM. Five articles were additionally excluded during the review of the actual PROM used in the manuscript for not assessing QoL. Accordingly, a total of 50 records from the initial searches met the selection criteria for Search A. A further 5 articles that met the selection criteria for Search A were added as a result of citation tracking, giving a total of 55 records.

Table 1 summarises the PROMs used to assess QoL in DMD from the full-texts meeting the selection criteria at Stage 1 (n = 55). A total of 40 PROMs used to assess at least one aspect of QoL in DMD were identified in published research articles through database searching (the two HUI classification systems use the same 15-item PROM). The majority of the PROMs were multidimensional (n = 32), designed to assess a range of different facets of QoL. The remaining unidimensional scales were designed to assess: activity limitations (CALI); anxiety (GAD-7); depression (BDI, DIKJ, PHQ-9); fatigue severity (FSS); life satisfaction (SWLS); or quality of life/health-related quality of life unidimensionally (KIDSCREEN, SOLE). Twenty-four of the PROMs had versions designed for completion by adult or young adult respondents, and 26 had versions designed for children. The most popular PROMs used in published research articles assessing QoL in people with DMD were the PedsQL 4.0 GCS (18 articles); PedsQL 3.0 NMM (10 articles); and the SF-36 (8 articles).

After removing duplicates, 92 records were identified through database searching for Search B. Of these, 51 had already been excluded during Search A. Eighteen unique records were found, 14 were excluded at title/abstract review stage, leaving 4 papers for full-text review. Of these 4 papers, 3 were excluded because they were not full-text primary research articles; and 1 did not meet the sample selection criteria. Of the remaining 23 records that had already met the selection criteria for Search A, 11 were excluded at full-text for containing information on measurement properties other than, and not including, content and structural validity; 4 were excluded as not containing information on measurement properties; and 3 for including a PROM that did not have a validated English copy that was free and/or available for review. The remaining 5 papers that met the selection criteria for Search B featured evidence on content validity (n = 3, of which one was classified as a development paper) and structural validity (n = 2). Finally, 33 PROM development papers were identified through a review of Google Scholar search results and 3 PROM development papers were identified through citation tracking, resulting in a final selection of 41 papers that met the selection criteria for Search B (see Fig. 1). These included 37 development papers, 2 content validity studies in DMD, and 2 structural validity studies in DMD.

The observed proportionate agreement between reviewers during selection, based on the primary database searches, was 92.4% at title/abstract, with Cohen’s κ = 0.51 or “moderate agreement” and is similar to other published reviews [84, 85]. At full-text review, the observed proportionate agreement was 93.5% with Cohen’s κ = 0.82 or “almost perfect agreement”.

Following the searches, 26 PROMs were taken forward for COSMIN quality assessment on content and structural validity in DMD (Table 2). The remaining 14 PROMs were not assessed for the following reasons: a copy of the PROM itself and/or necessary development papers were not freely accessible for review (CAPE, CHQ-PF50, DISABKIDS Smileys, OSIQ, SF-36 v2); no formally validated English copy of the PROM was available or in use (AUQEI, DIKJ, DUC-25, SOLE, TAAQoL, TACQoL); the PROM was no longer available or recommended for use (BASC 1st edition, which has been superseded by the BASC 2); or it was unclear from the study which of a large number of possible variants of a PROM were used (pediatric Neuro-QoL, Neuro-QoL).

Table 2 summarises key characteristics and COSMIN quality assessment of the development of the PROMs included in the review. Five PROMs were developed to be intended for use specifically within neuromuscular disorders (INQoL, PedsQL 3.0 NMM) or DMD (LSIA, MDCHILD, PedsQL 3.0 DMD module). Eleven PROMs either had no patients involved in their development, or it was unclear if patients were involved.

The joint most common COSMIN quality rating assigned to the PROMs for concept elicitation was inadequate (n = 12). This was primarily due to: the PROM development study not being performed in a sample of patients representing the target population (BDI, EQ-5D-3L, GAD-7, HADS, HUI 15Q, PedsQL 3.0 MFS, PHQ-9, SDQ, SF-36, and SWLS); or inadequacies within the details of the qualitative methods used (FSS, INQoL). The concept elicitation study of 11 further PROMs was rated as doubtful due to at least some unclear details/suspected problems within the qualitative methods used (CALI, DCGM-37, LSIA, MDCHILD, PODCI, PedsQL 3.0 NMM, PedsQL 3.0 DMD, PedsQL 4.0 GCS, PedsQL 4.0 SF-15, PSQI, WHOQOL-BREF). Only the KIDSCREEN family of measures (n = 3) received an adequate rating for concept elicitation and PROM design. However, the KIDSCREEN measures received a doubtful rating for the overall PROM development study, for failing to provide evidence that comprehensibility and comprehensiveness were assessed in the cognitive interview/pilot study of the PROM.

Only 2 published articles had independently assessed the content validity of the QoL PROMs in samples of people with DMD (Table 3). Neither of these studies were conducted in an English language context, and instead were cross-cultural validation studies. Hu et al. (2013) [67] assessed the relevance, comprehensiveness, and comprehensibility of the PedsQL 3.0 NMM in Chinese children with DMD. Simon et al. (2017) [56] assessed comprehensibility of the LSIA in Brazilian children with DMD, and comprehensiveness in professionals. However, both of these studies received ratings of doubtful due to at least some unclear details/suspected problems within the qualitative methods used.

The evidence from the PROM development papers and content validity studies was combined with reviewer ratings of the PROMs to produce a synthesis of the available evidence using the 10 COSMIN criteria for good content validity [13]. Most of the quality of the evidence was downgraded from High to Low or Very Low due to the assessment being based on development studies of doubtful or inadequate quality, respectively [13]. Only the LSIA and the PedsQL 3.0 NMM had moderate supporting evidence, featuring independent content validity studies as well as development papers. The KIDSCREEN measures and the LSIA were the only PROMs to receive satisfactory results for all three dimensions of content validity: relevance; comprehensiveness; and comprehensibility, based on the evidence available. Full synthesised results are presented in Table 4.

Two studies had assessed the structural validity of the PROMs included in this review in samples of people with DMD (Table 5). Both of these were conducted using English versions of the PROMs and either in the UK or USA. Lim et al. (2014) [72] assessed the structural validity of the PedsQL 4.0 GCS using an unspecified Rasch model in 63 boys with DMD. This study received a COSMIN quality rating of doubtful because it was doubtful that the sample size included in the analysis was adequate. Landfeldt et al. (2018) [66] assessed the structural validity of the PedsQL 3.0 NMM using a Rasch partial-credit model (PCM) in 278 people with DMD. This study received a very good COSMIN quality rating for its methodological content.

Of the 2 studies that assessed the structural validity of the PedsQL 4.0 GCS and PedsQL 3.0 NMM in people with DMD, neither provided satisfactory results (Table 4). First, the structural validity of the PedsQL 4.0 GCS in people with DMD received an indeterminate rating, as key details of the results from the Rasch model denoting good measurement properties were not reported. Due to the risk of bias assessment of Lim et al. (2014) [72] the quality of the evidence supporting this indeterminate conclusion was rated as very low. Second, the structural validity of the PedsQL 3.0 NMM in people with DMD received an unsatisfactory rating, as the psychometric criteria for good measurement properties were not met. The favourable risk of bias assessment for Landfeldt et al. (2018) [66] meant that the quality of evidence supporting this conclusion was graded as high.

The quality of this review was self-assessed against a newly derived COSMIN checklist [22], designed to evaluate the quality of systematic reviews of health-related PROMs. The results are displayed in Additional file 3.

In general, the review meets numerous quality indicators as defined by the COSMIN team, including the elements included in the research aim, search strategies, article selection, and assessment of measurement properties and quality. In a couple of instances, criteria have been partly met. For example, in this review all instruments were included where a validated English copy was freely available for review. It is possible that additional instruments could have been included if licenses were paid for to access the relevant PROMs and development materials, and this could be considered a limitation. Second, citation tracking (i.e. reference checking) was conducted on the final set of articles eligible at Stage 2 of the searches (n = 41), but not on results eligible for inclusion at Stage 1.