Understanding the impact of tuberous sclerosis complex: development and validation of the TSC-PROM

The construct to be measured is the impact of TSC on physical functions, mental functions, activity and participation, and the social support individuals with TSC receive, using the framework of the ICF (World Health Organization 2001).

The purpose of the TSC-PROM is to evaluate and monitor the impact of TSC on functioning, serving as a tool to facilitate detection and discussion of healthcare needs relevant to individuals with TSC before or during a clinical visit. Two versions of the questionnaire were developed: a self-rated questionnaire for individuals with TSC without ID or a mild ID and a proxy-rated questionnaire for parents and caregivers of adults with TSC who could not complete the questionnaire themselves due to ID severity as assessed by primary caregivers or legal representatives.

The TSC-PROM was developed for all adults (18 years or older) diagnosed with TSC. To fill out the questionnaires, individuals with TSC, parents, or caregivers were English or Dutch-speaking.

Relevant themes were identified by conducting interviews with adults with TSC and caregivers of adults [10]. The TSC-Associated Neuropsychiatric Disorders (TAND) checklist, Lifetime Version (TAND-L) [4], and TSC literature on adult manifestations were reviewed to identify additional themes [1, 10, 35‐37]. The TAND checklist was specifically designed as a screening tool for neuropsychiatric manifestations of TSC, and validated, showing sufficient internal consistency and external validity [38]. Additionally, representatives of patient organizations were asked to identify additional themes. Expert meetings with an expert group representing various disciplines including neurology, psychiatry, psychology, endocrinology, nephrology, ID physician, methodological experts, and representatives of individuals with TSC were held to identify and assess the relevance of the themes until consensus was reached (AvE, AR, MdW, LdG, ET, PJ, PdV, LtH, JvdE).

After identifying relevant themes, the expert group categorized TSC-relevant themes by using the framework of the ICF [39]. The ICF delineates several domains, including the components health condition, body functions and structure, activity, participation, environmental factors, and personal factors. These components were used to classify the TSC-PROM subscales. The component body functions and structure was divided into the domains (1) physical functions and (2) mental functions. The ICF components activity and participation were combined into one domain (3) activities and participation. The fourth subdomain social support was composed of the ICF component environmental factors.

From the identified themes, simple and quantitative questions with response options using 4-point Likert scales were formulated by the expert team. Higher scores indicated better situations. The response options and recall period of the past month were chosen based on the expert opinion. Additionally, visual analog scales (VAS) were included per domain. We also included one question about their health-related quality of life (HRQoL). The preliminary test versions of the questionnaires for individuals with TSC and proxies were reviewed and refined by the expert group. Based on their expertise, topics were added, altered, or removed.

Two questionnaires were initially developed in the Dutch language in the Netherlands and Belgium. After professional expert translation into English using back and forward translation, a pilot was performed in the USA with English-speaking individuals with TSC and representatives (at least five self-rated and five proxy-rated) in order to broaden the population. The pilot concluded with a discussion with the expert group, and final revisions were made to the questionnaires with a final consensus.

A cognitive interview study was performed in all participating languages (Dutch and English) to assess the comprehensibility of the questionnaire. Other than those who participated in the concept elicitation, individuals with TSC from the participating outpatient clinics were asked to provide qualitative feedback on the questionnaire. At least five participants were recruited per type of report (self or proxy) with a definite diagnosis of TSC [2] and a minimum age of 18 years for each participating country (the Netherlands, Belgium, and the USA), with an aimed minimum of 30 participants. We aimed to include participants with different ages, gender, and education to have a sample representing the target population. Individuals with TSC were excluded when an additional genetic disorder to TSC was diagnosed. The cognitive interview study included the “Think aloud” method [33, 40, 41] and the “Retrospective Verbal Probing” technique to assess comprehensibility of the instructions, all items, response options, and recall period [41].

E-mails with an invitation link and login code granting access to the questionnaires were sent to participants after answering a question about whether the questionnaire will be filled out by either the individual with TSC themselves or a proxy when (assisted) self-report was not possible, as indicated by the primary caregiver, legal representative, or clinician. A proxy who declared to know the individual with TSC well, such as the legal representative or primary caregiver, was allowed to fill out the questionnaires.

To be eligible for participation in this study, adults (18 years or older) with a definite diagnosis of TSC, molecularly or clinically confirmed according to recent recommendations [1, 2], should be English or Dutch-speaking. Participants were recruited from the outpatient TSC clinics at the University Medical Center of Utrecht (Utrecht, the Netherlands), Erasmus Medical Center (Rotterdam, the Netherlands), the University Hospital of Brussels (Brussels, Belgium), Le Bonheur Children’s Hospital Memphis (Memphis, USA), and Cincinnati Children’s Hospital Medical Center (Cincinnati, USA). Additionally, participants were recruited via the Dutch (STSN), Belgian (beTSC), and US (TSC Alliance) patient organizations.

In addition to the TSC-PROM, the SF-36 [24], including a proxy-report [42], and scales assessing emotional and behavioral problems from the Achenbach System of Empirically Based Assessment (ASEBA) [43, 44], i.e., the Adult Self Report (ASR), the Child Behavior Checklist (CBCL)/1.5–5 and CBCL/6–18 were used for assessing construct validity. Individuals who were mentally competent to fill out the questionnaires themselves received the ASR. For individuals with TSC who could not complete the questionnaires, caregivers or representatives indicated whether the developmental age was below or above the age of 6 years old, guiding the distribution of either the CBCL/1.5–5 or CBCL6-18 version. Information on measurement properties of these comparator instruments is provided (see Additional file 1) [24, 27, 38, 42‐62].

Item reduction was performed by selection based on frequency (at least 85% with response option “Not at all”), factor loadings, monotonicity, or local independence unless there was a clinical reason to include the item based on expert opinion. Additionally, items of the proxy version were reduced when frequency of the response options “Don’t know” and “Not applicable” was at least 30%.

Statistical analyses were performed using IBM SPSS Statistics 24 and R. Descriptive statistics were used to characterize demographics, clinical variables, and score distributions of the TSC-PROM. Domain scores were calculated as a percentage of the sum of the items within a domain. For the social support domain, the sum was divided by the number of items filled in other than “unknown” or “not applicable” times the number of response options to account for the “unknown” and “not applicable” response options. The TSC-PROM total score was the average of the domain percentages, excluding the social support domain. The social support domain was included as a scale to gather information on the type and quantity of the support someone is receiving, which is an important part of the PROM for the sake of completeness, as this could affect physical functions, mental functions, and activities and participation, but not directly a functioning component. A two-sided significance level of 5% was used.

Structural validity was assessed for the subscales (1) physical functions, (2) mental functions, and (3) activities and participation using a confirmatory factor analysis (CFA) (see Additional file 1).

With regard to internal consistency, Cronbach’s alpha was calculated for each TSC-PROM subscale, including the continuous HRQoL VAS. A Cronbach’s alpha between 0.70 and 0.95 was considered adequate [63].

Construct validity was examined by correlating the scores of the TSC-PROM with scores of other instruments that assess the same construct to be measured, also known as convergent validity. Regarding convergent validity, correlations were assessed between the TSC-PROM domain scores and the SF-36 physical component score, mental component score, and the total scores of the ASR, CBCL/1.5–5 or CBCL/6–18. Construct validity was considered sufficient if 75% of the hypotheses were met (see Additional file 1). To assess discriminative validity, analyses were performed using group dichotomization or categorization. A priori hypotheses were defined including (1) individuals with TSC2 pathogenic variants will show lower TSC-PROM scores on the physical domain, mental domain, and TSC-PROM HRQoL VAS compared to individuals with a TSC1 pathogenic variant [60‐62]; (2) individuals who reported a drastic life event in the past year will show a lower score on the mental functions domain; (3) individuals with a higher number of involved organ systems will show lower scores on the HRQoL VAS [27]; and (4) individuals with the presence of psychiatric diagnoses will show lower TSC-PROM scores on the mental functions domain, activities and participation domain, and HRQoL VAS [27] (see Additional file 1).

The questionnaires were digitally distributed using LimeSurvey [64]. As the survey did not allow for missing data, no specific missing item analysis was necessary. Only the principal investigator (AvE) and researcher (AM) had access to the code for each participant that was solely accessible in the secure network environment of the Erasmus Medical Center. Data were stored in LimeSurvey and exported to R for statistical analyses.

Concept elicitation resulted in a draft version of the TSC-PROM for both self and proxy-report (73 items from prior exploratory interviews [10]; supplementary 6 items from TSC literature on adult manifestations; 9 items from the TAND checklist [4]; 11 items from the expert group). After an expert meeting, some items were divided into separate items or combined, resulting in a total of 96 items (24 items within the physical functions domain, 43 items within the mental functions domain, 19 items within the activities and participation domain, and 9 items within the social support domain).

The TSC-PROM starts with 15 questions on demographic and clinical information, including clinical and sociodemographic information (age, sex, nationality, age of TSC diagnosis, genetic testing, organs involved, use of medication, epilepsy, level of functioning, educational level, other diagnoses or health conditions, life events). Visual analog scales from 0 to 100 were included on physical functions, mental functions, the ability to perform daily activities, and satisfaction with social support, and a HRQoL VAS was included. During item development, response options were defined using Likert scales with higher scores indicating overall less impairment. To illustrate, “a lot” (1), “somewhat” (2), “a little” (3), and “not at all” (4) were response options for items on the physical functions and mental functions domain, such as “During the past month I was bothered by [e.g. difficulty sleeping, skin abnormalities, seizures]” and “During the past month I [e.g. experienced restlessness/insecurity/difficulty in meeting new people, felt anxious, had mood swings, I worried about tumor growth/my financial independence].” Response options for the activities and participation domain included “always” (1), “often” (2), “sometimes” (3), and “never” (4) with items such as “During the past month I was limited in [e.g. learning something new, getting along with people I know well, participating in sport/physical exercise].” Response options for the social support domain included “not at all” (1), “a little” (2), “mostly” (3), and “completely” (4), “not applicable" with items such as “In the past month I was satisfied with [e.g. the support I received from my family/partner/mental healthcare professionals, how my medication is working].”

We recruited eleven participants (five self-rated and six proxy-rated) in the Netherlands, ten in Belgium (five self-rated and five proxy-rated), and ten in the USA (five self-rated and five proxy-rated), with a definite diagnosis of TSC [2] and an average age of 34.43 years (range 18–65 years). The questionnaires were completed for eighteen female participants and thirteen male participants. The level of ID differed from fourteen without ID, six with a mild ID, five with a moderate ID, and six with a severe ID. Based on the feedback received during the interview study, the following adjustments were made:

It took participants 16.53 (± 5.00, range 10–30) minutes to complete the questionnaire. Participants preferred a digital version and the lay-out was assessed as clear by 85.7% of the participants and 14.3% somewhat agreed.

Eighty-one percent of the participants found the instructions, the items, and the formulations clear, and 19% somewhat agreed. One participant indicated difficulties when complaints are always present. Small suggestions were made for clarification. 85.7% of the examples provided were clear and understandable. 66.7% indicated clear response options. Feedback included lack of the response option “not applicable” in the self-report version and difficulty to estimate the applicability of “not applicable” or “do not know” for the proxy-report. 85.7% of the participants agreed on the sequence of items.

Participants did not indicate other items or complaints and agreed on comprehensiveness, completeness, and relevance. All relevant questions were included, although not all questions were applicable to the different levels of functioning.

The TSC-PROM consisting of 96 items and the five visual analog scales was subjected to item reduction by applying the criteria defined in the method section, unless there was a clinical reason to include based on expert opinion. Two items were included based on expert opinion, namely the burden of seizures and kidneys over the past month. After item reduction, the self version contained 82 items and the proxy version 75 items (physical functions domain 18 items and 19 items with an additional item on side effects, mental functions domain 37 items and 28 items, activities and participation domain 13 items and 14 items, social support 13 items, for self-report and proxy-report respectively) (see Additional file 1). Items in the proxy version that relied on internal perception or were difficult to estimate as a proxy were removed, such as “the individual felt lonely.”

The mental functions and activities and participation self-report scales displayed sufficient unidimensionality and monotonicity according to the predefined criteria (Table 2). The physical self-report scale and the proxy-report scales did not or only partially satisfy the unidimensionality or monotonicity assumption. Some items within the self-report scale displayed local dependence (residual correlation > 0.20; physical functions domain: 4.90%, mental functions domain: 5.03%, activities and participation domain: 5.77%). Within the proxy-report scale, some items showed local dependence (physical functions domain: 8.77%, mental functions domain: 10.46%, activities and participation domain: 9.34%).

For the self-report, the corrected item-total correlations ranged from 0.00 to 0.62 (physical functions domain), 0.39 to 0.82 (mental functions domain), and 0.56 to 0.84 (activities and participation domain). For the proxy-report, it ranged from 0.00 to 0.72 (physical functions domain), 0.08 to 0.84 (mental functions domain), and 0.23 to 0.82 (activities and participation domain). Cronbach’s alpha value of the total TSC-PROM score was 0.819 and 0.775 for the self and proxy-report, respectively, which met the threshold criterion range of 0.70–0.95. Cronbach’s alpha of each subscale ranged from 0.81 to 0.97 (Table 3).

All hypotheses regarding construct validity were met for both the self version and proxy version (Table 4).

In the self-report, no significant differences were found between individuals with TSC1 and TSC2 pathogenic variants. Furthermore, individuals with TSC who experienced a life event showed a lower score on the TSC-PROM mental functions domain (p = 0.018, r =  − 0.26), patients with a higher number of organ manifestations showed a lower HRQoL VAS score (p = 0.021, r =  − 0.25), and individuals with TSC with the presence of psychiatric diagnoses showed lower TSC-PROM scores on the mental functions domain (p < 0.001, r =  − 0.47), the activities and participation domain (p < 0.001, r =  − 0.40), and HRQoL VAS score (p = 0.040, r =  − 0.22). In the proxy-report, individuals with TSC2 pathogenic variants showed a lower TSC-PROM score on the mental functions domain compared to individuals with a TSC1 pathogenic variant (p = 0.012, Cohen’s D = 0.69). No significant differences were found with regard to the experience of a life event, the number of organ manifestations, and the presence of psychiatric diagnoses.

For the self-report, all of the hypotheses were met, except for the hypotheses regarding the TSC1 and TSC2 pathogenic variants, resulting in a total of 63% of hypotheses met. For the proxy-report, one out of eight hypotheses was met (13%), which was the hypothesis regarding the effect of TSC1 and TSC2 pathogenic variants on the mental functions domain.