The ostomy leak impact tool: development and validation of a new patient-reported tool to measure the burden of leakage in ostomy device users

A systematic literature review was conducted to understand how leakage is currently defined and experienced by patients and clinical experts and to guide the development of interview guides to explore leakage and its measurement in qualitative interviews. The review was conducted in PubMed and Google Scholar. Search terms included “Ostomy,” “Ostomy/adverse effects,” “Stoma,” “Colostomy,” “Ileostomy,” “Quality of life,” “Patient preference,” and “Patient satisfaction”. Free text search terms were used where MeSH terms were unavailable. These included “Peristomal skin” and “Ostomy leakage.” Findings are reported in the Results section.

The testing of the instrument was conducted by doing cognitive debrief interviews with participants in each country. This involved the evaluation of the content validity and patient understanding of the new instrument, before conducting psychometric validation (Part 2). Content validity of the new instrument was evaluated during five cognitive debriefing/exploratory interviews conducted in each country (Denmark, UK, US, and France) with individuals with ileostomy or colostomy using stoma appliances. Participants were sent the instrument beforehand and asked to complete at home prior to the interview. Cognitive debriefing interviews were conducted by telephone in each country to evaluate item comprehension and interpretation, completeness of item coverage, item relevance, and clarity and readability of the new instrument [10]. Among the qualitative methods, cognitive interviewing allows direct input from respondents on the item content, format, and understandability. This method has emerged as an essential component in the development of a number of standardized measures. Participants had the instrument in front of them during the interview. Participants were asked to read each item separately, describe what it means to them, and how they answered it. They were asked if the item was clear, or if it should be revised in any way. Participants suggested removing “negative” statements, such as “I was not able to do things I wanted to,” as these were potentially confusing. Some items were reworded and final revisions were made based on feedback. Some negative items remained as the study team felt there should be a balance of positive and negative items. The items that participants had found confusing were reworded for clarity.

All participants completed the following sections in the online survey: a sociodemographic and clinical background form, Rand SF-36 (Physical functioning, Role limitations, Social functioning, and Mental health domains [11]), Ostomy-Q instrument (Confidence domain [12]), Ostomy Adjustment Inventory-23 (Anxious preoccupation and Social engagement [13]), and Global ratings for each domain on the instrument. SF-36 is a validated measure that has been used in several patient populations [11]. It consists of eight scaled scores, of which participants completed two four domains. The scale is from 0 to 100, where a lower score indicates worse functioning. The Ostomy-Q is a measure designed to assess quality-of-life outcomes for people using an ostomy. It was validated with ostomy patients [12] and includes four domains, of which participants completed the Confidence domain. The OAI-23 is a validated measure that has been used to measure psychological adjustment in patients with an ostomy. It includes four factors of which two were completed by participants. At baseline and follow-up, participants were also asked four additional global rating type questions regarding the overall impact on their lives of leakage or worry about leakage. The other three questions assessed the specific emotional impact, impact on usual activities, and impact on social activities. These were rated on a 5-point scale (‘not at all’ to ‘very much’), at baseline and follow-up. Those participants who provided the same rating at each time point were considered stable and were included in the assessment of test-retest reliability.

All instruments were scored and summarized for each participant. Sociodemographic/clinical data were summarized by descriptive analyses as appropriate.

Decisions regarding item reduction were based on information from item performance, the principal components analysis (PCA) and the reliability analysis. The plan was to develop a larger set of items than needed, and then through examining item performance the pool of items was reduced by removing those that functioned poorly.

Item variability determines the extent to which the range of potential responses for each item is selected by the participants. The total number of responses, percentage of total responses for each item, and floor and ceiling effects were calculated. The presence of a floor or ceiling effect was defined to occur when 50% of the responses were in the lowest or highest response category for any item [14, 15]. PCA was used to explore the “hypothesized factor structure/domains” of the leakage tool (see Additional file 1). This analysis tested several different issues. The instrument was designed such that each of the four domains had a number of different items reflecting aspects of that domain. The PCA tested the extent to which items loaded into their domains. It also tested whether items loaded into more than one domain (usually considered undesirable). Different approaches to the PCA were used in an attempt to better describe the patterns of variability in the data.

Internal consistency of each conceptual domain was measured using Cronbach’s alpha. A minimum alpha value of 0.70 was considered acceptable [16]. An alpha value between 0.80 and 0.95 indicated that the domain was internally consistent [17].

Test-retest reliability was assessed by correlating domain scores between baseline and 10–20 days after baseline, in participants reporting no change in leakage in the global ratings questions. (At reviewer request, we also assessed the test-retest reliability of the global ratings questions for all participants).

The intraclass correlation coefficient (ICC; two-way mixed, single measure (3,1)) was used to test for test-retest reliability [18]. For the purposes of this study, the following ICC thresholds were used: scale-level ICC > 0.70 was considered acceptable and an ICC > 0.80 was considered good. Questions were also used to identify participants who reported a change in their perception of the overall impact on them of leakage. Participants who reported a change on the global ratings from baseline (either better or worse) were separated out as different groups (Fig. 2; see also Additional file 2). The degree of change in their scores on the leakage tool was calculated and effect sizes and standardized response means were calculated.

Convergent validity of the new instrument was evaluated by benchmarking the new instrument with two validated instruments: the total score of the OAI-23 and domains of SF-36 (Emotional functioning, Physical functioning, Role physical, and Social functioning), and Ostomy-Q (Confidence domain).

Associations between these measures were explored using parametric and/or non-parametric correlations as appropriate. Convergent validity was considered supported if correlation coefficients between related scales were > 0.40. More specifically, for the Emotional impact domain we hypothesized correlations with total score (OAI-23); and Emotional well-being (but not Physical functioning or Role physical) (SF-36) and the Confidence domain (Ostomy-Q). The Usual and social activities domain was hypothesized to correlate with total score (OAI-23) and Physical functioning, Role physical, and Social functioning domains (SF-36). Coping and control was hypothesized to correlate with total score (OAI-23) and Confidence domain (Ostomy-Q), but not with Physical functioning (SF-36).

Different methods exist for determining the degree of change that would be considered a minimal important difference. These values support the interpretation of change data. The methods used here are based on statistical dispersion (standard error of measurement and ½ standard deviation) and degree of change indicated by the global ratings.

Of 213 titles, 11 full articles were considered relevant. The overall review of the peer-reviewed literature identified no standard accepted definition of “leakage.” In some studies, participants were asked to report experience of leakage without definition [5]. When described, the literature stated two general areas of leakage measurement: leakage “under” the baseplate and/or leakage “outside” the baseplate [5, 7, 19].

Two scales to measure leakage were identified: the Dialogue Study Ostomy Appliance Questionnaire [5] and the City of Hope – Quality of Life – Ostomy Questionnaire [8]. Other studies reported using subjective Likert scales to measure frequency or severity of leakage. Although these measures were used, there was no clear validation of the questions applied to measure leakage in a systematic way.

In the qualitative interviews, users described leakage in several ways: “when it has reached my clothes,” “stool leaking out from under the plate,” “leakage under the barrier or baseplate,” or “when it goes onto my skin (from under the skin barrier).” Participants were asked to describe the impact of leakage in four areas: emotional well-being, social, physical activity, and lifestyle. Participants described a “fear of leakage,” which felt as if they had loss of control. Even if participants hadn’t actually experienced leakage, the possibility of it would always be present. Due to the fear of leaks or experience of actual leaks, people felt embarrassed in public places, and shame and worry that leakage could happen at any time. Participants expressed feeling isolated and withdrawn if leakage occurred. A small number of people felt socially limited: most would go out regardless of fear of leakage. Some participants reported planning ahead for a specific activity such as travel or sport. People felt that they could not do certain activities such as swimming or golf, due to fear of leakage. However most people managed their condition and didn’t let their ostomy limit their activities. Some also expressed limitations in diet and having to wear loose clothing due to leakage.

In total, 340 people took part in the survey, but most analyses are based on a smaller sample as not all participants completed all questions. Participants’ demographic and background characteristics are given in Table 1.

The PCA showed that the proposed conceptual framework for the measure was not well supported. It was therefore agreed that the PCA should guide the development of a new conceptual framework that could be supported in terms of item content and statistical analysis. In the PCA analysis, factor solutions were forced or were allowed to emerge. Different approaches to rotation were explored, and after several rounds the use of promax rotation was settled on. Various methods of factor analysis were also explored, such as principal axis factoring. Items were removed if they cross-loaded (loaded above 0.400 on two different domains). Items were also removed if they did not appear to fit a domain conceptually. Through this process, 14 items were removed from the measure, leaving 22 items. The original four domains were not supported by the PCA. A three-factor solution was considered to be the best fit for describing the data. The pattern of missing data meant that the final PCA only included data from 93 participants, and so this analysis was repeated with a mean imputed value (as opposed to list-wise deletion) [20]. The final PCA results were very similar in both models.

These three factors were labeled Emotional impact; Usual and social activities; Coping and control. Some of the items had quite a high proportion of responses at the extremes, but these items were retained based on other aspects of their performance (data not shown). Many items in the Usual and social activities domains were removed, and what was left seemed to suggest a grouping of items around a single concept that reflected everyday or usual activities and included social activities. Lastly, a different set of items emerged that appeared to reflect issues related to coping and feelings of being in control. This led to some analyses to determine whether these domains are measuring something of importance or concern. To explore this, the distribution of the data for people who reported that they had experienced a leak in the last 7 days was compared with those who hadn’t (Fig. 3a for a histogram of this analysis for the Emotional impact domain). More detailed histograms are presented in Additional file 3. The box plots for the Emotional impact domain indicate that the scores are reasonably normally distributed. The box plots of the Usual and social activities domain (Fig. 3b) suggest a degree of skew in the data. There is a difference between the two groups in terms of distribution, but the scores in both groups are very high. Figure 3c shows the box plots for the Coping and control domain, which shows evidence of normally distributed scores and sensitivity to the experience of a recent leak. The histograms in Additional file 3 unpack these data in more detail. These analyses suggest that there are some limitations in the distribution of the Usual and social activities domain, but the other two domains seem to show normal patterns of distribution. The final tool is shown in Fig. 4.

The internal consistency estimates ranged from 0.77 (Coping and control) to 0.93 (Emotional impact). The internal consistency of the Coping and control domain may be slightly lower because the domain includes fewer items (Table 2).

In the global ratings, most participants reported the same answer at both time points, and the distribution of people who changed responses was fairly even (data not shown). The data suggest that the leakage measure is fairly stable over time (Table 2) and that the majority of participants didn’t report any change between baseline and follow-up (N = 169). The test-retest reliability (ICC) of the global rating questions all exceeded 0.70.

The validity of the three subscales of the leakage questionnaire was examined against different external criteria. These criteria included the experience of leakage, how frequently people had leaked in the previous 7 days, where the leak had happened, and the nature of the leak itself – under the baseplate only, or onto clothes. Table 3 shows the impact of a leak in the last 7 days on participants’ scores on the three dimensions of leakage tool. All three were significantly different.

Further analysis was conducted that explored the impact of leaks by location, i.e. whether the location of the participant at the time of the leak has a significant effect on a participant’s subjective experience. The data show that whether participants experienced a leak at home or not has very little impact on the scores on the leakage tool (Table 4). In contrast, those who report having had a leak in a public place outside the home report significantly lower scores on the Coping and control domain and a difference on the Usual and social activities domain that approaches significance (p < 0.05). For the Emotional impact domain, the trend was the same. In addition, the impact of the severity of most recent leak experienced by the participant was also explored. This was classified as either none, leakage under the baseplate only, leakage onto clothes, or leakage under the baseplate and onto clothes. All three domains were sensitive to differences between the patients, grouped in terms of leak severity. The most marked effect seemed to be related to the occurrence of leakage onto clothing (Table 5).

The performance of the three domains of the leakage tool was compared and benchmarked against other established measures of general HRQoL- and ostomy-related confidence, adjustment, and coping. The hypothesized relationships between measures (Table 6) was well supported. All hypothesized relationships exceeded the criterion of a correlation > 0.40; however, a number of relationships that were not predicted were also found – such as correlations between Emotional impact, Social functioning, and Coping and control. Emotional impact also correlated with Physical function (SF-36), which didn’t match our hypotheses.

Numerical estimates of sensitivity were estimated (Table 7) based upon the degree of change from baseline to follow-up in the leakage tool for those people who reported some level of improvement on the global ratings. The variables that were estimated include effect size. Effect size estimates suggested that the Emotional impact domain was the most sensitive. In contrast, Coping and control showed little sensitivity to self-reported change.

The degree of change that participants considered an important difference on the scale was estimated using a range of standard methods (as outlined above). The scores on the leakage tool reflected the improvement or deteriorations of users. There was an interesting effect whereby those reporting no change were also those reporting the least burden from leaks (data not shown). Table 7 shows the minimal important differences based on three methods.