Qualitative Development of the Allergan Satisfaction with Treatment Experience Questionnaire (ASTEQ) Instrument, a Patient-Reported Outcome Measure in Glaucoma and Ocular Hypertension

Development of the new PRO instrument, the Allergan Satisfaction with Treatment Experience Questionnaire (ASTEQ), was guided by a series of qualitative research interviews conducted among individuals with open-angle glaucoma or ocular hypertension who had received treatment with a sustained-release bimatoprost intraocular implant within the clinical trial setting. A preliminary conceptual framework capturing treatment satisfaction concepts pertinent to glaucoma (as identified by the literature review) was used to develop a qualitative, semi-structured interview guide for the initial (concept elicitation) interview. The purpose of this interview was to uncover specific aspects of the glaucoma treatment experience that were most relevant and important to these patients. Results from the interview were used to draft a PRO instrument, which was later refined based on input from a subsequent (cognitive debriefing) interview undertaken to test the interpretability and relevance of the instrument. Independent review board approval was obtained for all study documents and protocols. All subjects provided their written informed consent to participate in the study, and written authorization to access personal health information in accordance with the US Health Insurance Portability and Accountability Act (HIPAA).

Subjects with open-angle glaucoma or ocular hypertension were recruited from Allergan’s phase 3 ARTEMIS 1 and ARTEMIS 2 clinical studies of a sustained-release bimatoprost implant in dose strengths of either 10 µg or 15 µg (ClinicalTrials.gov, identifiers NCT02247804 and NCT02250651) [15, 16]. In addition to meeting the ARTEMIS study eligibility criteria, participants were required to (i) have received three scheduled intraocular injections of the 10 µg or 15 µg sustained-release bimatoprost implant at 16-week intervals as part of the ARTEMIS study; (ii) be available to complete a 60-min interview between 2 and 16 weeks after the third implant administration; (iii) be able and willing to understand and follow study instructions and complete all required visits and procedures; and (iv) be fluent in English. Those with a history of alcohol/drug abuse in the past 12 months, or a medical condition that would prevent participation in a 60-min interview, were excluded.

Concept elicitation interviews with study participants were conducted face-to-face or by telephone by trained personnel, using a semi-structured interview guide to probe and explore concepts associated with satisfaction with treatment experience in glaucoma and ocular hypertension through a series of open-ended questions. This approach was intended to reduce possible bias in soliciting responses, and to encourage participants to describe their experiences in their own words. Targeted probing questions were asked to elicit additional information if patients did not provide spontaneous responses. Each interview was audio recorded and lasted approximately 60 min.

Audio recordings were transcribed verbatim and anonymized before analysis. Interview analysis followed a grounded theory approach involving an iterative process of constant comparison of transcripts to one another and to existing codes and categories for the purpose of identifying patterns and relationships in the data [19, 20]. An initial code list compiled from the literature review findings, preliminary conceptual framework, interview guide, and research objectives was used to catalogue concepts reported spontaneously or following probing by the interviewer. This code list was updated as necessary to reflect the actual terms participants used to describe concepts, and to incorporate newly emerging data. Codes were applied to specific text within each transcript and queried for frequency across transcripts by constant comparative method using ATLAS.ti version 7.5.18 qualitative data analysis software (Atlas.ti GmbH, Berlin).

The adequacy of the sample size was confirmed through concept saturation, which refers to the point at which further data collection ceases to generate any new or distinctive categories, high-level concepts, or substantive codes [19]. Concept saturation was assessed by documenting concept emergence across sets of successive interviews [21].

Following analysis of the concept elicitation interview, a series of three concept generation meetings was held to identify items for inclusion in the draft PRO instrument, based on prespecified criteria and FDA guidance recommendations [17]. Item selection was guided by the frequency with which a concept was mentioned, the applicability of the concept to all potential respondents, and clear attribution to treatment. The preliminary structure and format of the PRO instrument, order of items, response options, and recall period were determined, and a draft version of the ASTEQ instrument was developed.

Cognitive debriefing interviews were conducted with a different sample of ARTEMIS 1 and 2 study participants; enrollment criteria for the cognitive debriefing interview were identical to those for the concept elicitation interview (as outlined in the Patient Recruitment section). The purpose of the cognitive debriefing interview was to assess how well study participants understood the instructions, items, and response options in the draft ASTEQ instrument; to determine whether its format and wording were appropriate; and to ensure that the instrument captured all concepts of importance and relevance to the study population. Given this objective, it was important that the cognitive debriefing interview should be conducted with a different sample of study participants to that used for the concept elicitation interview. To ensure a standardized interview technique, a Cognitive Interview Guide was developed, which included questions relating to content validity (how well the instrument captures participants’ overall experiences), ease of completion, comprehensiveness and appropriateness of the format, response scales, and recall period, as well as specific concept probes to determine whether participants consider conceptually similar items to be the same or different. To reduce the possibility of introducing interviewer bias, participants were encouraged to verbalize their thoughts while completing the draft PRO instrument and to identify words, terms, or concepts that they did not understand. Where necessary, more specific verbal probing was used to ensure the objectives of the interview were accomplished. As with the concept elicitation interview, the cognitive debriefing interview lasted approximately 60 min, and was audio recorded, transcribed verbatim, and anonymized before analysis.

Analysis of cognitive debriefing interview data was likewise founded on a grounded theory approach [19, 20]. An initial coding scheme was developed from the Cognitive Interview Guide and research objectives, and updated as new themes emerged from the interviews. Analysis of coding data patterns and frequencies was conducted using ATLAS.ti version 7.5.18 software (Atlas.ti GmbH, Berlin).

On completion of the cognitive debriefing interview, the draft PRO instrument was modified on the basis of participants’ feedback and all revisions were documented. In line with FDA guidance on PRO development [17], the draft PRO instrument was evaluated item by item for appropriate readability, as determined using the new Dale–Chall readability formula (developed specifically for evaluating health education materials) [22] and the Flesch–Kincaid readability formula [23]. If it was judged that an item would present difficulty in comprehension to a sixth-grader (an 11- to 12-year-old), the wording was revised as appropriate for the intended patient population.

As recommended by FDA guidance on PRO development [18], the preliminary conceptual framework was updated to illustrate how patients’ satisfaction with the intraocular implant treatment experience, as described during concept elicitation interviews and confirmed during cognitive interviews, mapped to the contents of the ASTEQ instrument.

Concept elicitation interviews were conducted among 19 study participants, all of whom had received treatment with one of the two dose strengths of the sustained-release bimatoprost intraocular implant during their participation in the ARTEMIS trials. Concept saturation findings indicated that this sample size was adequate. Interviewees ranged in age from 29 to 83 years, included similar numbers of males and females, were predominantly White (68.4%) and non-Hispanic/Latino (89.5%), and varied widely in educational level; most had used topical IOP-lowering medication prior to their enrollment in the study (84.2%) (Table 2).

When asked to describe which factors they considered when rating their satisfaction with implant treatment experience, participants most frequently mentioned “physical comfort during application of the implant” (n = 15, 78.9%), “feeling anxious about the procedure” (n = 15, 78.9%), “frequency of implant administration” (n = 13, 68.4%), “possible side effects of the implant” (n = 13, 68.4%), “convenience and accessibility of the implant” (n = 12, 63.2%), “relationship with the clinician” (n = 12, 63.2%), “degree to which the implant experience fits with my lifestyle” (n = 11, 57.9%), and “physical comfort during preparation for the implant procedure” (n = 10, 52.6%). Less frequently mentioned factors included: “ease of compliance,” “feeling worried or scared about procedural mistakes or side effects,” “impact on activities of daily living,” “impact on work,” “efficacy,” “side effects related to antiseptic drops,” “intention to use or continue,” and “social impact,” (Fig. 1).

With regard to physical comfort during implant administration, most participants reported both positive (no sensation) and negative (pain, pressure, or stinging sensation) experiences with needle insertion. Participants’ anxiety about the procedure stemmed from concerns about their lack of understanding of the procedure and/or the prospect of receiving an injection in the eye. Most participants who mentioned anxiety about the procedure reported that they experienced less anxiety during the second and third implant administrations. Comments on the frequency of implant administration were generally positive, focusing on the advantages of treatment every 4 months rather than twice daily, although one participant stated that they would not want to repeat the implant procedure every 4 months. Participants reported fewer positive than negative experiences with side effects (the latter included instances of ocular soreness [n = 1], pain [n = 1], increased sensitivity [n = 2], puffiness [n = 1], dryness [n = 4], and redness [n = 6] that participants attributed to the implant experience). Most participants commented positively on implant convenience and accessibility, noting that the procedure was quick, completed within a reasonable timeframe, and had minimal impact on their day-to-day routine; a minority stated that appointments occasionally involved long wait times. Physical comfort during preparation for the implant procedure was generally described in negative terms, with participants specifically citing exposure to bright lights, use of eyelid retractors, and application of topical povidone iodine antiseptic as causes of discomfort; in contrast, two participants found the preparation procedure to involve no discomfort. Participants frequently reported a positive relationship with the clinician when describing their satisfaction with the implant administration experience, noting that medical staff were very good at alleviating procedure-related anxiety. While relationship with the clinician was not recommended as an item for inclusion in the final PRO instrument since it is not a concept that can be modified by treatment, it is nevertheless an important aspect of the implant treatment experience.

When asked to provide overall satisfaction ratings with each of the three implant administrations, a larger proportion of study participants reported being “mostly” or “very” satisfied with the third implant (n = 5, 100%) than with the first (n = 6, 66.6%) or second (n = 3, 42.9%) implants. Table 3 summarizes representative quotes volunteered by study participants when questioned on details of their satisfaction with the implant experience during the concept elicitation interviews.

A draft version of the ASTEQ Implant Experience instrument was developed from qualitative analysis of the concept elicitation interview data. This version included 11 items: (1) satisfaction with the overall experience of receiving the implant, (2) satisfaction with the frequency of implant administration, (3) satisfaction with the convenience of receiving the implant, (4) satisfaction with how implant administration every 4 months fits with my routine or schedule, (5) satisfaction with the ease of following the implant treatment routine (e.g., remembering to schedule and attend the implant appointment), (6) bother about immediate side effects of the implant (i.e., side effects attributable to the implant procedure, occurring within the first few hours of administration), (7) bother about long-term side effects of the implant (i.e., side effects attributable to the sustained-release drug), (8) physical comfort during preparation for the implant procedure, (9) physical comfort during administration of the implant, (10) anxiety about receiving the implant, and (11) feeling scared or worried about potential risks or side effects of the implant. Two additional items, item 10a (did you feel anxious about the implant?) and item 11a (did you feel scared or worried about any potential risks or side effects of the implant?), were included in the instrument to explore alternative item wording and participant preference between items 10 and 10a, and 11 and 11a.

As the intention is to allow the ASTEQ Implant Experience instrument to be used at different timepoints during the course of a clinical study (i.e., after any implant administration), a non-specific recall period of “from the time you started this study until now” was chosen for the draft instrument. This provides flexibility as to when the instrument can be administered, and also enables participants to consider their cumulative experience with the implant.

Cognitive debriefing interviews to assess the draft ASTEQ Implant Experience instrument were performed among a new set of 20 study participants who had previously been treated with sustained-release bimatoprost implants in the ARTEMIS trials. This sample ranged in age from 37 to 81 (mean 59.5) years, was predominantly White (65.0%) and non-Hispanic/Latino (65.0%) and comprised similar proportions of college graduates and non-graduates; most subjects had used topical IOP-lowering medication before enrolling in the study (75.0%) (Table 2).

Based on the cognitive debriefing interview findings, modifications were made to several items in the draft version of the ASTEQ Implant Experience instrument. Item 3 (“satisfaction with the convenience of receiving the implant”) and item 5 (“satisfaction with the ease of following the implant treatment routine”) were removed on grounds of conceptual redundancy with item 4 (“implant fit with routine or schedule”), which was felt to be more specific and understandable. Response options for item 6 (“bother about short-term side effects of the implant”) and item 7 (“bother about long-term side effects of the implant”) were modified to capture both the presence/absence of side effects and the degree of bother caused by side effects. The wording of item 8 (“physical comfort during preparation for the implant procedure”), item 9 (“physical comfort during administration of the implant”), item 10 (“anxiety about receiving the implant”) and item 11 (“scared or worried about potential risks or side effects of the implant”) was revised to improve clarity. Items 10a and 11a (alternative wordings for items 10 and 11) were removed, as items 10 and 11 were deemed to be conceptually less complex and more readily understood. Participants did not report that any additional concepts needed to be added to the draft ASTEQ instrument.

Following implementation of recommendations arising from the cognitive debriefing interview, the revised ASTEQ instrument consisted of nine items assessing the following concepts: satisfaction with the overall experience of receiving the implant; satisfaction with how often the implant needs to be administered; satisfaction with how implant administration fits with one’s routine or schedule; occurrence of any immediate side effects following implant administration and the degree of bother they cause; occurrence of any long-term side effects following implant administration and the degree of bother they cause; degree of worry about receiving the implant; degree of worry about any potential risks or side effects of the implant; level of physical discomfort experienced during patient preparation for implant administration; and level of physical discomfort experienced during actual implant administration (Table 4). The non-specific recall period of “from the time you started this study until now” chosen for the draft PRO instrument was retained for the revised version. Readability tests on the draft ASTEQ instrument identified several difficult words (predominantly words of three or more syllables). However, since participant feedback during cognitive interviews revealed no issues in interpretation of the instrument items, revisions to the wording were not recommended.

The conceptual framework was updated to reflect the contents of the revised ASTEQ instrument and modified to reflect the hypothesized relationships among the ASTEQ items. Proposed scoring of the ASTEQ instrument envisages its division into two domains: pretreatment concerns and treatment experience satisfaction, to be further evaluated in future psychometric validation (Fig. 2).