Developing a patient-centered outcome measure for complementary and alternative medicine therapies I: defining content and format

In Phase Ia of the project, we utilized patient transcripts (n = 177) from six peer-reviewed externally funded studies of the outcomes of CAM therapies conducted between 2001 and 2004 [13, 14, 18, 34‐36]. While most of the interview data from these projects were not collected for the purpose of identifying shifts in well-being following CAM treatment, the transcripts provided a rich source of data on patient-reported experiences with CAM therapies including subjective accounts of treatment effect. The six CAM studies involved a broad range of study designs, clinical sites, CAM interventions, and disease states, summarized in Table 1. Quality criteria (typically identified as reliability) in qualitative research relates to efforts by researchers to assure faithful and credible representation of reality as observed or studied [37‐39]. All six original studies used several acceptable methods to increase the credibility, including respondent validation, audibility of data collection and analysis procedures, negative and deviant case analysis, triangulation across multiple researchers for each study, and close adherence to the emic (the subjects' own language and representations) perspective in the creation and reporting of outcomes.

A coding team at each of the six institutions where the data were originally collected completed transcript analysis. The lead analysis team located at the University of Arizona conducted weekly teleconferences with coders from all sites. A password-secured server was set up for the exchange of files, with only short excerpts from interviews shared across sites to protect participant anonymity. The University of Arizona Institutional Review Board (IRB) and other relevant institutional IRBs approved all procedures.

This resulted in the selection of 150 interview transcripts from 119 individuals in which participants spontaneously discussed shifts in well-being associated with CAM treatment. In the next step, a codebook was developed to facilitate the identification of dimensions of change. The coding utilized both deductively derived codes identified by the research team and informed by their understanding of previous studies, and inductively derived codes that emerged from the data and reflected the language of the participants. Initial codes were established in consultation with the entire research team by identifying the larger themes found in transcripts across the different studies. The coding team then used these initial codes to tag transcript segments. Coding was aided by the use of ATLAS.ti version 5.2 http://​www.​atlasti.​com qualitative data analysis software.

We generated a relatively large and rich pool of candidate items from this analysis, including items relating to states of "unwellness," the experiences of transitional states and processes, and states of greater well-being. Examples of the metaphorical language from the original interview transcripts, and sample simplifications, are shown in Table 2. This list of items was then shared with CAM practitioners (n = 30) who had previously participated in research studies (see Table 3 for a description of provider demographic and practice characteristics). They were asked to review and add to the pool any additional items that patients in their practices often reported, including descriptors of both negative and positive states. Items added by practitioners at this stage tended to focus on physical functioning, and included sleep, physical symptoms, slow/fast recovery, and "bouncing back."

From these data sources, we created a filtered list of relatively broad terms that captured the meanings of a range of words and phrases. At a two-day all investigator meeting, these items were further categorized into five areas of health and well-being (physical, emotional/affective, cognitive, social, and spiritual) to identify their distribution across these frequently used psychometric domains. In the process of categorization, we discovered a sixth domain that we termed "whole person" for items that seemed to bridge several domains. The resulting item pool and assigned categories generated through Phase Ia are shown in Table 4 in the left hand column (the numerical rankings in this table are described below in section 1b Results: Quantitative Analysis).

In order to test the fit of the list of shortened positive and negative phrases generated in Phase Ia to informant experiences of personal change and to capture other possible descriptors of positive and negative states, we created an innovative interview protocol to be used with a new pool of informants. Our goals with this phase were to identify a much shorter but widely endorsed set of markers of subjective states, and to obtain direct feedback on the wording of individual items (Table 4). An interview protocol was developed specifically to encourage informants to reflect on their states prior to and following CAM therapies, without requiring attribution of any changes to the therapies, and to select words and phrases which best captured their ranges of personal experiences.

We termed our interview strategy an "evocative card sort interview" in that it attempted to evoke both denotative and connotative meanings associated with words. Denotative language employs words or phrases to refer or point to a specific state or quality, such as a definitive symptom of an illness like fever or fatigue. Connotative language indexes a cluster of loosely associated images, schema and feelings about an experience that is particularly salient to an individual. For example, saying that one's energy has changed following a CAM treatment would be an example of connotative speech indexing a set of associations and feeling states. To the extent possible, we wanted to identify terms that captured widely endorsed evocative states, which were not highly idiosyncratic or culturally specific. We also wanted to identify descriptors that were scalable; that is, easy for many people to identify with as registers of change. We chose a "card sort" approach to interviewing subjects about their outcomes [40]. This was predicated on the recognition that for some individuals, their subjective shift may not have previously been articulated; that is, it may have been sensed internally but remained pre-verbal or pre-cognitive. Therefore, card prompts were used to trigger tacit knowledge and embodied memories as well as to provide frames of reference for experienced but thus far unspoken shifts in well-being.

The informants for this phase were recruited using a purposive sample approach at three of the sites that had been involved in Phase Ia of the project (Tucson, AZ, Portland, OR, and Vancouver, BC). Participants were recruited from two wellness centers frequented by cancer and HIV patients, from clinics, and from ads placed in local health magazines. We also asked CAM practitioners to refer patients to participate in interviews if they had reported significant shifts in well-being associated with CAM therapies (as defined above), as it would not benefit this part of the process to interview individuals who had not changed. We were careful to recruit a diverse set of individuals across multiple CAM systems and health conditions, as we were particularly interested in testing the relevance of the items for use with patients from a wide range of CAM therapies. After obtaining consent from individuals to participate in the interview, a letter was sent out prior to the interview asking the person to select a shift in well-being they had experienced following a CAM therapy and which they would be willing to share with the interviewer. Characteristics of the 34 participants are described in Table 5.

Because this interview protocol was innovative, interviewers required training in the card sort methodology. Each interviewer conducted four pilot interviews with people known to the research team using the evocative card sort method, thus providing them an opportunity to learn to work with the method and sensitizing them to how individuals might respond to the interview process. Interviewers were trained to allow informants sufficient time to "try on" the terms/phrases on the cards to determine if they fit their experiences. Importantly, interviewers were encouraged to be empathetic witnesses of the process.

At the onset of the interview, the interviewer explained that she was particularly interested in two stages that people encountered during the healing process: first, being in a tough spot (physically, emotionally, psychosocially, or spiritually), and second, a subsequent better place. Informants confirmed that they had this type of experience and were asked to share a specific story, both verbally and briefly in writing. If they subsequently shifted to another story while going through the cards, the interviewer would gently bring them back to the index event noted on the card as a form of an anchor.

The evocative card sort interview began by asking the informant to first reflect on the tough spot they had experienced. The interviewer presented the informant with 54 cards that contained short words/phrases derived from Phase Ia (shown in Table 4). Examples include "I was tired," "I felt betrayed by my body," "I was hopeless," "I felt out of control," "I felt vulnerable," and "I couldn't think clearly." The informant was instructed to go through the 54 cards and divide these largely negative descriptor cards into 3 stacks: "Applies to me (i.e., fits my experience)," "Not quite right," and "Does not apply." After the informant sorted the 54 cards, the interviewer reviewed the "not quite right" stack and asked the informant to suggest a modification of the item, if possible. Once modified, the informant was asked whether the item was then applicable to his/her experience and to place it in the appropriate stack (applies to me/does not apply). Next, the "applies to me" cards were sorted into domains by the interviewer, as a next step in further winnowing down the card choice. The interviewer picked up the selected cards in a particular domain and said: "These cards appear similar--which one(s) best describe your experience?" (e.g., cognitive domain: "I was unable to focus," or "I couldn't think clearly"). Some informants were able to identify a single card that best captured their experience, while others were unable to do so and viewed several cards as equally significant. Informants were also invited to alter the words on the cards to better fit their experience or to offer new words or phrases on blank cards. Few interviewees volunteered additional descriptors, suggesting that the list generated in Phase Ia provided reasonable coverage of the range of experiences.

Following the interview, the interviewer recorded the tally of all the cards endorsed, rejected, edited, and left as "not quite right" by the informant. These data were then computer-entered using a data entry program designed for this purpose. Once all interviews were completed, a tally was created from all participant responses summing how many individuals placed each item in the "applies," "best applies," "not quite right," or "does not apply" categories. The "applies" and "best applies" categories were subsequently combined to obtain a more stable metric. Tally results (see Table 4) were closely examined to identify those items that were consistently endorsed in positive and negative frames and thus were candidates for a directional scale item. A listing was also created that showed every item edit provided by participants. Thus the card sort process allowed us to quantify the level of endorsement for particular items among informants as well as to record comments and item edits, a process that guided the development of the final questionnaire.

Data analysis revealed that the majority of informants in this phase experienced the evocative card sort interview process as useful in understanding their own experience. Comments, which emerged organically at the close of the interview, reflected a range of insights including: "It got me thinking. I never thought about my experience like this before," "Now I understand how I got through this," and "I didn't know how far I had come." Querying how this process had occurred, several informants explained that sorting through the cards allowed them to verbalize feelings in a way that they had not done before, and that this afforded them a sense of clarity. Comments such as these were confirmation that the evocative card sort interview method had worked as intended and fostered reflexivity as well as enabled informants to put into words, states that had not previously been expressed.

Table 4 shows the levels of endorsement for the items, divided by domain for ease of evaluation. The items were assigned domains subsequent to their identification through the qualitative process in Phase Ia; the uneven distribution is an outcome of the process and was not planned. Further, the high number of "whole person" statements reflects the nature of the qualitative data. Overall, fewer of the negative items were rated as "applies/best applies" than positive items, with negative items receiving on average 20 endorsements, and positive items 27, in spite of almost equal numbers of cards (54 and 53 respectively). For both positive and negative cards, about 13% were initially put into the "not quite right" stack, and of these, about 40% were modified to "applies." This occurred for two reasons. Some participants never were able to modify the card appropriately and eventually put the card in the "does not apply" pile. Others had found more appropriate cards later in the pile and no longer wanted to work with the "not quite right" item. Notably, the majority of completed edits for both positive and negative items were directed toward making them less absolute.

In parallel with the identification of item content, the research team considered the types of response options that that would be most appropriate for the assessment of patient-centered outcomes in the context of a clinical trial. We reviewed the ways in which different objective and subjective phenomena or attributes (e.g., frequency, duration, severity, satisfaction, agreement, or change) are commonly quantified through the use of response sets/scales [31]. However, while reading the study transcripts from which the item content was being derived, it became clear that the traditional response sets/scales applied in a standard clinical trial model relying on baseline (pre-test) and subsequent serial assessments (post-tests) would likely be problematic. Early in the phase Ia data analysis, the issue of "surprise" began to appear in the transcripts. This surprise was in relation to the nature of the experiences in relation to CAM therapies ("I never knew that I could feel like this") or in the extent of the change ("I never imagined that I could feel so much joy"). In this evaluative context, where change from baseline is the efficacy endpoint, frame or response shift can be a significant concern and a threat to internal validity. As defined by Sprangers and Schwartz [41], response shift is a change in the meaning of one's self-evaluation of the construct of interest (e.g., quality of life) as a result of: (1) a recalibration of the respondent's standards of measurement; (2) a change in the respondent's values; or (3) re-definition or re-conceptualization of the construct. To avoid the measurement error associated with response shift, we chose the evaluative methodology called the retrospective pretest, which has been suggested to be valid when the subjective experience of change is most salient [42].

At the end of the card sorts, we showed informants different possible question formats to identify those that resonated with the interviewees' issues. Response sets such as "never-always" or "strongly agree-strongly disagree" did not address the interviewees' needs for lessened intensity. As noted earlier, many informants commented during the evocative card sort that the descriptors were too intense, and they requested modifications that would soften the intensity of the meaning. To address these comments, and to meet our goal of providing positive as well as negative directions on the final instrument, the study team developed, and then piloted with participants, the approach of creating pairs of words that anchored two ends of the same continuum (e.g., "hopeless-hopeful"). The intent was to allow the respondents to choose where they fell on that continuum "before" the treatment or intervention and "now." Participants easily grasped how to work with these word pairs and indicated that this format addressed the issue of gradation of intensity. Thus we moved from the lists of descriptors evaluated in the evocative card sort interviews to word pairs. We were also sensitive to the time needed for participants to consider and respond to these items, and in order to minimize participant burden we chose to work toward a target length of 15 to 20 item pairs.

There were several steps in the process of draft instrument design. First, investigators utilized the ranked tallies (Table 4) to create word pairs that identified continua, attempting to capture a set of pairs that represented the most highly endorsed positive and negative items, to minimize redundancy when several items had similar meanings, and to cover the domains indicated in Table 4. Second, new tallies were run with participants subdivided into important categories, including race/ethnicity, type of CAM therapy (practitioner-based or self-practice), and gender. The aim was to examine whether our shortened list of pairs lacked any specific pairs that were preferentially endorsed by a single group as a crosscheck for important items that might have been missed in the tally approach. The investigators reviewed the new tallies and the draft item list to assure that the list did not omit any items that were particularly important to a group of respondents. Pairs were added as necessary to meet this criterion. Third, practitioners reviewed item pairs to assure that dimensions that were considered highly important within particular CAM therapies were not omitted. Some items in the physical domain were added back at this stage.

At the same time as instruments were being developed, we had opportunities to pilot draft instruments in two clinical trials and chose to do so with instruments developed up to that point. This process of finalizing instruments for RCTs provided additional feedback from the investigators and staff of these clinical trials, and from some participants in those studies who were asked for feedback. Further input was sought from colleagues interested in potentially using the instrument. By the end of the process, 18 pairs were chosen for further refinement via cognitive interview [43] testing. This draft instrument is shown in Table 6 by domains. The level of endorsement (rank) of the comparable card sort descriptors is also shown. The draft instrument was piloted in several test environments to establish how the measurement axis should be displayed, and what instructions were adequate. The environments included a graduate-level medical anthropology class, a class for students learning spiritual healing, a small clinical trial of tai chi for cancer patients, and parents participating in a healing touch camp for families caring for children with severe disabilities.

The final draft instrument layout is shown in Table 7. The 100 mm blank line, without numbers or internal reference points, was the final consensus layout. Respondents indicate "before" and "now" on the same line; for data entry, the positions of the "before" and "after" points are measured as the distance in millimeters from the left edge of the line. We also successfully implemented this as a web-based data entry system in which the participants move a slider along the line to place the indicator.

In Phase II, reported in separate papers, the draft instrument was further evaluated following recommendations from the FDA guidance on developing PRO measures [31]. This included cognitive interviews [33, 43], and quantitative evaluation in five different settings to check construct validity, the psychometric properties of the items and overall instrument, face validity in relation to different types of CAM therapies and ease of use by different populations [manuscript under development].