Introduction

International health agencies, federal funders, and patient advocacy groups have long advocated the involvement of patients in clinical research. For example, the World Health Organization (WHO) established a Patients for Patients Safety program in 2005, with the vision “to engage, empower, encourage and facilitate patients and families to build and/or participate in (a) global network advocating for, and partnering with health professionals” focusing around safer and more people-centered health care services. [1] Although the WHO program was developed specifically to improve patient safety worldwide, the program also highlighted the importance of the patient perspective in health care. Over 25 years earlier, Read et al.[2]had argued that”…meaningful measures of health-related quality of life must be used to evaluate health care interventions.” [2] Whether originating in safety, advocacy, or health-related quality of life, there is agreement about the importance of including “measures of overall health” in the evaluation of health care interventions, because “overall health is the object of health care.”[2] [3]

Departing slightly from these earlier perspectives on patient engagement in clinical research, patient-reported outcomes (PROs) were defined by the US Food and Drug Administration (FDA) to be “any report of the status of a patient’s health condition that comes directly from the patient, without interpretation of the patient’s response by a clinician or anyone else.” [4] This definition (the final version of PRO guidance first drafted for comment in 2006 [4], see p. 1) signals the interest of FDA regulators in a focus on the patient’s experience with, or perspective on, interventions within the agency’s jurisdiction. The suggestion that regulation and labeling should (also) include patient perspectives only solidified the importance of the PRO in the biomedical, and particularly the clinical, research literature base. In 2010, the Patient-Centered Outcomes Research Institution (PCORI) was established by the US Congress, specifically to promote (and fund) research that is “…designed to improve patient care and outcomes through patient-centered comparative clinical effectiveness research.” [5]

Since the milestones of the FDA’s definition of the PRO⁴ and the establishment of PCORI in 2010, [5] there has been a notable increase in peer-reviewed publications relating to how to engage patients and other stakeholders in clinical research[6] and about other methodological considerations relating to comparative-effectiveness research. [7] Several models of how to engage patients throughout the research process have been published recently. [8–13] Among these recent publications is a conceptual framework intended “…to help guide the field (of clinical research) toward promising practices in research engagement”. [14] However, in addition to these manuscripts outlining “how to” obtain patient and stakeholder perspectives on research, there have also been others outlining the challenges that specific disciplines may face in engaging patients in clinical research within these fields (e.g., radiology,[15, 16] emergency medicine,[17] and neurosurgery[18]).

As appreciation for the importance of the patient perspective in clinical and health research has grown in the United States, the difference between a patient-reported outcome and one that is also patient-centered is often blurred. For example, Chen et al.[19] define “patient-centered outcomes research” as integrating “data from patients and other stakeholders to improve health care delivery and outcomes and guide health care decisions”[19]; here “patient-reported” is used as a synonym for “patient-centered,” although the FDA’s definition of a PRO can be interpreted as more specific to the patient him-/herself. While Chen et al.[19] incorporate the patient and other stakeholders explicitly in their definition of patient-centeredness, they do not provide details on how to achieve authentically patient-centered (PC) outcomes. Similarly, Frank et al. [20] state that, “(t)aking a patient-centered approach … is intended to produce research that looks beyond the questions and measures found ‘under the lamp post,’ within the researcher’s field of view.” [20] This perspective reinforces the idea that the “researcher’s field of view” does not necessarily encompass the patient’s experiences, but also does not provide details on how patient engagement or patient-centered outcomes can be developed–achieving research that “looks beyond …the researcher’s field of view” and also substantively contributes to the knowledge and practice of the field.

To address this gap, we argue that “patient-reported” and “patient-centered” are not synonymous, echoing the position originally advanced by Basch et al. [21] and Lasch et al.[22] A patient-reported outcome can be a patient’s report of any outcome, sign, or symptom that is perceived by the patient- whether it originates with him/her or with the individuals (clinicians, researchers, or clinician-researchers) who developed the outcome. By contrast, a patient-centered patient-reported outcome is the patient’s report of an outcome that represents the patient’s experience specifically–originating with the patient and not with clinicians, clinical researchers, or other stakeholders. With both PRO and patient-centered PROs (PC-PROs), the patient is doing the reporting; the concepts of PRO and PC-PRO differ in terms of the origins of what the patient is reporting about. Hence, PC-PRO is a subtype of PRO. Specifically, a patient-centered PRO (PC-PRO) is a type of PRO that explicitly maintains the primacy of the patient’s, vs. the clinician’s or researcher’s, perspective on the patient’s health status, condition, or experience. This distinction is consistent with Lasch et al.[22] and Basch et al.[21].

Asking patients to report on clinically-relevant or researcher-identified signs and symptoms is generally consistent with the FDA definition of a PRO, but it tends to maintain the perspective of what is clinically relevant because the clinical perspective first identifies what should be rated, and then patients rate their experience of that (and only that) set of signs and symptoms, typically using the rating scale specified by clinicians. This dominant paradigm, shown in Fig 1, is not consistent with the high-fidelity qualitative approaches to PRO development outlined in Lasch et al.[22], but is common for developing a patient-reported outcome.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Typical paradigm for developing patient-reported outcomes.

https://doi.org/10.1371/journal.pone.0171114.g001

The usual starting point for the development of new patient-reported outcomes is the clinical perspective of the target medical issue or context. Expert clinical opinion is traditionally obtained by interview, focus group, or as a result of an expert panel convened to articulate diagnostic or other criteria, which might be published as position papers or consensus practice guidelines. Although Chen et al.[19] correctly note that the development of a PRO begins with “direct patient input,” preparation for focus groups (or other methods to elicit information from patients) is not discussed. While Lasch et al.[22] note that qualitative approaches to the development of a new PRO include the collection of data via focus groups or interviews (among other methods), the interview structure/questions originate with the investigators (clinicians or researchers developing the instrument and/or planning the clinical study); patient input therefore generally takes place at the second step in Fig 1 (“validation”). That is, the essential preparation for focus groups or interviews with patients often represents—and maintains—the clinical or investigator perspective. Patient report and patient centeredness are conflated in this approach; for example, Chen et al.[19] describe patient validation as necessary “…to ensure that the instrument is clearly understood and that scores reflect the intended patient voice”.[19]

Fig 1 therefore represents a logical and systematic process of developing patient-reported outcomes that begins with an initial list of signs or symptoms—which is crucial in order to structure the interaction (e.g., interviews and focus groups) with patients or other stakeholders; these typically arise from clinicians or PRO developers (“initial elicitation,” Fig 1). This initial list is then sent/presented to a representative sample of potential patients for their “validation.”[19, 23] In this paradigm, the validation typically derives from/consists of ensuring that the patients can report on whatever the clinicians have specified; in some cases, these patients give feedback as to the clarity of either the signs or symptoms or the ratings that are expected.[19] Once patient input is reconciled with (or integrated into) the original clinical perspective, with or without further iteration with clinicians and/or patients, then the patient-reported outcome is ready for pilot testing or further validation.

Recent publications in patient reported outcomes and health-related quality of life stress the importance of partnership between/engagement with researchers and stakeholders,[7, 14, 20, 24–26] and Fig 1 is consistent with this emphasis because it shows that nothing results from the development process that did not go through the patients first. The challenge this model presents is that the origins of what patients are reporting on is clinician-, and not patient-, centered. As such, PROs developed following the paradigm in Fig 1 may present difficulties in the identification of “response” or “responders,” because the treatment “benefit” constituting a “response” would be defined by the clinical, and not the individual patient’s, perspective.[27]

Like Lasch et al.[22] and Basch et al.,[21] we define a patient-centered outcome as one type of PRO that starts with a focus on what the patients experience or what they prioritize about their experience, and not the patients’ report on what clinicians or investigators prioritize about the patients’ experiences. Unlike Lasch et al.[22], however, the approach shown in Fig 2 is iterative, with independent streams of input from patients and clinicians being integrated after inquiries (e.g. interviews and focus groups) have generated each source of input.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Framework for developing a patient-centered, Patient Reported Outcome (PRO).

https://doi.org/10.1371/journal.pone.0171114.g002

The model for PC-PRO definition shown in Fig 2 is achieved by executing the following steps.

1. Assemble an initial list of signs and symptoms based on literature review and independent discussions/experience with patients, clinicians, and researchers. Using this artifact as the point of departure, structure scripts that can support consistent focus groups and interviews around the phenomenon of interest.

2. Gather the independent perspectives on, and experiences of, the phenomenon of interest. Instruct all participants, clinicians and patients alike, that the output (the PRO) will be a list of signs and symptoms that reflect the patient’s perspective, to align clinicians and patients to that perspective. Ongoing formal qualitative analysis of interviews and focus groups separately from each perspective ensures the timely identification of saturation (further discussion or interviews cease to add new details or items to the existing list of signs and symptoms).

2a.1. Employ Consumer Experts (CEs, representatives of the target patient population or experienced caregivers) in the conduct (facilitation, transcription) of patient focus groups. The initial symptom list is a starting point and the CEs are the first patient perspectives elicited on this list to begin its evolution. The CEs will be conducting focus groups and for this, an overall script is required. Once the CEs finish their initial modifications to the symptom list and the focus group structure/scripts, they are able to begin focus groups.

2a. 2. Conduct a series of “pilot” focus groups to understand the functioning of this initial script and to specifically obtain input on how it should evolve to optimize the next series of focus group conversations. In all focus groups, the script is shared to provide cues and context for all participants as they respond to the script and to one another’s experiences. All focus groups are recorded and transcribed. CEs participate in ongoing discussions of focus group results relating to the symptom list and the script/presentation, making any modifications suggested by participant responses (e.g., to clarify prompts and the descriptions of signs or symptoms).

2b. Engage clinical experts in focused discussions and/or interviews to elicit their experiences during care of patients with the phenomenon of interest and their perspectives on the patients’ experiences. Detailed meeting notes, circulated to the participating clinicians between meetings for comment, serve as the content for developing and evolving the clinician-prioritized list of signs and symptoms. Alternatively, clinical experts can be engaged to conduct focus groups in a fully analogous method to that for the patients (2a.1-2a.2).

3. Iteration: reconciling and synthesizing patient and clinician perspectives.

Once both patient and clinician perspectives reach saturation, code the final results with at least two independent coders and a formal qualitative method. Iterative integration of the two perspectives by the research team begins with the identification and removal of duplicate items (i.e., that were identified by both perspectives; and that emerge within one group as two versions of the same sign or symptom). During this process, ambiguities that the investigators perceive in the results from either group are also uncovered, and all plausible interpretations of items are generated—for reconciliation/revision by the clinicians and patients. This is an iterative process whereby any changes made to the “unified results” are then presented to the clinicians for evaluation of whether the changes resulted in different meanings or a loss of specific items or nuance/meaning. The integration process focuses on identifying and eliminating duplications, and identifying and clarifying any vagueness so as to promote the most interpretable responses to any item in the resulting instrument. Additionally, when materials are returned by investigators to either of the other two groups, they are accompanied by an inquiry along the lines of, “is there anything missing from this list?”, to ensure/confirm that the saturation that initiated this integration step was correctly identified. The iterative integration stage ensures that the perspectives of both groups are represented in the integrated results. When the unified list stops changing at the integration step, the reconciliation and synthesis are complete and the result is the beta version of the new instrument.

4. Test the beta version of the instrument with a large, representative national sample of target population patients.

Several differences between the paradigms in Figs 1 and 2 are worth noting. First, the initial elicitation, the prompt to gather perspectives on the clinical topic of interest is shown explicitly in Fig 2 –it is implicit in Fig 1 (also, see Chen et al.[19]). That is, Fig 2 shows that inquiry in the area of interest must be framed, and prompts developed (scripted), to solicit the perspectives of both patients and clinicians, independent of one another. We also separate out “the investigators” from the clinicians whose clinical experiences and expert input we seek in parallel with the perspectives and lived experience of the patients. Instructions to both types of contributor should be explicit that their focus should not necessarily include what the “other group” might think about what they/their group generated. This independence approach ensures that differences and nuances in viewpoints and opinions between patient experts and clinical experts will be accounted for in the analysis of their respective focus groups and interviews. All participants should be informed of the simultaneous and independent processes by which “their group” will be generating its contributions to the structure of future focus groups in the development process (see Methods).

The remainder of this manuscript describes an example of PC-PRO development that uses the method shown in Fig 2. To do so, we explored the hypothesis that starting with, and prioritizing, the clinician/researcher perspective (as shown in Fig 1) leads to a clinician/researcher-centered patient-reported outcome, whereas the alternative approach (Fig 2) leads to a qualitatively different (patient-centered) patient-reported outcome. Although previous work has articulated the difference between patient-centered and patient-reported outcomes, this is the first study to our knowledge that tests the hypothesis that the models in Figs 1 and 2 will lead to qualitatively different instruments.

The domain in which this research is conducted is urinary signs and symptoms for people with bladder dysfunction caused by neurologic damage (neurogenic/neuropathic bladder, NB). This qualitative study was carried out as part of a larger initiative to develop a PC-PRO instrument in the domain of urinary signs and symptoms in NB. Existing patient-reported instruments for the domain focus on non-NB populations include women with recurrent UTI, women with urinary incontinence or men with prostatic hypertrophy–so existing instruments are not applicable, not even for developing an initial list of signs and symptoms. Clinical practice guidelines are also not useful to develop the initial elicitation scripts, because “urinary tract infection” (UTI) is common for persons with NB, but it is acknowledged in existing guidelines that people with NB have a different UTI experience than do those without NB.[28–30] Therefore, there is a need for a new instrument that is specific to individuals with NB.

Methods

Approval for all parts of this study was received from the MedStar National Rehabilitation Hospital Institutional Review Board (IRB # 2013–187). The research participants in this case were the focus group participants recruited from the region and nationally, and their informed consent to participate was obtained verbally via a Join Me [31] link (audio). CEs were contracted to be trained and participate as focus group facilitators. Clinicians who participated in face to face focus group meetings, moderated by a clinician with extensive experience in structured interviewing, were informed of the purpose of their participation and that they were free to not participate if they chose. This set of consent procedures was included in the protocol approved by the IRB.

Study investigators developed the model for PC-PRO definition shown in Fig 2, and followed it by executing the following steps.

Results

There are two sets of results presented here: the first outlines our findings representing the qualitative differences in the development process arising from patient-centered vs. patient reported (clinical/research centered) perspectives on developing this instrument (first from the development of our scripts for the focus groups, then from the focus groups themselves). The second set of results outline the differences in what the final instrument would have captured if we had used Fig 1 (using the clinical guidelines exclusively) vs. Fig 2 (integrating clinical and patient perspectives) to develop the PRO.

Results: Development of the instrument: Emergence of the focus group script

The table that follows presents an early point in the emergence of the final focus group script from thematic analysis (2.B.1, 2.B.2 above) with the integration by the research team. Table 1 captures the evolution of the script that was used in patient focus groups, derived from the initial symptom list, script elements, and prompts proposed by clinical experts as distilled by the research team (first column), juxtaposed with these same elements as initially (independently) proposed by our patient experts (CEs, second column). These responses have been sorted, but have not been edited. All of the qualitative data are presented in Table 1.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Comparison of perspectives of clinical and consumer experts with the final patient focus group script elements, organized by general domains of queries.

https://doi.org/10.1371/journal.pone.0171114.t001

In Table 1, several features bear highlighting.

1. The integrated “triggers” domain comprised questions regarding pattern of symptoms and contacting your care provider in response to these patterns. Both perspectives began with a question regarding the first symptom. Patients suggest there may be a pattern experienced, whereas clinicians want to know if the patient has an internal (explicit or implicit) definition for ‘UTI.’ These questions seem to be getting at the same idea, an experience or pattern (patients) or threshold (clinicians) that suggests to the patient that they do or might have a UTI and so need to go get treatment.
2. The contributions from patients and clinicians in the integrated “symptoms” domain was perhaps the most closely aligned; both clinicians and patients suggested questions to learn about the symptoms and individual experiences with UTI. All respondents (clinicians and patients) conflated “UTI” and “urinary signs and symptoms.”
3. The integrated “bladder emptying history” domain was a focus for both clinicians and patients. The clinician’s questions used broad terms such as ‘bladder management technique’ whereas patients proposed questions of a more specific nature. An example of this is ‘Do you clean…. Do you tape the catheter…’ The perceived focus on changes in bladder management technique was removed in the final script, however, in order to maintain the respondents’ focus on what they do “now.”
4. The integrated “prescribed/history” domain tended to represent the clinicians’ perspective with its questions that focus on who treats UTIs and how frequently antibiotics were required. Patients presented questions on who treats UTIs and management plans. Both clinicians and patients focus on prescribed management plans.
5. The “selecting antibiotics” domain of the script was focused differently in the different perspectives. Specifically, patients propose questions focusing on the specific antibiotic that can or cannot be taken and when they are taken. Patients suggest the idea of taking antibiotics prophylactically, whereas clinicians ask if a supply of antibiotics is maintained by the patient. These questions diverge because the clinician’s question does not differentiate between prophylactically taking antibiotics and self-management of urinary symptoms with antibiotics.
6. The integrated “preventative” domain comprised questions in the same vein as comment 4, above (“prescribed/history”), but with a focus on strategies with non-prescribed compounds. Patients and clinicians each articulated that consulting a clinician is unnecessary to utilize these strategies. Once again, clinicians use verbiage that is quite broad, such as ‘non-prescription strategies’. Patients do this as well, but also included questions with more detail such as ‘frequency and fluid intake’.
7. The integrated “impact” domain represents the impact these experiences have on a person’s life. The patients focused on the impact of treatment on a person’s life, not just on their symptoms. Patients also suggested asking about the social anxiety/stress that may be caused by these events, which was not a consideration for the clinicians. However, queries from clinicians tended to be similar to those identified by the consumer experts.
8. The integrated “Logistics” domain reflects on the steps in obtaining care. The clinicians limited their questions to access, focusing on the impact that the patient’s insurance may have on it. Patients took a broader approach, from experiencing a symptom, to transportation, to insurance coverage. Finally, the patients ended their series of questions connecting these problems back to the potential to use ‘in-home’ treatments as a substitute.

The early clinical perspective shown in Table 1 included queries about: a) the relationship with, and confidence in, the provider; and b) how patients obtained their information/education/training with respect to UTIs and their management. CEs suggested elimination of these from the script, as they were not aligned with the purpose of the instrument in our study or for future research on treatments that might result in observable changes in the PC-PRO that this project sought to develop. The investigators agreed and left them out of successive versions of the script. Clinicians also focused on the treatable—not the most bothersome (or common)—signs and symptoms. These differences in particular underscore the importance of a phenomenologic approach (see Lasch et al.[22], Table 1)- even in an approach that blends phenomenologic and Grounded Theory methods, as we employed—to capturing the patients’ experience in the development of a patient-centered PRO. Moreover, these results from the development of the focus group script highlight how, while following the paradigms in Figs 1 and 2 would each lead to a patient-reported outcome being developed, the patient-centeredness of an instrument developed following the paradigm in Fig 2 would be greater. We used the integrated results from clinician and patient experts, refined by experience in the first three focus groups of patients and their caregivers, to complete the script that was used in the final 10 focus groups to inform the content of the urinary symptom instrument.

Results: Representative national sample responses to instrument

Based on the results of our instrument development procedure outlined above, we had a 29-item list of signs and symptoms comprising our PC-PRO, and we sought to validate it with a nationally representative sample of respondents. As noted, we had coded each item with respect to whether or not an item arose from a specifically patient-centered (PC) perspective. This coding was not included in the instrument itself but remained part of our data analysis plan.

Of its 29 items, this instrument (the Urinary Symptom Questionnaire for people with Neuropathic Bladder, USQ-NB) included 13 signs/symptoms in the guidelines for diagnosing urinary tract infection set out by the Infectious Diseases Society of America (IDSA).[33] Our focus groups augmented these with 16 others, for a total of 29 signs and symptoms. Endorsement rates of our national sample on these 29 items were assessed (see Fig 3). The items that appear in the IDSA clinical practice guidelines are color-coded to show whether they represent those signs and symptoms included in clinical practice guidelines (CPGs),[33] shown in gray; items that were not included in the guidelines are shown in black. All items on the instrument represent signs/symptoms that were identified by our patient-led focus groups; the distinction in Fig 3 is that some of these items might not have been included if we had not encouraged the focus group participants to consider additional signs and symptoms beyond the CPGs.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Endorsement rates (percent of 301 respondents) for signs and symptoms.

https://doi.org/10.1371/journal.pone.0171114.g003

Fig 3 shows that the three most commonly endorsed items on this instrument were identified for inclusion in the instrument by patients (black), and would not have been included if we had structured the instrument without explicitly incorporating the patients’ perspective. The three least-commonly endorsed items were submitted by the clinicians/derive from the clinical guidelines (gray). Seven of the 15 most frequently endorsed (42%-82% of our national sample), and six of the 14 least frequently endorsed items (10–40% endorsement), would have been missed if the patient perspective had not been explicitly and independently sought and maintained.

Table 1 suggests that if items had been added to the CPG[33] items (shown in gray in Fig 3) based solely on the input of our clinicians, we might have added items about access and relationship with provider. Table 2 explores the types of symptoms that were obtained from the patient perspective that are not included in the CPG[33].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Origins of items on the USQ-NB: Patients or clinical practice guidelines[33].

https://doi.org/10.1371/journal.pone.0171114.t002

Table 2 shows that patients identified additional symptoms for inclusion in the USQ-NB that are both specifically urinary in nature (e.g., cloudy urine; increased incontinence) and also more general (e.g., fatigue; dizziness). Thus, their contributions to the instrument were not solely focused on one type of symptom. The instrument we developed (USQ-NB) includes queries about whether the patient feels the symptom is related to a UTI, but whether any of these is pre-infectious, infectious, or not related to an actual UTI is beyond the scope of this paper. Our research team is pursuing this line of investigation as we continue to document the utility of this new patient-centered PRO for research and clinical care in this domain.

Discussion

The purpose of this study was to demonstrate, and then explore the result of, the use of a new paradigm for developing patient-centered patient reported outcomes (PC-PROs). This approach allows patients to report equally on items that are derived from both the patient and the clinician perspective–i.e., this new paradigm for developing patient-centered, patient-reported outcomes is fully consistent with the definition of a PRO from the FDA, the objectives of PCORI, and with most models of patient and stakeholder engagement. However, although valid patient-reported outcomes can be derived by either the dominant paradigm (Fig 1) or by the new paradigm (Fig 2), our results suggest that starting from—and maintaining—the patient perspective will promote the ‘patient-centeredness’ of the resulting instrument while still maintaining signs and symptoms that are clinically relevant. Focus groups and interviews that, for example, used only the IDSA guidelines for structure could easily have included an open-ended, “anything else?” item; instead we began the development of our scripts from the phenomenological perspective. The IDSA guidelines were published in 2009, so the identification of items from these guidelines was essentially a priori; the data we collected following our new model are new and support the conclusion that this new paradigm engages patient and clinician contributions around populating the list of signs and symptoms that patients in the target population encounter most frequently. Only by explicitly including, and then maintaining, the primacy of the patient perspective can the patient-centered dimension of a patient-reported outcome be retained/included.

The instrument is more than twice as long as it would have been without the active involvement of the patients; this may have implications for participant fatigue in some studies/populations. However, the focus group participants and our patient experts were enthusiastic about the opportunity to represent their own experiences fully. The number of items suggests that our paradigm is consistent with the PCORI intention “…to produce research that looks beyond the questions and measures found “under the lamp post,” within the researcher’s field of view”.[20]

This new paradigm is consistent with principles of “valuing the patient perspective”, and maintaining a “culture of patient centeredness in research”[14] while representing the rigorous qualitative methodology that constitutes the “crucial foundation” of PRO development.[22] When we have pilot intervention research to report on, the patient-centeredness of our instrument will facilitate our dissemination of our results. This emphasis on urinary signs and symptoms that are most commonly experienced, as well as those that are most clinically relevant, will support our further engagement of our stakeholders, while also promoting valid scientific research in the future. However, in our implementation of the new model, clinician input was obtained in focused meetings, and not focus groups, and the clinician participants were a single (convenience) sample of four individuals. While careful notes were obtained from all of these meetings, and. However, in future iterations in this line of study, we are planning to design the clinician input to derive from focus groups with a clinician who is trained in focus group facilitation, and to generally treat the clinician input the same way as the patients. We did not perceive it to be at the time, but this is a key feature of balancing the perspectives from clinicians and patients, so we have added this to the discussion of the new model.

The results suggest that the model of PRO development that we describe here and followed (Fig 2) can lead to PROs that maintain the patient perspective and are patient centered. We obtained substantial endorsement rates of all items in the resulting instrument by individuals across the United States, suggesting that each item is recognized as part of the lived experience with urinary symptoms in NB. However, there are limitations to consider. Firstly, we did not issue invitations to individual patients for the focus groups or for participation in our national beta test. Therefore, we cannot be sure of our response rate, or even how representative of persons with SCI or SB (or caregivers of persons with SB) who have NB these results are. We are currently preparing a second manuscript to discuss and describe the results of the survey, where this is discussed more fully. For the current study, the self-selected nature of responses to our survey is not a limitation with respect to the demonstration of how the new model for developing a PC-PRO (Fig 2) can lead to instruments that differ from the traditional model. We also have not included in this paper any formal discussion of the focus group participants; again, this is not relevant for a demonstration/proof of concept about how the PC-PRO development model works and yields a different PRO than would have been obtained without our focus on maintaining the primacy of the patient perspective.

Our results support the conclusion that there are qualitative differences in outcomes that are developed using an approach that maintains the primacy of the patient’s perspective–which we argue are concretely “patient-centered”, versus those that incorporate an emphasis on the patients’ report alone, consistent with the argument put forward by Basch, Abernethy & Reeve.[21] The PCORI Methods Committee [7] argued that “(t)o be truly patient-centered, PCOR faces the challenge of how to best incorporate the patient and caregiver perspective at every step in the selection of research questions, design, conduct, and analyses of studies, as well as dissemination and implementation of findings”[7]. Our paradigm establishes the primacy of the patient perspective in a PRO so that, once the instrument is validated, it will bring the patient and caregiver perspectives to every step of the design, conduct, and analyses of clinical research within the target domain. A recent systematic review [35] extensively reviewed the desirable characteristics of patient reported outcomes, and encapsulates how the traditional model (Fig 1) is deeply entrenched in the literature currently. In their synthesis of the literature and interviews with experts on the desirable characteristics of the PRO, claims of “patient-centeredness” are made but not supported, and no patient input was obtained—not in the literature that was reviewed and not in the interviewing that was conducted. Therefore, while our results (and the model here) are consistent with the characteristics that Francis et al. (2016)[35] discuss, the new PC-PRO development model is a clear departure from this traditional model; and these results suggest that the PC-PRO model does yield a qualitatively different instrument than a PRO would.

Overall, comparative effectiveness research that is conducted using a validated instrument developed using our paradigm will be stakeholder-driven [25] in that the instrument to be used as the outcome/endpoint in such research will be strongly representative of the patient/patient perspective. Moreover, the impact of maintaining the patient perspective on the design and interpretation of comparative effectiveness, and other types of clinical research, can be studied using an approach similar to what was described here. We are also continuing to study the psychometric characteristics of the instrument described here, including determining whether all items on the instrument are responsive to clinical change (e.g., as verified by biomarkers and metagenomic changes). Future work will also be needed to determine whether the development of PC-PRO instruments can work in domains where PRO instrumentation has been identified as problematic fields (e.g., radiology,[15, 16] emergency medicine,[17, 18] and neurosurgery).

Acknowledgments

This work was supported by PCORI grant number: AD-1310-08215 to SLG.

The first author had full access to all of the quantitative data in the study and takes responsibility for the integrity of the data and the accuracy of the quantitative data analysis. The third author had full access to all of the qualitative data in the study and takes responsibility for the integrity of the data and the accuracy of the qualitative data analysis.