Background
Patient-reported outcome measures (PROMs) are questionnaires which measure patients’ perspectives on health outcomes [
1,
2]. PROMs may be used to inform patients of the performance of treatments and health care professionals, improve the care of individual patients, assist health care purchasers and reward performance in pay for performance schemes [
2‐
5]. PROMs are increasingly assigned these important tasks of measuring performance, treatment progress and quality of care in health care systems such as the English National Health Service (NHS) [
2,
4], the American Centers for Medicare and Medicaid Services (CMS) [
6], the Dutch and the Swedish health care system [
7].
As PROMs are meant to reflect the patient’s perspective, the aim should be to use valid PROMs which truly reflect this perspective. Patient involvement throughout the development of PROMs is therefore essential [
8‐
13]. Patient involvement can have a great impact on the questionnaire as the relevance of outcomes and the comprehensibility of the questionnaire can only be determined by patients [
12,
14‐
19]. Furthermore, patients may fail to complete the questionnaire if the questionnaire does not sufficiently reflect the patient’s perspective. A lack of patient involvement may have a negative influence on the validity, sensitivity and response of a questionnaire [
13,
20].
However, although patient involvement in PROM development has been recommended by researchers [
8‐
12,
21], and has also become a requirement of the US Food and Drug Administration (FDA) [
22], many PROMs appear not to have been developed with patient input [
9,
23,
24]. A scoping review of 193 PROMs looking at whether and to what extent patients are involved in the development of PROMs, suggests that in over a quarter of the included papers describing the development of a PROM, no patient involvement was recorded. Furthermore, the level of patient involvement in the remaining PROMs varied greatly, from feedback forms to focus groups, interviews and cognitive interviews [
24]. Another review of 42 PROMs for rheumatoid arthritis showed that PROMs only minimally covered personal factors which determine functioning and the individuals’ experience of disability according to the International Classification of Functioning, Disability and Health (ICF) and which were deemed important by patients [
25]. Other studies which investigated patient involvement in a small number of PROMs suggest a complete lack of patient involvement [
23], conceptual difficulties and questionnaire design problems. These problems might have been prevented by involving patients in the development process [
9].
Apparently, not all PROM developers choose to involve patients in the development of their PROM. Besides, if a developer does decide to involve patients, there is a great variety in the levels and methods of involving patients. Although many benefits may be gained by involving patients [
26], research suggests that developers also need to consider slightly less positive consequences such as the costs associated with patient involvement [
12,
26]. Additionally, there is no perfect method to involve patients, as fairly little is known about which methods are the most successful and which circumstances are needed for achieving optimal outcomes [
27]. Decisions that need to be made, such as whether patients should be involved, to what extent patients should be involved, and which methods should be used to involve patients, appear to be mostly based on the experiences and opinions of the researchers, the organisation they belong to, or the regulations or guidelines they adhere to such as the US Food and Drug Administration [
22] or PROMIS (Patient-Reported Outcomes Measurement Information System
) [
28].
However, as most professionals involved in PROM development deal with similar decisions, it is necessary that more insight is gained in how patients can be successfully involved in PROM development and what kind of benefits and costs of patient involvement are experienced by PROM developers. By looking at successful involvement, we are not necessarily interested in how patients were involved, as this was also recorded in an earlier scoping review [
24]. The focus lies instead on how developers chose their methods, and how they experienced using these methods. Such an overview may help PROM developers make a more informed decision regarding patient involvement. This qualitative study therefore aimed to gather the views of PROM developers, who did and did not involve patients, regarding patient involvement and its negative aspects and benefits. Our research questions are:
According to the interviewed PROM developers:
1
Why did certain developers involve patients and others did not?
2
How are patients successfully involved in the development of PROMs?
3
What do patients contribute to the development of PROMs?
4
What are negative aspects of patient involvement?
Discussion
As the importance of patients’ knowledge on health and health care as a source for improving the quality of care is increasingly recognised [
2,
4,
11], PROMs are used more and more [
1,
2,
4]. Although for capturing patients’ perspectives on health care patient involvement in PROM development is deemed necessary [
8‐
12], patients are still not always involved [
24]. There is not one correct procedure for involving patients [
27] and choosing between patient involvement and no patient involvement can be difficult as both can have consequences for the research project. This study therefore aimed to give more insight into PROM developers’ experiences with patient involvement by first investigating how patients are involved in the development process. Patients were mostly recruited via clinics or health professionals until data saturation was reached. Recruiting until data saturation has been reached is a method in qualitative research that is widely accepted and used by many researchers [
32,
33]. The developers mostly involved patients using interviews and focus groups. These methods are mentioned by several scientific standards or guidelines as fitting methods to involve patients [
28,
34]. Although some developers indicated that they adapted the recommended methods to their patient group or to how they thought it would work better, many followed the guidelines or used methods which were the most practical and/or cheapest, or based their choice on personal experiences and preferences.
Besides the more common methods used to involve patients, several developers involved patients especially in their more recent PROM developments as patient research partners or included patients in steering or advisory groups. Even though this more collaborative form of patient involvement may have some issues that need to be dealt with, such as the amount of experience and skills patients would need to actively participate [
35] and whether patients who are able to participate to such an extent are good representatives of the patient group [
36], it is considered an important step forward for patient involvement in PROM development [
12].
Another less common but increasingly used method concerns conducting qualitative research on the internet. As many people use the internet, conducting qualitative research online can be helpful in gathering the opinions of many patients [
37]. Patients with rare conditions can also be more easily reached [
38] and communicating online may offer patients more freedom to express themselves [
39]. Online qualitative research also has some drawbacks. For example, you are only able to study a select group of patients as patients need to have access to internet and certain skills to participate on the internet [
40,
41]. Additionally, it is much less clear what the rules are regarding informed consent and what is considered a private conversation and what is not [
37]. As both online and face to face qualitative methods have several pros and cons, a decision for either of these methods should be made separately for each study.
Besides how patients are involved in PROM development, the present study also investigated what patient involvement contributes towards the PROM development. Patients contributed by offering their experiences and views regarding living with a disease. These views not only helped increase the validity and comprehensibility of the PROM, but also helped to ensure that the PROM was relevant for other patients. The benefits of patient involvement were very important to PROM developers as most would not consider to refrain from involving patients, even if they did not really have the budget or the time to do it.
Although most PROM developers were overwhelmingly enthusiastic about patient involvement, negative aspects and costs of patient involvement were encountered at several levels. Most developers experienced some difficulties finding enough diverse patients and getting patients to fit research into their lives could be challenging. Furthermore, actually involving patients could be challenging as most developers encountered patients who dominated the conversation or easily went off-track. Overall, patient involvement was time consuming, made an impact on the budget and was logistically challenging. Although the costs of patient involvement are sometimes recognised in the literature as a challenge of qualitative research [
12], the challenge of actually recruiting and involving patients and the time it takes to do it well should also be acknowledged.
It should therefore come as no surprise that a lack of resources was one of the reasons why not all PROM developers involved patients. There were however also a few PROM developers who did not see any added benefit of patient involvement for their PROM. As patient involvement is increasingly required by organisations such as the FDA [
34], these PROM developers may not have this choice for much longer.
Limitations and strengths
Some limitations need to be taken into account. First, this study aimed to give insight into the views of both developers who involved patients and developers who did not involve patients. However, only a low number of PROM developers who did not involve patients were willing to answer our questions.
Second, only 58.5% of the invited PROM developers who did involve patients agreed to participate in our study. As we indicated in our invitation that we wished to discuss patient involvement in PROM development, it is possible that the PROM developers who agreed to participate were perhaps more enthusiastic about patient involvement than the PROM developers who did not reply. Whether we received a reply was in any case not related to how long ago the paper was published, as no difference was found between the publication dates of developers who participated and developers who did not participate. However, developers may have felt unsure about discussing their research in a language other than their native language, as all but one non-native English speakers declined our invitation. This makes it difficult to generalise the results beyond English speaking countries.
Third, results may be influenced by a confirmation bias. As patient involvement costs time, resources and effort, developers must have been very motivated to involve patients. This may also mean that they are more focused on the positives.
Fourth, although by including both co-authors in the analysing process efforts were made to guard the transparency and reliability, results may be coloured by views held by the first author who interviewed the developers and was the main coder.
An important strength of this study is that we were able to offer an insight into the views of many both experienced and less experienced PROM developers who worked with a number of patient groups and developed a PROM fairly recently. This study was therefore able to not only give insight into the methods used to recruit and involve patients, the many pitfalls and benefits of patient involvement, but also into the general opinion held by PROM developers regarding patient involvement in PROM development.
Implications
The results of this study have several implications for patient involvement in PROM development. First, this study indicates that patient involvement requires the availability of resources such as time and money. A lack of or limited resources may keep developers from optimally involving patients or it may keep them from involving patients at all. As patient involvement is included in some scientific guidelines [
28] and the Food and Drug Administration even requires it [
34], it is essential that the necessary resources are included in the costs and the duration of the study when applying for subsidies so this can no longer be a stumbling block.
This is especially important as not involving patients may have implications for the use of the PROM. It may impact the response and the validity of the PROM [
13,
20]. Perhaps more importantly, PROMs are used for important purposes, such as measuring the performance of health care providers [
2‐
5]. Not involving patients may mean that health care providers are judged on aspects of care which may not be important to patients, while important aspects may be forgotten. Furthermore, research shows that health care providers especially try to improve on the performance aspects they scored less well on [
42]. If performance measures are used which do not reflect what patients consider important, health care providers may neglect to improve on aspects of care which are important to patients. Because of the possible consequences of using PROMs which do not reflect the patient’s perspective adequately, patient involvement should be taken very seriously.
Furthermore, the results suggest that many developers keep to scientific guidelines for methodological decisions or alternatively pick methods to involve patients which are most practical to them or which suit their personal preferences. Guidelines may be a good place to start as it guarantees a certain level of patient involvement and practical solutions should certainly not be completely disregarded as it helps developers involve patients within their limitations. However, as there is no perfect method for involving patients [
27], it may be more beneficial that pros and cons of the available methods are considered. For example, based on our results, interviews may be especially fitting for discussing intimate or taboo subjects. Focus groups may be helpful to learn about the majority view or if you have a very tight budget. Both also have drawbacks such as the difficulty of keeping a focus group on track, or the single subjective view of a patient during interviews. It is therefore important that a method is picked based on what is most suitable for the patient group and the purpose of the study.
Fourth, several PROM developers mentioned the importance of good qualitatively trained and/or experienced staff for moderating focus groups or conducting interviews. They also highlighted the importance of good communication, a clear understanding of roles and expectations and awareness of patient’s limitations. These requirements combined with the difficulties one may experience while interviewing or conducting focus groups asks a lot of the persons who are hired to do so. Adequately training staff to deal with patients and conduct qualitative research may be imperative for successful patient involvement and optimal patient benefits for the PROM development.
Conclusion
Even though patient involvement may be difficult at times as it is costly, time consuming, logistically challenging and some patients can be hard to handle, all PROM developers who involved patients during PROM development agree that patient involvement in PROM development is necessary. Patient involvement results in a more valid and comprehensible PROM which is more relevant to patients. Although most developers are united in their enthusiasm for patient involvement, a lack of resources can be a stumbling block. Additionally, this enthusiasm for patient involvement does not appear to stretch out towards choosing the methods for involving patients. Most developers follow guidelines or standards or choose a method based on personal experience or practicalities. They do not always appear to actively consider pros and cons of different methods that can be used to involve patients. Although guidelines for PROM development, personal experience and practicalities may be a good place to start, to optimize patient involvement developers should explicitly think about which methods would suit their study and the patient population of interest.