Investigating the complementary value of discrete choice experiments for the evaluation of barriers and facilitators in implementation research: a questionnaire survey

There are numerous implementation strategies available that have proven to be at least moderately effective in bringing about change [1‐3]. Current insight in implementation research is that the choice of implementation strategy should be guided by a diagnostic analysis that starts with describing the gap between current care and optimal care [4]. An important part of the diagnostic analysis is the identification of barriers and facilitators to change. Until now, studies identifying barriers and facilitators to change have been carried out using a combination of qualitative and quantitative methods, such as case-specific questionnaires, semi-structured in-depth interviews, focus group interviews, and non-participating observation [5]. These methods have some limitations. First, they generally yield many barriers, but do not provide information with respect to the relative importance, or prioritizing, of the barriers. Second, existing instruments hardly differentiate between barriers, and consequently may overestimate the importance of less important barriers and underestimate the importance of more important barriers. Third, these traditional methods have a non-compensatory character, which may be problematic because in decision processes concerning the implementation of change it is often the case that facilitators can partly compensate for barriers. These limitations associated with the methods that are currently applied in implementation research have revealed the need for an alternative research methodology for the evaluation of barriers and facilitators. Discrete choice experiments (DCE) to investigate preferences are relatively new in the health care sector, and may be of value in the field of implementation research. DCE is a stated preference method that presents individuals with a number of choices. Each choice consists of two or more hypothetical profiles, and for each choice, people are asked which profile they would choose. Forcing people to make choices and trade-offs is a big advantage of DCE over the methods that are currently applied in implementation research. For us, this is a strong motivation to introduce DCE in implementation research. If DCE proves to have a complementary value for the evaluation of barriers and facilitators, the choice of implementation strategy will be based on factors that more accurately reflect individuals' preferences and trade-offs, and the strategies will be tailored to the preferences of those concerned with the actual implementation.

Respondents were expected to be unfamiliar with DCE, but also are actually involved in implementation processes and hence users of the results and responsible for getting the research findings into practice. Therefore, we asked respondents for their opinions about the feasibility of DCE. In particular, we asked for the completion time, the difficulty of the questions, and the appropriateness of DCE. The traditional questions are well-known, easy to administer and take little time to complete. Furthermore, the traditional method is often applied in implementation research, which is a strong indicator of its feasibility. DCE, on the other hand is new in implementation research, so we decided to evaluate only the feasibility and acceptance of this novel development.

To the best of our knowledge this is the first application of DCE in implementation research. We used DCE to identify barriers and facilitators to the implementation of the guideline for breast cancer surgery in day care. The objective of our study was to investigate the complementary value of DCE for the evaluation of barriers and facilitators in implementation research. To meet this objective we compared the results of a traditional questionnaire with the results of a DCE. In addition, we asked respondents for their opinions about the feasibility of DCE.

Neither DCE nor a traditional questionnaire was considered the 'gold standard' for identifying potential barriers and facilitators to the implementation of breast cancer surgery in day care. The reason why we conducted this study was that we would expect that DCE would provide implementers with more specific information to better fine-tune the implementation strategies. The results of a DCE provide implementation researchers and clinicians with a relative attribute importance ranking that can be used to prioritize potential barriers and facilitators to change. Prioritizing is useful to tailor the implementation strategies to the specific problems and challenges of a particular implementation process. Furthermore, the DCE method makes it possible to gain insight into the trade-offs people make between barriers to change on the one hand and facilitators on the other hand.

The traditional questionnaire was made up of statements that respondents answered on a five-point scale ranging from one (fully disagree) to five (fully agree). The questionnaire included both statements related to preconditions and statements related to expectations. Both kinds of statements cannot be compared to each other because the interpretation of the responses is considerably different. Statements phrased as preconditions measure whether the specified requirements should be fulfilled to successfully implement change. Statements phrased as expectations measure whether the specified barriers are expected to actually occur in respondents' own hospitals. Although the extent to which respondents agreed with the statements about the preconditions might be interpreted as a measure of the importance respondents attach to each of the requirements, it does not provide information about the relative importance of the barriers and facilitators and the trade-offs people make. Hence, a traditional questionnaire cannot easily be used to identify the most crucial barriers and facilitators to successful implementation and to leave aside the relatively unimportant ones. Furthermore, we must interpret the results of the traditional questionnaire with caution because the responses to the statements hardly differed from each other. One could argue that essentially respondents agreed with all statements to the same extent, which makes ranking less useful.

The results of our DCE and traditional questionnaire would possibly lead to different implementation strategies. This confronts those who are involved in implementation processes with serious difficulties, because we do not know which method is the best one. There are three major differences between DCE and the traditional questions that might challenge the comparability of the results. First, DCE is based on random utility theory that assumes that an individual acts rationally and always chooses the alternative with the highest level of utility. In our experiment, the 17 attributes (the independent variables in the utility function) are regarded as sources of respondents' utility because the attributes are more or less important for successful implementation of the guideline for breast cancer surgery in day care. The overall utility may therefore be described as an evaluation of how attractive it is to implement the guideline for breast cancer surgery in day care given the circumstances as described by the attributes. DCE is therefore a preference-based method in contrast with the traditional questionnaire that has no theoretical underpinnings. Second, the regular questions are more likely to measure the perceived barriers and facilitators that may not reflect the actual barriers and facilitators. The results of a DCE are likely to give a more objective view of the factors that are important for successful implementation. When responding to the regular questions, respondents are generally inclined to refer to the circumstances in their own hospitals ('which barriers would we encounter in our hospital?'). Whether respondents really answer the regular questions with their own hospitals' situations in mind depends to a great extent on the wording of the questions, which is not standardized. Discrete choice tasks present subjects with hypothetical scenarios; hence there is no link between the choices and subjects' current clinical practice. Third, in a DCE people are forced to make choices and trade off attributes, which is closer to the decision-making context in reality than the regular questions.

We encountered a low overall response rate of 10%. In accordance with Sitzia and Wood [14], sending reminders did little to increase the response rate. Response rates for mail surveys are approximately 62% [15]. Large variation is reported for response rates for DCE in the health care literature, ranging from 18% [16] to 98% [17]. We do not know if, to what extent, and how the low response has influenced the results. In the current study, there are four possible explanations for the low response. First, we investigated a complex multi-faceted health care decision – the implementation of the guideline for breast cancer surgery in day care – that moreover involves multiple professional disciplines. In implementation research, it is important to take into consideration this multidisciplinary nature of many health care decisions because different groups of health care professionals may express different opinions, interests, and preferences. Hence, the potential barriers and facilitators to the implementation of the guideline need not necessarily be the same for anaesthesiologists, surgical oncologists, and breast care nurses. To facilitate comparison between these groups, we developed an identical questionnaire for all three disciplines. Although the advantages of this approach are obvious, a disadvantage is that we could not completely fine-tune the questions to the specific circumstances of each professional group. Therefore, respondents may have considered the questions too broad and not fully applicable to their own situations. In particular, anaesthesiologists were expected to respond suboptimally because this group of health care professionals is less closely associated with the subject of the survey compared to surgical oncologists and breast care nurses. Respondents may thus have had less affinity with the questions. A second explanation for the low response may be the use of self-complete postal questionnaires. Other more personal data collection methods are available, and telephone or face-to-face interviews might well have increased the response rate, though at much higher costs. The phenomenal growth of internet technology in recent years has prompted many health care researchers to find ways of using this technology to communicate more effectively. Braithwaite et al. [18] examined whether internet-based surveys among health professionals can offer a valid alternative to traditional survey methods. They performed a systematic review of published internet-based surveys among health professionals and found response rates ranging from 9% to 94%. The authors also conducted an internet-based survey among general practitioners that explored attitudes about using an internet-based decision support system for the management of familial cancer. They achieved a response rate of 52.4% after five e-mail reminders. Brown and Kittleson [19] compared the response rates from an e-mail survey (43%) and a web-based survey (48%), and found no statistically significant differences between the two groups. An online survey among chiropractors yielded a response rate of 35.8% [20]. As online surveys become more feasible for more populations, it is worth considering the use of e-mail or the web as data collection tools in future research.

Third, the clinical subject may have been too complex to introduce DCE in implementation research. The implementation of the guideline for breast cancer surgery in day care is a complex process that involves changes on the organizational and management level, as well as the level of health care professionals and patients. The complexity of the clinical subject furthermore required that we had to include large numbers of potentially influential attributes. Therefore, we used HII, which is a more complex alternative to standard DCE.

Fourth, the questionnaire included both the traditional questions and the discrete choice tasks. All respondents first answered the traditional questions and then the discrete choice tasks. We do not know whether the response would have been better if respondents had been offered only the traditional questions or only the discrete choices. Several studies have shown that response rate is not correlated to questionnaire length [14, 15, 21]. Intuitively, we would nevertheless expect a higher response rate if the questionnaire included only the traditional questions for three reasons. First, our questionnaire was lengthy, and obviously completing only the traditional questions takes less time. Second, because respondents were likely to be unfamiliar with DCE, the discrete choice tasks were introduced thoroughly. Yet, this extensive introduction and warm-up task with fill-in instructions required a lot of reading, and thus time and motivation. Mean completion time of the discrete choice tasks was more than 25 minutes. Third, discrete choice tasks are without doubt more cognitively demanding than the traditional questions. The mean difficulty score was 4.8 on a scale from 1 to 9, which means 'somewhat difficult' to 'not difficult/not easy'. Eighteen respondents did not answer the discrete choice tasks. Although we did not systematically investigate why these respondents did not complete the choice tasks, we suppose that the two main reasons are the complexity of the DCE and the length of the questionnaire. We do not expect that these 18 respondents would have completed the DCE if the choices were presented before the traditional questions. The answers to the open question about the appropriateness of DCE are in support of this, and suggest doubt about the feasibility of DCE.

A considerable variety exists in the design and analysis of studies investigating barriers and facilitators to the implementation of change. In most cases, the analysis is constrained to descriptive statistics. Because of limitations associated with the methods typically applied in implementation research (see introduction), uncertainty exists with respect to the most useful research methodology. DCE should theoretically overcome the limitations of traditional methods, and provide implementers with more specific information, i.e., insight into the trade-offs people make and relative attribute importance. Because DCE probably reflects actual implementations decisions more closely, it is expected that the implementation strategies will become more tailored to the actual preferences, needs, and wishes of those who are involved in the actual implementation process. It is furthermore expected that this will increase the cost-effectiveness of implementation strategies. So, on theoretical and conceptual grounds it can be suggested that DCE should be considered the reference standard in our study [8]. From a practical point of view, however, our study revealed that DCE could not entirely fulfil the role of 'gold standard'. In brief, respondents considered the method too difficult and too time-consuming, which may partly explain the low response. The feasibility of any method is – at least partly – dependent on the study context such as the target respondents, study design, clinical subject, and data collection method. Further empirical applications should investigate whether DCE can really make a valuable contribution to the implementation science.

DCE was proposed as a tool to identify potential barriers and facilitators to the implementation of change. The results of a DCE and a traditional questionnaire would probably lead to different implementation strategies. Although there is no 'gold standard' for prioritising potential barriers and facilitators to the implementation of change, theoretically, DCE would be the method of choice. However, the feasibility of using DCE was less favourable; respondents considered the method too difficult and too time-consuming, which may – at least partly – explain the low response.

* Statements 1, 2, 4, 5, 11, 12, 13, 14, and 16 were phrased as preconditions; statements 3, 6, 7, 8, 9, 10, 15, and 17 were phrased as expectations.