The clinician crowdsourcing challenge: using participatory design to seed implementation strategies

A significant body of evidence supports the importance of evidence-based practice (EBP) in community mental health and health settings [1], yet these practices are largely underused by organizations and clinicians [2‐4]. Implementation strategies are the interventions used to increase adoption, implementation, and sustainment of EBP in health services [5]. The evidence base for implementation strategies has advanced significantly in recent years including the identification of a complex set of factors related to implementation success. These factors include organizational- and individual-level factors, as well as characteristics related to the intervention, and the economic, political, and social context [6‐8]. Although there are several proposed typologies of implementation strategies [5], no systematic approach to developing these strategies has been proposed [9]. Moreover, implementation strategy development has rarely systematically incorporated stakeholder involvement in the design [10], a critical oversight given research in healthcare suggesting that stakeholder input results in more effective outcomes [11, 12]. These weaknesses may explain why many implementation strategies fail to improve either implementation or clinical outcomes and why most strategies fail to engage their targeted mechanisms of action [13].

The importance of participatory approaches has been long recognized [14]. Multiple research traditions and disciplines such as implementation science, community-partnered research, and action research have incorporated provider input through stakeholder engagement methods and dynamic partnership [14‐18]. Participatory design approaches are lauded for creating effective implementation strategies because they provide systematic methods to include and empower stakeholders at the beginning stages [19‐21], and provide opportunities for stakeholders to be involved in design [14]. The current study utilizes an innovation tournament, a structured participatory design approach, in community mental health, as a springboard for implementation strategy co-design. Innovation tournaments are a form of crowdsourcing in which a host issues a call for ideas to address a specific challenge or problem within a system, and frontline providers and staff who work within the system are invited to submit their ideas for how to address the challenge. After multiple steps of screening and evaluation, a few ideas emerge as winners and others do not advance. Systematic review evidence points to the effectiveness of such crowdsourcing participatory design strategies in generating novel and useful solutions to complex and intractable problems in areas outside of healthcare settings [22‐24].

The aims of the current study were twofold: (a) to test the utility of an innovation tournament for developing participatory-informed implementation strategies that enhance the use of evidence-based practices (EBPs) within a large behavioral health system and (b) to generate ideas from clinicians about the best ways for organizations to facilitate EBP implementation. Our study adds to the literature in several ways. First, our study extends the current literature on participatory approaches in implementation science, specifically providing proof-of-concept of the utility of the innovation tournament approach. We are aware of only one other study that applied the innovation tournament methodology to problems in healthcare systems and that study included a single organization focused on improving patients’ experience of care [24]. Second, we extend this work by using an innovation tournament to address EBP implementation and by applying the method across an entire behavioral health system comprised of a network of 200 independent treatment organizations (of mental health and substance use disorders) which are affiliated only to the extent that they are subject to the same policy and funding constraints through a single policy and payer infrastructure. Third, we collected ideas directly from the stakeholders who will be targeted by implementation strategies and have the most intimate understanding of factors that could affect the implementation process. The innovation tournament provided a structured and efficient way to integrate insights from clinicians (who have unique expertise about the problem), system leaders (who can speak to feasibility and financing), and experts in behavioral economics who understand drivers of implementation-relevant behavior. Behavioral science, including psychology and behavioral economics, can contribute valuable information about the best way to structure implementation efforts to optimize the design of implementation strategies that target organizations, critically important given the scarcity of resources in the mental health system.

Due to past successful experiences with recruitment within this system, we anticipated high engagement and a successful tournament. We had one hypothesis regarding the content of ideas generated in the tournament: In our past work, financial challenges such as the scarcity of resources in the mental health system and the organizational financial investment required to support EBPs have been highlighted by all stakeholders [6, 25]. In addition, clinicians have previously reported that they do not feel incentivized by their provider organizations to implement EBPs [26]. We hypothesized therefore that the majority of ideas offered by clinicians would be related to compensation and financial incentives.

This study was approved by the Institutional Review Boards of the University of Pennsylvania and City of Philadelphia.

The present study is the first to utilize an innovation tournament to crowdsource ideas from community clinicians about how they can best be supported to use EBPs. The main purpose and benefit of an innovation tournament is to engage a community through participatory design to provide their best ideas. While many studies have included stakeholder elicitation of suggestions, preferences, and targets for intervention (e.g., [40]), an innovation tournament has a unique contribution as a participatory method for the following reasons. First, an innovation tournament can engage an infinite number of people in a fun and creative way for low incremental cost, while giving each participant an equal voice with the submission of an idea. A tournament allows for quick identification of potentially high impact ideas at every level of a system and can provide a training ground to empower stakeholders’ good ideas and improve their ability to express them. Lastly, a tournament provides public acknowledgement of people who come up with ideas, and this may increase buy-in to the ideas when ultimately implemented (the “Ikea effect”) and encourage future participation. Innovation tournaments are therefore a relatively inexpensive and efficient way to solicit and powerfully engage end-user input.

Our tournament was a success in this regard: we had 65 ideas submitted by 55 participants and 899 ratings of ideas by those participants. The number of submissions and overall participation exceeded our expectations, and we engaged many more in our community through our celebratory event. The study team was continuously struck by the high level of community stakeholder enthusiasm throughout the study period. While use of crowdsourcing in health research is nascent [22], this study demonstrates how an innovation tournament can be not only feasible to implement across an entire system, but also a successful and acceptable means to engage a community of service providers in sharing their expertise to generate useful implementation strategies that address local barriers, contexts, and populations.

The prompt for our innovation tournament was purposefully broad in order to encourage clinicians to share their unique local knowledge about targeted barriers that interfere with EBP implementation and their innovative ideas for overcoming those barriers. We wanted to capture the full spectrum of levers that support clinicians in implementing EBPs [34]. The most common theme reflected in the ideas involved training. This was surprising given that over the last 10 years, DBHIDS has supported implementation of several EBPs in over 50 mental health and substance use provider agencies by funding training, consultation, coordination, administration through EPIC, and in some cases reimbursing for lost therapist time, in addition to providing an enhanced reimbursement rate [26, 27]. Despite this, clinicians within this system overwhelmingly reported that additional (free) training would increase their use of EBPs with their clients. Obviously, active training is an important vehicle to change therapist behavior; training also impacts knowledge and attitudes [41]. Nonetheless, training is not the answer to every implementation problem: even clinicians trained via current gold standard approaches (i.e., workshop, manual, and clinical supervision) often do not demonstrate fidelity [42].

One alternative and more parsimonious explanation for well-trained clinicians’ thirst for additional training is that, put simply, EBPs are hard. Almost every EBP requires a skilled interventionist to deliver a complex, multi-component repertoire of behaviors, while responding to the client’s inputs and needs. It is striking that two thirds of the ideas put forth in the tournament target training and/or clinician support tools. There is also evidence from the research literature suggesting that trained clinicians do not feel confident delivering EBPs [43]. If we start with an assumption that clinicians want to deliver EBPs and would like to feel confident that they can enact effective treatment, we might accelerate implementation if we invest in implementation strategies that make our treatments easier to do, rather than additional training initiatives. Future research on implementation strategy design might draw on insights from behavioral economics such as “nudges” and changing the choice architecture; simple checklists can make multistep procedures easier [44‐46].

Consistent with our hypothesis that a significant portion of ideas would be financially motivated, over a quarter of the submitted ideas pertained to compensation and pay. As we found in our prior research, stakeholders note that EBPs are associated with higher marginal costs that need to be reimbursed and that existing reimbursement strategies rarely cover these higher costs [6, 25]. While there were ideas calling for additional lump compensation, there were relatively few ideas involving complex incentive structures such as those used in pay-for-performance or value-based payment models. This may reflect the “bottom up” approach of the innovation tournament; involving clinicians helped us identify new opportunities that would be overlooked if innovation was left to administrators and executives alone. Ultimately, a combination of “top down” and “bottom up” approaches may tell us most about how to motivate complex, expensive repertoires of behavior such as those required to implement EBPs.

There are limitations to the study design and methodology that deserve mention and highlight avenues for future research. First, because it draws from one system that is clearly aligned with promoting EBPs—this sample may not be representative of clinicians in this system or other public behavioral health systems. The specificity of implementation strategies that emerge from an innovation tournament conducted within a given service system is both an advantage and a limitation of this approach—the strategies developed through the tournament may be ideal for a particular context and system but may not generalize to other contexts and systems. Second, beyond “winning” the tournament, clinicians in our study were not promised that their ideas would be implemented. This may have restricted clinicians’ willingness to participate or encouraged ideas that were not “implementable.” Given evidence suggesting discordance between behavior and stated preferences [47], we framed tournament ideas as an “input” to design rather than relying entirely on stated preferences. Moving these inputs to design will require additional scholarship and analysis. Currently, we are re-analyzing the ideas emphasizing behavioral processes and barriers, not just elicited preferences, as part of a multifaceted approach towards implementation strategy design. Third, although our response rate was more than double that of the average innovation tournament, some tournaments have generated much higher participation rates, even in larger organizations. Future research should examine how participants’ expectations regarding the eventual implementation of their ideas influence engagement. Fourth, we elected to form a Challenge Committee of city administrators, agency stakeholders, and experts in behavioral economics to decide on the winners; a challenge committee comprised of different stakeholder groups (e.g., clinicians, patients) may have selected different strategies as winners. Future studies should compare whether different stakeholder groups select the same or different strategies and, most importantly, whether these choices predict the effectiveness of strategies used to improvement EBP implementation within the system.

Through a novel methodology for participatory design, findings from this study highlight the feasibility and utility of engaging clinicians—arguably the ultimate target of implementation strategies—through a structured, system-wide innovation tournament. Moreover, the innovation process does not have to end with the final and winning ideas [24]. Analysis of all of the ideas (or better yet, re-engagement with those who did not have winning ideas) can provide a fertile ground for future research. We believe the tournament succeeded in engaging stakeholders with critical expertise and provided valuable data that furthers the science of developing implementation strategies to improve the implementation of EBPs. More importantly, the success of our tournament was in engaging and empowering the community. Although we did not measure acceptability directly, the overwhelming enthusiasm for this project from clinicians, agency administrators, and city officials indicated to us that clinicians were pleased to be queried and felt validated that their ideas and viewpoints were not only heard, but celebrated. The clear implication to us is that involving clinicians in every stage of implementation (from strategy design to sustainability) consistent with community-participatory research is essential to designing effective implementation strategies that improve the quality of community-based care.