Progress in Community Health Partnerships: Research, Education, and Action

Participatory Design Components

The study's design, data collection, and interpretation were informed directly by policy, advocacy, and administration stakeholders, most of whom identify as people with psychiatric histories. We based our inclusive approaches on an existing framework for participatory research with people with disabilities.¹⁰ Stakeholders who contributed to project development were identified through the lead researcher's professional network, based on their history of contributions [End Page 46] to the field, rather than a singular "partner agency" with the university hosting the research project. Table 1 outlines the participatory design components¹⁰—as applied to study procedures—used in the survey.

Researcher Translation From Participating in Community Networks

Researcher translation refers processes in which researchers join community members in discussing important issues and potential research topics.¹⁰

Developing Specific Aims. The lead researcher previously worked on smaller survey research projects with members of the target population and was aware of issues that they might like addressed in a large-scale survey. Discussions within this community—via listservs, state/local organizing meetings, and national conferences—was the primary motivation for questions about the ACA and other policy priorities, and was consistent movement advocates' writing since the 1970s.²⁰ Other questions were related to organizational operations, such as staffing, financing, and support provided, because there has been no documentation of these characteristics of peer-run organizations since 2002.⁵

Click for larger view
View full resolution

Table 1.

Participatory Design Components and Research Project Component

Inventory of Organizations. The project capitalized on existing relationships in the national network of consumer/survivor advocates and peer-run organizations to create an inventory of all peer-run organizations in the country to conduct recruitment. At the outset, there was no comprehensive national list of peer-run organizations. Our search identified 895 organizations and programs. The primary strategy was [End Page 47] contacting statewide consumer networks and state offices of consumer affairs from August 2010 to June 2012. The National Coalition for Mental Health Recovery (NCMHR) and officials with SAMHSA's Statewide Consumer Networking grant program and the Center for Mental Health Services' Office of Consumer Affairs were consulted. Statewide networks were contacted by email and phone to obtain lists of peer-run organizations in each state. This search method was supplemented with lists from SAMHSA's three National Consumer Technical Assistance (TA) Centers. In states without a statewide network identified by SAMHSA or NCMHR, key local informants and organizations were contacted to provide missing information.

Agenda-Setting Survey

An agenda-setting survey approach involves reviewing literature and conducting interviews or focus groups with members of the target population to develop a community-oriented research agenda.¹⁰

Instrument Design. The survey was designed to collect current information on organizational operations, including funding, staffing, services, and activities. An additional goal was to assess attitudes toward upcoming changes associated with the ACA that may affect peer-run organizations, to make policy recommendations to advocates and policymakers.

Individuals working in research, government, managed care, advocacy, and program development contributed to the design of the instrument. Almost one-half of those (18 of 37) were people with psychiatric histories and professional qualifications. These key informants were interviewed by phone or in person in an unstructured manner. They were asked to identify topics and questions most pertinent to this population in the current policy context. Some sections of the survey were adapted from SAMHSA's 2002 Survey of Self-Help Organizations. Sections asking about consumer involvement in management and decision making were adapted for a web-based survey from the Consumer-Operated Services Program multisite evaluation fidelity instrument.²⁸

Click for larger view
View full resolution

Figure 1.

Study processes in temporal order.

The lead researcher designed the initial version of the instrument, [End Page 48] which underwent expert review, pretesting, and field testing before being launched online via Qualtrics survey software.

People With Psychiatric Histories as Research Team Members and Consultants

Researchers who identify as people with psychiatric histories had significant control over the research process and allocation of resources, and were paid team members and consultants. Experts identifying as people with psychiatric histories served as consultants on research process and content to improve social validity. These consultants had content experience, familiarity with interrelations among issues, and a nuanced understanding of the study content and implications for research.

Project Lead. The lead researcher identified as a person with a psychiatric history. She made determinations—in consultation with other researchers and team members—about funding allocations, hiring team members, data collection methods, and interpretation and dissemination.

Research Team. Research assistants were identified by On Our Own of Maryland, Inc. (a statewide consumer network) and hired by the lead researcher to assist in data collection. The research assistants identified as people with psychiatric histories and had experience in peer-run programs or as advocates. They were trained in a protocol for follow-up with nonresponders that included timing of follow-up contacts and a script.

Recruitment and Data Collection. Data collection was conducted from April to October 2012. The survey could be completed anywhere with an Internet connection, and respondents could pause during the survey and come back later with their answers saved. Funding was allocated by the lead researcher to pay research assistants for follow-up with nonresponders, provide an incentive for participation, and conduct public dissemination.

Organizations were recruited through a letter on university letterhead sent via the U.S. Postal Service introducing the study, its methods, and purpose, and the fact that it was being conducted by researchers with psychiatric histories. The letter, which included a $5 cash incentive, stated that participants would receive an email with a link to the survey, to be completed by the executive director or designee. The invitational letter was endorsed by the SAMHSA consumer TA centers and the NCMHR. Follow-up with nonresponders included multiple e-mail and phone contacts by research assistants.

Because research assistants were familiar with peer-run organizations, they were sensitive to the concerns and questions of the organizations' directors. For instance, research assistants could understand and respond nondefensively to concerns about the history of abuse in research on mental disorders or the cooptation of self-help initiatives by traditional medical care providers.

This study achieved a final response rate of 80%; 380 peer-run organizations were in the final sample for analysis. Response rates for web-based surveys of organizations range from 68% to 89%, and response rates for surveys of individuals average about 40%.^27,29

Consensus Panel. We convened a consensus panel of five advocates, TA providers, program directors, and researchers with psychiatric histories who are well-known advocates for fidelity to peer-run initiatives and meaningful inclusion of consumer/survivor voices in policy development. The consensus panel decided on final inclusion criteria, unanimously endorsed by all members. The panel confirmed the definition of a peer-run organization was confirmed by these community members and was based on existing user-led research on fidelity to organizational structure.²⁸ The consensus panel based their discussion on aggregate data provided by the research team on program structures and policies, and on the consumer/survivor movement's history of creating independent alternatives to traditional mental health services. This improved the validity and applicability of the definition, because it emerged from members of the community.

Qualification for Study Participation and Data Preparation. Organizations and programs that did not meet peer-run/peer-operated criteria were excluded from the sample. All organizations suggested by statewide consumer networks or other informant were recruited. However, we could not verify that all participants qualified for the study, so we empirically determined inclusion in analyses using the consensus panel's criteria.

Product Champions for Buy-in and Dissemination

Product champions are individuals or groups representing the target population who find the research results useful to them and their causes and, therefore, support the research process and dissemination of results.

Recruitment Buy-in. The co-signing of the invitational letter by the SAMHSA consumer TA centers and NCMHR [End Page 49] was intended to gain the trust of respondents and signal the importance of the study. However, not all respondents necessarily identified with these entities.

Dissemination. In addition to articles in academic peer-reviewed journals,^2-4 public reports on organizational characteristics and attitudes toward the ACA were created and disseminated immediately after data were analyzed. Participants were directly emailed reports, also available on the web, with a limited number of free paper copies available.

Immediately after completion of data collection and initial calculation of descriptive statistics, the lead researcher presented these preliminary results at the Alternatives 2012 conference, a SAMHSA-sponsored annual national conference of people with psychiatric histories, many of whom work in the mental health system or peer-run organizations. Attendees provided valuable input on interpreting results from their perspective as survey respondents, which was incorporated into written reports of study results.

Results were presented at the American Public Health Association 2013 annual conference by the lead researcher, a participant, and a member of the consensus panel, and by the lead researcher at the American Public Health Association 2014 annual meeting. The Association for Behavioral Health and Wellness (a membership organization of behavioral health managed care companies) hosted a webinar for members to learn about results related to the ACA and how they might implement recommendations.

Lessons for User-led and Participatory Research

The involvement of people with psychiatric histories in project design and implementation highlights important strategies that contributed to this study's success. Although the expertise of both researchers and community members is imperative to participatory research, the inclusion of individuals from the community as professional researchers may make user-led research a more advanced form of participatory research in mental health.^{12,13,15,16,30,31} Although it is not always possible to include community members in projects in substantial numbers, owing to availability of research expertise, appropriate partners, financial resources, and other facors,³² this project demonstrates additional benefits of user-led participatory research, including engagement of an array of advocacy and service organization representatives in research design and dissemination, and hiring and supervision of research assistants from the target population.

Project Development

The study's specific aims for data collection and analysis were motivated by the passage of the ACA and ensuing discussions among peer-run organizations and advocates about potential challenges related to Medicaid reimbursement. Although not commonly thought of as a service provider type that would be impacted by insurance reimbursement changes, discussions among peer-run organizations inspired the need for a national study on these issues. This approach may be useful in other policy research addressing regulatory and financing changes affecting peer-run organizations or peer support to solicit ideas from the impacted community, where the issues are well-understood by the researchers before embarking on the project.

Project Team

Existing relationships in the community and project staff who are part of the target population were key to achieving a high response rate. Team members' existing relationships helped to identify organizations and encouraged survey responses. Other research suggests that a shared history of marginalization is relevant to data collection, enabling better connection with respondents^13,33 who may be more likely to open up to interviewers that are their peers.³⁴ This reflects our experience.

The consensus panel was convened to determine inclusion criteria. A consensus panel may be useful when a study needs a definition of terms or constructs that is relevant to a community. The consumer/survivor movement is a close-knit community; members often have long histories of collaboration. This can make consensus easier (i.e., owing to commonly held beliefs) or more challenging (i.e., owing to political fragmentation common to social movements).³⁵ It might have been useful to have convened an advisory group for more substantial, formal roles in an ongoing manner throughout the project. For instance, although many knowledgeable individuals gave input into the survey design, members of the consensus panel decided that the values of peer-run organizations were not measured adequately. In the future, such a group would be helpful in brainstorming questions on "values," while limiting social desirability bias.³⁶ [End Page 50]

Engagement and Dissemination on a Broader Scale

This project did not use the common structure of a participatory research project that engages with a limited number of local or national community partners. Instead, we capitalized on the longstanding professional relationships of team members with established professionals in the consumer/survivor movement. This strategy also supported dissemination to national conferences and groups that could implement policy recommendations.

ACA-Related Advocacy in the States

An essential purpose of this study was to provide nationally representative data to advocates and policymakers to inform perspectives and make data-driven arguments in public policy deliberations about the future of peer-run organizations. Since the study's conclusion, there has been rapidly increasing interest in the states in Medicaid-reimbursable peer support. Although the focus has mainly been on peer support within traditional mental health services, peer support provided by peer-run organizations will also be affected by state laws. The survey results provide a broader perspective on the pros and cons of this issue, rather than just soliciting input from a small group who have the opportunity to influence policy.

Usefulness of Enumeration and Updating Database/Survey

Updating the database of peer-run organizations is valuable; it is important that these groups continue to be enumerated and monitored for changes in organization and practice associated with changing policies. However, there is currently no known effort to update or create ongoing national databases of these organizations.

Since completing the project, many statewide networks have expressed interest in receiving updated lists to use in organizing and coalition building. An updated database would be essential for a follow-up wave of the study. Even with buy-in from the community, the enumeration process is an intensive research process in itself.

Feasibility

The study found that web-based surveys are feasible in organizational research, even with hard-to-reach populations, provided researchers use complementary outreach and inclusion approaches. We believe that multiple contacts by research assistants, particularly repeated phone calls, resulted in the high response rate.

The study demonstrates the feasibility of monitoring a group of peer-run organizations using a web-based survey and facilitating nationwide assessment of organizational operations and attitudes toward policy changes. The 2012 National Survey of Peer-Run Organizations addressed some inherent challenges by marrying web-based survey research with participatory research. Web-based survey research provides adaptable question design and makes it affordable to recruit large numbers across a wide geographic area, making them a particularly good fit for studies like this.²⁷ We believe these factors contributed to the enthusiasm for the survey evidenced by the high response rate. Future research could test the effectiveness of user-led, participatory, and traditional research approaches more thoroughly to distill important elements.