Background
Applying RE-AIM to evaluate programming developed by community organizations
Methods
Program selection and recruitment
Design and operationalization of RE-AIM dimensions
Reach
Effectiveness
Adoption
Implementation
Maintenance
Procedures
Measures
Online survey
RE-AIM Elements | Descriptions | Numerator | Denominator | Other |
---|---|---|---|---|
Reach
National
Peer Mentors | The percentage of Canadians with SCI who are peer mentors | Total number of registered peer mentors | All Canadians with SCI [28] | |
Peer Mentor Representativeness | How representative registered peer mentors are compared to general Canadian SCI population | Demographic information of peer mentors compared to population estimates [28] | ||
Peer Mentees | The percentage of Canadians with SCI who have received mentorship | Total number of Canadians with SCI who have received mentorship | All Canadians with SCI [28] | |
Peer Mentee Representativeness | How representative are people who have received mentorship compared to general Canadian SCI population | Demographic information of peer mentees compared to population estimates [28] | ||
Provincial
Peer Mentors | The percentage of persons with SCI who are registered peer mentors in each province | Total number of registered peer mentors in a province | Total number of Canadians with SCI in the province [28] | |
Peer Mentees | The percentage of persons with SCI who received mentorship in each province | Total number of people who have received mentorship in a province | Total number of Canadians with SCI in the province [28] | |
Effectiveness
Provincial
Peer Mentor Outcome | The reported positive and negative outcomes of engaging in mentoring for peer mentors | Qualitative data from surveys | ||
Peer Mentee Outcomes | The reported positive and negative outcomes of engaging in mentoring for peer mentees | Qualitative data from surveys | ||
Adoption
Setting Level
Peer Mentor Programming | The % of organizations providing peer mentoring | The # of SCI organizations providing peer mentoring | The # of SCI organizations surveyed | |
Peer Mentor Programming – community | The % of organizations who provide community peer mentoring | The # of SCI organizations providing peer mentorship in the community | The # of SCI organizations surveyed | |
Peer Mentor Programming – hospital | The % of organizations who provide inpatient peer mentoring | The # of SCI organizations providing inpatient peer mentorship | The # of SCI organizations surveyed | |
Hospital adoption of peer mentoring | The % of hospitals where peer mentoring is occurring | The # of hospitals where inpatient mentoring is provided by a SCI organization | The # of SCI rehabilitations hospitals in Canada | |
Peer Mentoring Branches | The # of branches of peer mentoring services within each provincial organization | Information collected from surveys (# of reported peer mentoring branches) | ||
Implementation
National Level
Outcome data collection | The % of organizations that collect outcome data | The # of organizations that collect outcome data | The total # of SCI organizations | |
On-going Training | The % of organizations that provide continued/on-going training | The # of organizations that offer on-going training | The total # of organizations that offer training | |
Monitoring Practices | The % of organizations that monitor their mentor/mentee relationships | The # of organizations that track mentor-mentee interactions | The total # of SCI organizations | |
Provincial Level
Cost of Service | The % of money dedicated to peer mentoring | The amount of money allocated for peer mentorship program | Total amount of money in operation budget of an organization | |
Cost per member | Operational budget cost per mentor | The amount of money allocated for peer mentorship program | Total number of registered mentors in an organization | |
Peer mentors as intended | The % of peer mentors in the organization as intended | The total # of registered peer mentors in an organization | The # of peer mentors an organization wants available | |
Program efficacy | The % of mentors who actually end up mentoring | The # of mentors who have mentored someone | The # of mentors who have been trained | |
Staff Level
Peer mentoring staff | The % of staff dedicated to peer mentoring | The # of staff employed for peer mentoring (FTE equivalent) | The # of staff employed by an organization | |
Peer mentoring staff as intended | The % of staff in an organization dedicated to peer mentoring as intended | The # of staff employed for peer mentoring (FTE equivalent) | The # of staff (FTE equivalent) an organization wants available for peer mentoring | |
Peer mentoring volunteers | The % of volunteers dedicated to peer mentoring | The # of volunteers dedicated to peer mentoring | The # of volunteers employed by an organization | |
Maintenance
Setting Level
Long term sustainability | The long-term sustainability of the program | How long the program has been operating | ||
Individual Level
Growth of peer programming – Mentor | The growth of the peer mentoring program within an organization | The # of registered peer mentors in the organization | The # of registered peer mentors in the organization 5 and 10 years ago | |
Growth of peer programming – Mentee | The growth of the peer mentoring program within an organization | The # of people who have received mentorship in the past year | The # of people who received mentorship 5 and 10 years ago |
Telephone interview
Data analysis
Results
RE-AIM analysis of SCI peer mentorship programs
Reach
Effectiveness
Adoption
Implementation
Maintenance
Discussion
Operationalizing the RE-AIM framework to evaluate multiple programs
Evaluating the impact of SCI peer mentorship programs
Reach
Effectiveness
Adoption
Implementation
Maintenance
Recommendations for using the RE-AIM framework
-
Involve stakeholders from all participating organizations throughout the entire research process. Conceptualizing indicators that are reflective of the available data can only be accomplished by understanding the data monitoring methods of each organization. Furthermore, continuing to engage stakeholders during data collection can help inform qualitative follow-up questions to ask about the data that are being collected which will help facilitate a deeper understanding of results. Engaging with stakeholders will also allow you to facilitate conversation between organizations which could lead to collaborative health promoting initiatives between organizations. For example, organizations that serve people with chronic conditions (e.g. heart disease, diabetes) may choose to deliver similar health promoting initiatives (e.g. awareness runs) simultaneously to a) increase the potential reach of the initiative and b) reduce the financial cost to deliver the event.
-
Understand the funding model used by each organization and how it impacts their programming. Every non-profit organization will have its own unique funding model that typically includes funding from a variety of sources (e.g., donations, grants, fundraising). The outcomes and outputs that an organization delivers and monitors are often dictated to them based on the funding they receive. Thus, the data that are collected and the programs that individual organizations provide will likely vary. Having a complete understanding of the funding model for all participating organizations will aid in the development of indicators that are universal across those organizations. This recommendation is imperative if using RE-AIM to evaluate health promotion programming delivered by a combination of non-profit and for-profit organizations. For example, a for-profit gym may have more funding and staff available to track specific outcomes for a certain exercise class compared to a non-profit gym that may not have the resources to collect robust data. Understanding these funding differences would be crucial to develop indicators that are meaningful and relevant for both exercise programs.
-
Include indicators that may only be applicable to some programs. Evaluating the cumulative impact of several programs would require that all organizations collect the exact same data. As this may not always be the case, it is important not to exclude indicators that could still be used to evaluate the impact of some of the programs. For example, one organization may evaluate reach by examining the number of new members in a given year while the others may evaluate reach based on the number of resources distributed to the community. These sets of data could not be amalgamated, thus restricting our ability to evaluate the cumulative impact of these programs. However, the results from both reach indicators could still be compared, and a further understanding of why programs collect different data could be achieved through qualitative methods. Unique findings from one organization may be insightful for others about how they can improve their reach, monitor effectiveness, or implement their health promoting programs more effectively.
-
When conceptualizing indicators for each RE-AIM dimension it is important to not only consider individuals that may be receiving services from a program but also those who are providing the program, especially if they are members of that community organization. Careful consideration must be given when conceptualizing indicators to ensure that all potential “participants” are considered. For example, when using RE-AIM to evaluate a community-based summer camp as Burke and colleagues did [18], it may be prudent to conceptualize reach indicators for camp counsellors in addition to campers, especially if counsellors belong to the same organization.