A mixed-methods evaluation of the MORE^OB program in Ontario hospitals: participant knowledge, organizational culture, and experiences

We used a descriptive pre-post repeated measures design to assess the effect of the MORE^OB program on participants’ knowledge about obstetrical core content contained in the MORE^OB program and to assess the effect of the MORE^OB program on participants’ perceptions of their organizational culture.

We used data from two existing surveys administered routinely to all participants by Salus Global as part of involvement in the MORE^OB program:

The knowledge test and CAS were administered at four time points; pre-MORE^OB, post-Module 1, post-Module 2, and post-Module 3 (Fig. 1).

As a standard component of participation in the MORE^OB program, all MORE^OB participants from the 26 hospitals were asked to complete a knowledge test and the CAS at four time points while progressing through the program. In addition to participant scores, the database from Salus Global contained information about job title, hospital site, and a unique identifier which allowed for participant responses to be linked over time. Only those participants who completed the knowledge test and/or CAS at all four time points were included in the final analysis.

MORE^OB participants at the 26 sites completed the standardized knowledge-based assessment tool and CAS at each of the four time points as they progressed through the program. Data were collected by Salus Global on their MORE^OB secure on-line platform and released to the research team for analysis.

We summarized items from the baseline questionnaire and after each of the three modules using descriptive statistics (means and standard deviations, or frequencies and percentages, as appropriate). Changes in mean scores across the four time points were assessed using linear mixed-effects regression analysis. Repeated measures were accounted for using an unstructured covariance pattern matrix, and clustering at hospital sites was accounted for using a random intercept. Least square means and pairwise comparisons were assessed at the 1% level. All analyses were carried out using SAS v. 9.4.

We used a qualitative descriptive design [22, 23]. This design aligned with our evaluation objectives to provide an accurate and comprehensive summary of participants’ perceptions of the effect of the MORE^OB program on health care provider knowledge, organizational culture, and barriers and facilitators to implementing and participating in the MORE^OB program [22, 24].

A 14-question semi-structured interview guide (Additional file 2) was used to gather in-depth information about interprofessional team experiences with the MORE^OB program. The research team reviewed the initial draft interview guide, with modifications made to increase comprehensiveness and clarity (e.g., adding additional probes to elicit information on more concepts, changing wording to make the questions more easily understood). Minor modifications to the guide were made throughout the qualitative data collection period to reflect the timing of the interview in relation to program implementation and to probe new themes emerging from previously conducted interviews.

We used a purposeful sampling approach to recruit interview participants [20]. Specifically, individuals were eligible to participate in an interview if they were (1) health care providers (i.e., nurse, family physician, obstetrician, midwife) and/or administrators/leaders (i.e., nursing or medicine); and (2) participants in the MORE^OB program and/or involved in implementing the program as a member of the core team at one of the 26 sites during the two interview recruitment periods (January–April 2015 and April–July 2016).

Permission from each of the 26 hospitals was obtained for Salus Global to release the names and contact information of the MORE^OB core team chairs to the research team for the purpose of interview participant recruitment. Interviews were conducted at two time points (post-Module 2 and post-Module 3) to capture the variety of experiences as the MORE^OB program progressed. At each of the two time points, an email with an attached recruitment poster was sent by the research team to the MORE^OB core team chair at each of the 26 hospitals, with instructions to circulate the invitation to all team members participating in the MORE^OB program at their site. Interested participants were instructed to contact the principal investigator to learn more about the study, ascertain eligibility, and schedule a time for an interview. Reminders were sent by the research team to the MORE^OB core team chairs to maximize responses. Individuals could participate in interviews at one or both time points. We welcomed participants who volunteered to be interviewed a second time as an opportunity to learn about differences they experienced between modules.

While in qualitative research there are no standardized rules for sample size, we used the concept of information power for guidance [25]. For example, our study aims were relatively narrow (we aimed to explore key dimensions and implementation of the MORE^OB program), our purposive sample of participants were specific to our study aims (all had experience participating in and/or implementing the MORE^OB program), and we anticipated rich dialogue with participants due to the use of interviewers experienced in both qualitative research and the clinical practice area. These factors, in combination with our experience from past provincial program evaluations, led us to predict that we would need up to 20 participants to provide sufficient information power to answer our research questions.

Two research staff (one midwife, one nurse) experienced in qualitative interviewing conducted the telephone interviews. Informed written consent was obtained prior to each interview. Interviews lasted an average of 34 min. The first round of interviews (n = 9) was conducted between March and April 2015 to learn about participants’ experience with modules 1 and 2 of the MORE^OB program. The purpose of conducting the first round of interviews after module 2 was to allow sufficient time for participants to be able to experience and report on any perceived changes to culture and clinical processes in their setting. The second round of interviews (n = 10) was conducted between April and July 2016 to learn about participants’ cumulative experiences after completion of module 3 of the MORE^OB program and plans for sustainability. Four health care providers were interviewed during both round one and two, meaning that there were a total of 15 unique interviewees. Interviewing, transcription, and analysis proceeded concurrently to permit follow-up of issues emerging from the data, and allow probing of emerging themes in subsequent interviews [22, 23, 26].

Digital recordings of the semi-structured interviews were transcribed verbatim. Data were imported into NVivo 11™ [27]. In alignment with our qualitative descriptive study design, we used conventional content analysis to code and analyze our data [22, 28, 29]. Initially, interviews were analysed independently by two study team members. We used an inductive process whereby the codes emerged from the data itself, rather than a pre-existing framework. Specifically, the two study team members independently read each transcript, followed by assigning preliminary codes (i.e., labels that represent the content of words and excerpts in the transcripts [30]). The two coders met every three to five transcripts to compare their coding, reach consensus, and develop and revise the coding scheme. All discrepancies were resolved through discussion between the two coders. As analysis progressed, codes were continually reviewed to identify similarities and differences, and conceptually similar codes were grouped into broader categories [30]. Lastly, we looked for common underlying meaning between categories and created broader themes and sub-themes, which contributed to answering “how” participants experienced the MORE^OB program [30]. The analysis of data collected in phase 1 was used to inform revisions to the interview guide for phase 2. For example, questions were added to elicit feedback on perceived differences between modules and on plans for sustainability of the program, and additional probes on key emergent themes such as health care provider confidence were included. Research team members met to discuss the coding scheme, emerging themes and sub-themes, and to build consensus regarding study findings.

An important focus of the MORE^OB program was improving communication strategies. Participants identified specific communication strategies learned through the MORE^OB program implemented in practice, including ‘Take Five’ and ‘SBAR’ (Situation, Background, Assessment, Recommendation), which were implemented during debriefing about cases.

Participants valued the knowledge itself gained from the MORE^OB program. MORE^OB was described as a comprehensive, time efficient, and accessible resource as all information is contained in one place, both in-print and online. In addition, participants perceived that the content of MORE^OB had been well-researched, was evidence-based, and relevant to the local context.

Participants described how participating in the MORE^OB program allowed all team members to share the same knowledge and language, put everyone “on the same page,” and gave everyone a common reference point.

All participants described how MORE^OB increased their knowledge base and that of their colleagues. For some, this increase in knowledge was a reminder of the importance of ongoing training to maintain current knowledge and skills. The MORE^OB program was described as being particularly helpful in a small hospital setting with low birth volumes where it can be difficult for new obstetrical nurses to gain sufficient experience in obstetrics.

Participation in MORE^OB resulted in the perception that the health care team was providing safer care to patients. Participants identified concrete changes made on their units due to their participation in MORE^OB. Examples of concrete changes included development or updating of policies and procedures, creating disaster carts with appropriate equipment, changes to clinical practices, posting algorithms on walls for clinicians to view, conducting case audits, and utilizing debriefing methods.

Participants described how MORE^OB changed the way they delivered care. First, participants described improved collaboration with their colleagues, which resulted in better ‘team work, openness, and transparency’, which ultimately improves safety for patients. For those sites who described pre-existing hierarchical relationships between obstetrical care providers, participants described a decrease in hierarchy within their team. However, a couple of sites described still struggling with hierarchy and were still working to change the culture at their sites. In addition, participants identified that participation in the MORE^OB program improved the patients’ experience by improving their collaboration with women and emphasizing the importance of patient centred care.

Generally, participants could not speak definitively as to whether their hospital’s participation in the MORE^OB program had led to improved outcomes such as decreased maternal and infant morbidity and mortality, as they did not have data to refer to during the interview. One site had these data and stated they observed a reduction in critical events since participating in MORE^OB. Overall, participants believed that the increased knowledge and concrete changes were ultimately improving care, outcomes, and patient experience.

Participants described the impact MORE^OB had on their self-confidence, particularly nurses who were described as being empowered through the MORE^OB program by increasing their knowledge and skills and improving their team communication. In addition, participants described an increase in their confidence in the knowledge, skills, and judgments of their team members. As MORE^OB participants improved communication, increased their knowledge base, and improved their ability to provide safer care, their confidence in themselves and others increased.

Participants described different strategies used for implementing the MORE^OB program, including use of huddle boards, staff emails and staff newsletters. In general, participants described following the MORE^OB activities as intended, with sites modifying the activities to make it more relevant for their setting or to fit it into limited time frames as needed. Some participants stated that the MORE^OB program plan was not clear, even after attending the orientation, and felt that additional detail on upcoming activities would be beneficial.

Participants described several influencing factors that affected how easy or difficult it was to implement the MORE^OB program in their context. For example, buy-in was increased when team members were interested in the content of the module and “hungry for new knowledge.” Participants also identified organizational-level barriers including limited time of staff and lack of funding to pay staff for their time to participate in the program, and a baseline organizational culture that was more resistant to change. Organizational-level facilitators included engaged hospital administration and the presence of MORE^OB champions who were highly committed to the implementation of the program.

Participants described varying levels of awareness regarding the sustainability of the MORE^OB program at their hospital once their three-year funding ended. Participants were keen to keep MORE^OB active at their hospital, and many sites were pursuing continued involvement, either formally with Salus Global, or when funding was not available, through internally-led education programs and skills drills. Regardless of the specific strategy, sustainability of the knowledge and skills gained through MORE^OB was identified as a priority.

Staff turnover and new hospital priorities were identified as ongoing challenges to sustainability of the program. As new people joined the unit, they had not participated in the initial MORE^OB program modules with the remainder of the team which presented challenges as the team was “storming and forming” all over again. A key challenge for sustainability of the program was cost, with participants perceiving that the program should be kept at a reasonable price to ensure institutions and providers can benefit from it.