Health worker education during the COVID-19 pandemic: global disruption, responses and lessons for the future—a systematic review and meta-analysis

We conducted a systematic review and meta-analysis in accordance with the Measurement Tool to Assess Systematic Reviews-2 (AMSTAR-2) checklist [11] and the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) 2020 statement [12], based on a predesigned protocol registered with PROSPERO (CRD42021256629) [13].

We searched the MEDLINE (via PubMed), EMBASE, Web of Science, and CENTRAL databases, as well as ClinicalTrials.gov and Google Scholar (first 300 records) for randomized controlled trials (RCTs) or observational studies published from 1/1/2020 to 31/07/2022 in English, French or German (full search strategy available in Additional file 1). A snowball approach was also employed.

Our eligible population included Health Worker (HW) learners or faculty, as defined by the International Standard Classification of Occupations (ISCO-08) [14] group of health professionals, excluding veterinarians. Health care settings per the Classification of Health Care Providers (International Classification for Health Accounts, ICHA-HP) [15] and relevant educational settings (i.e., universities, colleges) were considered eligible. The included population was divided into undergraduate learners, postgraduate (e.g., residents or fellows) and continuing education (in-service) [16]. Any change(s) and/or innovation(s) that were implemented in health worker education in response to the COVID-19 pandemic (not before the COVID-19 pandemic or amidst other pandemics) were considered eligible. Online training methods were sub-divided into predominantly theoretical courses, courses with a practical component (i.e., practical skill, simulation-based training), congresses/meetings, interviews, and clinical experience with patients (i.e., clinical rotations/electives, telehealth-based training). Comparators included conventional/traditional practices existing prior to the pandemic.

The study outcomes are organized according to (1) impact of the COVID-19 pandemic on the educational process and mental health of learners; (2) policy responses (not included in the meta-analysis); and (3) outcomes of those policy responses (Table 1). Specific meta-analysis outcomes in the categories shown in Table 1 included: regarding axis 1, clinical training, mental health (i.e., anxiety, depression, insomnia and burnout), and learner career plan disruptions (e.g., redeployment), and concerning axis 3, satisfaction, preference and performance with new training and assessment modalities and volunteerism, including any social/community/institutional work. Regarding anxiety and depression, individuals whose symptom severity was deemed moderate or higher according to validated measurement scales were considered as affected. For the Generalized Anxiety Disorder-7 (GAD-7) and Patient Health Questionnaire-9 (PHQ-9) screening tools, this corresponded to a cut-off score of 10.

All retrieved records underwent semi-automatic deduplication in EndNote 20 (Clarivate Analytics) [17], and were then transferred to a Covidence library (Veritas Health Innovation, Melbourne, Australia) for title and abstract screening. Pairs of authors performed a blind scan of a random 15% sample of records. After achieving an absolute agreement rate > 95% (Fleiss’ kappa, 1st phase: 0.872, 99% confidence interval (CI) [0.846–0.898]; 2nd phase: 0.840, 99% CI [0.814–0.866]), single-reviewer screening was performed for the remainder of the studies, as per the AMSTAR-2 criteria [11]. Subsequently, pairs of independent reviewers screened the full texts of the selected studies for eligibility, and, if eligible, extracted the required data in a predetermined Excel spreadsheet. Screening and data extraction was carried out in two phases: the initial phase (1/1/2020 to 31/8/2021 by AD, ANP, M. Papapanou and MGS) and the updated living phase (1/9/2021 to 31/7/2022 by NRK, AA, DM, MN, CK, M. Papageorgakopoulou). After discussion with the WHO technical partner, we amended the extraction spreadsheet to further include descriptions of policies in the updated living phase. Satisfaction was extracted either from direct mentions of participants’ satisfaction by the authors or from questions surveying the participants’ perceptions on their satisfaction, the success, usefulness or effectiveness of the learning activity. Conflicts were resolved by team consensus. For missing data, study investigators were contacted. Studies for which the full text or missing data were unable to be retrieved were categorized as “reports not retrieved”. Studies on overlapping populations were also considered duplicates and subsequently removed if they related to the same study period and institution(s) and involved similar populations and author lines. The study with the most comprehensive report was retained.

Pairs of all aforementioned authors performed the risk of bias assessment, and any conflicts were resolved by team consensus. The quality assessment was performed using an adapted version of the Newcastle–Ottawa Scale (NOS) for cross-sectional studies (Additional file 1), the original NOS for cohort and case–control studies, and the Cochrane risk-of-bias (RoB2) tool (Version-2) for RCTs. Publication bias was explored with funnel plots and the Egger’s test [18]. Certainty of evidence was assessed using the Grading of Recommendations, Assessment, Development and Evaluations (GRADE) approach [19].

Categorical variables were presented as frequencies (%) and continuous variables as mean (standard deviation [SD]). To dichotomize ordinal data (e.g., Likert-type scales), we used the specific author provided cut-offs for the respective scales, or, if not provided, the 60th percentile (40th if the scale was reversed). Regarding mental health outcomes, we derived scale-specific cut-offs from the literature.

Analyses were carried out on learner and faculty population subsets separately. We carried out a meta-analysis of the Freeman–Tukey (FT) double-arcsine transformed estimates using the DerSimonian and Laird (DL) random-effects model [20‐22]. We used the harmonic mean in the back-transformation formula of FT estimates to proportions [23]. For each meta-analyzed outcome, we reported the raw proportion (%), pooled proportion (%) along with its 95% CI, the number of studies (n) and number of included individuals (N). When applicable, we pooled standardized mean differences (SMDs) with the method of Cohen [24]. Statistical heterogeneity was quantified by the I² [25], and was classified as substantial (I² = 50–90%) or considerable (I² > 90%) [26].

We performed subgroup analyses stratified by gender, continent, WHO geographical region, ISCO-08 occupational group, stage of training, and year of undergraduate studies, and computed p-values for subgroup differences (p_subgroup < 0.10 indicates statistically significant intra-subgroup differences) [26]. The potential effect of time on outcomes potentially exhibiting dynamic changes during the evolution of the pandemic, such as satisfaction and preference with learning formats, as well as mental health outcomes, was explored via additional subgroup analyses by year data collection was completed (2020 vs 2021 vs 2022). Only subgroups involving 3 or more studies are presented and taken into account for the p_subgroup calculation, so no subgroup analysis is presented for the 2022 study end year.

To better account for the anticipated substantial heterogeneity, two additional meta-analytical approaches were used: (i) the Paule–Mandel estimator to calculate the between-study variance [27]; and (ii) the Hartung–Knapp method for the CI calculation [28].

Statistical significance for all analyses was set at a two-sided p < 0.05. All analyses were conducted using aggregate data via the STATA software, version 16.1 (Stata Corporation, College Station, TX, USA). Further explanation of adopted statistical approaches is provided in Additional file 1.

The widespread disruption in undergraduate, graduate and continuing education of health workers due to closures and physical distancing has been clearly reported since the start of the pandemic [5]. There were references to complete or temporary cessation of in-person educational activities including classes and patient contact [29, 30]; and in many cases the temporary cessation of face-to-face learning, both pre-clinical and clinical. Especially for undergraduate learners, bedside education was initially halted to protect learners [31]. During residency training, the main disruptions identified were the reduction in case volumes [32, 33] especially in surgical training [34, 35], less time available for learners to spend in the hospital [36], or, conversely, increased workload, especially in COVID-related specialties. Other activities including in-person scientific conferences were discontinued [37]. Timely graduation was jeopardized [38], required examinations were canceled [39] and graduates were unable to apply for their next steps [40].

Several policy and management responses by governments, regulatory and accreditation bodies, schools, hospitals, clinical departments, health systems and student organizations were identified. A commonly cited response was the transition of face-to-face learning to online formats [44], including online videos [45], game-based learning [46], virtual clinical placements [34, 47‐49], virtual simulations [50], remote teaching of practical skills as well as augmented reality [51, 52]. Interviews also transitioned to virtual format after guidance by accreditation bodies [53, 54], and face-to-face conferences were replaced with large-scale virtual conferences [55]. There were also responses relating to online assessment [56].

COVID-19-specific learning was introduced in particular for in-service and postgraduate learners [57], such as workshops on the use of personal protective equipment (PPE) [58‐60] and simulations for COVID-specific protocols [61, 62]. Institutions published regulations and recommendations safeguarding learners’ health and continued learning [57, 63, 64], while there were interventions to specifically support learners’ mental health [65, 66]. Undergraduate learners were also involved in volunteering towards supporting the COVID-19 response [67, 68]. Another policy response was early graduation of final-year students who could work in a clinical capacity [69]. An overview of the institutions enacting these responses and policies as identified in the second phase of our systematic review is summarized in Table 6.

Our meta-analysis showed that 71% of learners reported their clinical training was adversely impacted by the pandemic. In a large study surveying medical students from South America, Japan and Europe, 93% of students reported a suspension of bedside teaching [70]. Trainees in surgical and procedural fields were severely affected, with 96% of surgery residents and early-career surgeons in the US reporting a disruption in their clinical experience, with an overall 84% reduction in their operative volume in the early phases of the pandemic [71]. Most included studies did not provide separate data on the type of surgery. In similar large-scale disruptions, achieving the difficult but crucial balance between patient and trainee safety with the necessary clinical training of health workers should be a priority for policymaking.

The extent of the impact on the mental health of learners is concerning and highlights the need for sufficient resources to support learners and faculty. Our meta-analysis revealed that about one in three learners suffered from at least moderate anxiety, depression, insomnia, or burnout. These appear to be higher than reported anxiety and depression in health workers, similar to the general population during the pandemic and similar to pre-pandemic studies. In an umbrella review of depression and anxiety among health workers (not learners) during the pandemic, anxiety and depression were estimated at 24.9% and 24.8%, respectively [72], although most meta-analyses also included mild forms of anxiety and depression. A different subgroup analysis estimated moderate or higher anxiety and depression in health workers at 6.88% (4.4–9.9) and 16.2% (12.8–19.9) [73], results much lower than ours. In the general population, anxiety and depression were estimated in one meta-analysis at 31.9% (27.5–36.7) and 33.7% (27.5–40.6) [74], similar to our estimate for health worker learners. Lastly, comparing our results with a 2018 meta-analysis, the prevalence of anxiety (33.7%, 10.1–58.9) and depression (39.2%, 29.0–49.5) might be similar among health learners before and during the COVID-19 pandemic [75], and warrants further study and policy interventions. Anxiety was significantly higher in 2021 studies compared to 2020, indicating a notable effect of persisting stressors on mental health and emphasizing the need for early intervention to prevent anxiety. Pharmacy learners were significantly more anxious, which may be associated with different backgrounds and levels of familiarity with the intense clinical environment at times of capacity, in comparison to medical and nursing colleagues.

Multiple studies showed female gender was a risk factor for increased anxiety and depression among health learners [71, 76‐79]. In studies that investigated underlying stressors, learners showed a high level of anxiety about their relatives’ health [41, 80‐83], getting infected with COVID-19 themselves [41, 80, 84, 85], lack of PPE [86, 87], failing their clinical obligations [88], the disruption of educational activities [89, 90], or financial reasons [88, 91, 92]. A UK study on the psychological well-being of health worker learners during the pandemic associated the educational disruption with a negative impact on mental health, estimating low well-being at 61.9%, moderate to high perceived stressfulness of training at 83.3% and high presenteeism at 50% despite high satisfaction with training (90%) [93]. Learners felt a lack of mental health resources and supports in some disciplines [93]. A US study found that lack of wellness framework and lack of personal protective equipment were predictors of increased depression and burnout in surgery residents and early-career surgeons, highlighting the importance of well-designed wellness initiatives and appropriate protection for learners [71]. A summary of protective and exacerbating factors identified from included studies is available in Table 16. An international study of medical students identified high rates of insomnia (57%), depressed mood (40%) as well as multiple physical symptoms including headache (36%), eye fatigue (57%) and back pain (49%) [70]. These important physical complaints were not included in our systematic review. Interestingly, time spent in front of a screen daily correlated positively with depression, insomnia and headache. Alcohol consumption declined during the pandemic, whereas cigarette and marijuana use was unchanged. Putting together these findings, trainees’ mental- and physical-health appears to be associated with multiple factors that should be targeted by policy interventions: gender disparities, lack of well-designed wellness frameworks, stressful training, lack of protective equipment and potential implications of increased screen time. It should be noted that variants of the MBI scale also tend to overestimate burnout rates [94], so these may be actually lower than reported by our study.

Learners’ satisfaction with the rapidly implemented policy of online learning was relatively high (76%), especially if it included patient contact or practical training, rather than a purely theoretical approach. However, although learners were relatively satisfied when the alternative was no education, their opinions seemed to change when presented with options for the future. Learners preferred face-to-face (49%) and blended (56%) over fully online education (32%). In addition, only a small percentage of students were willing to pursue an exclusively online learning format (35%) in the post-pandemic era, with their preference trending towards a blended model (68%). The “Best Evidence in Medical Education” series and other systematic reviews, including only studies published in 2020, showed that the rapid shift to online learning proved to be an easily accessible tool that was able to minimize the impact of early lockdowns, both in undergraduate and graduate education [105‐107]. Adaptations included telesimulations, live-streaming of surgical procedures and the integration of students to support clinical services remotely. Challenges included the lack of personal interaction and standardized curricula. All studies showed high risk of bias and poor reporting of the educational setting and theory [105]. Out meta-analysis of all relevant studies spanning from 2020 to mid-2022 showed that the integration of practical skill training into online courses led to higher satisfaction rates, solidifying a well-known preference for active learning among health workers. Satisfaction and preference for online learning was significantly increased in postgraduate and continuing learners compared to undergraduates, indicating it may be better suited for advanced learners with busy schedules. Higher convenience and ability to manage one’s time more flexibly and efficiently were frequently reported reasons for satisfaction and preference for online education [108‐111]. Ιn synchronous learning, interaction through interactive lectures or courses, quizzes, case-based discussions, social media, breakout rooms or journal clubs were associated with increased satisfaction [112‐116]. Conversely, in asynchronous learning, the opportunity for self-paced study and more detailed review of study material increased satisfaction [117‐119]. Limitations of online education included challenges in comprehending material in courses such as anatomy [120, 121], as well as lack of motivation among learners [122‐125]. A different systematic review found medical students appreciated the ability to interact with patients from home, easier remote access to experts and peer mentoring, whereas they viewed technical issues, reduced engagement and worldwide inequality were viewed as negative attributes of online learning [126]. Interestingly, one study comparing medical and nursing student satisfaction across India found high dissatisfaction (42%, compared to 37% satisfaction) which was not significantly different between the two fields, and higher in first-year students. Supportive faculty was important in increasing satisfaction [121].

We found that learners performed better in online assessments compared to prior in-person ones. It is unknown whether this represents lower demands, inadequate supervision, or changes in the constructive alignment between learning outcomes (e.g., theoretical knowledge) and assessment modality (e.g., multiple choice questions) [127]. However, online assessment has significant limitations in evaluating hands-on skills. Learners perceived online assessments as less fair, as cheating can be easier [128‐130], or felt unable to showcase their skills online [131]. Open-book assessments focusing on thinking instead of memorization were preferred by learners [132] and may be more appropriate for online assessment. A different systematic review including studies up to October 2021 reviewed adaptations in in-person and online clinical examinations of medical students. Overall, online or modified in-person clinical assessment was deemed feasible, with similar scores to prior in-person iterations, and well received by trainees [133].

Although 62% of learners reported a willingness to volunteer, one in three actually did. This could be due to health risks, lockdowns, lack of opportunity or time, or other factors. As expected, undergraduates had more time to actually volunteer than other groups, however willingness to volunteer was comparable between the different training levels. These activities made heavy use of technology and frequently involved telephone outreach and counseling of patients and the public [134‐137]. Students were also employed clinically in hospitals or other settings [138] and assisted with food and PPE donation and other nonclinical activities such as babysitting [139]. Some accrediting institutions responded by recommending that volunteering activities be rewarded with academic credit and supervised adequately [140].

To our knowledge, this is the largest systematic review and meta-analysis exploring the impact of the pandemic on the education and mental health of health worker learners. The vast amount of data allowed us to perform multiple subgroup analyses and explore the potential differences in training disruption, mental health and perceptions on educational innovations. We included health worker learners from all regions of the world, all occupations, and all levels of training. We also undertook sensitivity analyses by restricting our analyses to a homogenous sample of higher quality studies (e.g., by only pooling GAD-7/PHQ-9/MBI low risk of bias studies for anxiety/depression/burnout). These approaches demonstrated the robustness of our findings. Finally, we attempted to explore the effect of time on outcomes, given the dynamic character of the pandemic.

Although we excluded duplicate publications, there is still a risk for overlap, as learners may have participated anonymously in multiple cross-sectional studies. We attempted to minimize this with sensitivity analyses excluding very large datasets. Second, satisfaction was extracted from a variety of definitions among different studies leading to considerable heterogeneity. While prior experience with virtual learning might have affected learners’ or faculty perceptions, its inconsistent reporting did not allow us to account for it. For similar reasons, we did not manage to quantify mild mental health disruption for anxiety and depression. Although multiple significant subgroup differences emerged, heterogeneity remained largely unresolved. Heterogeneity is inherently high in meta-analyses of proportions, and the large sample of studies along with the subjective nature of many outcomes are in part responsible. The precision in point estimates (i.e., the observed narrow CIs) is therefore mainly a consequence of the large sample rather than true low variation. Therefore, we advise cautious interpretation and assess all our outcomes as very-low-certainty of evidence. Our sample mainly represented undergraduate students, learners in medicine and Asia, with reduced representation from Africa, South America and Oceania. Therefore, our results should be generalized with caution. However, subgroup analyses provide some insight into intra-group differences. Last, the authors were unable to include studies published in Spanish, which may in part reflect the scarcity of included studies from South America. We did, however, include studies in German and French.

Quality assessment revealed mostly observational studies and self-reported outcomes. RCTs were scarce and a considerable subset of them at high risk of bias. Publication bias was also evident in one-fourth of our analyses, leading to potential overestimation of proportions (e.g., higher satisfaction may be reported more eagerly). The above are consistent with the challenge in the education literature, which tends to capture mostly Kirkpatrick Level 1 data [141] (learner reaction), instead of objective learning assessments or behavioral changes. However, at the early stages of the pandemic, the literature is more likely to include lower-level immediate outcomes. Future studies will likely capture more objective outcomes and similar reviews should be repeated. Educational experiences are difficult to standardize and measure, making strict evidence-informed practice difficult [142]. However, quantitative evidence of any form can be a significant contributor to policy change.