Introduction
Methods
Eligibility criteria for studies
Study design
Population
Outcome
Intervention
Search strategy
Data collection and analysis
Study selection
Data extraction
Quality assessment
Data synthesis
Results
Study setting
D.1.1: Number of populations included | Frequency |
---|---|
Single (country, state, city, etc.) | 118 (41%) |
Multiple (countries, states, cities, etc.) | 167 (59%) |
D.1.2: Level of populations included | |
---|---|
National (country-level) | 117 (41%) |
Subnational (e. g. state-level) | 71 (25%) |
Both national and subnational (country- and e. g. state-level) | 97 (34%) |
D.1.3: Geographic areas covered\(^{\ddag }\) | |
---|---|
Asia | 144 (51%) |
Europe | 109 (38%) |
North America | 91 (32%) |
Middle East and Africa | 49 (17%) |
Central and South America | 46 (16%) |
Oceania | 42 (15%) |
D.1.4: Number of countries covered | |
---|---|
Multiple countries | 66 (23%) |
Single country (including multiple populations from a single country)\(^{\ddag }\) | 219 (77%) |
China | 54 (25%) |
United States | 43 (20%) |
India | 11 (5%) |
Italy | 11 (5%) |
Other | 100 (46%) |
D.1.5: Study period | |
---|---|
Start and end date span first epidemic wave | 161 (56%) |
One or more exceptions\(^{\ddag }\) | 124 (44%) |
End date in growth phase of wave | 44 (35%) |
Same end date for several populations with diverse epidemic trajectories | 38 (31%) |
End date at peak of wave | 16 (13%) |
End date could not be evaluated | 14 (11%) |
Other | 14 (11%) |
Population
Study period
Outcome
Raw outcome
D.2.1: Raw outcome\(^\ddag\) | Frequency |
---|---|
Epidemiological population-level outcome\(^\ddag\) | 223 (78%) |
Confirmed cases | 186 (83%) |
Deaths | 64 (29%) |
Recovered cases | 20 (9%) |
Hospitalizations | 18 (8%) |
Surrogate disease outcome | 10 (4%) |
Other | 24 (11%) |
Epidemiological individual-level outcome\(^\ddag\) | 23 (8%) |
Individual cases | 11 (48%) |
Individual cases and transmission chains | 8 (35%) |
Genome sequence data | 4 (17%) |
Behavioral outcome\(^\ddag\) | 55 (19%) |
Mobility | 50 (91%) |
Survey responses | 6 (11%) |
D.2.2: Time resolution of raw outcome | |
---|---|
Daily | 269 (94%) |
Other (weekly, biweekly, monthly, or not applicable) | 16 (6%) |
D.2.3: Computed outcome\(^\ddag\) | |
---|---|
None (only raw outcomes used) | 150 (53%) |
Measure of epidemic trend\(^\ddag\) | 34 (12%) |
Growth rate | 24 (71%) |
Doubling time | 11 (32%) |
Other | 1 (3%) |
Epidemiological parameter\(^\ddag\) | 89 (31%) |
Reproduction number | 78 (88%) |
Transmission rate | 6 (7%) |
Other | 16 (18%) |
Summary statistic | 8 (3%) |
Change points | 7 (2%) |
Other | 9 (3%) |
D.2.4: Method to obtain the computed outcome | |
---|---|
None (no computed outcome) | 150 (53%) |
One or several methods used\(^\ddag\) | 135 (47%) |
Simple computation (e. g. ratio, sum etc.) | 35 (26%) |
Exponential growth model | 11 (8%) |
Compartmental transmission model | 35 (26%) |
Statistical estimation of reproduction number | 43 (32%) |
Other | 29 (21%) |
D.2.5: Data source\(^\ddag\) | |
---|---|
Could not be evaluated | 10 (4%) |
Data from (sub)national authorities | 141 (49%) |
Data from publicly available cross-country selections | 77 (27%) |
Mobility data from corporate organizations | 40 (14%) |
Other | 54
(19%) |
D.2.6: Data availability | |
---|---|
Data access via source | 173 (61%) |
Data made available by the authors | 76 (27%) |
Data not accessible | 36 (13%) |
Computed outcome
Method to obtain the computed outcome
Data source
Data availability
Intervention
Terminology for non-pharmaceutical interventions
D.3.1: Terminology for interventions\(^{\dagger \ddag }\) | Frequency |
---|---|
Not applicable (only specific term for intervention type) | 22 (9%) |
Measures | 135 (54%) |
Interventions | 65 (26%) |
Policies | 16 (6%) |
Other | 14 (6%) |
D.3.2: Terminology for the specific type of non-pharmaceutical interventions\(^{\dagger \ddag }\) | Frequency |
---|---|
Not applicable (only general term for interventions) | 3 (1%) |
Non-pharmaceutical | 49 (16%) |
Control | 48 (16%) |
Social distancing | 45 (15%) |
Other | 159 (52%) |
D.3.3: Exposure types | |
---|---|
One single intervention | 43 (15%) |
Multiple separate interventions | 31 (11%) |
One combination of interventions | 84 (29%) |
Multiple combinations of interventions | 20 (7%) |
All interventions together | 70 (25%) |
Other | 37 (13%) |
D.3.4: Types of single interventions | |
---|---|
Not applicable (no single interventions analyzed) | 211 (74%) |
One or multiple single interventions analyzed (as defined in D.3.4 of the Documentation manual)\(^{\ddag }\) | 74 (26%) |
Stay-at-home order | 44 (59%) |
Other | 27 (36%) |
School closure | 25 (34%) |
Workplace closure | 20 (27%) |
International travel restrictions | 17 (23%) |
Declaration of a state of emergency | 13 (18%) |
Bans of large gatherings | 13 (18%) |
Venue closure | 12 (16%) |
Bans of small gatherings | 10 (14%) |
D.3.5: Coding of interventions | |
---|---|
D.3.6: Source of intervention data | |
Not applicable (no specific interventions analyzed) | 74 (26%) |
Not necessary (no joint analysis of interventions across multiple populations) | 137 (48%) |
Could not be evaluated | 98 (72%) |
Government or news websites | 30 (22%) |
Other | 9 (7%) |
Necessary (joint analysis of interventions across multiple populations) | 74 (26%) |
Could not be evaluated | 9 (12%) |
Coding done by authors | 20 (27%) |
Use of externally coded data | 45 (61%) |
D.3.7: Availability of data on exposure | |
---|---|
Not applicable (no specific interventions analyzed) | 73 (26%) |
Raw data documented in the manuscript | 136 (48%) |
Access to externally coded data via source | 32 (11%) |
Coded data
made available by the authors | 34 (12%) |
Coded data not available | 10 (4%) |
Exposure types and types of single interventions
Coding of interventions
Source of intervention data and availability of data on exposure
Methodological approach
D.4.1: Empirical approach | Total freq. | |||
---|---|---|---|---|
D: Descriptive | 151 (53%) | |||
P: Parametric | 94 (33%) | |||
C: Counterfactual | 40 (14%) |
D.4.2: Use of exposure variation | (D) | (P) | (C) | |
---|---|---|---|---|
Only variation over time for a single population | 78 | 23 | 24 | 125 (44%) |
Only variation over time for multiple populations | 63 | 22 | 10 | 95 (33%) |
Only variation between populations | 4 | 14 | 0 | 18 (6%) |
Both variation over time and between populations | 6 | 35 | 6 | 47 (16%) |
D.4.3: Method | ||||
---|---|---|---|---|
Description of change over time | 136 | — | — | 136 (48%) |
Description of time course | 49 (36%) | |||
Comparison of time periods | 87 (64%) | |||
Comparison of populations | 8 | — | — | 8 (3%) |
Comparison of change points with intervention dates | 7 | — | — | 7 (2%) |
Non-mechanistic model | — | 61 | 17 | 78 (27%) |
Generalized linear model | 51 (65%) | |||
Exponential growth model | 11 (14%) | |||
Other | 16 (21%) | |||
Mechanistic model | — | 30 | 13 | 43 (15%) |
Compartmental single-population transmission modl | 29 (67%) | |||
Compartmental meta-population transmission model | 4 (9%) | |||
Semi-mechanistic Bayesian transmission model | 5 (12%) | |||
Other | 5 (12%) | |||
Synthetic controls | — | — | 6 | 6 (2%) |
Other | 0 | 3 | 4 | 7 (2%) |
D.4.4: Code availability | ||||
---|---|---|---|---|
None (not available) | 121 | 66 | 33 | 220 (77%) |
Publicly available | 30 | 28 | 7 | 65 (23%) |
Empirical approach
-
(D) Descriptive approaches were used by the majority of analyses: These approaches provided descriptive summaries of the outcome over time or between populations, and related variation in these summaries to the presence or absence of different interventions. For example, some analyses compared changes in the growth rate of observed cases before and after interventions were implemented [55, 56]. Of note, descriptive approaches could involve modeling as part of an intermediate step, where a latent outcome was computed from the raw outcome (see Computed outcome), while, afterward, a descriptive approach was used to the link the latent outcome to interventions. For example, some analyses used a single-population compartmental transmission model to estimate the time-varying reproduction number and then compared the reproduction number before and after interventions were implemented [57‐59].
-
(P) Parametric approaches were used by a third of analysis: These approaches formulated an explicit link between intervention and outcome, where the association was quantified via a parameter in a model. Most frequently these were regression-like links between interventions and the reproduction number [2, 8].
-
(C) Counterfactual approaches were least frequently used: These approaches assessed the effectiveness of interventions by comparing the observed outcome with a counterfactual outcome based on an explicit scenario in which the interventions were not implemented. For example, the observed number of cases was compared with the number of cases that would have been observed if the exponential growth in cases had continued as before the implementation of interventions [60, 61].
Use of exposure variation
Method
Code availability
Effectiveness assessment
D.5.1: Reporting of effectiveness | Total freq. | |||
---|---|---|---|---|
QS: Qualitative statement | 53 (19%) | |||
CO: Comparison of outcome values | 73 (26%) | |||
QC: Quantification of change in outcome values | 159 (56%) |
D.5.2: Measure of effectiveness\(^\ddag\) | (QS) | (CO) | (QC) | |
---|---|---|---|---|
Change in reproduction number | 22 | 44 | 29 | 95 (33%) |
Change in confirmed cases | 16 | 15 | 38 | 69 (24%) |
Change in mobility | 9 | 6 | 28 | 43 (15%) |
Other | 18 | 29 | 100 | 147 (52%) |
D.5.3: Interpretation of results | ||||
---|---|---|---|---|
Associative | 111 (39%) | |||
Implicitly causal | 160 (56%) | |||
Explicitly causal | 14 (5%) |
D.5.4: Reporting of uncertainty | ||||
---|---|---|---|---|
Not applicable | 52 (18%) | |||
Yes | 154 (54%) | |||
No | 79 (28%) |
D.5.5: Sensitivity analysis (including computed outcomes) | ||||
---|---|---|---|---|
None (no sensitivity analyses w.r.t effect) | 217 (76%) | |||
One ore more sensitivity analyses\(^\ddag\) | 68 (24%) | |||
Model specification varied | 36 (53%) | |||
Epidemiological parameters varied | 29 (43%) | |||
Different or modified outcome used | 17 (25%) | |||
Same analysis with (sub)population excluded | 16 (24%) | |||
Different coding of interventions used | 10 (15%) | |||
Start or end date of study period varied | 4 (6%) |
D.5.6: Subgroup assessment | ||||
---|---|---|---|---|
None (no subgroups) | 250 (88%) | |||
One or more subgroups\(^{\ddag }\) | 35 (12%) | |||
Based on socioeconomic indicators | 23 (66%) | |||
Based on epidemiological indicators | 16 (46%) | |||
Based on public health response | 9 (26%) | |||
Based on geographic areas | 6 (17%) |
Reporting of effectiveness, measure of effectiveness, and reporting of uncertainty
Interpretation of results
Sensitivity analyses
Subgroup assessment
Discussion
Implications for future work
Recommendations for improving methodologies
Recommendations for improving comparability across studies
Box 1. Different types of analyses to assess the effects of non-pharmaceutical interventions | |
---|---|
(1) Observed outcome directly linked to interventions | A raw, observed outcome is analyzed directly by evaluating differences (1) over time with an interrupted time-series analysis comparing the outcome before vs. after an intervention, (2) between populations with a cross-sectional analysis comparing populations exposed vs. not exposed to an intervention, or (3) both over time and between populations with a panel data analysis. Mechanistic modeling is typically not involved in this type of analysis, with one exception, namely counterfactual approaches using a transmission model to project the observed outcome after intervention. |
(2) Computed, unobserved outcome linked to interventions | In contrast to type (1), the intervention effect is measured in terms of an unobserved outcome. This is computed from the raw outcome and then analyzed in a similar manner as in (1). Mechanistic modeling can be involved in computing the unobserved outcome, for example by using a model to estimate the reproduction number or transmission rate from the number of new cases. |
(3) Observed outcome linked to interventions via unobserved outcome in mechanistic model | Observed outcomes are used to fit a mechanistic model (e. g. compartmental transmission model) that includes a latent variable representing an unobserved outcome (e. g. the reproduction number), which in turn is parameterized as a function of interventions. For instance, a regression-like link is used within the mechanistic model to estimate the effect of interventions on the transmission rate as a latent variable. |
(4) Change points in outcome related to exposure | Change points are estimated in the time series of an observed or unobserved outcome. The estimated change points are then related to the implementation dates of interventions. If the estimated change points agree well with the actual implementation dates of interventions, this is interpreted as evidence for the effectiveness of interventions. |