Real-world effectiveness and factors associated with effectiveness of inactivated SARS-CoV-2 vaccines: a systematic review and meta-regression analysis

We searched without language restriction for studies published or preprinted by 31 August 2022 on inactivated SARS-CoV-2 vaccine efficacy or effectiveness in PubMed, Embase, Scopus, Web of Science, medRxiv, BioRxiv, and the WHO COVID-19 database VIEW-HUB website [23], which compiles searches of more than 100 databases. We searched for studies with multiple variations of the primary key search terms: [(“Effectiveness” OR “Efficacy” OR “Evaluation”) AND (COVID-19 OR SARS-CoV-2 OR Coronavirus) AND (Vaccine OR Vaccination) AND (CoronaVac or Vero Cell or BBIBP or WIBP or Inactivated). The full search strategy is shown in Additional file 1: Table S2. Additionally, the reference lists of the inactivated vaccine meta-analysis articles were hand-searched.

The selected studies met the following eligibility criteria: (a) observational study (prospective or retrospective cohort studies, case–control studies, and descriptive studies [mainly cross-sectional]); (b) assessing the effectiveness of inactivated SARS-CoV-2 vaccines (CoronaVac or BBIBP-CorV) to prevent SARS-CoV-2 infection or COVID-19-related hospitalization, severe/critical outcomes, and death; (c) reporting VE or related estimates from primary series or homologous booster vaccination at least 14 days after the last dose; and (d) with an unvaccinated reference group. We excluded randomized clinical trials, systematic reviews, and case series; we excluded studies that used only immunoglobulin M (IgM) antibody tests to diagnose COVID-19, serological studies, studies that did not report results of VE or estimates or data that can calculate estimates, and studies that only used a vaccinated reference group. Retrieved articles were exported to EndNote Reference Library, version X9.3.

Two independent reviewers (SX and JL) performed the data extraction and quality assessment, and discussed the discrepancies. A third investigator (HW) resolved the remaining discrepancies. We extracted and analyzed VE against SARS-CoV-2 infection and severe COVID-19. SARS-CoV-2 infection had to have been confirmed by reverse transcription-polymerase chain reaction (RT-PCR) or antigen testing in accordance with the WHO recommendations [24]; studies did not clarify the test method, but confirmed infection cases were also included. Due to differences in national policies for diagnostic testing, infection generally refers to testing after the onset of symptoms, but in countries like China that have large-scale RT-PCR testing, infection includes asymptomatic, test-positive individuals.

Estimates of COVID-19-related hospitalization, severe COVID-19 cases, COVID-19-related intensive care unit (ICU) admission, and death due to COVID-19 from articles were included in the severe COVID-19 outcome of our study. From included articles, severe COVID-19 cases generally defined according to WHO, by any of the following: (1) oxygen saturation < 90% on room air; (2) in adults, signs of severe respiratory distress (accessory muscle use, inability to complete full sentences, respiratory rate > 30; breaths per minute), and in children, very severe chest wall indrawing, grunting, central cyanosis, or presence of any other general danger signs (inability to breastfeed or drink, lethargy or reduced level of consciousness, convulsions) in addition to the signs of pneumonia [25]. Severe COVID-19 cases frequently require hospitalization and, if ventilatory support is required for acute respiratory failure, admission to the ICU. Additionally, if an article included more than one of the severe outcome subtypes and overlapped, we only chose the large-scale one to avoid double counting.

If an article specified the predominant SARS-CoV-2 variant during the study period or the specific variant in the study population, and if the variant was a VOC, we associated the reported VOC with the study estimates. If an article did not mention a predominant variant, had more than one predominant variant, or the variant was not a VOC, we classified the variant as “other.” Data in one article was of the WT strain, and the study showed that antibodies in vaccine-induced serum largely retained neutralizing response against Alpha [26], so we merged the evaluation of the WT strain with the Alpha strain data. If an article did not clearly state the time between vaccination and outcome, we estimated the duration from the start or completion of vaccination in the study setting, based on the information given in the article. Some studies reported vaccine effectiveness data on multiple groups; we combined relevant data in the same study according to the needs of meta-analysis to avoid double counting.

We extracted study data into a Microsoft Excel data extraction tool. Data were basic information, including title, first author, publication year, and study design; characteristics of the study population, including number of participants, country (for representation of regional characteristics and NPI use, we grouped countries by WHO region for the meta-analysis), population type, age range (populations were divided into different age ranges for meta-analysis, ± 5 years if the original age range was not completely specified); predominant SARS-CoV-2 variant during the study period; vaccination status, including vaccine brand, number of doses (primary [full series] and booster vaccination was defined as at least 14 days since two or three doses), and time since last dose; and vaccine effectiveness outcomes, including adjusted estimates, and their 95% confidence intervals (95% CIs).

We evaluated the risk of bias using the Newcastle–Ottawa Quality Assessment Scale for cohort and case–control studies. We assessed the quality of descriptive studies using a checklist recommended by the Agency for Healthcare Research and Quality (AHRQ) [27]. Cohort studies and case–control studies were classified as having low (7 scores), moderate (5–6 scores), or high (4 scores) risk of bias with an overall quality score of 9. For descriptive studies, we assigned values to each item of the AHRQ checklist with resulting scores that ranged from 0 to 11. We categorized these scores as low, moderate, and high risk of bias with scores of 8–11, 4–7, and 0–3, respectively.

We used DerSimonian and Laird random-effects models to calculate pooled estimates for subgroup analyses [28]: hazard ratios (HR), rate ratios (RR), or odds ratios (OR) with 95% CIs, comparing SARS-CoV-2 infection or severe COVID-19 in primary and booster vaccinated participants against different variants by time since vaccination. For data reported as frequencies or proportions, estimates were calculated directly. Vaccine effectiveness was (1-pooled HR/RR/OR) × 100%, together with 95% CIs. For vaccine effectiveness of 100% in which 95% CIs were not estimable, or if there was no event in either group in a trial, we adjusted estimates and approximated 95% CIs using study data, adding 0.5 cases to each group [29]. Negative VEs with study bias after review were excluded. We evaluated publication bias using funnel plots, Begg’s test [30], and Egger’s test [31]. The trim and fill method was used to identify and correct funnel plot asymmetry arising from publication bias [32].

We conducted multiple meta-regression with restricted maximum likelihood (REML) estimation and Knapp-Hartung adjustment to identify the factors associated with VE after adjusting for other explanatory variables [33, 34]. We used an information theoretic approach based on Akaike’s Information Criterion with the small-sample correction (AICc) for model selection, a valid model selection method outperforms other conventional methods for multiple regression or multiple meta-regression [35, 36], models with the smaller Δ_i (the difference units between the minimum AICc value model and the ith model) were more likely to be the potential best models, Δ_i values close to 0 have a lot of empirical support, and in the rough range, 4–7 have considerably less support [37], simply dropping models with ∆AIC will probably discard useful models [38]. Thus, we used multimodel inference to examine all possible factors combinations [36], and selected the top potential best models based on Δ_i values ranged 0–4, then decided the best model among potential ones by the coefficient of determination (R²) value of meta-regression, which represents the percentage of between-study heterogeneity explained by the factors of the meta-regression model [39]. We also conducted sensitivity analysis by dropping a small fraction of data to assess the robustness of meta-regression results [40], including the estimates from the moderate risk of bias studies, and the outliers identified by Cook’s distance. Factors for model selection were converted into dummy variables, factors, and their reference groups including study region (Western Pacific Region), VOC (Alpha), time since vaccination (14–90 days), vaccine brand (BBIBP-CorV), vaccine doses (primary series), age range (18–59 years old), population type (general), and study design (cohort study). Detailed factor groups can be found in Additional file 1: Tables S3 to S4.

Analyses were performed using the Meta-Analysis of Stata v17.0 and R software v4.2.2; the “metafor,” “dplyr,” “EnvStats,” and “ggplot2” packages were used for model selection, meta-regression, and visualization. A significance level of 5% was used for subsequent analysis.

Additional file 1: Table S5 shows the detailed characteristics of the 51 included studies [41‐91], and Additional file 1: Tables S3 to S4 and S6 show the summary of the key characteristics of the included studies and VE evaluations. The total study population was 142,236,816 subjects. Among included studies, 60.78% (31/51) only used the RT-PCR test as the infection identification method, while 35.29% (18/51) used both RT-PCR and antigen test to identify infection, and 2 studies did not specify the identification method of confirmed infection cases. We extracted 151 VE evaluations, among which 65 were evaluations against SARS-CoV-2 infection and 86 were against severe COVID-19.

The quality assessment showed that the risk of bias in most of the studies (48 studies) was low. The other three studies had a moderate risk of bias. Additional file 1: Tables S7 to S9 shows the quality assessment results.

Meta-analysis of 44 evaluations of primary series VE against SARS-CoV-2 infection and 57 evaluations of primary series VE against severe COVID-19 during Alpha, Gamma, Delta, and Omicron periods showed that VE against infection varied by VOC and time since vaccination and that VE against severe COVID-19 was higher than against infection and always greater than 60% against each variant (Tables 1 and 2).

The pooled estimate of VE against SARS-CoV-2 infection was higher against Delta (52.8% [95% CI, 43.3 to 60.7%]) and Gamma (44.2% [95% CI, 39.4 to 48.6%]) than against Omicron (16.4% [95% CI, 9.5 to 22.8%]). VE 14–90 days after vaccination was 87.0% (95% CI, 86.0 to 88.0%) against Alpha; 46.2% (95% CI, 38.4 to 53.0%) against Gamma, remaining stable during 90–180 days (42.1% [95% CI, 37.3 to 46.5%]); 56.3% (95% CI, 38.2 to 69.1%) against Delta, decreasing to 22.2% (95% CI, 4.9 to 36.3%) and 20.9% (95% CI, 8.1 to 31.9%) beyond 90 and 180 days respectively; and 34.1% (95% CI, 24.0 to 42.8%) against Omicron, decreasing to 7.2% (95% CI, 2.3 to 11.7%) after 180 days.

The pooled estimate of VE against severe COVID-19 was 84.0% (95% CI, 57.0 to 94.1%) against Alpha, similar to VEs against Gamma (73.3% [95% CI, 64.7 to 79.8%]), Delta (69.4, [95% CI, 63.1 to 74.6%]), and Omicron (66.0% [95% CI, 61.4 to 70.1%]). VE against severe COVID-19 14–90 days after vaccination was 90.0% (95% CI, 88.3 to 91.5%) against Alpha; 77.5% (95% CI, 56.1 to 88.5%) against Gamma, remaining stable during 90–180 days after vaccination (77.0% [95% CI, 74.7 to 79.1%]); 78.7% (95% CI, 59.6 to 88.8%) against Delta, 56.2% (95% CI, 40.0 to 68.0%) and 56.8% (95% CI, 35.7 to 71.0%) beyond 90 and 180 days respectively; 59.9% (95% CI, 46.1 to 70.1%) for Omicron, remaining stable after 180 days post-vaccination (64.5% [95% CI, 55.2 to 71.9%]). More detailed results of the subgroup analysis are shown in Additional file 1: Tables S10 to S12.

Nine evaluations of booster dose VE against SARS-CoV-2 infection showed that VE was 65.2% (95% CI, 48.3 to 76.6%) against Delta, similar to the primary series VE against Delta in 14–90 days. Booster dose VE decreased to 20.3% (95% CI, 10.5 to 28.0%) against Omicron, similar to the primary series VE against Omicron in 14–90 days.

In 14 evaluations of booster dose VE against severe COVID-19, VE was 79.2% (95% CI, 71.7 to 84.7%) against Delta and 87.3% (95% CI, 77.8 to 92.7%) against Omicron, and booster dose was more effective than primary vaccination against Omicron variant (Table 3). Due to the later starting time of booster vaccination, no VE estimates more distant than 14 to 90 days after booster vaccination could be determined. Comparison of primary series and booster doses VE against different VOC and time since vaccination is shown in Fig. 2.

For VE against SARS-CoV-2 infection, the lowest AICc value of potential models was 118.462; we selected eight top potential best models based on the ΔAICc among 2⁸ possible models (Table 4). After comparing the R² values of potential best models, we found that model 6 was the best model with the highest R² of 33.4%, and its ΔAICc was 3.451 compared to the lowest AICc (model 1). Variables in the best model were “study region,” “VOC,” and “time since vaccination,” and meta-regression results showed that each of these variables had a statistically significant association with vaccine effectiveness (Table 5). VE against Omicron variant was significantly different from VE against Alpha, and the risk of infection after vaccination during Omicron predominant period was 2.963 times compared with Alpha (the exponentiation of correlation coefficient exp(b) = 2.963, P = 0.021) after controlling for study region and time since vaccination. Risk of infection based on time since vaccination was not significantly different 90–180 days since vaccination (P = 0.067) compared to 14–90 days but was 1.772 times higher when the time since vaccination was greater than 180 days (exp(b) = 1.772, P = 0.022). Since vaccine doses were not the factor related to the change of VE against SARS-CoV-2 infection, and distance after booster vaccination for included studies could only be determined up to 90 days, meta-regression results also showed that booster doses restored VE to that seen 14 to 90 days after the primary series. VE against infection in the Eastern Mediterranean region was significantly different (exp(b) = 0.437, P = 0.014) from Western Pacific Region.

For VE against severe COVID-19, the lowest AICc value was 169.355 among 2⁸ possible models; we selected three top potential best models based on the ΔAICc (Table 6). Model 1 was the best model with the highest R² of 56.9% and the lowest AICc. Variables in the best model were “vaccine doses,” “age range,” “study design,” “study region,” “population type,” and “VOC,” and they all had a statistically significant association with VE against severe COVID-19 (Table 7). After controlling for other explanatory variables, the risk of severe COVID-19 after the booster dose was decreased (exp(b) = 0.541, P = 0.001) compared with the primary series. Compared with Alpha, risks of severe COVID-19 after vaccination during Gamma (exp(b) = 2.195, P = 0.034), Delta (exp(b) = 2.672, P = 0.001), and Omicron (exp(b) = 3.121, P = 0.001) were higher. VE against severe COVID-19 in Western Pacific Region was significantly different from the region of the Americas (exp(b) = 1.589, P = 0.034) and the European region (exp(b) = 2.587, P < 0.001).

Based on visual inspection of the funnel plot (Additional file 1: Figs. S1 to S2), and the results of Egger’s test for small-study effect, we found asymmetry for VE against SARS-CoV-2 infection (P < 0.001) and severe COVID-19 (P = 0.008). However, the results of Begg’s test for small-study effect showed no significant publication bias against SARS-CoV-2 infection (P = 0.865) or severe COVID-19 (P = 0.351). The results did not change after a trim and fill test, indicating that the impact of bias was likely not significant.

After sensitivity analysis (Additional file 1: Tables S13 to S16), meta-regression results did not change except for the model of VE against SARS-CoV-2 infection, factor “study region” related to the VE before it became insignificant after deleting the moderate risk of bias estimate, but results of other factors did not overturn or reverse, indicating the overall robustness of our meta-regression results.