Protocol for a systematic review of the effects of interventions for vaccine stock management

We aim to assess the effects of approaches used for vaccine stock management at facility level.

We registered the systematic review in the International Prospective Register of Systematic Reviews (PROSPERO) [15].

We will include individually randomised trials, cluster randomised trials, controlled before-after studies, interrupted time series studies and repeated cross-sectional studies. Eligible participants include the healthcare systems which deliver vaccines, healthcare facilities where vaccines are administered, healthcare workers involved in providing immunisation services and recipients of immunisation services.

We plan to include interventions targeting recipients or providers of immunisation services. Recipient-oriented interventions may include the involvement of end-users in monitoring vaccine availability at facilities, e.g. using mobile phone services or hotline platforms. Examples of interventions directed at providers of immunisation services include education or training, audit and feedback, prompts or reminders and supportive supervision. We will also include interventions targeting the health system offering immunisation services, e.g. action plans, re-designing (components) of the supply chain and integration with other services. Other interventions intended to ensure vaccine availability, including multi-component interventions, are also eligible for inclusion. We will consider the following as eligible comparisons: standard vaccine stock management practices in the study setting, alternative interventions and similar interventions implemented with different degrees of intensity.

Our primary outcomes are vaccine availability and vaccine stock-outs. We will measure vaccine availability as the proportion of vaccination days in which vaccines were available and no one eligible for vaccination was turned back for lack of vaccines; but will also consider other measures of vaccine availability used by the authors of included studies. Vaccine stock-out rates in the review will be measured as the percentage of facilities that experienced a stock-out of a specific vaccine that the site is expected to provide, at any point, within a defined period; or other definitions as used in included studies. Our secondary outcomes include acceptability, adverse events and cost of the intervention, as well as other outcomes as reported by included studies.

We will develop a comprehensive search strategy for both peer-reviewed and grey literature. We will search the following databases PubMed, Cochrane Central Register of Controlled Trials (CENTRAL), Embase, WHO Library Information System (WHOLIS), Web of Science, PDQ (Pretty Darn Quick)-Evidence and Scopus. We will also search the websites of WHO, Global Alliance for Vaccine and Immunisation, PATH Vaccine Resources Library and UNICEF. In addition, we will search the WHO International Clinical Trials Registry Platform, reference lists of included studies and related systematic reviews and citations of included studies. A preliminary search strategy developed for PubMed is found in the Appendix.

Two authors will independently screen the titles and summaries of records retrieved from the search for potentially eligible studies. We will obtain full-texts for all the potentially eligible studies. Two authors will assess these full-text publications for eligibility. Any disagreements between the two authors regarding study eligibility will be resolved by discussion and consensus. A third author will arbitrate any unresolved disagreements. We will provide a table with the characteristics of the included studies, and another of excluded studies with reasons for their exclusion. We will seek additional information, for studies with missing information, to assist us in our decision-making process.

For each included study, two authors will independently extract information using a piloted data extraction form. Extracted data will include study design, participant, intervention and outcome characteristics as well as outcome data. Any differences will be resolved through discussion and consensus. A third author will be consulted to arbitrate if disagreements persist between the two authors. If there are missing data, we will contact study investigators to obtain the missing information. Two authors will independently assess risk of bias in included studies using the appropriate tool for randomised trials [16] and non-randomised studies [17]. Differences in judgement will be resolved by discussion and consensus, with arbitration by a third author.

We will present study results as risk ratios for dichotomous data (e.g. frequency of vaccine stock-outs), and mean differences for continuous data (e.g. duration of vaccine stock-outs) will be presented as mean difference. We will combine data from clinically homogenous studies (in terms of designs, participants, interventions and outcomes) using random-effects meta-analysis. However, if we come across variation between studies, the findings will be summarised in a narrative format. We will analyse results of interrupted time series studies using regression analysis with time trends before and after the interventions [18, 19]. We will assess certainty of the evidence of effects of interventions using the Grading of Recommendations, Assessment, Development and Evaluation (GRADE) approach [16].

We will look out for and correct any errors made in the analysis of included studies. For example, if clustering is not addressed in an eligible cluster randomised trial, we will re-analyse the data if sufficient information is available. Otherwise, we will request necessary data from the authors or attempt to adjust data for clustering by inflating the standard errors by multiplying them by the square root of the design effect [16]. The adjusted effect will then be added in the meta-analysis. We will assess statistical heterogeneity among study results using the I² statistic. We will consider heterogeneity as substantial, if the I² is 50% or more. We will investigate the causes of substantial statistical heterogeneity using subgroup analyses. We will define subgroups based on participant and study design characteristics. We will use the chi-squared test for subgroup differences to assess for subgroup interactions. We will carry out sensitivity analyses, if applicable on aspects that could potentially affect the meta-analysis results such as study designs and overall risk of bias. We may also conduct a sensitivity analysis to explore the effects of fixed- versus random-effects analyses for outcomes with statistical heterogeneity [16].