Intervene before leaving: clustered lot quality assurance sampling to monitor vaccination coverage at health district level before the end of a yellow fever and measles vaccination campaign in Sierra Leone in 2009

Lot Quality Assurance Sampling (LQAS) is a technique used by the World Health Organization (WHO) Expanded Programme on Immunization (EPI) to evaluate vaccination programmes [1]. The LQAS test is based on two thresholds: the lower threshold (LT), which is the minimum vaccination coverage level considered to be acceptable, and the upper threshold (UT), which is generally the operational objective of vaccination activities (i.e. the vaccination coverage target) [2]. Based on these thresholds, a decision value (d), i.e. the maximum number of unvaccinated individuals allowed in the sample (N) to classify the lot as with acceptable vaccination coverage, is calculated. If the number of unvaccinated individuals detected is equal or below d, the lot is classified as having an acceptable coverage (i.e. “accepted”), meaning that coverage in the lot is probably equal or above the LT. If the number of unvaccinated is above d, the lot is classified as having an unacceptable coverage (i.e. “rejected”), meaning that coverage in the lot is probably below the UT. Alpha, calculated from the LT, is the probability of “accepting” a lot with low coverage; while beta, calculated from the UT, is the probability of “rejecting” a lot with high coverage. When the real coverage in the lot falls in the “grey area” delimited by UT and LT, alpha and beta increase, making the LQAS classification less precise [3‐5]. The LQAS rule can be summarized by the operative characteristic (OC) curve (Figure 1).

LQAS requires smaller sample sizes and offers the advantage that the survey can be stopped if d is exceeded in the lot. It is therefore quicker than the standard vaccination coverage cluster-sampling survey[6]; however, it does not allow estimation of vaccination coverage with 95% confidence interval at the lot level. The "classic" LQAS approach is based on simple random sampling (SRS) rather than cluster-sampling [7]. Using SRS on large lots, such as health districts, introduces logistic challenges and engenders costs as the sample is scattered over a wide geographic area [8]. Clustered Lot Quality Assurance Sampling (C-LQAS) is a modified design of LQAS, which allows for the lots to be divided into smaller clusters, considerably increasing rapidity of implementation in the field [2, 9]. C-LQAS employs the same statistical principles as the standard LQAS, with the difference that N is divided in clusters (k) of n individuals. Statistical precision is then recalculated assuming that the true coverage in each cluster would vary from the mean lot coverage according to a binomial distribution with preset standard deviations (SD) [2]. Several authors have explored the applications of dividing the sample in smaller clusters, while applying sequential sampling techniques, such as LQAS, to different fields, from the assessment of global acute malnutrition [10‐12] to applications in clinical audit [13], veterinary medicine [14], or agriculture [15, 16]. WHO has been piloting C-LQAS with Ministries of Health (MoH) in west and central Africa to monitor coverage while vaccination campaigns are in progress in order to identify areas that need mop-up activities before leaving the field [17].

From 24 to 29 November 2009, Sierra Leone conducted a preventive yellow fever (YF) vaccination campaign among individuals aged nine months or older in six of its 14 health districts: Western Area Urban, Western Area Rural, Koinadugu, Kambia, Tonkolili and Port Loko. The campaign was integrated with a measles follow-up campaign throughout the country, among children aged 9–59 months. For both campaigns, the operational objective was to reach 95% of the target population.

We used C-LQAS to identify areas in Sierra Leone with unacceptably low yellow fever and/or measles vaccination coverage during the last two days of the ongoing vaccination campaign in order to improve coverage in the areas of weakness before the end of vaccination activities.

Using C-LQAS to assess vaccination coverage near the end of the integrated measles and YF campaign in Sierra Leone, we identified five areas (lots) with unacceptably low measles vaccination coverage and three with low YF coverage, based on documented proof of vaccination (card). This allowed the health authorities to put in place control measures to increase vaccination coverage before the end of the campaign in the rejected areas. Relying on verbal confirmation of vaccination status, fewer lots would have failed, as it is expected in these settings [17, 22]. With one exception, verbally confirmed district vaccination coverage estimated through the post-campaign cluster-sampling surveys was above the minimum acceptable threshold (75%) used for the end-campaign C-LQAS survey, suggesting that control measures to raise coverage may have been effective in the areas of weakness identified. The fact that one district (Port Loko), failing at the end of the campaign for both low measles and YF coverage, presented post-campaign measles and YF coverage figures not significantly above 75%, suggests that it may have structural problems in reaching the target population with vaccination activities.

We used C-LQAS for end-campaign monitoring. While this approach allows time to implement mop-up activities, undertaking an assessment before the end of the time allocated to achieve the target (95%) may lead to underestimation of coverage [17]. For this reason we set the UT of the C-LQAS plan to 90% instead of 95%, the actual coverage target of the campaign. We then decided to use 75% as LT based on previous experiences showing that using a “grey area” of 15% is a possible solution to keep statistical errors reasonably low if N = 50 (5x10) [17]. By setting alpha (≤8%) lower than beta (≤19%), we gave priority to the classification based on the LT (75%), rather than the UT (90%). We wanted to be more confident that we were not accepting lots with low coverage, which is a risk for the population (alpha is also known as “consumer risk”); rather than failing lots with high coverage, which is a risk for the vaccination programme (beta is also known as “provider risk”). Results of the C-LQAS surveys need to be interpreted as an indication that, towards the end of the campaign, at least 92% (1-alpha) of the accepted lots were probably reaching the minimum acceptable coverage level of 75%; while at least 81% (1-beta) of the failed lots were probably not reaching the desirable coverage level of 90%. In other words, when we accept a lot we are 92% confident that real coverage is at least 75% (i.e. when we reject we cannot be confident that coverage is 75%).

Intense efforts from WHO and UNICEF (United Nations Children’s Fund) to encourage 90% coverage for first dose measles vaccine by 12 months of age and offering a second opportunity for vaccination have contributed to a substantial reduction of measles cases in the African region [23, 24]. In this respect, the results (i.e. 15 lots out of 20 accepted for measles coverage at least at 75%), albeit before the end of the campaign, are far from optimal. Although achieving 75% vaccination coverage may be enough to stop transmission of YF, it is definitely not enough for measles, for which the threshold believed to confer herd immunity is 95% [25]. If we would have used the other, stricter, decision value proposed in Table 2 (d = 4; UT = 95%; LT = 80%) we would have accepted even fewer lots (i.e. 10 out of 20) as with measles coverage at least at 80%. By allowing individuals to present only the vaccination card of the current campaign as evidence of measles vaccination we may have underestimated the actual measles coverage in the country. Even if there was an underestimation of the true coverage rates, it is important to note that three weeks before the vaccination campaign the largest measles outbreak over the last decade started in the country and lasted until July 2010, confirming that measles vaccination coverage was low in Sierra Leone [26].

The C-LQAS approach works better if there is indication that the lots are homogeneous in terms of coverage [2, 9, 17]. Where possible, we attempted to divide districts into smaller lots to increase the likelihood of homogeneity. The distribution of vaccination coverage between the clusters in the lots exceeded the maximum assumed 0.1 SD always in rejected lots, apart from one occasion, Lot 8 in Port Loko, which had adequate YF coverage. Such a high ICV of vaccination coverage indicates that some of the clusters in those lots had many unvaccinated individuals, while others had few (i.e. the lot is not homogeneous in terms of vaccination coverage). This kind of distribution of coverage may indicate that some of the EAs may have been missed or inappropriately covered by vaccination activities (a good reason to reject the lot for poor quality of vaccination).

We showed that assuming that vaccination coverage would vary according to a SD of 0.1 in the clusters is a similar approach to assuming a DEFF of approximately three (if coverage is at 75%) or approximately six (if coverage is at 90%). We believe that this is more conservative than assuming DEFF = 2 as it is done in a standard manner while designing cluster surveys when there is no previous information to guide this decision [6]. The fact that the assumed DEFF increases proportionally to the assumed coverage is explained by the fact that assuming a variance of 0.1 SD when coverage is high (i.e. 90%) has a greater magnitude than assuming the same variance when coverage is low (i.e. 75%). Therefore, our approach should be more robust as the levels of real coverage get higher.

This study is subject to a number of limitations. First, because of limited resources we conducted the C-LQAS surveys at sub-district level only in the areas of the country targeted by both vaccines. In the part of the country targeted only for measles vaccination, each district was a lot. These districts have a mean total population of approximately 350,000. It may have been difficult to implement targeted control measures in such large districts. To overcome this limitation we advised the surveying teams to discuss the results of the C-LQAS surveys during the daily campaign meetings chaired by the district medical officers, in order to facilitate the implementation of control measures using all the information available also from other sources (e.g. vaccination teams or campaign supervisors). Second, by using PPS to select clusters, we inevitably placed data collection in the most populous communities. These are areas that generally are easier to cover also by vaccinators, so vaccination coverage may have been overestimated. PPS does not yield a spatially even sample as can be seen in Figure 2. Alternative methods based on geographic sampling could be used to allow that sampling locations are more evenly spread across the lots [27‐29]. Third, as previously seen with the C-LQAS approach, the high inter-cluster variability (SE > 0.1) seen in some lots, reaching a maximum level of SE = 0.21, may indicate that the plans may have had errors (alpha and beta) above the levels defined in some lots [9, 17], although this finding should be interpreted with caution, since it is not possible to support it statistically given the small sample sizes per lot used to calculate the SE. One way to reduce the inter-cluster variability and consequently increase the precision would be to increase the number of clusters sampled to more than five [22]. Finally, when we aggregated several lots to obtain a district level LQAS classification, we were not able to exactly assess how the unequal probabilities of selection (i.e. different lot population sizes) may have affected alpha and beta. On the other hand, the increased sample sizes at district level combining more than one lot would probably have errors that are lower than the ones assumed at lot level.

Identifying lots with low vaccination coverage before the end of the campaign allowed the health authorities to put in place timely control measures to increase vaccination coverage while vaccination activities were still ongoing. Post-campaign coverage estimations indicated that control measures to raise coverage were likely to have been effective in the areas of weakness identified, although areas of low coverage still remained in the country. Dividing the districts into smaller lots increased the particularity of the assessment, enabling us to locate areas with unvaccinated populations in smaller geographic or administrative units. The decision to use 75% as LT should be revised when assessing measles vaccination coverage. Further studies could compare C-LQAS results with those of cluster-sampling surveys conducted in the same areas at the same time, to validate statistically the use of C-LQAS in alternative.