Background
The success of the Global Polio Eradication Initiative (GPEI), launched after a World Health Assembly resolution of 1988, was remarkable and nearly 5 million children were protected from being paralyzed by 2003 [
1,
2]. About 10 billion doses of polio vaccine have been administered since 1988 on hundreds of national and sub-national immunization days (NIDs/SNIDs) at a cost of US$ 4.5 billion [
3,
4]. But four countries - Afghanistan, Pakistan, India and Nigeria - never interrupted wild polio virus (WPV) transmission and re-infection of 26 countries in 2006-07 raised questions about the feasibility of polio eradication. By the end of 2007, polio incidence had decreased by 35% and polio transmission had stopped in all but 6 re-infected countries [
5], yet polio eradication seems illusory as endemic countries are missing targets and 1648 cases were reported worldwide in 2008 [
6].
Pakistan adopted the Polio Eradication Initiative (PEI) within the Expanded Program on Immunization (EPI) in 1994 remarkably decreasing the number of cases to 32 by 2007 including 19 WPV type 1 and 13 WPV type 3 cases with virus found in 18 of 120 districts [
2,
6‐
8]. But polio resurged in 2008 infecting 49 districts with 118 cases including 81 WPV type 1 and 37 WPV type 3 cases - highest amongst the previous six years [
2,
6‐
8]. Pakistan reportedly achieved all targets set in the GPEI strategic plan but failed to interrupt virus transmission [
5,
9‐
11], probably due to sub-district coverage gaps, low routine coverage, operational weaknesses in the quality of services and large numbers of children missed during NIDs/SNIDs. The scenario calls for immediate action.
As health programs mature, large-scale surveys become less useful. Instead, data from small-scale studies are required to evaluate different aspects of a program [
12]. The aim of this study, in one district of Pakistan's Punjab province, was to establish valid and reliable estimate for the effectiveness of polio eradication services and to ultimately provide scientific evidence for tailoring future interventions. Specific objectives were the evaluation of routine immunization services including oral polio vaccine (OPV) coverage and the characteristics of vaccinated and unvaccinated children, the evaluation of logistics management and the quality of monitoring systems, and the assessment of national immunization days (NIDs) including NIDs coverage and the quality of NIDs service delivery.
Methods
Design and Setting
A cross-sectional study using lot quality assessment techniques was conducted between March 02, 2009 and March 22, 2009 in District Nankana Sahib, located in the central Punjab, with about 100 health facilities and 2000 health personnel obliged to provide curative and preventive services to 1.8 million inhabitants of the district (administrative data, 2009). [Additional files
1 and
2]
Sample
The sample was taken by the lot quality assessment sampling (LQAS) technique, using World Health Organization (WHO) guidelines, Brixton health software packages and Epi Info 6 [
13‐
15]. A sample was separately taken for assessment of routine and NIDs coverage, logistics management and the quality of monitoring systems, and the quality of NIDs service delivery. For random selection of the lots of the catchment areas of primary health centers, primary health centers, static centers and mobile teams; a line listing of all the respective areas/facilities was obtained from the District Department of Health and the sample was randomly selected by using two-digit random number table generated by Epi Info 6. [Additional file
3]
Sample for routine and NIDs coverage
For routine and NIDs coverage, 20 lots of the catchment areas of 20 primary health centers (PHCs) were randomly selected - having a population of 516,918 and approximate population in each lot was 20,000 - 30,000.
Study subjects for the routine OPV coverage assessment and the characteristics of routine immunization status included 1080 children aged 12 - 23 months from 20 lots. Statistical parameters included an accuracy level of ± 3, a confidence level of 95%, a total sample size of 1080 children, a total number of 20 lots, a lot sample size of 54 children, a low threshold level of 80%, a high threshold level of 99% and a decision value for lot rejection of >3 unimmunized children (P:99% = 100%, P:80% = 100%, Error = 0%).
Subjects for the NIDs coverage assessment were all children under-five present in 420 households in 20 lots. Statistical parameters included an accuracy level of ± 5, a confidence level of 95%, a total sample size of 420 households, a total number of 20 lots, a lot sample size of 21 households, a low threshold level of 90%, a high threshold level of 99% and a decision value for lot rejection of >0 households having one or more unimmunized children (P:99% = 81%, P:90% = 89%, Error = 30%).
Sample for logistics management and the quality of monitoring systems
Out of 70 primary health centers (PHCs) in District Nankana Sahib, 20 randomly selected PHCs were evaluated for logistics management and the quality of monitoring systems in health facilities. With a low threshold level of 80% and a high threshold level of 99%, the decision value for lot rejection was >0 (P:99% = 86%, P:80% = 96%, Error = 18%), and based on the compliance score for logistics management and the quality of monitoring systems (scale = 0-15)), therefore acted as equivalent to the lot sample size.
Sample for quality of NIDs service delivery
Static centers at each of the 20 randomly selected PHCs and one randomly selected mobile team from their catchment area were evaluated for the quality of NIDs service delivery. With a low threshold level of 80% and a high threshold level of 99%, the decision value for lot rejection was >0 (P:99% = 85%, P:80% = 97%, Error = 18%), and based on the compliance score for the quality of NIDs service delivery (scale = 0-16), therefore acted as equivalent to the lot sample size.
Data Collection
Study instruments were designed according to WHO guidelines [
13,
16‐
18], pre-tested in the field and modified accordingly. [Additional file
4]
For evaluating routine OPV coverage, a standardized questionnaire consisting of sociodemographic information, immunization status and related characteristics was used in face-to-face interview in household settings. It took 10 minutes to fill out the questionnaire by response of child's caretaker (mother, if possible). Coverage was measured by "card" and by "card plus history". A child who received all immunizations required by the EPI schedule was considered fully immunized. Valid immunization meant an immunization given at the appropriate age (within six months), after an appropriate interval of time (according to EPI schedule) and recorded on an immunization card.
A list of all the villages/wards was obtained from the District Department of Health and three villages/wards in each lot were randomly selected as sampling point areas. As the villages/wards have different settlement designs in Pakistan, the mosque was considered the center of each village/ward. Interviewers went to the mosque, spun a bottle on level-ground, and the first household was selected by the direction in which the bottle pointed. The household was included in the sample if a child aged 12-23 months was present. Subsequent households were chosen by proximity. One child was seen from each household and in case of more than one children aged 12-23 months in a household, the youngest child was seen. In this way, 18 households were randomly selected from each sampling point area; therefore, 54 children were seen from each lot, making a total sample of 1080 children from the district.
For evaluating logistics management and the quality of monitoring systems, structured forms were filled out by observation and inspection of PHCs. Data were obtained from respective vaccination point (the center from where vaccine was collected) for PHCs with a non-functioning refrigerator.
For NIDs coverage, a standardized questionnaire was used in face-to-face interviews in household settings. It took 5 minutes to fill out the questionnaire. Coverage was measured by "finger-mark". A household was considered unimmunized if one or more of the children present were not immunized.
The sampling point areas selected for routine coverage were surveyed again during NIDs (March 16, 2009 - March 20, 2009). The 7 households were randomly selected from each sampling point area and all children in each household were seen. The 21 households were visited in this way from each lot, making a total sample of 420 households from the district.
For evaluating the quality of NIDs service delivery, structured forms were filled out by observation, inspection and exit interviews in static centers and mobile teams.
Verbal informed consent from respondents was deemed sufficient. Approval for the study was granted by the Ethical Review Board of Allama Iqbal Medical College, Lahore, Pakistan and the Department of Health, District Government Nankana Sahib, Punjab, Pakistan. Data were collected by medical students trained to carry out surveys and the entire process was monitored by a field coordinator. Health education for children's caretakers and technical assistance for health workers was also provided during the study.
Data analysis
Data were entered and analyzed using SPSS version 17 (SPSS Inc. Chicago IL, United States: 2008). Weighted immunization coverage was calculated by using WHO guidelines for lot quality surveys [
13]. For logistics management and the quality of monitoring systems in health facilities and the quality of NIDs service delivery, a weighted percentage was obtained. Each characteristic was equally weighed, a compliance score based on present/maintained characteristics was computed for each lot, and a mean compliance score was then obtained. A compliance percentage was calculated as sum of correct answers/total observations × 100. Bivariate analysis using the chi-square test was employed to independently correlate variables. Statistical significance was considered at P < 0.05.
Discussion and Conclusions
The historically used LQAS design is still an appealing technique for rapid evaluation of preventive programs, especially in rural areas of developing countries [
19‐
23]. It is very efficient in determining the performance of individual subunits in a specific area [
22,
23]. It was observed that most health personnel were unaware of this technique and its effectiveness in local evaluations.
Five out of twenty lots were rejected for unacceptably low routine immunization coverage, while the validity of routine overage was questionable to the extent that all lots were rejected. Weakness in routine immunization is the main constraint in polio eradication [
21]. A national immunization card program could significantly increase coverage and the validity of coverage [
22,
23]. Knowledge gaps underlie low compliance with vaccination schedules, and the quality of interaction between health workers and caretakers is essential to ensure compliance with vaccination schedules [
24].
Nearly half of the children sampled could not present a card. Coverage was significantly higher among those having cards, consistent with previous studies [
19,
23,
25,
26]. Card availability and immunization status were significantly associated with socioeconomic factors, consistent with previous studies [
23,
26‐
28]. Health workers were the major source of immunization and information. Outreach services have been recommended as the most efficient way to increase coverage [
27‐
29] and there is a need for interventions aimed at fostering the communication skills of health workers [
30,
31]. The main reasons for routine immunization failure were educational constraints and problems of accessibility and availability, in agreement with previous literature [
19,
21,
23,
32].
All 20 lots were rejected for poor compliance in logistics management and the quality of monitoring systems. Previous compliance studies have also indicated poor performance regarding cold chains and logistics [
32‐
37]. The functioning of refrigerators was poor due to security and electricity problems. Maintenance of refrigerator temperatures within the correct range and vaccine storage were also poor, which can be attributed to the lack of technical skills among health workers and poor attitudes towards maintaining cold chain charts. Maintenance of frozen icepacks and cold box/vaccine carriers was poor. This merits necessary action because in case of electricity failure, it would be the only way to maintain cold chains, as power generators were not available at most of the facilities. Previous studies in India revealed vaccine carriers were non-compliant in about one-third facilities, and power failures were also indicated as major constraint [
34,
36]. Poor use of safety boxes indicates a lack of awareness about safety measures. A holistic rather than logistic approach should be used for vaccine safety [
38].
The quality of monitoring systems and data was poor in all aspects, in agreement with previous studies [
32,
39‐
41]. This was attributed to the lack of supervision and knowledge among those who were responsible, consistent with previous studies [
42]. There is a need for monitoring systems to be viewed with a broader perspective, not focusing only on technicalities but also on the support mechanisms [
32,
41,
43]. In Pakistan, EPI/PEI records are not computerized; however, the feasibility of linked immunization database systems has been established elsewhere [
44], therefore, computerizing EPI/PEI records and linking them with health management information systems already in place might improve data quality. The implications of a poor quality data system are reflected in the efficiency of health services. Previous experience has shown that significant improvements in data quality and monitoring systems can be made by data quality self-assessment (DQS) and the use of data for action [
45‐
49]. This strategy is recommended in the Reaching Every District (RED) approach and Global Framework for Immunization Monitoring and Surveillance (GFIMS) by the World Health Organization [
48‐
51].
The 15 out of 20 lots were rejected for unacceptably low NIDs coverage by finger-mark. Poor coverage during NIDs which had been designed to deliver supplementary OPV doses to all children has been indicated as an underlying factor for the continued transmission of polio [
9,
52].
All 20 lots were rejected for poor NIDs service delivery. Poor technical skills of health workers, logistic problems, poor planning, and deficient communication skills were highlighted as the main problems. Similar findings were previously reported and associated with increased risk of non-vaccination [
36,
53‐
56]. In particular, communication with the clients was very poor. Previous studies have revealed the importance of effective communication to improve the coverage [
57,
58]. Decisions have generally not been based on studies of populations' knowledge and attitudes about immunization. Had this datum been strategically used, interventions could have been more effective in reaching zero-dose children [
54]. Poor planning and lack of technical skills accounted for service disruptions [
59], and supportive supervision could significantly increase the performance of immunization sessions [
36,
46,
60].
The study does have limitations. Since the status and validity of routine immunization obtained from cards was likely to be an underestimate, immunization by card plus history was included. It is an established practice to estimate coverage by maternal history [
61,
62]. Some characteristics evaluated in the study were of subjective nature that might be a limitation in itself. Central selection rather than the random walk method was followed in coverage estimation because of different settlement designs in villages/wards in Pakistan and that may be a limitation. The per lot error for NIDs coverage was high because of the low sample size as a higher sample was not logistically feasible; however, the error for overall coverage was not affected by that.
Although the study was conducted in a rural district of Pakistan's Punjab province, some of findings may be generalized to other areas with similar health system infrastructure, socio-cultural environments and topography. The findings suggest that LQAS studies could be conducted in other areas to assess performance on a sub-national level.
Short-comings revealed in polio eradication services are potentially important due to lack of similar studies in the region and the failure to achieve polio eradication despite continuous efforts by the global community over the past two decades. Following interventions are recommended:
-
training of mid-level health managers in LQAS,
-
administrative measures to improve routine coverage and its validity, including the appraisal of excellent performance, making flexible field plans to track missed children, initiating Immunization Card Crash Program, constituting law for birth registration with health centers, and improving linkage between preventive programs,
-
consideration of socioeconomic status in prioritizing interventions especially focusing on the illiterate, those living in poor houses and having large families (indirect predictors of poverty) and those not seeking health care at a public health facility,
-
focus on advocacy and communication including involving local communities in designing strategies, decentralizing resources, and training health workers in communication skills,
-
up-gradating cold chain equipment including ensuring on-going availability and maintenance of equipment, addressing electricity and security problems, making a vaccination point at each PHC, and maintaining reserve stocks at district stores,
-
capacity building of staff including establishing a training school for vaccinators, supportive supervision, and focused training programs held bi-annually for EPI/PEI staff at all levels, which should be mandatory for their promotion,
-
computerizing EPI/PEI records integrated with district health management information system and encouraging realistic reporting,
-
practically-oriented training and micro planning for NIDs and accessing the quality of NIDs service delivery, which is often overlooked in rapid campaign evaluations,
-
measures to address the extreme shortage of public health professionals in Pakistan, which affects all preventive programs including the provision of incentives to medical graduates for joining public health/preventive medicine and establishing national public health services.
Further studies in the region to evaluate findings are suggested. Polio continues to be a public health problem and we need to have a better understanding of the factors involved in achieving polio eradication.
Acknowledgements
We are highly grateful for the valuable comments of J Partridge, Kathryn Alberti and the editorial team on the earlier versions of the manuscript. We would like to thank Dina Kakar, MD, MSc Global Health for her invaluable efforts. We are indebted to staff of Department of Health, District Nankana Sahib, in particular Liaqut A Virk CDC Officer, Riaz Ahmad District Sanitary Inspector, Mukhtar A Dogar PSO to Executive District Officer Health, A D Cheema Tehsil Sanitary Inspector, Tanvir Virk CDC Inspector, and Muzammil Hussain. We acknowledge untiring efforts of our research associates especially M Qais Luqman, M Rizwan Afzal, Mahar M Shafique, Syed Razi Abbas, Fatima Omer, Sana Kifayat, Zara Zaheer, Anum Saeed Khan, Rohma Ahmad, Sarah Rashid, Tanzila Saleh, Komal Mushtaq and Sania Moin. We would also like to thank volunteers who worked with us especially M Hammad Mushtaq and Hammas Liaqut. Mushtaq MU likes to thank Andrea Lyman, MD, MSc, MS. We express our appreciation for the members of sampled households for sharing information, and staff at primary health centers who generously devoted their time and resources to facilitate the work of the survey team.
FUNDING
We acknowledge partial financial support from Allama Iqbal Medical College, Lahore, Pakistan, and assistance from Department of Health, District Government Nankana Sahib. Authors have all rights for the use of primary data.
Competing interests
The authors declare that they have no competing interests.
Authors' contributions
All authors contributed significantly in all phases of the project in accordance with uniform requirements established by International Committee of Medical Journal Editors (ICMJE) and contributors who do not meet authorship criteria are listed in acknowledgements. All authors read and approved the final version of the manuscript.