Figures
Abstract
Background
Retention of patients on antiretroviral therapy (ART) over time is a proxy for quality of care and an outcome indicator to monitor ART programs. Using existing databases (Antiretroviral in Lower Income Countries of the International Databases to Evaluate AIDS and Médecins Sans Frontières), we evaluated three sampling approaches to simplify the generation of outcome indicators.
Methods and Findings
We used individual patient data from 27 ART sites and included 27,201 ART-naive adults (≥15 years) who initiated ART in 2005. For each site, we generated two outcome indicators at 12 months, retention on ART and proportion of patients lost to follow-up (LFU), first using all patient data and then within a smaller group of patients selected using three sampling methods (random, systematic and consecutive sampling). For each method and each site, 500 samples were generated, and the average result was compared with the unsampled value. The 95% sampling distribution (SD) was expressed as the 2.5th and 97.5th percentile values from the 500 samples. Overall, retention on ART was 76.5% (range 58.9–88.6) and the proportion of patients LFU, 13.5% (range 0.8–31.9). Estimates of retention from sampling (n = 5696) were 76.5% (SD 75.4–77.7) for random, 76.5% (75.3–77.5) for systematic and 76.0% (74.1–78.2) for the consecutive method. Estimates for the proportion of patients LFU were 13.5% (12.6–14.5), 13.5% (12.6–14.3) and 14.0% (12.5–15.5), respectively. With consecutive sampling, 50% of sites had SD within ±5% of the unsampled site value.
Citation: Tassie J-M, Malateste K, Pujades-Rodríguez M, Poulet E, Bennett D, Harries A, et al. (2010) Evaluation of Three Sampling Methods to Monitor Outcomes of Antiretroviral Treatment Programmes in Low- and Middle-Income Countries. PLoS ONE 5(11): e13899. https://doi.org/10.1371/journal.pone.0013899
Editor: Landon Myer, University of Cape Town, South Africa
Received: June 25, 2010; Accepted: October 19, 2010; Published: November 10, 2010
This is an open-access article distributed under the terms of the Creative Commons Public Domain declaration which stipulates that, once placed in the public domain, this work may be freely reproduced, distributed, transmitted, modified, built upon, or otherwise used by anyone for any lawful purpose.
Funding: Funding for this collaborative analysis using existing databases was provided by the World Health Organisation. The World Health Organisation participated in study design, data analysis, decision to publish, and preparation of the manuscript.
Competing interests: This collaboration was initiated by the World Health Orgnanisation which funded it. The World Health Organisation is part of the agences co-signing this paper.
Introduction
At the end of 2009, more than 5 million people were receiving antiretroviral therapy (ART) in low- and middle-income countries,[1] out of 33.4 million people (estimated range: 31.1 million–35.8 million) living with HIV [2], [3]. This represents a 30% increase in one year and a 13-fold increase in ART uptake in six years. Monitoring of ART programmes is critical for understanding when sites are under-performing and estimating the potential impact of treatment, at the population level and for program management at different levels of the health system. In addition, reporting of such indicators helps to sustain national and global commitment to monitor quality of care while expanding access to ART and its growing use.
Many countries are still struggling to report national programme indicators. In 2009, 70 out of 149 low- and middle-income countries (47%) reported statistics on patient retention on ART at 12 months and 30 (20%) at 48 months [4]. This outcome indicator is one of several core indicators recommended to monitor the Declaration of Commitment on HIV/AIDS during the United Nations General Assembly Special Session on HIV/AIDS (UNGASS) [5]. Although some countries have highly automated information systems, many ART sites within countries have difficulty in maintaining the registers/databases necessary to produce these statistics. A number of factors may explain the difficulties in generating good quality information. Many ART programmes are relatively recent, yet facing large and rapid increases in the number of patients starting therapy. They have thus logically focused their attention on putting new patients into care and reporting baseline information rather than on follow-up. In addition to ART, HIV care includes various other components (e.g. prevention and treatment of opportunistic infections including tuberculosis or integration with reproductive health services) that require regular monitoring. With the rapid expansion of ART services and decentralisation to peripheral health centres, data management has to occur at all levels of health service provision and should be designed to be as simple and user-friendly as possible in order for it to be adopted universally. Finally, ART treatment is life-long, leading to continued workload increase due to inclusion of new patients and continued follow-up.
In this paper we evaluated the performance of three sampling approaches to produce two clinic and higher level indicators at 12 months: retention on ART [6] and proportion of patients lost to follow-up, using existing databases from ART programmes in low- and middle-income countries.
Methods
Sources of data
ART-LINC of IeDEA is a large collaborative network of HIV/AIDS treatment programmes in low- and middle-income countries in Africa, South America and Asia, originally funded by the United States National Institutes of Health (Office of AIDS Research) and the French Agence Nationale de Recherches sur le Sida et les hépatites virales (ANRS). It is now part of the International epidemiological Database to Evaluate AIDS collaboration of the NIH (http://www.iedea-hiv.org). This network was established in 2003 to characterize the prognosis of HIV-infected patients treated with ART in resource-limited settings, to compare the experience between different settings, delivery modes and types of monitoring; and to compare outcomes with those observed in industrialized nations [7]. Staff at the sites filled in a detailed abstraction form using their own medical records, or downloaded the required data from their own electronic medical records (EMR) systems. Data were merged anonymously at the University of Bern, Switzerland and the University of Bordeaux, France [8].
Since 2001, Médecins Sans Frontières (MSF) has provided ART in 26 countries with a high HIV prevalence, most of them in Africa and Asia. Basic patient individual clinical, laboratory and treatment information are routinely collected at every clinic visit using standardized forms. Data are continuously entered into the Follow-up and Care of HIV Infection and AIDS (FUCHIA) EMR software (Epicentre, Paris, France). Capacity building and maintenance of the databases are funded by MSF and data centralized anonymously in Epicentre – Paris, France, the epidemiological support office of MSF [9], [10].
Ethics statement
International review boards in each country have approved the use of routinely collected programme data at all ART-LINC sites. This study was approved by the research ethics review committee of the World Health Organisation and the ethics review board of Médecins Sans Frontières. The data analysed are primarily collected for patient management and program monitoring, with the agreement of the ministries of health of the countries. Because of this reason, patient informed consent is not routinely requested in all sites. All ethic boards were aware of this and approved the secondary use of data for this analysis.
Selection of sites and indicators
For this analysis we selected all ART sites with more than 260 ART naive adults aged ≥15 years old starting ART in the year 2005. All databases were updated more than 12 months after the inclusion of the last patient in 2005. We analysed the patient status at 12 months, classified as followed on ART, dead, stopped treatment, and lost to follow-up (LFU). Patients LFU were defined as patients with no recorded visit for ≥90 days from the last visit within the first year. Patients transferred to another ART programme within the first 12 months of treatment were excluded from the analysis (n = 896 or 3% of the overall database).
We studied two indicators measured 12 months after ART initiation, the proportion of patients LFU and the proportion retained on ART.
Sampling methods
For each ART site, the required sample size was calculated, assuming a retention at 12 months of 75% as n≥[(u2α*N*p*(1-p))/(Δ2 * (N-1)+u2α*p*(1-p))] with n sample size, N total population of each site, p expected proportion estimated at 75%, Δ precision estimated at 0.05 and uα estimated at 1.96. We used the sample required to ensure that the 95% (i.e., 1-alpha) confidence interval of the 75% proportion had a width at least 2 times Delta where Delta is the required precision. Three sampling methods were considered: the random selection, systematic and consecutive sampling. In random sampling each patient of each clinic database had an equal probability of selection and each patient was selected using a random number allocation. In systematic sampling, the first patient was randomly selected and the others were drawn from the clinic database according to a sample interval defined as N/n until achievement of the desired sample size. In consecutive sampling, the first patient was randomly selected, then patients consecutively registered were selected until the achievement of the required sample size.
Statistical analyses
We first compared estimates of the retention on ART at 12 months obtained using Kaplan-Meier methods (taking into account the exact duration of follow-up of each patient before treatment discontinuation) with the proportion of patients retained on ART at 12 months (as recommended by UNGASS [5]). The outcomes were either death, stopping ART or LFU within the first 12 months, while for patients alive and on ART follow-up was censored at 12 months.
Thereafter all estimates were generated as proportions at 12 months as it is the usual method in routine programme monitoring. We calculated the two indicators using first the overall dataset and compared the proportions with those resulting from computation in sub-samples obtained with the three sampling methods. Indicators obtained using the full dataset are referred to as “unsampled” values. For each site and for each sampling method, 500 samples were simulated. We used the mean of the 500 results and the 2.5th and 97.5th values to determine the 95% sampling distribution (SD) and compared it to the unsampled site value. Indicators obtained using the full dataset were also compared to those obtained with the sampling methods according to cohort size taking the median value (870 patients) as a threshold, type of setting (rural/urban) and the proportion of patients LFU in the cohort (<10% versus ≥10%).
Combined indicators for all sites were then generated by aggregating site specific sampling results; the estimate of the mean was a weighted average of the proportion at each site weighted by the sample proportion (i.e., the number of patients selected divided by the total number of eligible patients at each site). Statistical analyses were performed using Statistical Analysis System software (SAS, version 9.1).
Results
Twenty-seven ART sites, 22 located in Africa and five in Asia, were included with a total of 27,201 patients treated. The number of patients per site ranged from 378 to 4111. After 12 months on ART 2,036 (7.5%) patients had died (range 1.9% to 16.7% across sites), 688 (2.5%) had stopped ART (range: 0% to 8.5%) and 13.5% were LFU (range 0.8% to 31.9%) (Table 1). Estimates of retention on ART at 12 months calculated with the proportion and Kaplan-Meier methods were similar, at 75.9% (range 58.7% to 88.6%) and 76.5% (58.9% to 88.6%) respectively. All following results were generated as proportions as it is the usual method in routine programme monitoring.
A total of 5,696 patients (20.9%; range 6.6% to 51.1% across cohorts) were sampled. Estimates for 12-month retention on ART, from random, systematic and consecutive sampling were 76.5% (95% SD 75.4–77.7), 76.5% (95% SD 75.3–77.5) and 76.0% (95% SD 74.1–78.2), respectively, compared to 76.5% for the unsampled value (Figure 1). The sample distribution for the 500 sample iterations varied across sites. Overall, sample distribution was wider when using consecutive sampling; the 2.5th value was within minus 5% of the unsampled value for 14/27 sites (51.8%) and the 97.5th value was within plus 5% for 21/27 sites (78%) and ranged from −10.0% to +10.6%. Variability in sample distribution was independent of cohort size (P = 0.98), urban/rural location (P = 0.99) or the proportion of patients LFU in the cohort (P = 0.81).
Similar results were observed for the proportion of patients LFU indicator (Figure 2). The consecutive sampling method slightly overestimated the estimates (14.0%; 95% SD 12.5–15.5) compared to 13.5% using the full dataset, and the sample distribution was wider than intervals obtained with random (13.5%; 95% SD 12.6–14.5) and systematic sampling (13.5%; 95% SD 12.6–14.3).
Discussion
When paper records are used at ART sites, programme monitoring often relies on transferring key information from patients' medical records into paper registers or electronic databases and aggregating the information at regular periods for reporting and interpretation of findings at all levels of the health system. Maintaining accurate medical records for all patients at every contact is essential to ensure quality of care and patient management, but the subsequent transfer of information for the purpose of programme monitoring could be limited to a sample of patients rather than the full cohort to reduce workload. Sampling has already been used for HIV/AIDS care program monitoring [11] but precision of the results obtained had not been assessed. Sampling has also been used to determine outcomes among patients LFU by tracing in the community [12], [13].
We compared two monitoring indicators, proportions of patients retained on ART and of patients LFU at 12 months. These statistics were obtained first with the overall dataset without sampling, then after applying three sampling strategies. Many international indicators, including the UNGASS indicators, are indeed calculated only at a national level and do not require exhaustive site-specific data. Our sampling strategies performed well on the dataset combining patient data collected in 27 sites. Overall estimates were similar independently of the sampling method used. Sampling performed particularly well when indicators were calculated in the full dataset while differences in estimates and sample distribution were more variable when analysed at site level. For 12-month retention on ART, the sample distribution obtained by consecutive sampling was wider compared to random and systematic sampling methods with a maximum distribution of ±10% of the unsampled value. Whereas not directly comparable, the cluster sampling method used to monitor immunization coverage in children was developed three decades ago for sampling results to range within ±10% from the population value and has always been recommended since then; although 17% of sampling results fell outside these limits, the method was considered to perform well enough for programmatic purposes [14].
Patient life-long retention on ART is of growing concern in the rapid scale-up of large treatment programmes[4], [15]–[17]. An analysis in South Africa showed a deterioration in retention on ART over time with an increasing proportion of patients LFU among those enrolled during the most recent years. It was in part related to the large increase in patients but the authors also discussed the burden on health informatics systems and administrative errors leading to misclassification in LFU [15]. Improving and maintaining simple and standardised monitoring systems capturing true treatment outcomes is part of the strategies recommended to better document and therefore improve patient retention on ART [17].
The workload required to ensure good quality of ART cohort monitoring is substantial for both paper-based and electronic systems. It also increases with the size and follow-up of the cohort. EMR systems allow the systematic and sometimes automated production of statistical indicators using information from all patients, but they are resource-intensive, as they require data clerks, training, and system maintenance[18]. A review of EMR systems used in 21 ART sites in low- and middle-income countries reported a median proportion of missing data for key information of 10.9% [19]. Missing data declined with training on data-management and with the number of hours spent by data-clerks in the maintenance of the databases. The number of hours necessary to reduce the proportion of missing information below 10% was estimated at 10 hours per week per 100 patients on ART. Time is also required to set the system up and expand it throughout a country. Some countries are moving towards a national EMR system; SmartCare is an EMR system currently deployed in Zambia, Ethiopia and South Africa [20]. However, in the current situation where international donors are not providing additional financial support, countries where an insufficient proportion of persons in need of ART are receiving treatment may prioritize employing and training additional clinical and laboratory staff to provide patient care over investing resources to develop and maintain an EMR system.
In the absence of EMR that would support the analyses of full cohort data, sampling approaches could thus potentially limit the workload in longitudinal monitoring and reinforce the long-term sustainability of monitoring systems at local and national levels. Moreover, retention on ART is to be analysed not only at 12 months but also for subsequent years of follow-up to document trends in retention on ART over time. The number of yearly end-points to analyse will therefore increase with the maturity of the programs. Sampling might be particularly useful in ART sites with a large number of patients initiating ART annually. Based on our analysis, the minimum sample size for a cohort of 500 patients would be 184, 224 for a cohort of 1000 and 252 for a cohort of 2000 patients. Consecutive sampling would probably be the most practical approach for the selection of patients. A number of countries are currently collecting retrospective patient data to generate longitudinal indicators once or twice a year; abstraction of information and calculation of indicators could potentially be performed using a sample of patients starting on ART in each calendar year. Sampling could be also piloted as an interim strategy while implementing EMR and/or could be used for purpose of quality control.
The present evaluation was based on existing databases in well resourced sites that receive support to maintain complete, quality-assured medical records. Thus the performance observed for sampling is not directly generalisable to a national programme. Performance of sampling strategies will depend on the accuracy or completeness of medical records and/or registers and on the level of organisation of the filing system. This evaluation did not address the issue of additional support that may be required in many ART sites where data are missing or inaccurately recorded. Sampling may also be difficult to implement in routine monitoring and needs a standardised and sustainable method for the long-term comparability of results. Personnel at site level may not be confident with results produced by sampling. Pilot projects to produce and validate monitoring indicators from sampling and to quantify related workload are therefore needed to complement this work.
In conclusion, generation of the two longitudinal indicators we studied in a sample of patients appeared to be a potentially useful method for programme monitoring at national level, based on available data from the ART-LINC of IeDEA and MSF cohort collaborations. However, the feasibility of this approach needs to be evaluated at country level and the accuracy of estimates based on sampling should be evaluated at site level in countries interested in this approach.
Acknowledgments
We are grateful to MSF partners in the Ministry of Health of participating countries, MSF field teams, Epicentre FUCHIA team, AIDS working group of MSF and the patients followed in the programs.
We would also like to thank Marthe-Aline Jutand for statistical advices, Loretxu Pinoges for the preparation of the MSF databases and Keith Sabin for his useful comments on the manuscript.
This paper was presented in part at the 14th International Workshop on HIV Observational Databases, 25–27th March, 2010, Abstract #85 and at the XVIII International AIDS Conference in Vienna, Austria, 18–23 July, 2010, Abstract #THPE0410.
The ART-LINC of IeDEA cohort network was organized as follows:
Central coordinating team: Martin W.G. Brinkhof, Eric Balestre, Claire Graber (project manager), François Dabis (principal investigator), Matthias Egger (principal investigator), Mauro Schechter (principal investigator).
Participating centers for this analysis (site investigators): Moi Teaching and Referral Hospital, Eldoret, Kenya (Silvester Kimaiyo, Winston Nyandiko Mokaya, John Sidle); Immune Suppression Syndrome clinic, Mbarara, Uganda (David Bangsberg); Centre de Prise en Charge, de Recherche et de Formation sur le VIH/SIDA (CEPREF), Abidjan, Côte d'Ivoire (Eugène Messou, Siaka Touré); Lighthouse Trust Clinic, Lilongwe, Malawi (Mina Hosseinipour, Sam Phiri, Ralf Weigel); Gugulethu ART Programme, Gugulethu, South Africa (Robin Wood); Connaught Clinic, Harare, Zimbabwe (Ruedy Luthy, Margaret Pascoe); YRG Care, Chennai, India (N. Kumarasamy).
The Epicentre FUCHIA team:
Serge Balandine, Megan McGuire, Sarala Nicholas, Loretxu Pinoges, Elisabeth Poulet, Mar Pujades-Rodríguez.
The medical HIV advisors of the AIDS Working Group of MSF:
Line Arnould (MSF-Belgium), Suna Balkan (MSF-France), Esther Casas (MSF-Holland), Cecilia Ferreyra (MSF-Spain), Marianne Gale (MSF-Australia), Johnny Lujan (MSF-Switzerland), Marcio Silveira (MSF-Holland), Elisabeth Szumilin (MSF-France).
Author Contributions
Conceived and designed the experiments: JMT KM MPR EP DB ADH MM MS YS FD. Analyzed the data: KM. Wrote the paper: JMT KM MPR EP DB ADH MM MS YS FD.
References
- 1.
WHO , UNAIDS , UNICEF (2010) Toward universal access. Scaling up priority HIV/AIDS interventions in the health sector. Progress report. Available: http://whqlibdoc.who.int/publications/2010/9789241500395_eng.pdf. Accessed October 15, 2010.
- 2.
UNAIDS , WHO (2009) AIDS epidemic update 09. Available: http://data.unaids.org/pub/Report/2009/JC1700_Epi_Update_2009_en.pdf. Accessed October 15, 2010.
- 3. Mahy M, Tassie JM, Ghys PD, Stover J, Beusenberg M, et al. (2010) Estimation of antiretroviral therapy coverage: methodology and trends. Curr Opin HIV AIDS 5: 97–102.
- 4. Tassie JM, Baijal P, Vitoria MA, Alisalad A, Crowley SP, et al. (2010) Trends in Retention on Antiretroviral Therapy in National Programs in Low-Income and Middle-Income Countries. J Acquir Immune Defic Syndr 54: 437–41.
- 5.
UNAIDS (2009) Monitoring the declaration of commitment on HIV/AIDS -Guidelines on construction of core indicators 2010 reporting. Available: http://data.unaids.org/pub/Manual/2009/JC1676_Core_Indicators_2009_en.pdf. Accessed October 15, 2010.
- 6. Bennett DE, Bertagnolio S, Sutherland D, Gilks CF (2008) The World Health Organization's global strategy for prevention and assessment of HIV drug resistance. Antivir Ther 13 : Suppl 21–13.
- 7. Braitstein P, Brinkhof MW, Dabis F, Schechter M, Boulle A, et al. (2006) Mortality of HIV-1-infected patients in the first year of antiretroviral therapy: comparison between low-income and high-income countries. Lancet 367: 817–824.
- 8. Dabis F, Balestre E, Braitstein P, Miotti P, Brinkhof WG, et al. (2005) Cohort Profile: Antiretroviral Therapy in Lower Income Countries (ART-LINC): international collaboration of treatment cohorts. Int J Epidemiol 34: 979–986.
- 9. Pujades-Rodriguez M, O'Brien D, Humblet P, Calmy A (2008) Second-line antiretroviral therapy in resource-limited settings: the experience of Medecins Sans Frontieres. AIDS 22: 1305–1312.
- 10. O'Brien DP, Sauvageot D, Olson D, Schaeffer M, Humblet P, et al. (2007) Treatment outcomes stratified by baseline immunological status among young children receiving nonnucleoside reverse-transcriptase inhibitor-based antiretroviral therapy in resource-limited settings. Clin Infect Dis 44: 1245–1248.
- 11. Alemayehu YK, Bushen OY, Muluneh AT (2009) Evaluation of HIV/AIDS clinical care quality: the case of a referral hospital in North West Ethiopia. Int J Qual Health Care 21: 356–362.
- 12. Yiannoutsos CT, An MW, Frangakis CE, Musick BS, Braitstein P, et al. (2008) Sampling-based approaches to improve estimation of mortality among patient dropouts: experience from a large PEPFAR-funded program in Western Kenya. PLoS ONE 3: e3843.
- 13. Geng EH, Emenyonu N, Bwana MB, Glidden DV, Martin JN (2008) Sampling-based approach to determining outcomes of patients lost to follow-up in antiretroviral therapy scale-up programs in Africa. JAMA 300: 506–507.
- 14. Henderson RH, Sundaresan T (1982) Cluster sampling to assess immunization coverage: a review of experience with a simplified sampling method. Bull World Health Organ 60: 253–260.
- 15. Cornell M, Grimsrud A, Fairall L, Fox MP, van CG, Giddy J, et al. (2010) Temporal changes in programme outcomes among adult patients initiating antiretroviral therapy across South Africa, 2002-2007. AIDS 24: 2263–2270.
- 16. Fox MP, Rosen S (2010) Patient retention in antiretroviral therapy programs up to three years on treatment in sub-Saharan Africa, 2007–2009: systematic review. Tropical Medicine & International Health 15: 1–15.
- 17. Harries AD, Zachariah R, Lawn SD, Rosen S (2010) Strategies to improve patient retention on antiretroviral therapy in sub-Saharan Africa. Trop Med Int Health 15 : Suppl 170–75.
- 18. Douglas GP, Gadabu OJ, Joukes S, Mumba S, McKay MV, et al. (2010) Using touchscreen electronic medical record systems to support and monitor national scale-up of antiretroviral therapy in Malawi. PLoS Med 7: e1000319.
- 19. Forster M, Bailey C, Brinkhof MW, Graber C, Boulle A, et al. (2008) Electronic medical record systems, data quality and loss to follow-up: survey of antiretroviral therapy programmes in resource-limited settings. Bull World Health Organ 86: 939–947.
- 20.
SmartCare (2010) SmartCare home page. Available: http://www.smartcare.org.zm/. Accessed October 15, 2010.