Imputing HIV treatment start dates from routine laboratory data in South Africa: a validation study

In many low- and middle-resource settings, clinical records are often incomplete, missing, and/or poorly archived [1]. Where patient records are hand-written, implementation and maintenance of electronic records can be expensive and still not impervious to error [2‐4]. Accurate clinical record keeping is increasingly important to clinical care and appropriate monitoring and evaluation in such settings [5]. Many developing country health systems were set up to provide maternal services, vaccinations, and acute care for injury and infections. However, the rising global burden of chronic disease requires health systems to manage patients’ health longitudinally, a challenge in the absence of accurate clinical information. Information gaps also make it difficult to monitor large health care programs and allocate resources optimally [6]. Improved record keeping from treatment sites or cohorts is particularly challenging in settings of high patient mobility across sites, providers, and sectors (public/private), and yet it is precisely in such settings that complete and accurate records are critical to the coordination of patient care [1, 3, 7].

For many diseases, laboratory testing is a meticulously specified element of clinical protocols, as it is the basis for many diagnostic and treatment decisions in low-resource settings. Laboratory tests are often conducted off-site at central laboratories and test results may be available directly from the laboratories, bypassing two steps prone to human error: manual entry of data into patient charts and manual transcription of chart information into electronic databases. Where data from routine laboratory tests are systematically available, such information may be used to impute missing data in clinical records.

As with other chronic diseases, HIV/AIDS requires lifelong clinical management and decision-making guided by routine laboratory monitoring. South Africa has the largest number of people living with HIV/AIDS worldwide, with 7 million adults living with HIV in 2015 [8] and approximately half of these receiving antiretroviral therapy [9]. According to South African national ART treatment guidelines [10‐12], patients initiating ART should have certain laboratory tests conducted prior to starting treatment, hereafter referred to as “ART workup”. Clinicians use this combination of blood tests to assess liver function, anemia, and other factors. This allows the clinician to accurately stage the patient’s disease, treat any co-morbid conditions, and guide the choice of treatment regimen.

We propose using routine laboratory data from South Africa’s National Health Laboratory Service (NHLS) to impute dates of antiretroviral treatment (ART) initiation among patients in South Africa’s public sector HIV care and treatment program. South Africa has detailed national guidelines for initiation of HIV care and treatment [10‐12] (Fig. 1). Following a positive result from an HIV test, a CD4 count test is done to determine immune function and, historically, eligibility for initiation of ART. As of September 2016, all patients are eligible for ART at diagnosis, but CD4 counts are still done at enrollment. Once eligibility and intent to initiate is established, the ART workup blood tests are conducted to guide regimen choice. Patients then attend approximately three once-weekly counseling and preparatory sessions before initiating ART (under guidelines during the study period). We thus proposed to impute ART start dates by identifying ART workup dates and adding 21 days. To determine the success of the approach we assessed its validity with respect to sensitivity, specificity, positive predictive value, and negative predictive value. Sensitivity refers to the proportion of true ART start dates accurately identified using our approach. Conversely, positive predictive value captures the proportion of imputed start dates that accurately identify a true start date. Specificity and negative predictive value describes the accuracy of our approach in identifying non-initiators.

NHLS provides laboratory services to all public-sector facilities in South Africa (the province of KwaZulu-Natal joined NHLS in 2010), including all laboratory testing required for ART monitoring with over 26 million CD4 counts and 13 million viral loads resulted between 2004 and 2015. If successful, our imputation approach has the potential to improve the capabilities of laboratory data to monitor and evaluate South Africa’s national HIV care and treatment program.

We analyzed data on 57,401 patients in the Hlabisa Cohort; 38% (n = 21,766) of these had a documented ART initiation date in the clinical database while 33% (n = 18,836) had a documented ART workup. An additional 493 (0.9%) had an Hb/ALT result without a CD4 count. Currently, the RTC cohort contains data on over 133,000 patients, of whom 102,070 have initiated ART. Of these, we analyzed the 69,283 (68%) with a documented ART initiation date, of whom 63,914 (92%) had a documented ART workup and an additional 1153 (1.6%) had an Hb/ALT without a CD4 count. The cohorts were similar with respect to age and gender distribution while those in the RTC cohort initiated at slightly lower CD4 counts (median 123 vs. 152 cells/mm³) compared to those in the Hlabisa cohort (Table 1).

We calculated the number of days from the ART workup date to the known date of ART initiation for each patient (Fig. 2). Very few patients (2.5% in the Hlabisa Cohort, 4.4% in the RTC cohort) had their first recorded ART workup after ART initiation. These patients may have transferred into care and not been identified as such in the cohort dataset. The vast majority of patients who initiated ART did so after the workup date and 95% of patients in both the Hlabisa Cohort and RTC cohorts who initiated ART did so within 91 days after the ART workup. The median interval from ART workup to ART initiation was 16 days (IQR 7 – 31) in the Hlabisa cohort and 21 days (IQR 8 – 43) in the RTC cohort.

Median times from lab workup to ART initiation in the two cohorts were not only very similar to each other but also remarkably close to our hypothesized interval of 21 days from ART workup to initiation based on the initiation process outlined in national guidelines. While this suggests that, in general, sites implement procedures as outlined in the guidelines, practices could vary considerably by facility. We thus stratified this analysis by each of the facilities contributing data towards the larger Hlabisa and RTC cohorts. While there was some variation in the median number of days from ART workup to initiation between certain sites and we could not conclude there was no clinic-level variation, the overall consistency of the interval between workup and ART start dates by site was reassuring (Additional file 1: Table S1).

We also considered that there might be an association between calendar year and time from workup to treatment initiation as treatment guidelines have evolved somewhat over time and clinics may have become more efficient at initiating eligible patients as the rollout of the treatment program matured. We stratified the median number of days to initiation by calendar year and noted a decreasing trend in the number of days to initiation over our period of study (Additional file 1: Table S2). This decrease in the interval to initiation was remarkably similar for both cohorts and consistent with known programmatic changes in the treatment initiation guidelines towards faster ART initiation. Of note, 2008 PMTCT guidelines mandated that all pregnant women should be fast-tracked on ART [15], though no exact timeline was stipulated and clinic-level implementation may have been formalized somewhat later. In 2010, updated guidelines specified that pregnant women, patients with drug-resistant tuberculosis, and patients with nadir CD4 cell counts <100 cells/mm³ should be fast-tracked onto ART within 2 weeks of determining eligibility [11]. In 2012, fast-tracking was further expanded to include initiation within 2 weeks for those with CD4 counts < 350 cells and same-day initiation for those with CD4 counts <200 and all pregnant women regardless of CD4 count (Communication, National Department of Health, 2012). Thus, while workup date plus 21 days is a reasonable estimate for dates of initiation historically, analyses of patients initiating ART in more recent years and in the future might consider downward revisions to the 21-day interval. We note that differences of 2-3 weeks are unlikely to affect most analyses, and indeed our estimates of sensitivity, specificity, and positive and negative predictive values were virtually unchanged when imputing using the estimated annual median time to initiation in place of the 21-days value.

Having established 21-days as a reasonable estimate of time to initiation for patients with both workup dates and ART start dates, we turned to an assessment of the validity of using workup date plus 21 days to impute ART start dates. We estimated the sensitivity of our imputed start date for each of the cohorts based on whether the imputed date was considered a “match” to the known start date as defined above. Our results demonstrated high sensitivity of the imputed date. Among those with known ART start dates, very similar proportions (82.6% in Hlabisa and 88.2% in RTC) had a matched imputed start date (Table 2), indicating this imputation method will correctly estimate the ART start date as defined within a 6 month window period for roughly 85% of those who initiated ART. The sensitivity of our imputation approach was consistently high over time ranging from 78 to 93% over the 10 calendar years evaluated (Additional file 1: Table S2), and across facilities, ranging from 77 to 97% in all but two sites (Additional file 1: Table S1).

Next, we restricted the analysis to the Hlabisa cohort and evaluated the PPV, NPV, and specificity (Table 3). Among all patients with an ART workup — including those that never initiated — the probability that a patient went on to initiate ART within the six-month window was PPV= 95.4%, implying that fewer than 5% of patients with an ART workup would be falsely identified as initiating ART using our imputation strategy. Specificity and NPV were also both high (99.8% and 92.2% respectively). Sensitivity and NPV increased slightly after removing the requirement of a CD4 count in the previous 12 months from the definition of an ART workup (Tables 2 and 3, rightmost column).

As highlighted previously, we used a guideline-driven initial window period of 6 months to define a “matching” period for the imputed initiation dates. However, this period is arbitrary, and as we show in Fig. 2, the vast majority of patients initiated within a much shorter window. We thus created several alternate matching window periods varying from 1 month to 9 months and estimated sensitivity of the imputation approach with these changes in window period (Additional file 1: Table S3). Sensitivity dropped to about 60% when the window was restricted to one month but was over 80% at 3, 6, and 9 months.

Our analysis has a number of implications. The results provide guidance on whether the date of an ART workup can be used to infer dates of ART initiation in South Africa, and with what degree of certainty. We demonstrate high sensitivity and high positive predictive value of this approach, and performance appears to be consistent across diverse geographic and facility settings and over time. Laboratory-imputed initiation dates could be used as part of an effort to create a national clinical cohort from laboratory databases such as the NHLS (i) to monitor and evaluate South Africa’s ART program and (ii) to improve clinical records, patient care and accountability at all levels of the healthcare system. The implications for the evaluation of South Africa’s national program could be significant. The ability to impute dates of initiation within the national NHLS database will enable evaluation of key aspects of the national program, providing strong evidence (in a resource efficient manner) beyond what existing surveillance systems and cohort estimates have been able to do to date. These data could also be used to monitor facility performance and to improve accountability, although vigilance is required against the possibility of adverse behavioral consequences (e.g., conducting unnecessary Hb/ALT tests to boost numbers).

Laboratory testing is used routinely in determining HIV disease progression a nd assessing ART eligibility (historically), determiningART regimen, and monitoring patients on therapy in many southern African countries, though the specific protocols for how laboratory testing is used vary across countries and coverage is uneven [16‐23]. Although our approach — imputing clinical data using laboratory results — may have broad applicability, the specific imputation algorithms used may differ in other countries and will have to be validated in those contexts. Our analysis yields a validated algorithm for South Africa and a proof of concept that could be applied in other countries, particularly as laboratory infrastructure improves.

Our findings also have implications for understanding the continuum of HIV care and treatment. Although significant losses have been described between HIV testing and ART initiation both in the Hlabisa cohort [24] and elsewhere [25], our results on positive predictive value imply that the losses from date of ART work-up to ART initiation are modest. This implies that losses to initiation are occurring primarily within the first few clinic visits, between HIV testing and collection of workup bloods. Future interventions could target these early stages in the cascade of care, where most attrition occurs.

The methods described here to leverage routinely collected laboratory data may have other applications. First, improving the accuracy and completeness of clinical records may be an important and cost-effective step in overcoming obstacles to effective chronic disease management in low resource settings. Integration of laboratory data into clinical records could help manage highly mobile patient populations. Second, this methodology may offer several opportunities to enhance existing monitoring and evaluation strategies. Laboratory data could be used to triangulate against aggregate (e.g. district-level) information on new ART initiates based on clinic reporting. Because sensitivity is lower than PPV, the number of ART workups is expected to be lower than the number of ART initiators. However, the number of ART initiators can be estimated easily by multiplying the number of workups in a given population by a factor equal to PPV/sensitivity, equal to 1.15 in the Hlabisa cohort.¹ Of course, the generalizability of this adjustment factor depends on the generalizability of our sensitivity and PPV estimates. In settings where nationally complete registration data is not available, linkages between various sources of “big” data can significantly improve monitoring disease burden and health system response. South Africa has a national vital registration system, the National Population Register, and linkages between this register and several HIV cohorts have successfully improved ascertainment of mortality among those on ART [26, 27]. More recently, linkage to the South African National Cancer registry has also been completed [28]. Imputation of treatment start dates using the NHLS laboratory data creates a promising additional source of “big” data for linkage opportunities.

Our results must be considered in light of their limitations. First, despite the large database used, the method was tested in just two well-resourced cohorts with well-documented laboratory results. Though these cohorts do represent very diverse populations and geographic regions in South Africa, they may not represent all populations and regions in the national HIV program. Additionally, changes to ART initiation protocols or different practices in different countries would limit the generalizability of these findings; we encourage further validation in settings using different initiation protocols. Researchers recently showed the potental to impute ART start dates using laboratory-measured repeat viral loads in New York City, where (unlike South Africa) viral loads are assessed routinely both before and after initiation [29]. This example illustrates the broad potential of lab-based imputation as well as the necessity of tailoring imputation algorithms to local clinical protocols. Second, our estimates of PPV and sensitivity were computed in a patient population known to be HIV-infected. It is possible that implementing our approach in a population of patients with unknown HIV status would have a lower PPV due to the presence of false positives. However, the requirement of a CD4 count in the previous 12 months is expected to screen out HIV-uninfected patients and minimize false imputation of start dates. In populations including HIV-uninfected patients, NPV and sensitivity are expected to be much higher than reported here. Third, the trend towards decreasing time to initiation from workup by calendar year may require adjustment of the 21-day interval assumption by calendar period. Fourth, our identification of ART workup dates relied on the ability to identify a patient’s first ALT/HB, which requires longitudinal linkage of identifier in the national NHLS laboratory data are currently under way by members of this research team. Fifth, some patients may transfer out of the cohort facilities after their work up and initiate elsewhere. However, this implies that the high PPV demonstrated in the Hlabisa cohort (95%) is actually an underestimate and that true PPV may be even higher. Sixth, we may slightly underestimate PPV for late 2012 as some patients may have initiated ART after the end of the study period in December 2012 (Table S2). Lastly, if some laboratory tests were not captured into the cohort databases, then our estimates of sensitivity (82.6%, 88.2%) would also be underestimates of the true values.

Chronic disease management in low-resource settings is typically informed by routine laboratory monitoring and guided by standardized clinical protocols. We have demonstrated that routine laboratory testing data can be used to impute information about clinical decision-making that may not be complete in patient charts or aggregate statistics. The ability to impute ART start dates using lab workups has concrete implications for program monitoring and patient care in the South African national HIV CCMT program. Our focus on one disease, one treatment decision, and one country imply that caution should be exercised in generalizing from our results to other diseases, clinical decisions, and settings. However, our results offer a proof of concept. We are optimistic that this analysis may illuminate other opportunities to leverage routinely collected laboratory data to improve the accuracy and completeness of clinical records and improve chronic disease management in low resource settings in South Africa and beyond.