Treating loss-to-follow-up as a missing data problem: a case study using a longitudinal cohort of HIV-infected patients in Haiti

HIV programs are often assessed by the proportion of patients who are alive and retained in care, which has direct consequences for funding and programmatic services offered [1, 2]. However, among individuals who initiate antiretroviral treatment (ART), the reported rate of lost to follow up ranges from 5 to 53% [1, 3‐10]. Clinically, these LTF patients are at risk for adverse outcomes such as medication resistance, transmission to others, lack of care, or at best, incomplete medical records when they transfer care to another clinic [1, 6, 7]. Programmatically, lost to follow up leads to underestimates of retention which could be mis-interpreted as under-performance on program outcomes [1, 5, 6, 11].

The category of lost to follow-up (LTF) is not a homogeneous outcome—e.g., “dead” or “alive”—but rather a heterogeneous category of three disparate health states: undocumented deaths, undocumented or silent transfers to another source of HIV care, or alive and complete disengagement from HIV care [12‐14]. Alive and being retained in care is synonymous with the proportion of patients who are neither dead nor LTF. The fact that LTF is part of the definition makes this outcome complex and problematic.

In reality, LTF is a marker for missing data on vital status. We argue that LTF should not be treated as a legitimate outcome category because it’s meaning can easily change over time and across sites. For example, patients who silently transfer to another provider, move domiciles or die outside of a healthcare facility could all be classified as LTF. Thus, studying predictors of LTF should be avoided. Instead, LTF should be considered a missing data problem that needs to be solved. We present a unique application of MICE to impute both missing outcome (vital status) and missing covariates, simultaneously, using a large longitudinal cohort of patients from Haiti who were treated for HIV infection, and compare the results with MICE to the more traditional analytic methods of using complete cases and inverse probability weights. We also evaluated associations that were predictive of death using three different methods: complete case, inverse probability weights and multiple imputation with chained equations.

Among the first cohort of HIV patients who initiated antiretroviral therapy in Haiti from 2003 to 2014, we aimed to find associations that were predictive of death using three different methods: complete case, inverse probability weights and multiple imputation with chained equations. These three procedures have different assumptions and differed in the number of observations included in the adjusted model due to how missing values for co-variates were addressed. Although the point estimates were similar across the three models, for statistically significant factors we found as much as a 20% difference in odds ratio values. For statistically significant factors, such as severe poverty and WHO stage, the odds ratios in the MICE models were farther away from the null compared to the CC and IPW models. Severe poverty was a statistically significant predictor of death in the MICE model (OR 1.80; 95% CI: 1.28–2.52). In a similar cohort from the same clinic in Haiti, income was associated with a higher odds of attrition (OR 1.65; 95% CI: 1.25–2.19) [45]. Additionally, these estimates are similar to those from an intensive contact tracing program performed in Malawi on HIV positive patients, which found about 70% of people who were initially categorized as LTF were alive and 30% were dead [13].

Worldwide, LTF rates for patients who have initiated ART treatment for at least one year range from 5 to 53% [1, 3‐10]. Patient characteristics associated with becoming LTF include being clinically ill, as measured by CD4 count or WHO symptom staging, low socioeconomic status, and concern for stigma, as well as structural factors such as transportation issues [3, 7‐10, 45‐47]. Several studies have reported high rates of re-engagement in care by patients who were previously labeled as LTF [3, 4, 7, 8, 11, 45]. A study in South Africa found that up to 50% of patients who disengaged from care will re-engage within 3 years including care received at a hospital or emergency department visit [7]. Contemporary studies that were able to determine the true status of LTF patients—which is a small number—most had transferred care to clinics closer to their home or newer clinics that provide different services; or alternatively, were alive and not engaged in care [3, 4, 7, 11, 45]. Forster et al. found a strong correlation between clinics with high LTF rates also had high rates of missing data for patient characteristics [1]. Ideally, a formal tracking system that “follows” patients when they receive care at other institutions would be an optimal way to track silent transfers; however this is still in development in most countries [3, 4, 7, 10, 48]. With these findings that most LTF patients are actually alive, our method of imputing LTF status and missing covariates, at the same time, is a cost effective method to estimate true mortality and to study risk factors for HIV.

Each of the described methods in this article has different assumptions for LTF, as well as limitations and strengths (Table 4). For complete case analysis, the loss of statistical power by automatically excluding observations that have missing information is a concern for many researchers [15, 29]. This automatic exclusion leaves room for bias depending on the types and patterns of missingness [28, 29]. Many HIV studies have found that the underlying assumption that LTF is unrelated to mortality is an incorrect assumption and thus survival estimates and associations of death to be biased and incorrectly estimated [17, 21, 25, 49]. Clinicians report that those who were LTF back in the early 2000’s were later found to be dead compared to more contemporary cohorts whose LTF participants are more likely to be alive [13, 22, 25, 50, 51].

With regards to IPW from tracing data, there are many limitations associated with this methodology. IPW from tracing techniques assume that the traced participants are a representative sample of all LTF. With this assumption in mind, a random sample of LTF participants is selected for tracing [13, 20, 21, 52‐55]. In this cohort, tracing was attempted on all participants who were LTF and was performed with telephone and in-person follow up. Additionally, in this cohort, tracing was done at the end of the 10 year follow-up period, and those who were more recently lost were more likely to be found compared to those lost at the beginning of the follow-up period. Another limitation, inherent in most IPW analyses, is the non-inclusion of several observations because of automatic case-wise deletion by the analysis software due to missing data. With this in mind, estimates might be biased and a loss of statistical power might occur when utilizing this method [22, 56].

Unlike IPW, MICE is able to use all the observations in a dataset by imputing the missing values, resulting in robust results. However, it too has assumptions and is prone to limitations. One major assumption is that the risk of death among patients who are LTF is constant over time. This may not be the case as mortality is known to be highest in early periods after ART initiation and decreases over time [33, 34, 45, 57]. Additionally, MICE relies on a good prediction model and requires data to be missing at random (MAR) [29, 31]. Although MAR is difficult to ascertain, recent publications have explored the application of MICE in non-MAR situations and found that a small amount of bias might be present in the results. However, compared to the other methods, the small amount of bias that might be present is offset by the gains of using all observations present in the dataset and the robust standard errors calculated by the procedure [29, 30, 58]. Several studies have incorporated MICE as a method to estimate associations due to attrition or lost to follow up in longitudinal studies [59, 60]. Regardless of the method used, one must diligently explore patterns of missingness before performing any analyses [10, 25, 28‐31]. We believe that, despite some limitations with MICE, the benefits of using all available data and the subsequent calculation of robust standard errors outweigh the limitations. Therefore, the approach of imputing both the outcome and covariates seems better than more traditional methods.

Although we describe a statical approach to approximating survival rates, implementation research is needed to determine the effectiveness and scalability of interventions to keep patients engaged in care and to return them into care [3, 44, 45, 48]. HIV programs should consider including sensitivity analyses or other methods for estimating the vital status among those categorized as lost, as traditional methods, such as CC, IPW, Kaplan Meier and Cox proportional hazards models,do not consider that patients who are lost re-engage in care. The multiple imputation method that we describe in this paper provides an estimate that is closer to the actual outcome rates. Further research is needed to test this method in other countries and HIV programs to see if it provides outcome estimates close to actual rates.

In the last ten years, there has been an increase in the number of journal articles citing multiple imputation as a method used for filling in missing values or as a secondary analysis [53, 61, 62]. MICE might be a cost efficient mathematical alternative that can be employed in resource limited settings such as Haiti to impute outcome status estimates for program evaluation to estimate survival. However, data should be evaluated for patterns of missingness. Currently, MICE is underutilized in public health research—especially of HIV-infected cohorts. Because the benefits of MICE outweigh the potential for erroneous use, we encourage the use of MICE among our HIV research colleagues.