Using value of information methods to determine the optimal sample size for effectiveness trials of alcohol interventions for HIV-infected patients in East Africa

To be concordant with a prior high evidence level alcohol study among HIV-infected patients in East Africa [36] and four recent National Institute of Health(NIH) funded RCTs in East Africa [37‐40], we studied a hypothetical randomized-controlled trial (RCT) that aimed to investigate the effect size of an alcohol intervention for HIV infected patients in East Africa with following assumptions: (1) Participants will be equally randomized to an intervention arm and a control arm; (2) Participant eligibility criteria are hazardous or binge drinkers (score ≥ 3 on the Alcohol Use Disorders Identification Test (AUDIT-C), a brief screening for heavy drinking and/or active alcohol abuse or dependence [36, 41‐43]) and being ART-eligible or ART-initiated in the past 12 months; (3) The trial could be funded for 6–36 months; (4) Baseline trial costs, including program costs, marginal treatment costs, and reporting costs, were aggregated into a marginal cost per sample that was also based on the four RCTs (Table 1). Note that the primary interest of this work was to investigate evidence necessary to reduce uncertainty regarding the effect size of an alcohol intervention. The effect size was measured as the relative risk reduction of unhealthy alcohol consumption.

Prior information provided by the alcohol trials that test the effect size of an alcohol intervention is defined as the information already available before a new trial, which may not be sufficient to support optimal decision-making. We specified the certitude regarding an alcohol intervention’s effect size prior to additional information (e.g., “prior distribution”) based on results of Papas and colleagues who evaluated a cognitive behavioral therapy-based intervention for HIV-positive outpatients with unhealthy alcohol use in Kenya [28, 36]. The intervention was based on social cognitive theory [44], and was designed to increase alcohol abstinence by teaching skills to mitigate substance use circumstances caused by stress or other problems. The intervention had been proven effective during the treatment and follow-up phases in Papas’ study, and effect size information reported in the study was used to estimate the prior distribution below. Specifically, in prior distribution, e₀ is the intervention effect size sampled from the prior beta distribution; \( \overline{e} \)is the expected prior effect size; and n₀ is sample size of the prior study (Table 1).

Prior distribution:\( {e}_0\sim Beta\left({n}_0\overline{e},{n}_0\left(1-\overline{e}\right)\right). \)

If a new trial with sample size n is initiated to gain additional information about the intervention effect size, in order to predict the sample statistics α_D for the new trial, we used a conjugate pair of the prior distribution to construct a predictive distribution. The predictive distribution is summarized below.

Predictive distribution: α_D~Binomial(e₀, n)

In order to utilize both of the prior and the new information on intervention effect sizes, an updated distribution was obtained by integrating the prior distribution and the predictive distribution through Bayesian Updating. e₁ is updated effect size.

Updated distribution: \( {e}_1\sim Beta\left({n}_0\overline{e}+{\alpha}_D,{n}_0+n-{n}_0\overline{e}-{\alpha}_D\right) \)

Given a specific simple size n for the new trial, we ran I iterations to generate statistics \( {\alpha}_D^{(i)}\ \left(i=1,\dots, I\right) \). For each \( {\alpha}_D^{(i)} \), we ran J iterations to generate \( {e}_1^{(j)}\mid {\alpha}_D^{(i)}\ \left(j=1,\dots, J\right) \). We used a total number of one million (I × J) iterations to output a set of updated effect sizes to ensure the robustness of our VOI results. More details about the approach can be found elsewhere [31, 32].

As previously discussed, alcohol interventions among HIV infected patients might reduce HIV transmission and improve health, and additional information on the effectiveness of such interventions comes with the opportunity cost of not spending those funds on the interventions themselves, which are potentially cost-effective. We used a previously published and validated HIV model to test the impact of the updated distribution of intervention effect size e₁ on the decision of adopting (s = 1) versus not adopting (s = 0) the alcohol intervention [45, 46]. The model outcomes necessary for the subsequent cost-effectiveness evaluations and VOI calculations were quality-adjusted life years QALY(s, e₁) and costs cost(s, e₁) for the intervention scenario (s = 1) and the null scenario (s = 0) respectively. A detailed description of the HIV model is reported elsewhere [45, 46]. Briefly, the HIV model contains an HIV progression module and an HIV transmission module. The HIV progression module approximates the health outcomes by evaluating the change of CD4 cell counts and HIV-1 viral load for each HIV-infected individual. This module also simulates the effect of ART and takes the major causes of ART failure into account, such as ART non-adherence, non-adherence related genotypic resistance, and medication toxicity [46]. The individual-based HIV progression module interacts with a population-based compartmental HIV transmission module which simulates heterosexual transmission among the population in East Africa. The model compartments are differentiated based on health characteristics as well as behavioral risk characteristics. A hypothetical population switches compartments based on change to their health and behavior status. Probability of transmission is a function of multiple factors, including rate of acquiring new partners, duration of partnership, frequency of sexual contact within a partnership, and likelihood of condom use. The people who are in the compartments of unhealthy alcohol consumption were modeled as having these three major factors: increased risk of condom nonuse, increased risk of ART non-adherence, and increased STI prevalence (Table 1). The alcohol intervention reduces the HIV infection rate by transferring people from the compartments representing unhealthy alcohol consumption to those without unhealthy alcohol consumption.

In value of information (VOI) methods, expected value of perfect information (EVPI) represents an upper bound estimate on the potential benefit of a new trial if we assume new trial information would be perfect to help policymakers eliminate all decision uncertainty. Therefore, EVPI is used in economic evacuation to determine the necessity of conducting a new trial. Mathematically, it is the difference between the value of the decision made based on perfect information (e.g., after an RCT without bias and with infinite sample size) and the value of the decision made based on prior information (e.g., before a RCT) [31, 32]:

A positive EVPI is a necessary condition for conducting a new RCT. Net benefit was calculated using the QALYs (s, e₁) and costs (s, e₁) generated by the HIV model using the following equation [31, 32]:

WTP is the decision maker’s willingness-to-pay for incremental health benefit. The World Health Organization’s (WHO) Choosing Interventions that are Cost-Effective (CHOICE) program recommends benchmarking willingness-to-pay (WTP) based on gross domestic product (GDP) per capita [47], in particular between 1 and 3 times the annual GDP per capita. Alternatively and in greater accord with economic theory, WTP may be inferred from a desired program that is not fully implemented because of resource constraints (e.g., routine viral load testing).

While EVPI estimates an upper bound benefit when information is perfectly known after a new trial (i.e., sample size of the new trial approaches infinity), expected value of sample information (EVSI) estimates the potential benefit of a new trial with a finite sample size. We then calculated the expected value of sample information (EVSI) which compares the net monetary benefit of a decision made with the updated information (e.g. after the RCT) and the benefit of a decision made with the prior information (e.g., before the RCT):

\( {E}_{\alpha_D}\mathit{\max}{E}_{e_1\mid {\alpha}_D} NB\left(s,{e}_1\right) \) is the expected net benefit of the decision made based on the updated information (e.g., after the RCT), and \( \mathit{\max}{E}_{e_1} NB\left(s,{e}_1\right) \) is the expected net benefit of the decision made with the prior information (e.g., before the RCT) [31, 32].

In additional to evaluating expected trial benefit, costs associated with conducting such a trial should be weighed against potential benefit in trial design. Expected net benefit of sampling (ENBS) values the expected net benefit of a new trial as the difference between the monetized expected value of sample information (EVSI) and the marginal investment of such a trial [31, 32]:

In the ENBS equation, CS is cost of sampling; \( \overline{c} \) is marginal cost per sample (Table 1); and n is the sample size of a trial. As sample size grows larger, the incremental certitude from information represented by EVSI grows smaller (i.e., diminishing returns) whereas the incremental cost of conducting the trial may not. As long as EVSI, or the value of conducting the new RCT, exceeds the corresponding study cost (ENBS > 0), the return on investment would be positive. An optimal sample size could be obtained at which the expected net benefit of sampling (ENBS) is maximized.

Since ENBS is greatly influenced by policymakers’ willingness-to-pay (WTP), we performed a sensitivity analysis to evaluate how optimal results change over a series of WTP benchmarks, where WTP was measured using the standard metric of US$ per additional quality-adjusted life year (QALY), where QALY is a measure that aggregates additional quality and quantity of life. We also performed a sensitivity analysis across three intervention duration scenarios: long, medium, and short (20 years, 10 years, and 5 years respectively).

Given QALY and cost outcomes estimated by the HIV model, we applied cost-effectiveness acceptability curves [48] to identify where the decision uncertainty was greatest and how it varied by willingness-to-pay threshold. In Fig. 1a, when WTP threshold was greater than $3200/QALY or smaller than $900/QALY (shaded area), there would be no decision uncertainty with regard to choosing a more cost-effective strategy: prior evidence suggested that policymakers should adopt the intervention when WTP > $3200/QALY or should not adopt the intervention when WTP < $900/QALY. In Fig. 1a, decision uncertainty existed in the unshaded area, and the greatest decision uncertainty arose when WTP = $1710/QALY. In this case, additional evidence from a new trial might be necessary to increase the certitude. Similarly, if we implement a study for 10 or 5 years (Fig. 1b and c), the greatest uncertainty exists when WTP = $8000/QALY or $31,000/QALY respectively. Given the distributions of decision uncertainty in unshaded areas in Fig. 1, EVPI can be used to quantify the consequence of the decision uncertainty (or the health benefit of addressing the uncertainty) in a monetary format.

We used EVPI (Fig. 2) to quantify the uncertainties in Fig. 1, and the EVPI varied substantially over different WTPs and different durations of intervention implementation. When implementation duration was assumed to be 20 years, EVPI was maximized ($14.8 million) at a WTP of $1710/QALY, well within the suggested range of WTPs for East Africa ($1014/QALY to $3042/QALY), suggesting that a new RCT could yield valuable information. However, at shorter durations, the minimum WTP required to produce a positive EVSI fell outside the recommended range rendering the prospect of conducting a trial too expensive and not a good return on investment. For a duration of 10-years, EVPI was positive when WTP was greater than $4500/QALY and maximized ($5.8 million) at a WTP of $8000/QALY. For a duration of 5-years EVPI was positive when WTP was greater than $18,000/QALY and maximized ($3.4 million) at a WTP of $31,000/QALY. Since EVPI is the upper bound estimate on the potential benefit of conducing a new trial, our EVPI results in Fig. 2 indicated that it might be worthwhile to conduct a new trial when the intervention implementation duration is 20 years. However, when intervention duration was set to 10 or 5 years, EVPI was zero across the WHO recommended range of WTPs for East Africa ($1014/QALY to $3042/QALY) [47], suggesting that a RCT should not be conducted under these two scenarios.

EVSI and ENBS were calculated to estimate an optimal sample size for a new study after the necessary condition was satisfied (EVPI > 0). At baseline intervention duration of 20 years and baseline WTP equal to $2473/QALY, the incremental cost per QALY of implementing routine viral load testing for HIV-infected patients in East Africa, incremental cost of sampling for the trial grew larger whereas EVSI produced by the trial was diminished as the sample size of the trial grew larger in Fig. 3. Thus, the optimal sample size was found at the place where ENBS was maximized in Fig. 3. Specifically, the optimal sample size for the new RCT was 200 (100 per arm) and the corresponding maximum ENBS was $67 thousand US dollars (Fig. 3) for the base case scenario.

For a scenario of using a conservative WTP (1 X GDP/capita of East Africa, $1014/QALY) and assuming an implementation duration of 20 years (Table 2), conducting a new trial was not recommended since the return on investment was negative (ENBS < 0). Conducting a new trial was also not suggested even when policymakers were willing to pay more for an additional health gain and thereby using an upper bound WTP of 3 X GDP/capita ($3042/QALY) and also assuming an implementation duration of 20 years (Table 2). However, setting WTP equal to the incremental cost-effectiveness ratio (ICER) of the alcohol intervention ($1710/QALY, within the lower and upper WTP bounds but distinct from our base case assumption), ENBS rose to more than $11 million and corresponding optimal sample size reached 500 (250 per arm).