Constructing treatment selection rules based on an estimated treatment effect function: different approaches to take stochastic uncertainty into account have a substantial effect on performance

To compare these principles we introduce some basic notations. Let X be the continuous covariate representing the biomarker value. Let Y be a continuous outcome and T the treatment indicator, randomized with a 50 percent chance to 0 or 1, and indicating a treatment with the standard or the new treatment, respectively. The treatment effect θ(x) is defined as the difference between the expected outcomes: We assume that higher values of Y represent a higher treatment success. Thus, a positive treatment effect characterizes superiority of the new treatment.

A treatment selection rule can be regarded as the choice of a subset C of all possible values of X. Patients with covariate values in C should receive the new treatment instead of the standard treatment in future. A construction method is an algorithm to transform the data (Y_i,X_i,T_i)_i=1,...,n observed in an RCT into a set C. Since the result of a construction method depends on random data, we consider it as a set-valued random variable \(\mathcal {C}\). We can study the performance of the construction method by considering the distribution of \(\mathcal {C}\).

We start by defining quality measures for a single set C. Since this set C determines the treatment selection for future patients, we introduce a new random variable X^∗ denoting the biomarker value for future patients. We consider three quality measures:

Sensitivity and specificity focus on the correct classification of patients by the treatment selection rule. Sensitivity measures the ability to select those patients who can expect to benefit from the new treatment. Specificity measures the ability to avoid recommending the new treatment to patients who cannot benefit from it. The overall gain is a summary measure taking into account also the magnitude of the treatment effect. It represents the change in the average outcome (i.e. in E(Y)), when we apply the proposed treatment selection rule in future, i.e. patients with x^∗∉C receive the standard treatment and patients with x^∗∈C receive the new treatment. It takes into account that θ(x^∗) may be actually negative for some patients selected by the rule. The gain can be also seen as one specific way to balance between sensitivity and specificity, or – to be precise – between true positive and false positive decisions. A patient with θ(x)>0 correctly selected to receive the new treatment gets a weight equal to his or her individual benefit. A patient with θ(x)<0 incorrectly selected to receive the new treatment gets a weight equal to his or her individual, negative benefit. All patients selected for standard treatment get a weight of 0.

We have chosen these three measures, as they cover important characteristics. The different construction principles mentioned in the introduction can be regarded as attempts to control the specificity at the price of a reduced sensitivity. The overall gain measures the success of obtaining a sufficient balance in the sense that a low specificity decreases the overall gain by including too many patients with a negative θ(x^∗), and a low sensitivity decreases the overall gain by excluding too many patients with a positive θ(x^∗). However, it takes also into account that it is most favourable to include patients with large positive values of θ(x^∗) and least favourable to include patients with large negative values of θ(x^∗). Measures similar to the overall gain have been considered in the literature, but mainly with respect to the optimal rule C={x∣θ(x)≥0} as a measure of the benefit we can expect from a new biomarker. See [2] and the references given there. In the presentation of the results we will also indicate the maximal possible overall gain as a benchmark, defined as \(E (\theta (X^{*}) {1}\hspace {-.1cm} \mathrm {I}_{\theta (X^{*}) \geq 0})\).

To describe the performance of a construction method for treatment selection rules, we study the distribution of these three quality measures when applied to \(\mathcal {C}\) under the assumption that X^∗ follows the same distribution as X. In this paper we will only consider the mean of this distribution, i.e. the expected sensitivity, the expected specificity, and the expected overall gain. In the context of comparing different subgroup analysis strategies, the expected overall gain has also been considered by [12].

As mentioned above, we will consider four different construction principles for the treatment selection rule. All of them are based on the assumption that we have some statistical method providing us with an estimate \(\hat \theta (x)\). Three principles assume that we can also perform certain types of statistical inference in order to construct pointwise or simultaneous confidence bands of the treatment effect or confidence intervals for the roots of θ(x). In the sequel, let l_p(x) and l_s(x) denote the value of the lower bound of a 95 percent pointwise and simultaneous confidence band, respectively. Let CI(x_r) denote a confidence interval around any root x_r, i.e. \(x_{r} \in \hat \theta ^{-1}(0) =\{x \mid \hat \theta (x)=0\}\). Then, the construction principles can be described like shown in Table 1.

There is a close conceptual relation between the two principles POI and CIR. Both aim at excluding marker values x for which θ(x)=0 is "likely". POI tries to identify these values by considering the uncertainty in \(\hat \theta (x)\). CIR tries to identify these values by considering the uncertainty in determining the root(s) of θ(.). (There can be several roots when θ(.) is chosen as a non-linear function, resulting in the somewhat technical definition shown above). Moreover, there is a direct mathematical relation. If a pointwise 1−γ confidence band for θ(.) is given, we can interpret it not only vertically, but also horizontally in the following sense: If for a given θ_t we consider all values of x such that (θ_t,x) is within the confidence band, then these values define a 1−γ confidence interval for θ⁻¹(θ_t). A proof is outlined in Additional file 1.

We will nevertheless consider POI and CIR as different approaches, as there are a variety of methods to obtain confidence intervals for θ⁻¹(0). In particular we will consider a simple application of the delta rule to obtain standard errors of θ⁻¹(0), as it has been also used in [1].

In the general set up of the simulation study we generate a random variable X∈[0,1] representing the biomarker. T is generated as a Bernoulli random variable with a probability of 0.5. The continuous outcome Y follows a normal error model: Y=α(X)+θ(X)T+ε, where ε∼N(0,1). As the error variance is fixed to one, the value of θ(x) can be interpreted roughly as an effect size. We chose to investigate three shapes for the treatment effect function θ(x), a linear, a concave and a convex shape, see Fig. 1. Within each shape we have a scaling parameter β reflecting the steepness of the function. For the linear case we chose to investigate two different distributions of the biomarker, \(X \sim \mathcal {U}(0,1)\) or \(X \sim \mathcal {T}(0,1,1/3)\), while we only look at a uniformly distributed biomarker for the other two shapes. Here \(\mathcal {T}(a,b,c)\) denotes a triangular distribution on the interval (a,b) with a mode in c. We do not consider the case of a normally distributed X, as the theory behind the methods we use to construct simultaneous confidence bands applies only to bounded intervals. Thus, in total we are investigating four scenarios summarized in Table 2. Without loss of generality we will assume α(x)=0 in generating the data. This is justified if we assume that the analysis models used are correctly specified with respect to α(x), such that the estimates for θ(x) are invariant under the transformations Y^′=Y+α(X).

In estimating θ(x) we use linear regression assuming a linear or a quadratic model for α(X) and θ(X):

We will focus on using the “correct” analysis model, i.e. we apply the quadratic analysis model if θ(x) is concave or convex, and the linear model otherwise. The mathematics for building the pointwise and simultaneous confidence bands and the confidence intervals for the roots are outlined in Additional file 2. Candidate sets are constructed as described above for each of the four principles. However, this step is only performed in case of a significant interaction test, i.e. if H₀:β₁=0 or H₀:β₁=β₂=0, respectively, could be rejected at the 5 percent level. In case of no significance all candidate sets are empty, i.e. \(\mathcal {C} = \emptyset \).

In addition to the performance characteristics expected sensitivity, expected specificity, and expected overall gain, we also consider \(P(\mathcal {C} \not = \emptyset)\), i.e. the probability to select at least some patients for the new treatment. We refer to this probability as the power, as it reflects the chance to get a “positive” result from the investigation of interest. It will also allow to judge the relevance of a chosen β value. The numerical computation of the performance characteristics is outlined in Additional file 3.

All calculations were performed using Stata 13. We used the available built-in procedures for generating random numbers, performing linear regression, construction of pointwise confidence bands (lincom) and application of the delta rule (nlcom). The calculation of the simultaneous confidence intervals were performed with self-written Stata programs and self-written functions in Mata, a programming language integrated in Stata. Source code for reproducing the results of the simulation can be viewed as Additional file 4 which also includes the data sets produced by the simulation.