Surveillance for Hepatocellular Carcinoma: Development and Validation of an Algorithm to Classify Tests in Administrative and Laboratory Data

Abstract

Background

The purpose of alpha-fetoprotein (AFP) and abdominal ultrasound (US) cannot be discerned in administrative data.

Aim

We developed an algorithm to identify AFP and US used as surveillance tests for hepatocellular carcinoma (HCC).

Methods

We evaluated 300 AFP and 301 US tests from a VA database. Surveillance predictors in the administrative files (diagnoses, labs) were examined in logistic regression models. We calculated model-based probabilities of HCC surveillance status, and developed classification procedures using single and multiple imputation methods.

Results

The predictors of surveillance intent for AFP were absence of alcoholism, abdominal pain, ascites, diabetes and high AST levels. For US, the predictors of surveillance were prior AFP testing and HIV status and absence of abdominal pain, ascites, or drug dependence. For AFP classification, single imputation compared favorably with multiple imputation, both showing robustness in discrimination and calibration. For US both approaches were less robust in discrimination and calibration which was more moderate in multiple imputation than single imputation.

Conclusions

Predictive algorithms in administrative files can be used to identify AFP performed for HCC surveillance, however, the intent of US is more difficult to identify.

Appendices

Appendix A

See Table 6.

Table 6 Variables for consideration in model development

Appendix B

Direct Multiple Imputation

In the direct multiple imputation approach, surveillance counts are taken to be expected values of those counts as random variables with respect to the probability distribution induced by the model-predicted surveillance probability values. Estimates of expected values of these and other random variables are obtained by taking means of the imputed counts over the imputation iterations. In particular, estimation of parameter estimates in multiple imputation approaches to generalized linear modeling proceeds in this same way [12].

Decision-Theoretic Framework for Dichotomization

The decision-theoretic approach involves treating the predicted probabilities as a continuous scale, on which, thresholds are determined which minimize the total misclassification costs of true and false positives and negatives.

Costs are assigned per unit to true and false positives and negatives:

$$ \begin{aligned} {\text{Total}}\,{\text{cost}} \,=\, & \alpha *\left( {\# {\text{TP}}} \right) + \beta *\left( {\# {\text{FP}}} \right) + \gamma *\left( {\# {\text{FN}}} \right) + \delta *\left( {\# {\text{TN}}} \right) \\ \,=\, & {\text{N}}*(\left( {\alpha - \gamma } \right){\text{sens}}\left( {{\text{true}}\,{\text{screen}}\,{\text{rate}}} \right) + \left( {\delta - \beta } \right)*{\text{spec}}*\left( {{\text{1}} - {\text{true}}\,{\text{screen}}\,{\text{rate}}} \right) \\ & +\, \gamma *\left( {{\text{true}}\,{\text{screen}}\,{\text{rate}}} \right) + \beta *\left( {{\text{1}} - {\text{true}}\,{\text{screen}}\,{\text{rate}}} \right) \\ \end{aligned} $$

This cost function can be applied directly to the vertices of the ROC curve for the predictive model. Those vertices result in minimal yield of the desired dichotomization thresholds. Slopes of level curves of this cost function can assist in this process:

The slopes of the level curves of the total cost function on the ROC plane (with x = (1 − spec) and y = sens) are equal to

$$ \frac {(\delta - \beta)*(1 - {\text {true screen rate}})}{(\alpha - \gamma)*(\text {true screen rate})} = \frac {(\delta - \beta)/(\alpha - \gamma)}{\text{odds of true screen}}$$

In case, as with most applications, the costs for true positives and true negatives are zero, this is simply

$$ {\frac{{{\frac{\beta }{\gamma }}}}{{{\text{odds}}\,{\text{of}}\,{\text{true}}\,{\text{screen}}}}}. $$

The cost-minimizing ROC vertices will be tangency points for these slopes.

In particular, equal unit misclassification cost assignments to false negatives and false positives (β = γ) result in cost-minimization, which coincides with maximization of agreement (#TP + #TN).