Development and validation of classifiers and variable subsets for predicting nursing home admission

The data consisted of the records of 7259 home care customers between January 2011 and September 2015 in the city of Tampere, Finland. These data were linked to records that contained information regarding all social and health care service usage during the same period. Nursing home admission (model outcome) was indicated by whether the customer admitted to a nursing home or not, and coded as a binary indicator. The data were linked on the customer level using unique encrypted identifiers. We excluded clients with recorded home care episode shorter than 12 months between January 2011 and September 2015 (n=3192) and those whose RAI-HC (Resident Assessment Instrument - Home Care [17]) values (n=981) were missing. In total, we had 3056 customers (539 NHA is “true” and 2544 NHA is “false”) for analysis.

All the variables were calculated 3–12 months before the evaluation day t _ev . In addition, the variables were calculated 6–12 months before the day t _ev for additional analyses. The main variables are listed in the column “variable” of Table 1. The variables were selected by the experts of elderly care services. Figure 1 shows a time scale in which t _s,i is the starting day and t _e,i the ending day of home care according to the home care service data for customer i (i=1,…,3056). The variables were the numbers of events or boolean value [t r u e/f a l s e] that an event occurred between times t _k and t _k+1 (k=1,2,3) or t ₁ and t ₄. For example, variable j (a blue box in Fig. 1) for customer i was calculated from time period t _{2,i j} −t _{3,i j} . The interval between times t ₁ and t _ev was set to be 12 months and t ₁<t ₂<t ₃<t ₄<t _ev . If NHA variable was “true” for the customer i, that is the customer i was admitted to nursing home at time t _e,i , time t _{e v,i} was set to be the admission day (→ t _{e v,i} =t _e,i ). If the NHA variable of customer i was “false”, then time t _{e v,i} was a random day between times t _s,i and t _e,i , st. t _{e v,i} >t _s,i +12 months.

Table 1 presents general characteristics of the study sample (n=3056), of which 539 (17.6%) were admitted to nursing home. The table includes the results of t-test of significance difference for continuous variables and chi-squared test for categorical variables between the groups of home caring customers and nursing home residents.

The aim of this study was to find efficient variable subsets X _sub ={x _i |i=1,…n} from a large variable set X={x _j |j=1,…k} for predicting the NHA when n and k are the numbers of variables and n<k. Let Y be the binary vector of NHA variable, F(·) is a classifier and Y=F(X _sub ). That is, we predict the state of Y _i for customer i at time t _{e v,i} (Fig. 1), when the variable vector X _{s u b,i} is calculated from time range t _1,i −t _4,i (3–12 months before t _{e v,i} ) or t _1,i −t _3,i (6–12 months before t _{e v,i} ).

Variable selection is a mature research topic and has been used for many applications [18]. In this study we applied sequential forward selection (SFS) method [19] for variable subset generation. SFS starts with an empty set and adds one variable at a time from the original set X for classifier by maximizing the performance measure. Our primary performance metric was classification accuracy:

where y _{p r e d,i} is the predicted NHA class of the i-th sample, y _i is the corresponding true NHA class, n _samples is the number of samples and L(·) is the indicator function (L=1 if y _pred =y; L=0 if y _pred ≠y) [20]. We additionally calculated the average area under the curve (AUC) and true-positive rate (recall) values for classifiers. AUC values correlated almost perfectly with acc values, but we decided to report them, because in some research areas they are more familiar than the acc values. Recall of a classifier is calculated by dividing the correctly classified positives (true positives) by the total positive count (true positives + false negatives) [21]. That is, recall is the probability that a risk customer is found.

The strength of the accuracy metric, compared to the other common metrics, is that the accuracy metric is easy to understand. It should be noted, that our data set is highly unbalanced. We use a random operator to form balanced data sets and the performance results are reported on those balanced data sets instead of the original set. Otherwise, the accuracy metric would be biased and not suitable.

An alternative of SFS would be sequential backward elimination (SBE). SBE starts with X and eliminates one variable at a time by maximizing the performance measure. Our selection of SFS instead of the SBE method is justified by the ratio of relevant (# r) and all (k) variables. According to Liu et al. [18], if # r is small, then the SFS strategy should be used, and if the number of irrelevant variables (k−# r) is small, then the SBE strategy should be used. According to the pre-tests, the original variable set X includes many irrelevant variables (low univariate prediction power); thus, we prefer the SFS strategy.

Different classifiers have different performance for different data sets. In this study, we evaluated the performance of three classifiers: logistic regression (LR) [22], Gaussian naive Bayes (GNB) [23] and support vector classifier (SVC) [24]. That is, SFS was run three times using the classifiers of LR, GNB or SVC. Figure 2 shows the components of variable subset selection process. The subset generation component (SFS) feeds candidate variable subset X _sub to subset evaluation component. Evaluation component trains and validates classifier and calculates the accuracy values for the subset X _sub .

It should be noted, that we use a random operator to form balanced data sets for the analyses. Let A={X|Y=1} and B={X|Y=0} be the data sets. That is, the set A contains the data of customers with the values of “true” of NHA variable and the B with “false”. Because the set A is smaller (n=539) than the set B (n=2517), the balanced data set C was formed, st. C=A∪R(B) where R is a random operator for selecting 539 random samples from B, thus setting the level of chance at 50%. To be sure that the selection did not bias the results, data set C was formed 100 times.

Furthermore, classifier algorithms were trained and validated using a ten-fold cross validation method. That is, we formed sample C _i (i=1,…,100) from the data sets A and B, and split it into 10 equal-sized parts P _ik (P _ik ∈C _i and k=1,…10). The classification accuracy value, a c c _ik , was calculated by Eq. 1 for the part P _ik of the data set C _i when the parameters of the classifier were trained with the other K-1 parts of the data set C _i . The process was repeated for k=1,2,…10. The overall classification accuracy, CA, for the subset X _{s u b,n} of size n (n=1,…15) was calculated as

SFS calculated the best variable subsets for all balanced data sets C _i . That is, we have 100 variable subsets of size of 1–15 variables. The (average) importance of each variable was measured by a rank metric:

where r(i,j) is the rank of variable j based on sample C _i and # F is the size of the largest subset that was formed by SFS [25‐27]. In this study # F=15. Higher R(j) indicates that variable j is more important according to SFS, because it was selected for smaller size variable subsets. That is, variable has higher prediction capability according to SFS and its NHA classification ability is high.

We used four Python packages: sklearn [20], mlxtend [28], numpy [29] and pandas [30] to implement the classifiers and compute, acc, AUC, recall, C A _Xsub and R(j). SFS was computed by the function “SequentialFeatureSelector” in the package mlxtend. The classifiers of LR, SVC and GBN were implemented using the functions from the sklearn.linear_model, sklearn.svm and sklearn.naive_bayes packages. The packages of numpy and pandas were used for data reading and processing.