How to predict the electronic health literacy of Chinese primary and secondary school students?: establishment of a model and web nomograms

This study was designed as a cross-sectional study.

A total of 1300 students from seven primary and middle schools in Shaanxi Province, China, were surveyed from June to August 2021. In this study, cluster sampling was used to randomly select two primary schools, two middle schools, and three high schools in the main urban areas of Yulin City and Ankang City of Shaanxi Province. Four classes were randomly selected from each primary school, and four classes were randomly selected from each middle school and high school. The inclusion criteria were public schools, elementary students in grades 2–5, middle school students in grades 1–2, and high school students in grades 1–2. The exclusion criteria included private schools, first and sixth graders, junior middle school students, and senior high school students.

Two to four researchers were responsible for each study. To ensure the quality of the questionnaire, the students were guided by the researchers during the questionnaire-filling process. After explaining our study, informed consent was obtained from all participants or their legal guardians for those below 16 years old. Of the 1300 students interviewed, 65 were excluded from the analysis because of the large number of missing values in the questionnaire. We then randomly divided them into training and testing datasets at a ratio of 70:30, with 872 students assigned to the training database and 363 students assigned to the testing database.

All methods were performed in accordance with the transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD) [25] guideline and regulation.

We conducted a systematic review of HL in Chinese students [26], identifying all published observational studies in both Chinese (CNKI, Wan Fang, CQVIP) and English databases (PubMed, Embase, Web of Science, Cochrane Library) between January 2010 and September 2020 on factors that affect HL in Chinese students. The significant influencing factors for Chinese students were sex, location of the household grade, good academic performance, race, health information concerns, online game time, parental education, whether they were a single child, family monthly income, health education, if they were majoring in medicine or attending medical school. Therefore, we identified the following potential predictive variables for this study: sex, age, race, grade, family size, only child, employment status, household location, mother’s education, father’s education, and gaming time. We did not consider health information concerns, majoring in medicine, and medical school attendance because the influence group of these variables is college students in the systematic review. Academic performance was not included in the analysis because China’s current policy regards student performance as very important and private and is therefore difficult to obtain in the data collection process. Most primary and middle school students do not know their family income. Therefore, we did not include family income in the analysis. In addition to the factors mentioned above, another study found that self-efficacy and parental phubbing behavior were closely related to HL [27, 28]. Therefore, these two variables were included. General self-efficacy was measured using the general self-efficacy scale (GSES) [29], and parental phubbing behavior was measured using the parental phubbing scale (PPS) [30].

We used the eHEALS prepared by Norman et al. [2] to evaluate the EHL of primary and secondary school students (with vs. without). Students who scored above 80% were judged to have EHL [31]. We used 80% of the scoring nodes because we borrowed the Chinese HL classification method. There have also been other studies [15, 32] that have determined EHL using the 80% threshold. See Additional file 1 for more detailed information and the reliability and validity analysis of the scale.

Bivariate analysis was performed using the Mann–Whitney U test for continuous and ordinally distributed variables and the chi-squared test for categorical variables. For further analysis, a nomogram was formulated based on the machine learning results.

Random forest, a classical algorithm in machine learning, was selected for learning and prediction. The basis of random forest is a decision tree, which is a basic classification and regression method. The decision tree model takes the form of a tree. A classification problem represents the process of classifying instances based on their features. Random forest is an algorithm that combines the results of multiple decision trees for classification or regression. The number of decision trees constructed in this study was 500, and three variables were randomly selected for each node of the decision tree. Random forests select or exclude variables based on the importance of the features. Validated variables were used to create a simplified model rather than a complete model with all variables. Similar to other machine learning models, the random forest algorithm consists of training and testing steps. The computer first uses a training set to select the optimal model and then uses a test set to evaluate the model. The area under the curve (AUC) was used as an assessment tool, and AUC values between 0.6 and 0.8 were considered acceptable [33].

The least absolute shrinkage and selection operator (LASSO) is a regression analysis method used for simultaneous feature selection and regularization. This adds an L1 norm as a penalty in the calculation of the minimum residual sum of squares. When lambda is sufficiently large, certain coefficients can be accurately reduced to zero. LASSO has excellent feature selection ability. Therefore, we also conducted LASSO regression and compared the results with random forest.

The receiver operating characteristic (ROC) curve is drawn on a two-dimensional plane. It was drawn with sensitivity as the ordinate and specificity as the abscissa. Any point on the curve represents the corresponding sensitivity and specificity for the observed sample. The AUC refers to the size of a part of the area under the ROC curve, which is a standard used to measure the quality of a classification model and reflects the accuracy of the model. Typically, AUC values range from 0.5 to 1.0, with a larger AUC representing better model performance.

Decision curve analysis (DCA) reflects outcome variables and can be used to evaluate and compare different prediction models. The AUC only measures the accuracy of the prediction model and does not consider the actual utility of a particular model, whereas the DCA integrates the preferences of the object or decision-maker into the analysis.

To facilitate the application of the prediction model, we developed a web page based on a prediction model using Shinayapp. Statistical analysis was performed using R version 4.0.5 for Mac (R Foundation for Statistical Computing).

We found that 415 (33.6%) students were e-health literate, and 820 (66.4%) were not. Table 1 summarizes the characteristics of the total population of Chinese students (N = 1235). The univariate analysis showed that age (P < 0.001), grade (P < 0.001), employment status (P < 0.001), household location (P < 0.001), parental phubbing behavior (P < 0.001), and general self-efficacy (P < 0.001) were significantly associated with EHL.

Table 2 summarizes the characteristics of the Chinese students in the training dataset (n = 872). Moreover, we found that 287 (32.9%) students were e-health literate, and 585 (67.1%) were not. The univariate analysis showed that age (P < 0.001), grade (P < 0.001), household location (P < 0.001), parental phubbing behavior (P < 0.001), and general self-efficacy (P < 0.001) were significantly associated with EHL.

Table 3 summarizes the characteristics of the Chinese students included in the testing dataset (n = 363). In the testing dataset, 128 (35.3%) students were e-health literate, and 235 (64.7%) were not. The univariate analysis showed that general self-efficacy (P < 0.001) was related to EHL.

Thirteen variables measured at school (Tables 2 and 3) were included in the random forest. The process and results of feature selection by random forest are shown in Fig. 1, which identifies seven variables that were confirmed to be important: age, grade, employment status, father education level, game time, parental phubbing behavior, and general self-efficacy. Five variables were confirmed as unimportant: sex, mother’s education level, being an only child, number of people, and race. Furthermore, the variable of “household location” was excluded. To consolidate the results of the random forest feature selection, we performed a LASSO regression, as shown in Fig. 2. As expected, there were only two variables (grade and general self-efficacy) left in the LASSO regression model, far fewer than in the random forest model, due to the strong shrinkage capability of LASSO regression. These variables overlapped exactly with the variables identified in the random forest.

We then developed a prediction model using the seven identified key factors selected by the random forest. In the internal verification of the training dataset, the ROC showed that the model had high recognition ability, with an AUC of 0.975 (Fig. 3). The validation cohort included 363 students with a mean (SD) age of 13.5 (3.9) years, 188 (51.8%) males, and 128 (35.3%) students with EHL. In the independent validation database, the model showed satisfactory discrimination with an AUC of 0.738 (Fig. 3).

The EHL prediction score was constructed based on random forest. We used the model to build nomograms (Fig. 4). To further facilitate the use of our findings by policymakers and parents, this study presents nomograms in the form of a web page; that is, a web calculator was generated that can automatically calculate the probability of students having EHL according to seven key variables (https://​xietao.​shinyapps.​io/​DynNomapp/​).

In Fig. 5, the three lines in the training and test parts represent the different conditions. Smoothed net benefit Pr (EH) represents the prediction model used in this study. The other two lines represent two extremes: net benefit: Treat none represents a situation where none of the samples have EHL, and the net benefit is zero. Net benefit: Treat indicates that all samples have EHL, and the net benefit is a negative slope of the backslash line. As shown in Fig. 5, the model in this study is higher than the extreme curve in a large threshold range. Therefore, the model in this study has a relatively large optional threshold range and is relatively safe. For example, in the training data set, assuming that we choose a prediction probability of 30%, 15 out of 100 students who use the model will benefit from it without affecting anyone else.