Solitary pulmonary nodule malignancy predictive models applicable to routine clinical practice: a systematic review

A systematic review of studies for the construction of predictive models of malignancy of SPN applicable in routine clinic settings. The study protocol was registered with the University of York Centre for Reviews and Dissemination International prospective register of systematic reviews (PROSPERO Record CRD42020161559, http://​www.​crd.​york.​ac.​uk/​PROSPERO/​).

We performed a search for scientific articles from the first available date in the following databases until October 2020: PubMed, SCOPUS and Cochrane Central. The final search equation was developed for use in MEDLINE, adapting it to the rest of the databases consulted, leaving the following: (“solitary pulmonary nodule” [MeSH Terms] OR (“solitary” [All Fields] AND “pulmonary” [All Fields ] AND “nodule” [All Fields]) OR “solitary pulmonary nodule” [All Fields]) AND (prediction [All Fields] AND model [All Fields]) AND ((“lung” [MeSH Terms] OR “lung” [All Fields] OR “pulmonary” [All Fields]) AND (“neoplasms” [MeSH Terms] OR “neoplasms” [All Fields] OR “malignancy” [All Fields])). We also completed the search with an assessment of the bibliographic list of the articles selected, including in the analysis studies that had been identified, but had not been detected in the digital.

The studies in this review were described considering the following data: first reference author and year of publication, where the study and follow-up were carried out, type of study, characteristics of the population, number of participants, prevalence of malignancy, prevalence of former smokers or active smokers, statistical analysis and predictor variables (Table 1).

In Table 2, we present the clinical and radiological variables of the 15 predictive models evaluated. These were described according to the recommendations of the Fleischner guidelines 2017 [7]. The clinical characteristics included: sex, race, emphysema, fibrosis, apical scarring, multiplicity and perifissural nodules; the radiological characteristics included: nodule size, growth rate, morphology, consistency and location. For SPN growth rate, the volume doubling time (1 VDT is equivalent to a 26% increase in diameter) is recommended, being in the 100–400-day range for the majority of solid cancers and on the order of 3–5 years for subsolid cancerous nodules.

Additional files 1 and 2 show the external validations; those carried out by the authors themselves and by other authors, respectively. In turn, Additional file 1 describes the results of applying models developed by other authors to the same sample. Furthermore, in both Appendices, we use the Transparent Reporting of a multivariable Prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement [14] to describe the type of predictive model included in each study included in our review, as well as the results of the discrimination and the calibration of these.

Finally, in Additional file 3 we describe the predictive mathematical models of each study evaluated.

The Prediction model Risk Of Bias ASsessment Tool (PROBAST) was used to assess the quality of the selected articles [11], with the aim of providing a structured judgement of the risk of bias, thereby allowing the analysis of the applicability and transferability of predictive models to clinical practice. It contains 20 items on potential biases distributed in 4 domains/dimensions (participants, predictors, results and analysis). Applicability is analysed for participants, predictor and outcome domains. The response templates for each model are reflected in Additional file 4.

In Table 3, as in Fig. 1 and Fig. 2, following the table format suggested by PROBAST [11], we have presented the quality results, representing the risk of bias, the applicability and the final global assessment, respectively.

The models risk of bias was assessed using the PROBAST tool [11] (Table 3, Figures 1 and 2). Regarding the Participants dimension, only 3 of the 15 studies were rated as appropriate by PROBAST, as case–control studies nested in a cohort [21, 27, 28]. The rest were non-nested case–control studies [15‐17, 19, 20, 22, 24‐26]; in 3 [18, 23, 29], the type of study was not clear. Only two studies [21, 28] included all patients in routine clinical practice; the other studies selected those who underwent surgery/biopsy or who had suspected malignancy. Regarding the Predictors dimension, the possibility in any study of the result of malignancy being known prior to evaluation as recommended by PROBAST could not be ruled out; only one [20] specified that the results were unknown. Furthermore, in one study [27] predictors were not evaluated similarly because radiographs were studied by different radiologists. Regarding the Results dimension, the method of determining malignancy (surgery, biopsy or follow-up) was not adequate in 8 studies [15‐17, 19, 20, 22, 23, 26] and was ambiguously estimated in 5 [18, 24, 25, 27, 29], only being correct in two [21, 28]. None reported whether the measurement of the results was performed without the predictors analysed being known. Furthermore, the time interval between the evaluation of the variables and obtaining the result was not adequate in 3 studies [21, 27, 28], in the rest it is unknown. Regarding the Analysis dimension, only 8 presented an adequate number of participants providing relevant results [20, 29] or an adequate number of events per variable [15, 16, 22, 24, 27, 28], as 3 could not be specified due to lack of data [18, 21, 23] or inadequate data in 3 [17, 19, 25]. In 5 studies [17, 18, 23, 26, 29], continuous variables were categorized/ dichotomized, and in only one [17] were data imputation techniques used for missing values.

Discrimination (AUC) and calibration (using the calibration slope ± Hosmer–Lemeshow test) were evaluated in 5 studies [17, 21, 23, 24, 27] , while in 4 others [15, 20, 22, 25], only the calibration (Hosmer–Lemeshow test) was evaluated; and in the rest [16, 18, 19, 26, 28, 29], only discrimination was evaluated. Only in 2 [17, 24] were bootstrapping techniques used to avoid overestimation of the model. In 8 of the 15 models [15‐17, 19, 21, 23, 27], the predictor weights of the models were assigned according to the results obtained from the multivariate analysis; of the rest, in 3 it was not clear due to lack of information [18, 28, 29] and it was not correct in 4 others due to errors in the mathematical equation [18] and because the assigned weights did not coincide with the multivariate analysis [22, 24, 26] (see Additional file 3).

All studies were classified with a high risk of bias compromising their applicability (Fig. 2).

We followed the PROBAST guidelines on potential biases distributed in 4 domains (participants, predictors, results and analysis) to set out several methodological deficiencies of the studies included [11].

There is clear disagreement between the prevalence of SPN malignancy found in the models included in this review (between 23 and 77.45%) and the prevalence in daily clinical practice (between 12.1 and 18.2%) [5]. This is probably due to the fact that most models are based on the population referred for surgery/biopsy, with consequent selection bias, since there is an important group of the population attended to in routine clinic settings—those considered to be at lower risk of malignancy and less likely to be sent to surgery/biopsy—not included in most of the models studied. This selection bias occurs in all the studies except three [21, 27, 28], which used a case–control design nested in a cohort study, also including those that only required radiological follow-up. The rest describe themselves as retrospective cohort studies [15‐17, 19, 20, 22, 24‐26], and in three, the type of study is not well established [18, 23, 29].

According to PROBAST [11], the prospective cohort study is considered the optimal design [11] with low risk of bias, since it allows all the information on the potential predictors (exposures) to be collected before the potential outcome, thus reducing selection or interviewer biases. Non-nested case–control studies in a cohort select a population from a study designed for another purpose, and therefore have a higher risk of bias. In line with the results obtained by Collins et al. [53], the models are seldom prospective and usually use information from populations intended for a completely different purpose.

Nodule consistency (solid, subsolid) is a determining factor when predicting SPN malignancy. The stability of solid nodules is estimated over a period of 2 years [6, 7], whereas in subsolids, it is 5 years [7]. Thus, longer initial follow-up intervals and longer total follow-up periods are recommended for subsolid nodules than for solid nodules. Bearing this in mind, this was insufficient in the 3 studies that followed up [21, 27, 28] with 2 years of follow-up, respectively. The remaining studies [15, 16, 18‐25, 27, 29] did not specify whether they followed up.

In some models, there was categorization of continuous variables: in one [17], the values of the biomarkers were dichotomized; in others, it was the smoking history (≥ 30 pack-years) [29], (≥ 400 pieces-year) [18]; and in one, it was the age (≥ 70 years) [26]. This establishes an arbitrary cut-off point, from which a different risk level is established, causing loss of information, so that predictive capacity is lost [11].

In most of the studies, the analysis does not mention patients with missing data. These are interpreted as having been omitted, meaning that the analysis performed is an “available/complete case analysis”. This is the most frequent type of analysis in predictive models and is the one which we suppose was in 14 of the 15 studies in which this information was not reported. The exclusion of missing data leads to biases in the association of the predictors with the result and skews the performance of the model because after the exclusion of cases with incomplete information, the selected subpopulation may not be representative of the population. Only one study [17] took into account the missing data, and used the multiple imputation technique as recommended by PROBAST, with a lower risk of bias, and is considered the best method described [11].

The optimal sample size for binary prediction models is considered to be a minimum of 100 events (preferably ≥ 200) for external validations [53], with 10–15 events per variable (EPV) (better ≥ 20) [54] for development models [11]. This was the case in only 8 studies [15, 16, 20, 22, 24, 27‐29].

The external validation of any development model in an independent sample is essential to demonstrate its satisfactory performance, i.e. applicability and transferability in clinical practice. One of the most important limitations of the models created so far is the lack of external validations. External authors have only validated 3 models [19, 21, 27] (Additional file 2), the most frequently evaluated being that of Swensen et al. [21], which has presented good discrimination in all of them, with values greater than 0.75 [55]. Although there are studies that have created models and have externally validated them with very promising results [17, 20, 25], there are no studies as yet that corroborate the results obtained.

In some studies [15, 20, 22, 25], the Hosmer–Lemeshow Test was the only calibration method used. However, it is not without limitations: large sample sizes can generate erroneous results and it does not reveal the magnitude of the difference between the predicted values and the observed values [55]. This does not happen with the calibration slope (the method most recommended by PROBAST), which was performed in only 5 articles [17, 21, 23, 24, 27].

The 15 models analysed showed low clinical applicability due to the high probability of bias. In normal practice, models that do not present selection biases are required, ones that reflect all possible malignancy risk profiles (from none to all) that may occur in a patient with an SPN found incidentally. Some models are not explicit in the exclusion of patients with a recent history of cancer (the last 5 years) [16, 24, 25, 27, 29]; possibly, they are more likely to experience a tumour recurrence/metastasis, thus overestimating the predictive values. In other cases, only solid nodules are included [17, 24] and cannot be applied to patients with subsolid nodules, and vice versa. Other recommendations on predictors and their measurements are that they should be standard and applicable to the clinical setting; specifically, biomarkers may not always be available.

The heterogeneity of the studies did not allow for a meta-analysis. Only studies in English, Spanish or Portuguese were included. These languages allow a wide coverage of more than 90% of articles in the literature, however, we discarded 6 articles written in other languages that could have been relevant.

Additionally, when we used the search equation, there were a large number of articles that were not ultimately relevant to the study objective. This may have been due to the lack of specific descriptors (MeSH), which meant that we had to use Pubmed to search for Title and Abstract fields.

Another limitation is that our literature search was carried out only in three databases (Pubmed, Scopus and Cochrane Central) not including for example important databases such as Embase. However, Scopus is a good alternative having the largest number of health articles which constitutes approximately 90% of the articles processed by PubMed, and more than 97% of the total titles processed by Embase [56]. Therefore, we believe that if there was a risk of publication bias from missing other key databases this was minimal.

The strengths of this review are the rigorous use of standard tools of proven methodological quality to evaluate the proposed models, and the independent selection and review of the articles included their quality, with high concordance among researchers in a relevant area of knowledge, as reflected in the Burton–Kebler Index and the Price Index. We attempted to minimize bias in the review by adhering to a registered protocol and following the PRISMA statement [57].