Poor reporting of multivariable prediction model studies: towards a targeted implementation strategy of the TRIPOD statement

To cover a wide range of clinical domains, we started with 37 subject categories (2012 Journal Citation Reports®) [14] from which we selected the 10 journals with the highest journal impact factor (Additional file 1). After deduplication, 341 unique journals remained. We performed a search in PubMed to identify prediction model studies published in these journals before the launch of TRIPOD (May 2014), using a validated search filter for identifying prognostic and diagnostic prediction studies (Additional file 2) [15].

Eligible publications described the development or external validation of a multivariable prediction model (either diagnostic or prognostic) or evaluated the incremental value of adding a predictor to an existing model [1‐5, 16]. We excluded so-called prognostic factor or predictor finding studies, as well as studies evaluating the impact of the use of a prediction model on management or patient outcomes [3, 7, 17]. We excluded prediction model studies using non-regression techniques (e.g. classification trees, neural networks, and machine learning) or pharmacokinetic models. Titles and abstracts of the retrieved publications were screened by one of two authors (JAAGD or PH). After reading the full text report, they judged whether to include or exclude a potentially eligible publication. Any doubts regarding definitive eligibility were discussed, if necessary, with a third author. If we were not able to retrieve the full text of a publication via our institutions, it was excluded.

For each included publication we recorded the journal impact factor (2012 Journal Citation Reports®) [14], clinical domain, and whether the purpose of prediction was diagnostic or prognostic. Furthermore, we classified publications into four types of prediction model studies: development, external validation, incremental value, or combination of development and external validation of the same model. A publication could be categorised as more than one type of prediction model study. For example, if a publication reported on both development and external validation, but of different models, it was classified as development as well as external validation. If a publication included multiple prediction model studies of the same type, e.g. if two models were developed, we extracted data for only one model. If there was no primary model, we used the model that was studied in the largest sample. Information about study design, sample size, number of predictors in the final model, and predicted outcome was extracted for all included prediction models.

To judge the completeness of the reporting, we transformed items of the TRIPOD statement (Box 1) into a data extraction form, which was piloted extensively to ensure consistent extraction of the data. The TRIPOD statement consists of 22 main items, 10 of which are divided in two (items 3, 4, 6, 7, 14, 15, and 19), three (items 5 and 13), or five (item 10) subitems [12, 13]. For TRIPOD items (main or subitems, hereafter just called items) containing multiple reporting elements, we extracted information regarding each of these elements. For example, for item 4b, ’Specify the key study dates, including start of accrual, end of accrual, and, if applicable, end of follow-up’, we used three data extraction elements to record information regarding (1) the start of accrual, (2) end of accrual, and (3) end of follow-up. The data extraction form including all data extraction elements can be found on the website of the TRIPOD statement (www.​tripod-statement.​org/).

For each data extraction element we judged whether the requested information was available in the publication. If a publication reported both the development and external validation of the same prediction model, we extracted data on the reporting of either separately, and subsequently combined the extracted information for each data extraction element.

Three authors extracted data (JAAGD, PH, RP). If the authors disagreed or were unsure about the reporting of a data extraction element, it was discussed in consensus meetings with the other co-authors.

Based on the extracted data elements, we first determined whether the reporting of each TRIPOD item was complete (completeness is defined in the following subsection). We then calculated overall scores for completeness of reporting per model, per publication, and per item of the TRIPOD statement (across models).

Overall, publications adhered to between 16 and 81% of the items of the TRIPOD statement with a median of 44% (P₂₅–P₇₅ 35–52%) (Fig. 2). The reporting quality for prognostic and diagnostic prediction models was comparable, with a median adherence of 44% (P₂₅–P₇₅ 35–53%) and 41% (P₂₅–P₇₅ 34–48%), respectively. The most complete reporting was seen for the combined reporting of development and external validation of the same model (47%; P₂₅–P₇₅ 35–54%), followed by the reporting of model development (43%; P₂₅–P₇₅ 35–53%), external validation (43%; P₂₅–P₇₅ 37–54%), and incremental value (38%; P₂₅–P₇₅ 33–49%). No associations were found between completeness of reporting and sample size, journal impact factor, number of predictors in the final model, and prospective study design (data not shown).

Six TRIPOD items were reported in 75% or more of the 170 models, and 10 items in less than 25% (Table 1).

Completeness of reporting of individual TRIPOD items is presented in Fig. 3 and Additional file 3 over all 170 models, and per type of prediction model study. The most notable findings for each section of the TRIPOD statement (title and abstract, introduction, methods, results, discussion, and other information) are described below.

Our main finding of inadequate reporting in the majority of publications within 37 clinical domains is comparable to the findings of systematic reviews of prediction model studies performed in general medicine or specific clinical domains [6‐11]. Inadequate reporting is considered to be a form of research waste [18, 19]. Therefore, for many study types, reporting guidelines were published in the last 20 years, such as the Consolidated Standards of Reporting Trials (CONSORT) statement in 1996 (updates in 2001 and 2010), the Standards for Reporting of Diagnostic Accuracy (STARD) statement in 2003 (update in 2015), and Reporting recommendations for tumour marker prognostic studies (REMARK) in 2005 [20‐24]. Completeness of reporting before the introduction of these reporting guidelines was similar to our result of 44% adherence. Moher and colleagues (2001) evaluated 97 reports of randomised trials before the introduction of CONSORT and found adequate reporting for just over half of the items (58%) [25]. In a systematic review of 16 studies evaluating the adherence to STARD, overall, 51% of items were adequately reported [26]. For six included studies with quantitative data before publication of STARD, a range of 44–61% adherence was reported. An assessment of the reporting of prognostic studies of tumour markers was done shortly after the introduction of REMARK [27, 28]. Ten (out of 20) items were evaluated, and, overall, articles adhered to 53% of these.

With this literature review we cover a broad literature base by including three major types of prediction model studies, both prognostic and diagnostic, across 37 clinical domains. Despite the use of a validated search strategy, we may have missed publications on prediction models. It is likely that the completeness of reporting of prediction models in these studies would have been worse. Furthermore, we selected studies from high impact journals. Therefore, our results on the completeness of reporting might be an optimistic representation of the reporting of prediction model studies in general.

In accordance with the TRIPOD statement, we included prediction models based on regression modelling approaches [13]. Although most TRIPOD items would apply, transparent reporting of prediction models using non-regression modelling techniques may require additional details, especially regarding model-building procedures, and specific guidance might be desirable.

We were strict in scoring adherence by requiring complete information on all elements of a TRIPOD item; e.g. complete reporting of model performance required the provision of both discrimination and calibration measures. This is in line with the nature of TRIPOD as having essential items needed to appraise and utilise a prediction model. However, authors might have good reasons not to provide specific details regarding an item. For example, if they believe that their model should not be validated or used in clinical practice, they may have decided not to present the coefficients of the full model. In the current study we would have scored TRIPOD item 15a as ’incompletely reported’. Although strict scoring potentially leads to poorer adherence results, it is needed for reasons of consistency.

We used two different denominators in our analyses, the number of publications (n = 146) and the number of models (n = 170), which implies that in the ’model’ analysis a number of publications were included multiple times. It is likely that results from the same publication, although based on the reporting of different models, are correlated. Given the descriptive nature of our analysis, we did not adjust for such a possible correlation.

We present results from studies that were published 4 years ago; nevertheless, we expect these findings to still be applicable and relevant to current publications of prediction models. From evaluations of other reporting guidelines, like CONSORT and STARD, we know that it takes time to demonstrate the impact of a reporting guideline on completeness of reporting, and changes over several years might be small [25, 26, 28‐33]. In our opinion, therefore, it is too early for a before-after comparison at this moment, and the focus should first be on optimal implementation of TRIPOD.

Inadequate reporting impedes the use of all available evidence regarding a prediction model. First, as title and abstract were among the least well-reported items, identifying publications of prediction model studies might be challenging. In addition, we found the reporting of model development often insufficiently detailed, which makes external validation almost impossible. As a consequence, a new model might be developed, rather than making use of an existing model. Also, without model specifications it is impossible to use the model in clinical practice. Finally, inadequate reporting hinders critical appraisal and, thereby, the possibility of methodological investigation of sources of variation and bias in prediction model studies.

Experiences from other research areas indicate that the improvement in reporting after the introduction of a guideline is often slow and might be subtle [25, 26, 28‐33]. Improving the completeness of reporting of prediction models is probably even more challenging, as it is a relatively young, less well-known research field, with methodology still in development and not yet strongly embedded in education. Moreover, the multivariable nature of prediction model studies and their focus on absolute probabilities rather than on comparative measures require the reporting of many details on methods and results. In addition, practical issues, like word limits or journal requirements, could act as barriers for complete reporting.

The introduction of the TRIPOD statement was the first step in improving the reporting of prediction model studies. However, more activities should be undertaken to enhance the implementation of the TRIPOD statement. Active implementation involves a collaborative effort of developers of a reporting guideline and other stakeholders within the academic community, like journal editors and educational institutions. Apart from raising awareness and providing training, possible post-publication activities that are recommended are encouraging guideline endorsement, asking for feedback, and evaluating the impact of the reporting guideline [34].

By highlighting the flaws in the reporting of prediction model studies, our results enable a targeted implementation strategy for the TRIPOD statement. Possible future activities are the development of educational materials and training regarding specific aspects of the reporting of prediction model studies. The examples of both adequate and suboptimal reporting within our data set can be used in the training of different stakeholders. An initiative that already has been started by the TRIPOD Group is the development of specific guidance on informative reporting of prediction model studies in abstracts [35]. Furthermore, as TRIPOD is periodically being reappraised and will be updated if necessary, our study will provide useful input for modifications of specific TRIPOD items, related to content, phrasing, or more detailed explanation [12]. Finally, our study will serve as a baseline measurement for future studies evaluating the impact of the introduction of the TRIPOD statement.