Key messages
-
Selective reporting of association and non-standardisation of cut points make evidence synthesis difficult
-
There was a negative correlation between improved discrimination and the baseline performance of Framingham Risk Score
-
No evidence was found between poor reporting practice and the magnitude of categorical net reclassification index
Background
Methods
Screening Process & Study Selection
Data extraction
Critical appraisal
Data analysis
Results
Included studies
Quality of included studies
The types and calculation of Framingham risk score
No. of Studies (n = 35) | |||
---|---|---|---|
Ordinal outcomes | |||
Items of Alteration (n = 18) | Yes | No | Unclear |
Addition | |||
Item 1. Antihypertensive | 16 | 15 | 4 |
Item 2. Weight related measures, e.g. BMI | 0 | 35 | 0 |
Item 3. Race/ethnic groups | 2 | 33 | 0 |
Item 4. Triglycerides | 2 | 33 | 0 |
Item 5. Alcohol | 0 | 35 | 0 |
Item 6. Previous cardiovascular disease | 1 | 34 | 0 |
Item 7. Others (family history, PVD & stroke) | 5 | 30 | 0 |
Deletion | |||
Item 8. Diastolic blood pressure | 9 | 21 | 5 |
Item 9. aDiabetes | 0 | 19 | 0 |
Item 10. HDL cholesterol | 4 | 28 | 3 |
Modification | |||
Blood pressure | |||
Item 11. Systolic blood pressure | 3 | 23 | 9 |
Item 12. History of hypertension/self-reported hypertension | 7 | 23 | 5 |
Item 13. Other blood pressure definition modification | 3 | 31 | 1 |
Lipid levels | |||
Item 14. History of hyperlipidaemia/ self-reported hyperlipidaemia | 14 | 14 | 7 |
Diabetes | |||
Item 15. aFasting glucose > 126 mg/dL or 7.8 mmol/L | 0 | 19 | 0 |
Item 16. aSelf-reported diabetes/ use diabetic medication | 3 | 10 | 6 |
Smoking | |||
Item 17. Pack years of smoking | 0 | 34 | 1 |
Item 18. Use of ex-smoker category | 11 | 23 | 1 |
Thresholds and reporting of association
Reference groups, n | Subgroups, n | Reported subgroups, n (%) | Missing p-value, n (%) | |
---|---|---|---|---|
All effect sizes | 92 | 381 | 328 (86) | 85 (26) |
OR | 4 | 27 | 27 (100) | 9(33) |
RR | 8 | 37 | 37 (100) | 18 (49) |
C-index | 0 | 22 | 22 (100) | 0 (0) |
HR | 80 | 295 | 242 (82) | 58 (24) |
Intended population for Framingham risk score
Documentation of regression, discrimination & AUC analysis
Part 1. Documentation of multivariable regression (n = 35) | No. | (%) |
a. Information on whether additional predictor is significant at <.05 level | 24 | 68.6 |
b. Results of a test that penalises for the inclusion of additional predictor | 8 | 22.9 |
Appropriate documentation (1a or 1b) | 26 | 74.3 |
Part 2. Documentation of AUC in ROC analysis (n = 33) | ||
a. Described method used to compare ROC curves | 13 | 39.4 |
b. Presented the AUC values with and without the additional predictor | 31 | 93.9 |
c. Presented CIs of AUC values with and without additional predictor | 9 | 27.3 |
d. Presented P value for comparison | 26 | 78.8 |
f. Availability or enable calculation of Δ AUC CIs | 30 | 90.9 |
Appropriate documentation 1 (2a and 2b and [2c or 2d]) | 11 | 33.3 |
Appropriate documentation 2 (2a and 2b and [2c or {2d or 2f}]) | 12 | 36.4 |
Part 3. Documentation of calibration (n = 35) | ||
Documentation of Hosmer-Lemeshaw test (n = 7) or Schoenfeld residuals (n = 1) | 8 | 22.9 |
Part 4. Documentation of reclassification analysis (n = 35) | ||
Report using table or text | ||
Not reported | 19 | 54.3 |
Partial | 5 | 14.3 |
Complete | 11 | 31.4 |
Standard of reporting of reclassification analysis (n = 16) | ||
a. Use of standard categories of risk | 11 | 68.8 |
b. Justified use of other categories of risk | 15 | 93.8 |
c. Reported the number of patients changing categories | 9 | 56.3 |
Appropriate documentation ([4a or 4b] and 4c) | 9 | 56.3 |
Inadequate | 7 | 43.8 |
Part 5. Documentation of NRI (n = 23) | ||
Type of NRIs | ||
Continuous/ category-free NRI | 4 | 17.4 |
Categorical NRI | 16 | 69.6 |
Reported both continuous & categorical NRIs | 1 | 4.3 |
Reported relative NRI | 1 | 4.3 |
Unclear | 3 | 13.0 |
Standard of reporting of categorical NRI (n = 16) | ||
a. Report censor handling | 5 | 31.3 |
b. No extrapolation | 7 | 43.8 |
c. Categorical NRI reference available | 14 | 87.5 |
d. Justification of risk categories | 14 | 87.5 |
e. Report NRI components | 5 | 31.3 |
f. Availability of reclassification table showing event and non-event | 8 | 50.0 |
g. Reclassification table enables the calculation of NRI components | 7 | 43.8 |
h. Combined NRI reported as a sum not a percentage | 8 | 50.0 |
i. The proportion of correctly reclassified subjects available | 7 | 43.8 |
j. Reported NRI not used to construct strong summary | 5 | 31.3 |
Adequate reporting of categorical NRI (> 5 items listed 5a–j)a | 11 | 68.8 |
No. | AUC FRS (median) | IQR | P value | No. | AUC FRS + CT (median) | IQR | P value | No. | Δ AUC (median) | IQR | P value | |
---|---|---|---|---|---|---|---|---|---|---|---|---|
1. Alteration of Framingham model | ||||||||||||
Major | 31 | 0.64 | 0.62–0.68 | 31 | 0.74 | 0.71–0.77 | 30 | 0.07 | 0.05–0.15 | |||
Minor | 42 | 0.7 | 0.64–0.74 | 0.0006 | 45 | 0.76 | 0.68–0.79 | 0.7271 | 46 | 0.05 | 0.02–0.09 | 0.015 |
2. Coronary heart disease measured | ||||||||||||
Yes | 58 | 0.68 | 0.62–0.72 | 61 | 0.75 | 0.71–0.78 | 61 | 0.06 | 0.04–0.11 | |||
No | 15 | 0.66 | 0.64–0.68 | 0.5208 | 15 | 0.72 | 0.68–0.75 | 0.2452 | 15 | 0.05 | 0.04–0.08 | 0.5393 |
3. Explore analysis model | ||||||||||||
Yes | 13 | 0.75 | 0.72–0.76 | 16 | 0.77 | 0.71–0.80 | 16 | 0.05 | 0.03–0.06 | |||
No | 60 | 0.65 | 0.62–0.71 | < 0.0001 | 60 | 0.74 | 0.71–0.78 | 0.4559 | 60 | 0.07 | 0.04–0.13 | 0.0274 |
4. Population as intended for Framingham | ||||||||||||
Yes | 45 | 0.68 | 0.64–0.72 | 48 | 0.75 | 0.71–0.77 | 48 | 0.06 | 0.04–0.11 | |||
No | 28 | 0.64 | 0.63–0.71 | 0.1841 | 28 | 0.74 | 0.68–0.78 | 0.5901 | 28 | 0.05 | 0.04–0.12 | 0.8546 |
5. Calibration reporting | ||||||||||||
Yes | 19 | 0.69 | 0.64–0.72 | 19 | 0.74 | 0.67–0.77 | 19 | 0.04 | 0.01–0.06 | |||
No | 54 | 0.67 | 0.62–0.72 | 0.1427 | 57 | 0.75 | 0.71–0.78 | 0.1465 | 57 | 0.07 | 0.05–0.12 | 0.0007 |
6. Validation reporting | ||||||||||||
Yes | 22 | 0.65 | 0.62–0.70 | 22 | 0.74 | 0.71–0.76 | 22 | 0.08 | 0.05–0.13 | |||
No | 51 | 0.68 | 0.36–0.74 | 0.0433 | 54 | 0.76 | 0.68–0.78 | 0.7267 | 54 | 0.06 | 0.04–0.09 | 0.1231 |
7. Multivariable documentation | ||||||||||||
Adequate | 52 | 0.64 | 0.62–0.71 | 52 | 0.74 | 0.71–0.78 | 52 | 0.07 | 0.04–0.13 | |||
Inadequate | 21 | 0.72 | 0.68–0.75 | 0.003 | 24 | 0.76 | 0.69–0.78 | 0.6588 | 24 | 0.05 | 0.03–0.08 | 0.1002 |
8. AUC documentation | ||||||||||||
Adequate | 28 | 0.72 | 0.64–0.75 | 28 | 0.77 | 0.69–0.80 | 28 | 0.05 | 0.01–0.08 | |||
Inadequate | 45 | 0.66 | 0.62–0.70 | 0.0018 | 48 | 0.74 | 0.71–0.77 | 0.3431 | 48 | 0.07 | 0.05–0.13 | 0.016 |
9. Reclassification analysis documentation 1 | ||||||||||||
Adequate (reference) | 14 | 0.69 | 0.62–0.72 | 17 | 0.76 | 0.74–0.78 | 17 | 0.06 | 0.05–0.11 | |||
Inadequate or not reported | 59 | 0.67 | 0.63–0.72 | 0.3924 | 59 | 0.74 | 0.70–0.74 | 0.2539 | 59 | 0.06 | 0.03–0.11 | 0.2032 |
Inadequate | 17 | 0.64 | 0.63–0.67 | 0.095 | 17 | 0.73 | 0.70–0.76 | 0.1678 | 17 | 0.07 | 0.05–0.11 | 0.9035 |
Not reported | 42 | 0.68 | 0.63–0.72 | 0.7189 | 42 | 0.74 | 0.71–0.78 | 0.3885 | 42 | 0.05 | 0.02–0.11 | 0.0772 |
10. Reclassification analysis documentation 2 | ||||||||||||
Inadequate | 17 | 0.64 | 0.63–0.67 | 17 | 0.73 | 0.70–0.76 | 17 | 0.07 | 0.05–0.11 | |||
Not reported | 42 | 0.68 | 0.63–0.72 | 0.0443 | 42 | 0.74 | 0.71–0.78 | 0.6452 | 42 | 0.05 | 0.02–0.11 | 0.0877 |
11. Types of incremental value threshold | ||||||||||||
> 2 | 40 | 0.68 | 0.63–0.72 | 43 | 0.74 | 0.68–0.76 | 43 | 0.05 | 0.03–0.08 | |||
< 2 | 33 | 0.67 | 0.62–0.75 | 0.731 | 33 | 0.77 | 0.73–0.83 | 0.0013 | 33 | 0.08 | 0.05–0.15 | 0.0034 |
12. Types of incremental value threshold 2 | ||||||||||||
> 3 | 12 | 0.72 | 0.69–0.74 | 12 | 0.76 | 0.74–0.78 | 12 | 0.05 | 0.04–0.07 | |||
< 3 | 61 | 0.66 | 0.62–0.71 | 0.0158 | 64 | 0.74 | 0.69–0.78 | 0.2884 | 64 | 0.06 | 0.04–0.11 | 0.192 |
Documentation of reclassification and NRI analysis
No. | NRI (median) | IQR | P value | |
---|---|---|---|---|
1. Reporting of censor handling | ||||
Yes | 13 | 0.18 | 0.14–0.43 | |
No | 33 | 0.26 | 0.19–0.35 | 0.4869 |
2. No extrapolation | ||||
Yes | 20 | 0.28 | 0.14–0.49 | |
No | 26 | 0.23 | 0.16–0.34 | 0.4186 |
3. Category NRI referencea | ||||
Quoted | 38 | 0.25 | 0.18–0.39 | |
Not quoted | 8 | 0.18 | 0.00–0.38 | 0.1922 |
4. Justification of NRI categoriesa | ||||
Yes | 40 | 0.25 | 0.15–0.41 | |
No | 6 | 0.21 | 0.18–0.29 | 0.7691 |
5. Reporting NRI components | ||||
Yes | 22 | 0.24 | 0.13–0.34 | |
No | 24 | 0.28 | 0.19–0.48 | 0.1907 |
6. Reclassification table showing the number of events and non-events | ||||
Yes | 32 | 0.27 | 0.18–0.37 | |
No | 14 | 0.21 | 0.14–0.43 | 0.3396 |
7. The availability of reclassification table with sufficient information to enable the calculation of event and non-event NRI | ||||
Yes | 30 | 0.25 | 0.18–0.35 | |
No | 16 | 0.23 | 0.14–0.48 | 0.9265 |
8. Describing the combined NRI as sum not a percentage | ||||
Yes | 25 | 0.28 | 0.14–0.47 | |
No | 21 | 0.22 | 0.18–0.30 | 0.256 |
9. Provide any indication of the proportion of correctly reclassified | ||||
Yes | 15 | 0.35 | 0.14–0.53 | |
No | 31 | 0.23 | 0.16–0.34 | 0.0916 |
10. Strong conclusion based on the reporting of NRI | ||||
No | 20 | 0.29 | 0.19–0.48 | No |
Yes | 26 | 0.23 | 0.13–0.34 | 0.08 |
11. NRI adequate documented | ||||
Adequate | 39 | 0.25 | 0.14–0.39 | |
Inadequate | 7 | 0.24 | 0.20–0.49 | 0.7949 |
12. Types of incremental value | ||||
> 3 | 18 | 0.28 | 0.2–0.44 | |
< 3 | 28 | 0.23 | 0.13–0.37 | 0.2464 |