Discussion
The method of adaptive monitoring was previously published by Jacke et al. on a breast cancer query database from two 1-year episodes (1996/1997, 2003/2004) [
21,
22]. In total, 877 cases were included in the study. Instead of an actual SDV, a secondary database was taken, and distributions of data were compared. This approach is suitable for large data sets, yet somehow questionable due to selection bias from the primary to the secondary database. Partly, a similar approach was used when scoring the data quality of the BFCC registry, when the length of stay was oriented on 49.778 orthopaedic and trauma patients analysed by Chona et al. [
20]
Jacke et al. could reach an improvement of data quality from 51.7 to 67.7% [
21], after adjusting the parameters, similar to the scoring the data quality of the BFCC fracture registry. Initially, when crude registry data was taken to calculate the score, a medium data quality with a scoring result of 50 was calculated. During the SDV, it was found that the actual difference between the registry and source was a mere 5.48%, in contrast to 50.8% of missing data elements. Consequently, a new score was calculated, including the individual weight of 3 for the optional data element of “height and weight” (Table
1). An adjusted score value of 75 was the result and placed the data quality at the upper end of ‘good’ (Table
2). For future scorings of data quality, a different item for the indicator ‘optional data elements’ is recommended.
Since no two registries use matching methods for data quality evaluation, reproducibility and comparability between registries are hardly possible, which yet again shows the strength of the method of adaptive monitoring. The score of the data quality has a direct consequence on the sample size for SDV. On top, partly biased by the selection of parameters investigated and possible modification of thresholds, attempts for comparability between data quality in registries can be made. The items chosen for scoring can vary hugely from registry to registry, leading to procedure bias.
The BFCC project chose to implement a further modification, by splitting up the indicator “compliance with procedural rules” into two investigated items. The choice was made to split the relatively high individual weight of 6 into two times 3 (Table
3). Hence, the compliance with a legal procedural rule “patient age” at registration and a registry specific procedural rule of “fracture age” could be individually taken into consideration for scoring.
The Anglo-American date format of ‘month/day/year’ used in the registry’s software probably caused faults in both fracture age and length of stay recordings, since the date format of ‘day/month/year’ is used in Germany. As the pilot phase of the registry was conducted from November 2017 until February 2018, outliers in the data set are likely, as the first 12 days of single digit months, like January and February, were prone to error when entering data.
The registration of comorbidities hinted towards an under-registration, as the majority of patients (52.5%) had 3 or more comorbidities. A change from a categorical variable (0, 1, 2 and > 3) to a numerical variable (0, 1, 2, …, n) could improve precision.
For the systematic evaluation of registry entries, the data format in the source data evaluation needs to be chosen diligently to enable statistical analysis. Certain faults (e.g. missing ICD-10 GM codes) or necessary additional options for fixation methods (e.g. the use of an external fixator followed by internal fixation) were identified, corrections recommended, and the items added by the IT section of the BFCC project.
The graphical method
geom_count for displaying proportions of agreement between registry data and source data (Fig.
1) proved to be suitable for finding systematic faults. By using R Statistics [
19] as a software tool to analyse registry data, automated reports can be created using a carefully written statistical script. This lowers the threshold for the re-evaluation of registry data and facilitates continuous improvement of the registry. As freeware, it is readily available and cost-effective for institutions to use. Furthermore, the software proved excellent when using multiple, large data sets.
Funding of the BFCC project stopped by March 2019. Implementations of suggested improvements were only carried out to a limited extent. For example, missing ICD codes were added, but the date format could not be changed by the end of the project.
Despite having a deep mathematical foundation, the method was developed with the emphasis to be used by non-mathematicians to allow for a wide application. This has been proven by this publication, as it was applied by a clinician and non-mathematician, supporting its user- friendliness and potential for broad application.
Conclusion
Scoring the data quality of a registry is a unique feature to Nonnemacher et al.’s method of adaptive monitoring and demanding in its execution, but its applicability has been proven by this publication. To tap the full potential of the method, a repeated application on an established registry would be desirable. The tested graphical method helps improving the data quality.
An outlook to monitoring data quality in the future
As the application of the method of adaptive monitoring has yet only been published for two registries, possibilities for further research are vast. Its application on different projects could further test its reliability, with the aim to make it the gold standard for evaluating the data quality of registries.
To limit transfer and human error and safe time, an automated data capture should be considered in the future. The excessive manpower needed to acquire sufficient amounts of data for the BFCC registry was outdated. Solving this issue was subject of a different branch of the BFCC project, focusing on import and export solutions from the Hospital Information System (HIS) to the registry’s database. It could not be fully executed by the end of the project for reasons of software and data format incompatibilities. It is advisable for any new registry to meticulously care for data formats prior to setting it up or evaluating the data quality.
A shortcoming of the project was a selection bias, as only patients able to consent were included in the registry. As orthopaedic and trauma departments often deal with fragility fractures of older and not contractually capable patients, a bypass through an opt-out system, as used in Scandinavia [
23,
24] or the Netherlands [
25] would facilitate including patients. A drop-out rate caused by this was unfortunately not tracked and is suggested to be recorded by future projects.
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.