Development and evaluation of a customized checklist to assess the quality control of disease registry systems of Tehran, the capital of Iran in 2021

Clinical registries are interactive real-time databases recording the detailed information of patients, including specific diagnoses, clinical conditions, and procedures [1]. A clinical registry is typically customized to fulfill its major purposes, including describing the natural history of diseases, treatments, medications, and their clinical efficacy and cost-effectiveness, as well as disease outcomes and safety and quality of care issues [2, 3]. In 1974, the World Health Organization (WHO) pioneered these registries in epidemiological and clinical research [4], and since then disease registry systems (DRSs) have turned into organized systems to improve the quality of care in the healthcare system and progress in medical research [5, 6]. Disease registry systems provide opportunities to conduct high-quality medical research, advance diagnostic and therapeutic clinical practice guidelines, and improve the quality of healthcare services, patient outcomes, and resource and financial management. In addition, DRSs can show us the best way to strategic purchasing and conduct cohort and clinical trials based on real data [7, 8], especially for rare diseases [9].

However, using databases for research and audit and answering specific questions require high-quality data and resolving the weaknesses of DRSs [10]. Quality control (QC), as an integrated system, is an important component of the quality management of data registries, helping in dynamic monitoring and evaluating the effectiveness of the construct and process of data registration according to predetermined goals. As with any project, DRSs should be evaluated by a supervisor appointed by the founder. This process helps DRSs improve themselves and correct their errors, such as missing data, delayed follow-up, and data duplication [4, 11, 12]. Therefore, clinical DRSs need to be validated and improved as their quality assessment results are provided to policymakers and health insurance companies and are used for making public-heath related decisions.

In Iran, DRSs started to grow 30 years ago when the cancer registry and then other national registries such as trauma, spinal defects, and newborn anomalies were established [13‐16]. In 2014, a national DRS program was created aiming to integrate at least 20 DRSs in Iran. Until November 2018, a total of 71 clinical DRSs were active in the medical universities and health institutions affiliated with Iran’s Ministry of Health and Medical Education, among which Shahid Beheshti University of Medical Sciences (SBMU), with six DRSs, ranked fifth until then. This university now hosts around 50 DRSs for different medical specialties. The pillar of dynamic monitoring of these DRSs is to validate their different QC dimensions and integrate them into a meaningful whole to provide comprehensive coverage while keeping these dimensions organized and integrating them with the health information system (HIS).

Because it is inapplicable to interpolate all internationally used QC dimensions into all DRSs, we here developed a checklist to monitor the most important QC dimensions including comparability [17], reliability, and validity [18], completeness [19], timeliness [11], accessibility [20], efficiency, and duplication [2]. The second purpose was to evaluate the quality of 49 DRSs.

The present study had a mixed-method design and was conducted in two consecutive qualitative and quantitative phases. In the qualitative phase, a checklist was developed to assess the QC of 49 active DRSs established by the research centers, hospitals, and educational departments affiliated with the Shahid Beheshti University of Medical Sciences (SBMU). In these DRSs, the data were launched using unique and standard software, and crude data were transferred from actively supervised registries approved by SBMU. Registries with no recorded data were excluded.

During the study’s qualitative phase, a total of 29 items were extracted corresponding to the goals of the Ministry of Health and according to the literature in order to rank the quality of each registry. A satisfactory level of agreement was observed among the panelists with regard to the final 23 QC items. Regarding content validity assessment, the CVI and CVR of these 23 items were 0.79 and 0.58, respectively, suggesting the good content validity of the checklist items. Cronbach’s alpha coefficients for all QC domains were higher than 0.69, indicating acceptable internal consistency. This checklist was used to evaluate six QC domains for DRSs, including completeness (Q1, Q2, Q3, Q5, Q6, Q7, Q10, Q17, and Q18), timeliness (Q11, Q21, and Q22), accessibility (Q12, Q20, and Q23), validity (Q9, Q13, and Q14), comparability (Q16), and interpretability (Q4). These dimensions were ranked based on the total scores from the summing of related questions. Furthermore, data duplication, confidentiality, and understanding of the memorandum (MOU) were separately evaluated by the questions of Q8, Q15, and Q19, respectively (Table 1).

During the study’s quantitative phase, the registries were assigned specific serial numbers. An overview of the 49 registries approved by SBMU has been shown in Table 2. Most of the registries (81.6%, n = 40) were focused on diagnosis/treatment; six of them (12.2%) recorded treatment outcomes and other registries were related to different scopes of diagnosis (n = 1,#49), procedures (n = 1, #47), and side effects of treatments (n = 1, #30). As illustrated in Fig. 1, most of the registries were in the field of neurology (n = 8), followed by pediatrics (n = 7) and cancer management (n = 6).

Table 3 shows the ranking of the registries studied based on their QC scores. The highest rank (96.1%) belonged to registries #22 and #42, and the lowest ranking (20%) was related to registry #49.

Table 4 presents the mean score of each QC domain for DRSs based on the total scores from the summing of related questions. Regarding the acceptable quality cut-point (i.e., > 70% of the mean score of each domain), out of 49 DRSs evaluated, 48 (98%), 46 (94%), 41 (84%), 38 (77.5%), 36 (73.5%), and 32 (65.3%) registries obtained quality scores in the domains of interpretability, accessibility, completeness, comparability, timeliness, and validity, respectively. In this study, the rate of recording duplicated data was low (12.2%), which can be explained by the development of electronic registries with unique national ID numbers.

In this study, we developed a customized checklist to assess six quality domains in 49 DRSs established by SBMU in 2021. Our results demonstrated that all the registries had acceptable interpretability, accessibility, comparability, and completeness. In addition, the timeliness and validity domains acquired the lowest quality ranks.

There are several methodological problems with data quality assessment, one of which is the lack of a comprehensive standardized method for this purpose. The quality of data varies between different practices, and data quality needs to be assessed based on unique requirements in various fields [21, 23, 24]. Comparability, completeness, and validity are considered key elements during data quality assessment [21, 23]. Faulconer and de Lusignan suggested an 8-step statistical method for assessing the quality of the data of patients with chronic obstructive pulmonary disease [24]. Therefore, we developed a specific checklist to assess the QC of our DRSs in different fields.

Only a high degree of completeness will ensure that the incidence and prevalence rates estimated in DRSs are close to their ‘true’ values. Most data QC dimensions overlap with each other, and their interpretations are vague due to ambiguous definitions or even a lack of standard definitions. Two of the most frequently cited data QC dimensions are “accuracy” and “completeness”. It may be difficult to accurately estimate the completeness of registries because this entity is influenced by the proportion of patients introduced to registries by healthcare centers, the ratio of those refusing to be referred, and the total number of patients in the study population [25]. In our investigation, the overall rate of completeness was obtained at 84%, which was in line with the study conducted by Fung et al. on Singapore’s cancer registries, reporting a completeness rate of 98.1% [26]. One possible reason for lower completeness in some of our DRSs may be the short time passing from their establishment (#11, #19, and #27). Despite all the limitations such as the relatively short period of the study, in a study by Lee et al., completeness of 90–100% was reported for a registry of operative sectors (e.g., operating surgeons, consulting surgeons, and the hospital). Interestingly, auditing revealed that the registry’s completeness reached 100% after resolving deficiencies [27]. Another explanation for this variation in completeness may be differences in the number of patients with specific disorders such as Parkinson's disease (#1), SMA (#2), pediatric migraine (#8), tracheal stenosis following intubation (#15), and pediatric nephrotic syndrome (#21). In addition, nationwide recruitment for a number of our registries (#42, #43, and #44) could have contributed to their high completeness rates. Also, some medical procedures should be registered before their costs can be reimbursed by insurance companies (registries #12, #23, and #24). Another factor increasing the completeness of data recording can be the proven utility of this practice amongst health professionals in the registeration centers (#10, #18, #22, #36, #37, and #41). It is worth noting that using this checklist, we were unable to determine the proportion of eligible patients who decided not to be enlisted in relevant registries, increasing the likelihood of overestimating completeness in these registries.

Using standard internationally approved definitions for recording and reporting data boosts the level of comparability of registries [11]. In our study, there was limited standardization regarding the definitions used in registries, leading 11 out of 49 registries to have unacceptable comparability in terms of diagnostic and therapeutic elements. It is worth mentioning that in our checklist, only one question (Q16) was related to comparability, limiting the ability of this checklist to reliably assess this quality domain compared to other dimensions investigated by multiple questions. Comparability has been reported to vary considerably in different registries. In a study in Russia, only four cancer registries out of 10 studied registries met international standards [28]. However, cancer registries in Singapore were reported to have a high level of comparability [26]. Low comparability is the main barrier to achieving an interoperability framework, and one potential solution to this problem is to develop a team of specialists and experts to standardize definitions across all DRSs.

High timeliness allows for the real-time recording of diagnoses, procedures, and other relevant data in DRSs. Although there are currently no international regulations for assessing timeliness, timely reporting of information is a foremost priority for all registries [1, 11]. There are rare reports on the timeliness of DRSs [29], and the definition of this term in the context of data registries should be exactly determined. In general, timeliness refers to the rapidity at which a registry can collect, process, and report reliable and complete data [30]. In our checklist, timeliness was defined as the date on which the database was ‘frozen’ to calculate annual statistics for issuing an official report. This period comprised two intervals: the time until receipt announcement (ie., from the date of diagnosis to the day of receiving the report) and the processing time (i.e., from the date of report receiving until data availability). In our report, in 36 out of 49 registries (73.5%), physicians or nurses reported the data at the time of patient visits or shortly afterward, but five registries had delays in submitting annual reports. Also, eight of the registries preferred to postpone the publication of their results to attain better completeness to the cost of undesirable timeliness. Evaluating the timeliness and several other quality dimensions of a pediatric mortality surveillance system in Iran showed that this system successfully fulfilled timeliness criteria mainly because the managers were committed to holding a monthly committee for monitoring childhood mortality and the immediate reporting of infectious diseases [31]. Moreover, an evaluation of timeliness at the Cancer Registry of Norway during 1953–2005 showed that the median time for diagnosis of a new case reduced from over 525 days in 2001 to 261 days in 2005 [29]. Another study showed that the timeliness of the diseases was low based on the national reporting of the disease surveillance system [32]. Therefore, implementing electronic data recording and employing dedicated and well-trained staff can improve the timeliness of registries in reporting their data.

Accessibility is defined as the ease of access to data for users, rendering the data more available and making it possible for others to confirm the registry’s results [33]. As almost all of our registries used the same unique software, they acquired high accessibility (94%). This unique software facilitates the generation of meaningful and credible information from diverse sources, decreasing the occurrence of potential errors during the data entry process and facilitating access to the data. In line with our results, a study by Azadmanjir et al. aimed to identify and address hurdles to data accessibility at the National Spinal Cord Injury Registry of Iran, a registry relying on primary data sources. Their expert panel selected 174 data quality items, including accessibility and usefulness in quality-of-care assessment in emergency settings [34].

Validity is the extent to which the data registered can be assessed in terms of accuracy and relevant rules or definitions [35]. A high validity rate (91.9%) was reported for Singapore’s cancer registries, which could be attributed to the fact that these registries were focused on a specific field [26]. In the present study, nearly one-third of our registries (35%) had low validity, mainly due to the lack of uniform definitions for items due to the variety of DRS fields. This observation highlights the importance of employing uniform, transparent, and accurate definitions by all registries according to existing guidelines and classifications.

Interpretability is defined as the ease of understanding of data for users, presenting one of the main challenges faced by data registries and requiring a clinical framework for data interpretation and scoring [2]. Registered data can be interpreted only if they are specifically assigned to exact endpoints [22]. In our checklist, this quality domain was assessed by a single question (Q4), asking about the presence or absence of a data dictionary at each registry, and almost all (98%) of the registries investigated obtained a score higher than the cut-point. There are no previous reports on the interpretability of registered data in DRSs, so we could not compare our observation with other studies in terms of this QC domain.

The main strength of the present investigation included the development of a customized checklist to assess data quality in Iran’s DRSs in various fields in terms of all main QC standard indicators. One of the limitations of this study was that we confined our search to DRSs established by SBMU in Tehran, which could decrease the generalizability of our findings. Second, most of these DRSs mainly rely on the reports generated by governmental hospitals, and it is possible that the private hospitals or health service centers in different regions might not be covered.

In this study, a customized checklist was developed to assess the quality of the data recorded in DRSs, which could be considered a proof of concept for future investigations. Our results demonstrated that most of DRSs had high degrees of interpretability, accessibility, comparability, and completeness. However, their timeliness and validity needed to be improved. As the DRSs of the SBMU acquired a high degree of quality control in most of the studied domains allows for greater confidence in the use of the qualified data to improve the healthcare system and the possibility of integrating data with national healthcare data.