Assessing the Quality of Randomized Controlled Urological Trials Conducted by Korean Medical Institutions

Evidence-based medicine (EBM) was introduced by the EBM Working Group in 1992 and is defined as the conscientious, judicious, and explicit use of the best evidence in making decisions for the care of individual patients [1]. Randomized controlled trials (RCTs) have been acknowledged as important standards for EBM. The RCT is a study design that can minimize bias and produce the most dependable data among study methods [2]. However, even the RCT cannot eliminate all bias, which can occur at the designing, conducting, or reporting phase and can yield misleading results [3]. Peer review is an essential method for preventing such results from being used as the basis of clinical applications [4]. Moreover, objective methodological quality assessment of published studies is essential because these studies can influence the quality of medical care received by patients [5]. The methodological quality of an article can represent its overall quality and should therefore be assessed at the design, conduct, and analysis steps [6, 7]. In addition, any erroneous data can be identified throughout the assessment process, thereby eliminating the erroneous data from clinical care and saving medical expenses [8]. When meta-analyses or systematic reviews are conducted with the use of data from low-quality RCTs, the reliability of the conclusions drawn can be impaired [3].

Several methods can be used to assess the methodological quality of clinical trials, such as scales, individual markers, and checklists. The scales method is easy to apply to interstudy comparisons and is more applicable to performing quantitative assessments on the quality of a clinical trial than are the other methods. Randomization, blinding, and dropout are the factors of scale that directly relate to reducing bias. The Jadad quality assessment scale (Jadad scale) is a representative quality assessment tool consisting of these three items [9]. The Jadad scale has been widely used because of its simple questionnaire and capacity to make assessment easy, but it does not include an assessment item for concealment of allocation. However, there is an individual marker method that assesses allocation concealment, which is a way to randomize the allocation sequence to avoid any selection bias in the allocation of patients for treatment [10]. The van Tulder scale and the Cochrane Collaboration risk of bias tool (CCRBT) include an assessment item for concealment of allocation.

Some recent studies have examined issues in the quality assessment of RCTs. For example, Lee et al. [11] analyzed all RCTs published in the Korean Journal of Urology (Korean J Urol). The present study assessed the quality of urological RCTs that were conducted in Korea by using the Jadad scale, the van Tulder scale, and the CCRBT. The results of the present study can be used to assess the past and present status of urology research in Korea. Moreover, the results of this study can suggest guidelines for future studies and thus improve the medical practice of urology in Korea.

In the present study, the quality of published urological RCTs conducted in Korea from 1986 to 2011 was assessed. Whereas the number of RCTs published has gradually increased, there was no significant difference in the quality of RCTs according to publication year. The small number of methodological descriptions of concealment of allocation and the lack of double-blind studies were the greatest factors in preventing studies from receiving high-quality assessments.

There have been few qualitative analyses of RCTs. Uetani et al. [2] analyzed medical RCTs conducted in Japan by using consolidated standards for reporting of trials (CONSORT) statements. The CONSORT statement was announced in 1996 with an aim to decrease the number of poorly conducted RCTs by detailing appropriate standards. Since the release of the CONSORT statement, various journals and organizations have played a leading role in enhancing the quality of RCTs [14]. Uetani et al. [2] showed that of 98 RCTs conducted in Japan from January to March 2004 that had been published, only 11 were in accordance with the CONSORT statement. However, because the CONSORT statement is not a quality assessment tool, it cannot be used to quantify the qualitative analysis.

Various types of methodological quality assessment tools for RCTs are available, including the Campell, Chalmers, CCRBT, Jadad, Moher, Newell's, and van Tulder methods. However, assessment of the quality of trials remains controversial, and no consensus exists on which tools are highly accurate and valid [15]. The present study was able to overcome such limitations through the use of three different tools. These three tools are representative assessment tools and are commonly used both nationwide and worldwide. The Jadad scale is a simple and easy method of performance assessment, but does not include assessment items for concealment of allocation. Therefore, additional analyses were performed by using the van Tulder scale and the CCRBT to supplement the Jadad scale. Patient, caregiver, and observer blinding as well as allocation concealment are considered in the van Tulder scale. Also, the term for intention-to-treat analysis is included in van Tulder scale. Although the CCRBT has a limitation in that it cannot quantify the quality of RCTs, it can be used to perform more objective analysis of RCTs because of its more detailed interpretation of each classification.

Kim et al. [16] analyzed the quality of RTCs published in five academic journals (the Korean Journal of Internal Medicine, the Journal of the Korean Surgical Society, the Korean Journal of Obstetrics and Gynecology, the Korean Journal of Pediatrics, and the Korean Journal of Family Medicine) by using the Jadad scale. They found that the number of RCTs with a Jadad score of more than 2 points increased in the 1990s compared with the 1980s. In addition, Chung et al. [5] analyzed RCTs published in the Korean Journal of Family Medicine from 1980 to 2005 and reported that the number of RCTs increased according to publication year. This increase in the number of RCTs may have been a result of the growing influence of EBM [17]. In the present study, as in these previous reports, the number of RCTs was found to have increased over time. However, the quality of the RCTs did not significantly change according to publication year.

Lee et al. [11] previously analyzed the quality of RCTs published in the Korean J Urol. According to their study, there were 28 RCTs out of 3,156 original articles presented from 1991 to 2010. The mean Jadad scale score was 1.75 and 8 RCT articles were determined to be high-quality articles. They found that the number of RCT articles and their quality improved over time. Moreover, they suggested that the descriptions of allocation concealment in the RCTs published in the Korean J Urol needed improvement. In the present study, among the urological RCTs conducted in Korea, only three articles had proper descriptions of allocation concealment. Hewitt et al. [18] showed that 46% of RCTs published in four different medical journals (the British Medical Journal, the Journal of the American Medical Association, the Lancet, and the New England Journal of Medicine) had inappropriate descriptions of allocation concealment. According to Schulz and Grimes [19], not incorporating allocation concealment into a study can impair the effects of randomization and blinding. The omission of allocation concealment could distort the results of the intervention by more than 40%. The quality of Korean urology reports would be improved if clinical studies included adequate blinding and allocation concealment.

Double blinding can be applied to drug studies more easily than to nondrug studies, such as those investigating surgery. Therefore, studies investigating drugs were scored as higher quality in comparison with studies not investigating drugs. In the present study, double-blind studies and drug studies were of better quality than other studies.

The present study confirmed that studies that were under IRB review were of better quality than RCTs that were not. To the best of our knowledge, no previously published study has analyzed the association of IRB review and the quality of articles. IRB review partially serves to assess the feasibility of the study design and the performance of the study protocol. The system by which RCT study protocols are validated and approved by IRBs has played a major role in raising the quality of articles, because IRB approval is considered an international quality standard. Clifford et al. [20] reported that there was no association between the funding source and the quality of the article. However, in the present study, funded studies were of better quality than were those that were not funded. This result may indicate that receiving financial support helps investigators to design well-organized studies and perform orderly research, resulting in higher quality articles. Moreover, the results of the present study indicated that most RCTs were published by selected universities and most RCTs were published in Korean J Urol.

This study had at least one limitation. It is possible that some RCTs were not considered in the present study. The extraction of RCTs and their quality assessments were based on the subjective judgment of the researchers. Therefore, two medical doctors independently extracted the RCTs. The assessments were performed independently by two reviewers and the outcomes were adjusted to ensure objectivity and reliability.

For the first time, the quality of urological RCTs conducted in Korea was assessed. The number of RCTs conducted by Korean medical institutions has increased over time. However, no qualitative improvement of RCTs was observed over time. In present study, areas for improvement were identified, and the findings of this study may contribute significantly to the overall quality of urological medical research in Korea.