Assessing the Quality of Randomized Controlled Trials Published in the Journal of Korean Medical Science from 1986 to 2011

Randomized controlled trials (RCTs) have increasingly become acknowledged as important standards of evidence-based medicine. RCTs follow a study design that can reduce bias and produce the most valuable data among study methods, making them the most reliable assessments of the effectiveness of medical treatments (1). However, even the RCT study design cannot eliminate all bias, which can occur at the designing, conducting, reporting, or application phase and can lead to derivation of incorrect results (2). Peer review within journals is an indispensable prevention method against error and can assess the validity of studies so as to prevent incorrect information being used as the basis of clinical application (3). Moreover, objective assessment of articles on methodological quality is essential because it can heighten the quality of medical care (4). An article's methodological quality can represent its overall quality and should therefore be assessed at the design, conduct, and analysis levels (5, 6). In addition, throughout the complete assessment process, any unnecessary or erroneous data can be identified, thereby eliminating it from clinical relevance and saving medical expenses (7).

There are several methods to assess the methodological quality of clinical trials, including scales, individual markers, and checklists. The scales method allows for easy inter-study comparison and is more advantageous than other methods when performing quantitative assessment on the quality of a clinical trial. Randomization, double blinding, and drop out are the three factors of scale that directly relate to reducing bias. The Jadad quality assessment scale (Jadad scale) is a representative quality assessment tool consisting of these three items (8). The Jadad scale has been widely used because of its simple assessment questions and capacity to make assessment easy, but it does not include an assessment item for allocation concealment. There is however an individual marker method that assesses allocation concealment, which is a way to randomize the allocation sequence to evade any selection bias in the allocation of patients for treatment (9). The van Tulder scale and Cochrane collaboration risk of bias tool (CCRBT) include an assessment item for allocation concealment. Recent studies have examined issues in quality assessment in RCTs. For example, Kim et al. analyzed all RCTs published in five Korean medical journals (10). The Journal of Korean Medical Science (JKMS) is the flagship journal of the Korea Academy of Medical Sciences and is the only science citation index Medical Journal in Korea. It was launched in 1986 and includes evidence-based, scientifically written articles aimed at introducing Korean medical sciences to the world and facilitating international medical information exchanges. However, there has been no quality assessment of RCTs published in the JKMS. Yonsei Medical Journal (YMJ) is a clinical science citation index expanded journal which has been published since 1960 by the Yonsei University College of Medicine and it covers all the subjects related to medicine based on either clinical or basic research. Korean Journal of Internal Medicine is an international medical journal published by the Korean Association of Internal Medicine. To suggest the direction of future studies and to improve medical practice in Korea, this study assessed the quality of RCTs presented in the JKMS, YMJ and Korean Journal of Internal Medicine using the Jadad scale, van Tulder scale, and CCRBT.

In this study, the quality of original RCT reports published in the JKMS, YMJ and Korean Journal of Internal Medicine was assessed. While the number of RCTs published has gradually increased, no statistically significant increase in the quality of the articles was observed. The small number of double blinded reports and the absence of methodological details described for concealment of allocation have hindered high quality assessment.

There had been three previous analyses on the quality of RCTs published in Korean journals. Kim et al. performed an analysis on RTCs published in five different national academic journals: the Korean Journal of Internal Medicine, the Journal of the Korean Surgical Society, the Korean Journal of Obstetrics and Gynecology, the Korean Journal of Pediatrics, and the Korean Journal of Family Medicine (10). According to that report, the number of RCTs with a Jadad score of ≥ 2 points increased in the 1990s compared to the 1980s. In addition, the number of papers with high quality articles increased from two to seven during the same time period. In another study, Chung et al. analyzed RTCs published in the Korean Journal of Family Medicine from 1980 to 2005 (4). Among the 1,290 original articles published, there were 23 RCT articles (1.8%), and their mean Jadad scale score was 1.87. The authors reported that the proportion of RCTs increased from 1.09% of the original articles in the 1980s to 2.63% in the 2000s, with a corresponding 1.17-point increase in the Jadad scale from 1.00 in the 1980s to 2.17 points in the 2000s. These two studies clearly indicate that the number and quality of RCT articles increased over time. This is considered a result of the growing influence of evidence-based medicine. Therefore, the overall number of RCTs that provides a high quality level of evidence is increasing (12). In the present study, the number of RCTs increased as in these previous reports. However, the quality status of the RCTs analyzed here did not significantly change over time. Specifically, the quality status did quantitatively increase over time, but a significant difference was observed only from the assessment by the van Tulder scale in terms of quality in the JKMS. However, when the qualities of articles were compared in 1-yr units, rather than 5-yr units, no statistically significant differences were observed.

Recently, Lee et al. (13) analyzed the quality of RCTs published in the Korean Journal of Urology using the Jadad scale. Their study showed that there were 28 RCT articles (0.89%) out of 3,156 original articles presented since 1991 and during the following 20 yr. The mean Jadad scale score of those RCT articles was 1.75. In addition, eight high quality articles were found out of those 28 articles. The number of RCT articles, which totaled five before 2000, increased to 23 by 2000, and their quality improved over time. Moreover, Lee et al. (13) showed that only one article in the Korean Journal of Urology had adequate allocation concealment. In the present study, among the RCTs published by the JKMS, the majority did not even comment on concealment of allocation. In 2006, an article contained the context related to the concealment of allocation for the first time, but only eight articles (18.18%) in total provided details. There is the only one article contained the context related to the concealment of allocation in the Korean Journal of Internal Medicine. Schulz et al. (14) explained that without concealment of allocation, randomization tends to be damaged in the process of study performance, even if the randomization is well conducted. This omission could distort the effects of intervention by ≥ 40%. Hewitt et al. (15) presented that cases showing inappropriate or uncertain concealment of allocations constituted almost 46% of RCTs published in four different major medical journals (British Medical Journal, the Journal of the American Medical Association, Lancet, and New England Journal of Medicine) in 2000. If appropriate planning and preventions are taken by the medical academics of Korea on double blind and concealment of allocation in RCTs, the quality of such reports would improve.

There have been few studies conducted worldwide on qualitative analysis of RCTs. Uetani et al. (1) analyzed whether RCTs conducted in Japan and published in medical journals met the consolidated standards for reporting of trials (CONSORT) statements, which provide guidance on reporting RCTs and is comprised of a checklist, flow diagram, explanations, and extensions (16). They showed that 98 RCT articles had been published in various academic journals from January to March in 2004, and only 11 RCT articles were in compliance with the CONSORT statements. However, since the CONSORT statement was not a quality assessment tool, it was not possible to digitize the qualitative analysis.

There are various types of qualitative assessment tools for RCTs including Campell, Moher, Chalmers, Jadad, van Tulder, Newell's, and Cochrane. The interesting point in this study is that by using three different tools, we found differences in the qualitative analysis outcomes of RCTs. The assessment of the quality of trials remains controversial, and there is no consensus on highly accurate and valid tools (17). However, in this study, efforts were made to overcome such limitations by using three different tools: the Jadad scale, van Tulder scale, and CCRBT. These are representative assessment tools used most commonly nationwide and worldwide. In particular, the Jadad scale has advantages in the simplicity of the assessment questions and ease of assessment performance, but it does not include assessment items for the most important item of RCT assessment: concealment of allocation. Therefore, additional analyses were performed using the van Tulder scale and CCRBT to supplement in this regard.

Furthermore, in this study, articles that were reviewed and approved by an institutional review board (IRB) were, on average, of higher quality. To the best of our knowledge, no previous studies have been conducted in connection to the association of IRB review with the quality of articles. The review of IRB serves to obtain acknowledgement of the feasibility of the study design and performance in the study protocol. The recent system to validate study protocols in RCTs in order to obtain IRB approval has played an important role in raising the quality of articles since IRB review is considered an international quality standard. Studies that used placebo and drug products easy for double blinding were considered high quality in comparison to the studies with non-drugs. However, there were no significant intergroup differences observed from the results of this study in JKMS. However, overall quality assessment of RCTs in three journal, there was statistically difference according intervention type. Among studies that analyzed RCTs previously published by the Korean Journal of Urology and the Journal of Korean Academy of Family Medicine, those receiving financial support were able to establish well-organized study designs and perform orderly research, resulting in many high quality articles (4, 13). By contrast, Clifford et al. analyzed 100 RCT articles published in five different peer-reviewed, high impact, general medical journals (Annals of Internal Medicine, the British Medical Journal, the Journal of the American Medical Association, Lancet, and New England Journal of Medicine) and found no association between the funding source and the quality of the article (18). This study also did not reveal significant differences in the qualitative assessment of articles depending on the presence/absence of funding except van Tulder scale scoring. This result is not considered to be an accurate conclusion, however, because the number of analyzed articles was small in this report. Therefore, reassessment through analyses of more RCTs is required.

The limitations of this study include a probability for subjective judgment of the researcher likely to intervene in extraction of RCTs and their quality assessments. Therefore, two medical doctors independently extracted RCTS while the assessments were performed independently by two reviewers; thereafter, the outcomes were adjusted to secure objectivity and reliability. The assessment of RCTs published by the JKMS, YMJ, and Korean Journal of Internal Medicine was implemented for the first time in this study. Suggestions for the qualitative improvement of medical research in the Republic of Korea would be a significant contribution of this study.

While the number of RCTs published in the JKMS, YMJ, and Korean Journal of Internal Medicine has gradually increased according to publication time, the quality of these reports has remained unchanged. Therefore, national medical academics should focus more efforts in performing high quality studies to ensure appropriate randomization, reviews by IRB, financial support, and inclusion of allocation concealment during study performance.