An assessment of the quality of randomised controlled trials conducted in China

Randomised controlled trials published in 2004 were identified through two broad sources:

For both sources, reference lists of included studies were checked. No language or other limitations were imposed. Chinese text was translated into English by two authors fluent in Chinese (DZ and PY). Titles were initially scanned for relevance and abstracts read if titles were unclear. The full text of papers with no abstract was viewed and checked for eligibility.

We included any papers reporting randomised controlled trials on all disease groups and all types of interventions, which were published in 2004, took place in China and included Chinese citizens. We excluded reports that did not include any participants from Mainland China. We excluded papers from Hong Kong and Taiwan where research and clinical practice are different from those in the Mainland.

The CONSORT statement is an internationally agreed standard for reporting RCT[11]. It includes recommended items designed to report the methodology and conduct of a study that are common to many standard quality assessment checklists. We used or adapted a subset of the CONSORT indicators in order to assess both the quality of reporting and, in those studies where information was provided, the actual conduct of the study. We also added some customised indicators in order to extract basic descriptive information specific to the Chinese papers (Table 1). We did not use overall quality scores or categories to judge each paper because the use of summary scores could be problematic and often obscures individual aspects of quality [21].

One reviewer extracted data from all included papers. A second reviewer independently checked a random sample of 26% of the papers. Discrepancies were resolved where possible by discussion, and the sample results compared with the full results using Kappa scores. Data on the quality of the included papers were presented in tabular format accompanied by a critical description.

Figure 1 describes the results of the search and the identification of eligible trials. Among 716 identified papers, 29 were initially excluded as they were duplicate publications of the same study. Only one paper was included for each study. Twelve were excluded on screening of the titles. Of the remaining 675 studies, 36 studies were excluded as they either were not, or could not be confirmed as, RCTs (Table 2), and 296 were excluded after reading the abstract. The full text of the remaining 343 papers was obtained, and finally 307 papers were included as confirmed RCTs. The full article list can be obtained from the contact author.

Agreement between the reviewers was good with a kappa score of greater than 0.7 for the main indicators (funding source, disease area, choice of comparator interventions, ethical committee approval, informed consent from participants, sample size, randomisation, allocation concealment, blinding, baseline characteristics, primary outcomes, loss to follow-up, length of follow-up, statistical reporting).

Study of trial quality is rare in developing countries, and tends to focus on limited clinical areas [22, 23]. Although among Chinese publications there are a few paperswhich describe trial quality in specific journals or fields [13‐19], this is the first systematic study to evaluate the quality of trial conduct and reporting in a sample which is likely to be more representative of Chinese RCTs in general.

Our review revealed that the standard of reporting of trials was generally poor, which concurs with the other published reports on Chinese trials [16‐19]. For example, nearly two-thirds failed to report any information on their methods of randomisation, reinforcing previous work [18, 19]. In the remainder there were various methods of random allocation, of which about a quarter reported using a computer-generated method or a random number table, which are the usually acceptable ways of randomisation. None of the trials discussed allocation concealment. If the allocation of the patient is not adequate and fully independent of the enrolling investigator, then this may allow either conscious or unconscious selection of participants into the trial, or into particular arms of the trial, thus introducing selection bias and undermining the randomisation. The internal validity of a randomised controlled trial has been shown to be directly associated with a clear description of appropriate methods of random allocation of participants, and concealment of their allocation [24].

Over 80% of trials provided no information about blinding of either participants or investigators. This confirms the result observed in a review of RCTs of traditional Chinese medicine[19]. Without blinding the groups may have been treated differently by the investigator and the outcomes not measured objectively, thus creating further assessment bias. Participants aware of their treatment may behave differently or have particular expectations [8], thus affecting the results.

Interestingly, among the included Chinese studies in this review, over half stated that none of their participants dropped out. This is unusual compared with trials in countries with more established research programmes, where a drop-out rate of below 5% is generally considered a very good result. Over 60% of the trials in this review reported a drop-out rate of less than 5%, and two-thirds less than 10%. The reasons behind these very low rates warrant further investigations.

The reporting of ethical issues was inadequate in the Chinese RCTs. Fewer than 11% of the trials reported having ethical committee approval, although the latter is a legal requirement in China [25]. Also, only a minority of the Chinese studies (17.4%) gave adequate details about informed consent procedures; a few mentioned that participants attended of "their own free will" but the remainder made no mention of consent. However, this level appears better than in a recent review of traditional Chinese medicine trials [19].

Compared with many published trials in developed countries[26], the standard of reporting in China is lagging behind, although there are still many fields in Western countries which have inadequate standards of reporting [27]. However, the application of the CONSORT statement has demonstrated benefits in improving reporting [28] and could be expected to do the same in China.