Background
Positron emission tomography (PET) is a noninvasive imaging technique used for measuring the concentration of positron-emitting radioisotopes within tissue in malignant and benign disease and provides a three-dimensional image of functional changes in the body. PET can be used to assist management decisions related to diagnosis, staging/restaging, recurrence, and treatment planning and response as well as prognosis. Recently, an increasing number of clinical applications of PET have been observed, particularly in oncology, and mostly with the use of fluorine-18 fluorodeoxyglucose (18 F-FDG) as the PET tracer [
1,
2].
PET, however, is a high-cost technology, and it is therefore important for health policymakers to systematically assess not only the clinical indications of PET but also its cost-effectiveness in comparison with other competitive diagnostic technologies [
3]. To this end, various countries have evaluated the efficacy and efficiency of PET since its introduction and diffusion into clinical practice [
4-
6].
In this evaluation, both diagnostic accuracy and methodological quality of diagnostic studies are important elements of this evaluation. However, diagnostic studies have several unique features in terms of quality, which are not addressed by the traditional approach to evaluating controlled trials. A validated quality assessment tool is currently not available [
7]. In 2003, the Standards for the Reporting of Diagnostic Accuracy (STARD) statement was developed to help authors improve reporting [
8], and the quality assessment of diagnostic accuracy studies (QUADAS) tool was developed in the same year [
9]. Despite these advancements and the development of study selection tools for systematic reviews [
4,
6,
10,
11], comprehensive quality assessment of PET diagnostic studies has been very limited.
In Japan, while PET has been introduced and diffused without a systematic health technology assessment, the total number (i.e., 466) of PET units installed in Japan was the second highest in the world following the United States (i.e., 1,450) in 2011 [
12]. A considerable number of diagnostic studies for PET have been conducted, but as they are primarily published in Japanese, they have not been widely reported in international journals and databases.
Therefore, even though Japanese studies have made a contribution to the evaluation of diagnostic accuracy of PET both in Japan and internationally, comprehensive information about PET studies conducted in Japan is greatly lacking. We conducted a systematic review to assess the characteristics and quality of PET studies in Japan and analyze the factors related to their quality.
Methods
Search strategy
All papers reporting diagnostic efficacy studies for PET conducted in Japan as original articles were identified through two databases: the international database, MEDLINE, and Ichu-Shi, the domestic Japanese database of the Japan Medical Abstract Society. MEDLINE includes Japanese papers written in English, as well as only English summaries for a very limited number of papers written in Japanese. We searched this database from inception to 15 July 2011. Ichu-Shi includes studies written in Japanese and published in Japanese journals and occasionally features an English summary. We searched this database from inception to 23 August 2011. The search strategy used the following general terms, expanded and appropriately modified for each database: ‘positron emission tomography’ or ‘positron emission computed tomography’ or ‘PET’, and ‘sensitivity’ and ‘specificity’. The search terms are presented in Additional file
1: Table S1 and Additional file
2: Table S2.
Inclusion and exclusion criteria
All papers reporting diagnostic efficacy or accuracy studies of PET conducted in Japan that used data from PET scans performed at institutions in Japan and were published as original articles until the end of 2010 were included. The following types of PET studies were excluded: 1) non-diagnostic studies, such as studies of treatment planning, and response and prognosis; 2) studies that did not explicitly describe sensitivity and/or specificity of PET, and which could not be derived from the data provided in the paper; 3) studies written in languages other than Japanese or English, and 4) case reports, systematic reviews and meta-analyses.
Study selection and data extraction
Two reviewers independently screened titles and abstracts of studies identified by the search. The text of all potentially relevant studies was evaluated in detail and assessed against eligibility criteria. The following data were extracted and checked independently by two reviewers: database indexed, publication year, index test, study type, sample size, target disease, comparator, study subjects, outcome (i.e., sensitivity and specificity), conclusions, and funding source. Any disagreements were resolved by consensus between the reviewers.
Assessment of quality of studies
The QUADAS tool was used to assess the methodological quality of studies. This tool is a validated quality checklist containing 14 items that address the most important sources of bias and variation in diagnostic accuracy studies [
7,
9,
13]. QUADAS was also adopted by the Cochrane Collaboration in their handbook of diagnostic systematic reviews [
14]. The detailed explanations of the 14 items were listed in Additional file
3: Table S3.
The reviewers who assessed and extracted data were trained in the use of the QUADAS checklist. Each item in the checklist was categorized as ‘Yes’ for low risk of bias, ‘No’ for high risk of bias, or ‘Unclear’ if there was insufficient information to make a judgment. We also calculated a quality score defined as the total number of items categorized as ‘Yes’ among 14 items.
Data analysis
Firstly, we identified the number of diagnostic efficacy studies of PET conducted in Japan from 1990 to 2010 and examined their characteristics. Then we assessed the quality of these studies using the QUADAS tool. Secondly, we evaluated the quality of these studies by comparing studies indexed in MEDLINE with studies published in Japanese journals that were not indexed in international databases. We used Fisher’s exact test for proportions and t-test for quality scores in the statistical analysis.
Thirdly, the factors affecting the quality of Japanese PET studies were analyzed using a multiple logistic regression model. The dependent variable was whether the quality of the study was high (=1) or low (=0) [
15-
17]. We defined studies as high quality when the quality score exceeded eight (i.e., more than a half of total score), based on a definition from a previous systematic review [
18]. The independent variables were target disease, publication year, sample size, study design, funding source, international indexing, and whether or not comparative statistical analysis was conducted. All analyses were performed with STATA/MP 13.
Finally, the results and conclusions of the Japanese studies were compared with those of international HTA reports and systematic reviews in clinical medicine, in relation to PET and other competitive imaging technologies. We made this comparison to primarily examine the correspondence of the results and conclusions of Japanese studies with those of the international assessments. In the comparison, instead of an integrated form, only a proportion of positive conclusions, as well as a quality score and a proportion of comparative studies, are presented for the reader’s own consideration since there is no explicit or standardized guideline for integrating conclusions or recommendations of accumulated studies. In addition, we assessed the coverage of different diseases among both the Japanese studies and international assessments to examine whether Japanese studies covered some diseases that international assessments did not. HTA studies related to PET were comprehensively identified by database searches (i.e., the Centre for Reviews and Dissemination (CRD) database, a health services research center based at the University of York). Of the 49 reports identified and retrieved, two Belgian Health Care Knowledge Centre (KCE) reports were selected as a reference [
6,
10], which are the latest and most comprehensive assessments of high quality according to the INAHTA checklist for HTA reports [
19]. We found only one systematic review that comprehensively covered areas of disease [
11]. Note that as the number of Japanese studies was limited, we analyzed target diseases that were examined in more than three of the Japanese studies.
Discussion
This study is the first systematic review of the characteristics and quality of diagnostic accuracy studies of PET conducted in Japan. Although a total of 138 Japanese studies involving PET were identified, half of them were not indexed in MEDLINE. Although a potential overlap of study subjects may exist among several studies, this could not be taken into consideration due to a lack of information about the study participants in some studies. Also, papers with different aims and methods were considered as independent studies. In contrast, the total number of studies reviewed in a HTA report in the UK was 158 and included 6 non-English studies indexed in international databases [
4]. Also, languages in the primary study selection were limited to several European languages in the Belgian report [
6,
10]. Therefore, non-indexed Japanese studies or studies written in Japanese would likely be missed from international HTA reports.
Malignant neoplasm was the target disease most frequently covered by Japanese studies (Table
1). This is a similar finding to previous international studies [
4,
10]. Fifty-eight percent of Japanese studies had a sample size less than 50. The estimates of accuracy in small studies are often inexact and their results have little generalizability for target patients [
20]. Also, Bachmann et al. estimated that the median number of patients with or without a target condition necessary to calculate valid sensitivity and specificity of diagnostic accuracy is 49 and 76, respectively. The sample size for most international PET studies was also less than 100 [
4,
6,
10]. In addition, approximately 90% of Japanese studies did not include information about funding sources. A systematic review of conflict of interests highlighted that systematic biases support products created by the funder [
21], thus implying that hidden conflicts of interest may be present among the Japanese studies.
Our study showed that the mean quality score was 6.7 (e.g., a full score is 14), and 33% of Japanese PET studies were of high quality, as indicated by the quality score of more than 8. These results were similar to those of several recent systematic reviews [
22,
23]. Also, a high risk of bias was observed in six items including adequate spectrum, adequate reference standard, and absence of verification bias, among others (Figure
2). This result indicates that the Japanese studies have numerous biases and are of relatively low quality, which is limitedly applicable to PET use in clinical settings. For example, PET studies of low quality were excluded from international health technology assessments (i.e., quality score less than eight) [
18], or critically examined in clinical recommendations [
4,
6,
10]. Therefore, studies of low quality will neither be used nor reflected in clinical guidelines and health policies. Moreover, the quality of test studies is extremely important as a basis for further evaluation for clinical decision making and health outcomes [
24,
25]. Greater improvement of the quality of test studies is urgently needed.
Factors related to the methodological quality, target disease, publication year, and study design were determined by multiple logistic regression analysis (Table
2). Prior to 2002 when PET examination was first included in the National Health Insurance, the quality of studies was high. This may be because that prior to the application of insurance coverage, PET researchers, related academies and industries proactively and rigorously conducted and published many PET studies to encourage and persuade the government to include PET testing in the insurance scheme, as well as to promote the utilization of PET testing after its inclusion in the insurance coverage. These efforts to promote PET testing seem to have had a positive influence on maintaining the methodological quality of the studies and overcoming critical assessments from the government. As PET studies have been conducted mainly in the area of oncology, particularly in respiratory cancer, the quality of studies based on target disease will improve over time as research applications of PET expand into other areas. Our results are consistent with previous research which highlighted that prospective studies are favorable for reducing biases [
26].
Our assessment of the characteristics and quality of Japanese PET studies demonstrates that efforts to educate researchers, provide incentives, and establish systems for conducting diagnostic studies are needed to encourage investigators to comply with existing methodological standards. Low quality of reporting was found to be a significant obstacle in the evaluation of quality, and therefore the risk of bias remained unclarified in this study. As many studies have limited applicability in clinical practice and health policy, their inclusion might be misleading in some cases. In this study, a high proportion of ‘unclear’ results were observed in several items of risk of bias (Figure
2), which proved difficult for reviewers to evaluate the actual quality of the studies. This result is also reported in several systematic reviews [
18,
19]. Concerns about the quality of reporting of diagnostic studies led to the endorsement of the STARD statement [
27]. Since the publication of this statement, the quality of reporting of diagnostic accuracy studies has slightly improved [
28]. To advance the quality of reporting in Japan, efforts are required to raise awareness of the STARD statement and to encourage publishers of Japanese scientific journals to adopt the statement in their instructions to authors.
Conversely, 34.9% of studies not indexed in MEDLINE and 30.4% of the indexed studies were of high quality with a quality score of eight or higher. There was no significant difference in the total quality score between the two groups, even though a significant difference was observed in several items between them (Figure
3). After adjusting for other factors in a logistic regression model, the overall quality was not significantly different between indexed PET studies and those not indexed in MEDLINE (Table
2). This result suggests that non-indexed Japanese studies should be included in systematic reviews as well as both international and Japanese databases in order to prevent the exclusion of high-quality Japanese PET studies. In addition, excluding Japanese studies may introduce a language bias and lead to erroneous conclusions.
The search and collection of non-English language papers is important to minimize language bias [
29]. In conducting systematic reviews, international collaboration in the area where language bias might occur could be a practical and feasible solution for minimizing language bias. On the other hand, non-English-speaking researchers should also be encouraged to publish original studies in English in a journal indexed in international databases.
Finally, only 47.8% of Japanese studies employed comparators (i.e., competitive diagnostic technologies such as MRI or CT) to evaluate diagnostic accuracy of PET. Of this percentage, only 23 studies performed statistical analysis. As the diagnostic accuracy of non-comparative studies often differs to that of comparative studies [
30], the conclusions of the Japanese studies should be carefully interpreted. However, there has been no mention of this issue even in systematic reviews and HTA reports of PET studies [
4,
22,
23,
30]. In addition, only 6.9% of non-comparative studies performed simple comparisons with the results obtained from literature surveys.
These issues might influence the discrepancy of conclusions between Japanese studies and international assessments. For discrepancies found in the coverage of disease areas, Japanese studies could serve as supplementary information for the conclusions or recommendations of international assessments to prevent language bias, since international assessments do not include most Japanese studies. However, further systematic examination would be needed to integrate the information and assess the influence on conclusions and recommendations, since there is no explicit or standardized guideline for integrating these conclusions or recommendations.
On the other hand, in regard to the disease areas where only Japanese studies were available, the application of the Japanese National Health Insurance was based on a small number of studies with relatively low quality scores. In the case of uterine cancer, there is no positive conclusion. In Japan, since there has been neither comprehensive HTA nor guidance based on systematic reviews, further examinations would be required for health and clinical policy.
Conclusions
The Japanese studies covered a wide range of target diseases, which were not evaluated by the HTA and the systematic review. For a practical solution of these issues, Japanese studies or other language studies should be comprehensively included and simultaneously evaluated when conducting HTA or systematic reviews.
Diverse factors such as study design, conduct, analysis, and reporting of PET studies are related to the quality, number, and progress of PET studies in Japan. Greater efforts are required to set and implement feasible strategies for improving Japanese PET studies under the collective action of all stakeholders.
Competing interests
The authors declare that they have no competing interests.
Authors’ contributions
AH initiated and oversaw the project. All authors were responsible for the study concept and design. SN and MS acquired the data. SN, KM, DY, EO, and AH conducted the quality assessment. The manuscript drafted by SN was discussed and critically revised by AH and KS. KS gave administrative and technical support and supervision. All authors approved the final draft of the manuscript.