Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

The MicroArray Quality Control (MAQC) project shows inter- and intraplatform reproducibility of gene expression measurements

Abstract

Over the last decade, the introduction of microarray technology has had a profound impact on gene expression research. The publication of studies with dissimilar or altogether contradictory results, obtained using different microarray platforms to analyze identical RNA samples, has raised concerns about the reliability of this technology. The MicroArray Quality Control (MAQC) project was initiated to address these concerns, as well as other performance and data analysis issues. Expression data on four titration pools from two distinct reference RNA samples were generated at multiple test sites using a variety of microarray-based and alternative technology platforms. Here we describe the experimental design and probe mapping efforts behind the MAQC project. We show intraplatform consistency across test sites as well as a high level of interplatform concordance in terms of genes identified as differentially expressed. This study provides a resource that represents an important first step toward establishing a framework for the use of microarrays in clinical and regulatory settings.

This is a preview of subscription content, access via your institution

Access options

Rent or buy this article

Prices vary by article type

from$1.95

to$39.95

Prices may be subject to local taxes which are calculated during checkout

Figure 1: Repeatability of expression signal within test sites.
Figure 2: Signal variation within and between test sites.
Figure 3: Concordance of detection calls within and between test sites.
Figure 4: Agreement of gene lists.
Figure 5: Agreement of log ratios across platforms and test sites.
Figure 6: Correlation between microarray and TaqMan data.

Similar content being viewed by others

Accession codes

Accessions

Gene Expression Omnibus

References

  1. Lesko, L.J. & Woodcock, J. Translation of pharmacogenomics and pharmacogenetics: a regulatory perspective. Nat. Rev. Drug Discov. 3, 763–769 (2004).

    Article  CAS  Google Scholar 

  2. Frueh, F.W. Impact of microarray data quality on genomic data submissions to the FDA. Nat. Biotechnol. 24, 1105–1107 (2006).

    Article  CAS  Google Scholar 

  3. Dix, D.J. et al. A framework for the use of genomics data at the EPA. Nat. Biotechnol. 24, 1108–1111 (2006).

    Article  CAS  Google Scholar 

  4. Tan, P.K. et al. Evaluation of gene expression measurements from commercial microarray platforms. Nucleic Acids Res. 31, 5676–5684 (2003).

    Article  CAS  Google Scholar 

  5. Ramalho-Santos, M., Yoon, S., Matsuzaki, Y., Mulligan, R.C. & Melton, D.A. “Stemness”: transcriptional profiling of embryonic and adult stem cells. Science 298, 597–600 (2002).

    Article  CAS  Google Scholar 

  6. Ivanova, N.B. et al. A stem cell molecular signature. Science 298, 601–604 (2002).

    Article  CAS  Google Scholar 

  7. Miller, R.M. et al. Dysregulation of gene expression in the 1-methyl-4-phenyl-1,2,3,6-tetrahydropyridine-lesioned mouse substantia nigra. J. Neurosci. 24, 7445–7454 (2004).

    Article  CAS  Google Scholar 

  8. Fortunel, N.O. et al. Comment on “'Stemness': transcriptional profiling of embryonic and adult stem cells” and “a stem cell molecular signature”. Science 302, 393 author reply 393 (2003).

  9. Miklos, G.L. & Maleszka, R. Microarray reality checks in the context of a complex disease. Nat. Biotechnol. 22, 615–621 (2004).

    Article  CAS  Google Scholar 

  10. Frantz, S. An array of problems. Nat. Rev. Drug Discov. 4, 362–363 (2005).

    Article  CAS  Google Scholar 

  11. Marshall, E. Getting the noise out of gene arrays. Science 306, 630–631 (2004).

    Article  CAS  Google Scholar 

  12. Michiels, S., Koscielny, S. & Hill, C. Prediction of cancer outcome with microarrays: a multiple random validation strategy. Lancet 365, 488–492 (2005).

    Article  CAS  Google Scholar 

  13. Ein-Dor, L., Zuk, O. & Domany, E. Thousands of samples are needed to generate a robust gene list for predicting outcome in cancer. Proc. Natl. Acad. Sci. USA 103, 5923–5928 (2006).

    Article  CAS  Google Scholar 

  14. Petersen, D. et al. Three microarray platforms: an analysis of their concordance in profiling gene expression. BMC Genomics 6, 63 (2005).

    Article  Google Scholar 

  15. Dobbin, K.K. et al. Interlaboratory comparability study of cancer gene expression analysis using oligonucleotide microarrays. Clin. Cancer Res. 11, 565–572 (2005).

    CAS  PubMed  Google Scholar 

  16. Irizarry, R.A. et al. Multiple-laboratory comparison of microarray platforms. Nat. Methods 2, 345–350 (2005).

    Article  CAS  Google Scholar 

  17. Larkin, J.E., Frank, B.C., Gavras, H., Sultana, R. & Quackenbush, J. Independence and reproducibility across microarray platforms. Nat. Methods 2, 337–344 (2005).

    Article  CAS  Google Scholar 

  18. Kuo, W.P. et al. A sequence-oriented comparison of gene expression measurements across different hybridization-based technologies. Nat. Biotechnol. 24, 832–840 (2006).

    Article  CAS  Google Scholar 

  19. Shi, L. et al. QA/QC: challenges and pitfalls facing the microarray community and regulatory agencies. Expert Rev. Mol. Diagn. 4, 761–777 (2004).

    Article  Google Scholar 

  20. Shi, L. et al. Cross-platform comparability of microarray technology: intra-platform consistency and appropriate data analysis procedures are essential. BMC Bioinformatics 6 Suppl. 2, S12 (2005).

    Article  Google Scholar 

  21. Ji, H. & Davis, R.W. Data quality in genomics and microarrays. Nat. Biotechnol. 24, 1112–1113 (2006).

    Article  CAS  Google Scholar 

  22. Canales, R.D. et al. Evaluation of DNA microarray results with quantitative gene expression platforms. Nat. Biotechnol. 24, 1115–1122 (2006).

    Article  CAS  Google Scholar 

  23. Shippy, R. et al. Using RNA sample titrations to assess microarray platform performance and normalization techniques. Nat. Biotechnol. 24, 1123–1131 (2006).

    Article  CAS  Google Scholar 

  24. Patterson, T.A. et al. Performance comparison of one-color and two-color platforms within the MicroArray Quality Control (MAQC) project. Nat. Biotechnol. 24, 1140–1150 (2006).

    Article  CAS  Google Scholar 

  25. Tong, W. et al. Evaluation of external RNA controls for the assessment of microarray performance. Nat. Biotechnol. 24, 1132–1139 (2006).

    Article  CAS  Google Scholar 

  26. Guo, L. et al. Rat toxicogenomic study reveals analytical consistency across microarray platforms. Nat. Biotechnol. 24, 1162–1169 (2006).

    Article  CAS  Google Scholar 

  27. Mecham, B.H. et al. Sequence-matched probes produce increased cross-platform consistency and more reproducible biological results in microarray-based gene expression measurements. Nucleic Acids Res. 32, e74 (2004).

    Article  Google Scholar 

  28. Carter, S.L., Eklund, A.C., Mecham, B.H., Kohane, I.S. & Szallasi, Z. Redefinition of Affymetrix probe sets by sequence overlap with cDNA microarray probes reduces cross-platform inconsistencies in cancer-associated gene expression measurements. BMC Bioinformatics 6, 107 (2005).

    Article  Google Scholar 

  29. Draghici, S., Khatri, P., Eklund, A.C. & Szallasi, Z. Reliability and reproducibility issues in DNA microarray measurements. Trends Genet. 22, 101–109 (2006).

    Article  CAS  Google Scholar 

  30. Irizarry, R.A., Wu, Z. & Jaffee, H.A. Comparison of Affymetrix GeneChip expression measures. Bioinformatics 22, 789–794 (2006).

    Article  CAS  Google Scholar 

  31. Pruitt, K.D., Tatusova, T. & Maglott, D.R. NCBI Reference Sequence (RefSeq): a curated non-redundant sequence database of genomes, transcripts and proteins. Nucleic Acids Res. 33, D501–D504 (2005).

    Article  CAS  Google Scholar 

  32. Thierry-Mieg, D. & J, T.-M. AceView: a comprehensive cDNA-supported gene and transcripts annotation. Genome Biology 7, Suppl. 1, S12 (2006).

    Article  Google Scholar 

  33. Bammler, T. et al. Standardizing global gene expression analysis between laboratories and across platforms. Nat. Methods 2, 351–356 (2005).

    Article  Google Scholar 

  34. Harr, B. & Schlotterer, C. Comparison of algorithms for the analysis of Affymetrix microarray data as evaluated by co-expression of genes in known operons. Nucleic Acids Res. 34, e8 (2006).

    Article  Google Scholar 

  35. Tusher, V.G., Tibshirani, R. & Chu, G. Significance analysis of microarrays applied to the ionizing radiation response. Proc. Natl. Acad. Sci. USA 98, 5116–5121 (2001).

    Article  CAS  Google Scholar 

  36. Thompson, K.L. et al. Use of a mixed tissue RNA design for performance assessments on multiple microarray formats. Nucleic Acids Res. 33, e187 (2005).

    Article  Google Scholar 

  37. Baker, S.C. et al. The External RNA Controls Consortium: a progress report. Nat. Methods 2, 731–734 (2005).

    Article  CAS  Google Scholar 

  38. Reid, L.H. The value of a proficiency testing program to monitor performance in microarray laboratories. Pharm. Discov. 5, 20–25 (2005).

    Google Scholar 

  39. Ball, C.A. et al. Standards for microarray data. Science 298, 539 (2002).

    Article  CAS  Google Scholar 

  40. Tong, W. et al. ArrayTrack–supporting toxicogenomic research at the U.S. Food and Drug Administration National Center for Toxicological Research. Environ. Health Perspect. 111, 1819–1826 (2003).

    Article  CAS  Google Scholar 

  41. Tong, W. et al. Development of public toxicogenomics software for microarray data management and analysis. Mutat. Res. 549, 241–253 (2004).

    Article  CAS  Google Scholar 

Download references

Acknowledgements

All MAQC participants freely donated their time and reagents for the completion and analysis of the MAQC project. Participants from the National Institutes of Health (NIH) were supported by the Intramural Research Program of NIH, Bethesda, Maryland. D.H. thanks Ian Korf for BLAST discussions. This study utilized a number of computing resources, including the high-performance computational capabilities of the Biowulf PC/Linux cluster at the NIH (http://biowulf.nih.gov/) as well as resources at the analysis sites.

Author information

Authors and Affiliations

Consortia

Corresponding author

Correspondence to Leming Shi.

Ethics declarations

Competing interests

Many of the MAQC participants are employed by companies that manufacture gene expression products and/or perform testing services. J.C.W. is a consultant for and has significant financial interest in Gene Express, Inc.

Supplementary information

Supplementary Table 1

cRNA Size and Yields Collected from Microarray Platforms (XLS 45 kb)

Supplementary Table 2

Genes-to-NM Accession List for All Probes (TXT 2629 kb)

Supplementary Table 3

Most 3′ Probe-to-NM Accession List for 23,971 Common Probes (TXT 3272 kb)

Supplementary Table 4

One Probe-to-One Gene List for All Genes (TXT 1718 kb)

Supplementary Table 5

One Probe-to-One Gene List for 12,091 Common Genes (TXT 1211 kb)

Supplementary Methods

BLAST Analysis for the MAQC Project (PDF 67 kb)

Supplementary Notes

README for Probe Mapping Files (PDF 11 kb)

Supplementary Data

Supplementary Information on MAQC Study (PDF 663 kb)

Rights and permissions

Reprints and permissions

About this article

Cite this article

MAQC Consortium. The MicroArray Quality Control (MAQC) project shows inter- and intraplatform reproducibility of gene expression measurements. Nat Biotechnol 24, 1151–1161 (2006). https://doi.org/10.1038/nbt1239

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/nbt1239

This article is cited by

Search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing