Skip to main content

Advertisement

SpringerLink
Log in

Challenges to evidence-based medicine:

A comparison of patients and treatments in randomized controlled trials with patients and treatments in a practice research network

  • ORIGINAL PAPER
  • Published:
Social Psychiatry and Psychiatric Epidemiology Aims and scope Submit manuscript

Abstract

Background

The practice of evidence-based medicine depends on the availability of clinically relevant research, yet questions have been raised about the generalizability of findings from randomized controlled trials (RCTs).

Objectives

The aim of this study was to quantify differences between RCT patients and treatments and those in day-to-day clinical practice.

Research Design

Data from published reports of two key RCTs underlying recent treatment advances in psychiatry were compared with data on routine psychiatric practice collected through a Practice Research Network (PRN).

Setting

Hospital inpatient units (RCT) and the full range of psychiatric practice settings in the United States (PRN).

Subjects

Adults with bipolar I disorder and adults with schizophrenia.

Measures

Demographic (age, gender, race), clinical (principal diagnoses, comorbid conditions, psychosocial functioning, and histories of hospitalization), and treatment (medication name and dosage) characteristics.

Results

PRN patients had more comorbid conditions and were more likely to be white, female, and older than RCT patients. In all, 38% of PRN patients with schizophrenia and 55% of PRN patients with bipolar I disorder would have been ineligible for the corresponding RCT. Most PRN patients receiving an RCT study medication were also receiving other medications not allowed by the RCT protocol.

Conclusions

Findings support the assertion that RCT patients and treatments are not typical of those in clinical practice, and most patients in clinical practice are receiving treatments that do not have direct empirical support. Research is needed to determine the extent to which RCT findings should be used to guide routine clinical decisions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Sackett DL, Rosenberg WMC, Gray JAM, Haynes RB, Richardson WS (1996) Evidence based medicine: what it is and what it isn’t. BMJ 312:71–72

    CAS  PubMed  Google Scholar 

  2. Sackett DL, Haynes RB,Guyatt GH,Tugwell P (1991) Clinical Epidemiology: A Basic Science for Clinical Medicine. Boston: Little, Brown and Company, p 281

  3. Starfield B (1990) Quality-of-care research: Internal elegance and external relevance. JAMA 280:1006–1008

    Article  Google Scholar 

  4. Zarin DA, West JC, Pincus HA, McIntyre JS (1995) The American Psychiatric Association Practice Research Network. In: Sederer L, Dickey B, eds. Outcomes Assessment in Clinical Practice. Baltimore, MD: William and Wilkins, pp 146–155

  5. Begg CB (1987) Cancer clinical trials in the USA: Patient eligibility, generalizability of results and technology transfer. Bull Cancer 74:197–203

    Google Scholar 

  6. Chlebowski RT, Butler J, Nelson A, Lillington L (1993) Breast cancer chemoprevention. Tamoxifen: current issues and future prospective. Cancer 72(Suppl 3):1032–1037

    Google Scholar 

  7. Tannock IF (1992) Assessment of study design in clinical trials for bladder cancer. Urol Clin North Am 19:655–662

    Google Scholar 

  8. Lee KL, Califf RM, Simes J, Van de Werf F, Topol EJ (1994) Holding GUSTO up to the light. Global Utilization of Streptokinase and Tissue Plasminogen Activator for Occluded Coronary Arteries. Ann Intern Med 120:876–881

    Google Scholar 

  9. Mulrow CD, Cornell JA, Herrera CR, Kadri A, Farnett L,Aguilar C (1994) Hypertension in the elderly. Implications and generalizability of randomized trials. JAMA 272:1932–1938

    Article  Google Scholar 

  10. Egglin TK, Horowitz RI (1996) The case for better research standards in peripheral thrombolysis: poor quality of randomized trials during the past decade. Acad Radiol 3:1–9

    Google Scholar 

  11. Koch A, Bouges S, Ziegler S, Dinkel H, Daures JP, Victor N (1997) Low molecular weight heparin and unfractionated heparin in thrombosis prophylaxis after major surgical intervention: update of previous meta-analyses. Br J Surg 84:750–759

    Article  Google Scholar 

  12. Saver JL (1995) Coping with an embarrassment of riches. How stroke centers may participate in multiple, concurrent clinical stroke trials. Stroke 26:1289–1292

    Google Scholar 

  13. Schneider LS, Olin JT, Lyness SA, Chui HC (1997) Eligibility of Alzheimer’s disease clinic patients for clinical trials. J Am Geriatr Soc 45:923–928

    Google Scholar 

  14. Cunningham WE, Bozzette SA, Hays RD, Kanouse DE, Shapiro MF (1995) Comparison of health-related quality of life in clinical trials and nonclinical trial human immunodeficiency virus-infected cohorts. Med Care 33(Suppl 4):AS15–AS25

    Google Scholar 

  15. Kramer MS, Shapiro SH (1984) Scientific challenges in the application of randomized trials. JAMA 252:2739–2745

    Article  Google Scholar 

  16. Greil W, Ludwig-Mayerhofer W, Steller B, et al. (1993) The recruitment process for a multicenter study on the long-term prophylactic treatment of affective disorders. J Affect Disord 28:257–265

    Article  Google Scholar 

  17. Licht RW, Gouliaev G,Vestergaard P, Frydenberg M (1997) Generalisability of results from randomised drug trials. Br J Psychiatry 170:264–267

    Google Scholar 

  18. Leber PD, Davis CS (1998) Threats to the validity of clinical trials employing enrichment strategies for sample selection. Control Clin Trials 9:178–187

    Article  Google Scholar 

  19. Miller CA, Hooper CL, Bakish D (1997) A comparison of patients with major depressive disorder recruited through newspaper advertising versus consultation referrals for clinical drug trials. Psychopharmacol Bull 33:69–73

    Google Scholar 

  20. Amori G, Lenox RH (1989) Do volunteer subjects bias clinical trials? J Clin Psychopharmacol 9:321–327

    Google Scholar 

  21. Taylor KM, Feldstein ML, Skeel RT, Pandya KJ, Ng P, Carbone PP (1994) Fundamental dilemmas of the randomized clinical trial process: results of a survey of the 1,737 Eastern Cooperative Oncology Group Investigators. Clin Oncol 12:1796–1805

    Google Scholar 

  22. Antman K, Amato D, Wood W, Carson J, Suit H, Proppe K, Carey R, Greenberger J, Wilson R, Frei E 3rd (1985) Selection bias in clinical trials. Clin Oncol 3:1142–1147

    Google Scholar 

  23. Skerritt U, Pitt B, Armstrong S, O’Brien A (1996) Recruiting patients for drug trials: a difficult task. Psychiatr Bull 20:708–710

    Google Scholar 

  24. Layde PM, Broste SK, Desbiens N, Follen M, Lynn J, Reding D, Vidaillet H (1996) Generalizability of clinical studies conducted at tertiary care medical centers: a population-based analysis. Clin Epidemiol 49:835–841

    Article  Google Scholar 

  25. Smith P, Arnesen H (1998) Non-respondents in a post-myocardial infarction trial: characteristics and reasons for refusal. Acta Med Scand 223:537–542

    Google Scholar 

  26. Torgerson DJ, Sibbald B (1998) Understanding controlled trials: what is a patient preference trial? BMJ 316:360

    Google Scholar 

  27. Llewellyn-Thomas HA, McGreal MJ, Thiel EC, Fine S, Erlichman C (1991) Patients’ willingness to enter clinical trials: measuring the association with perceived benefit and preference for decision participation. Soc Sci Med 32:35–42

    Article  Google Scholar 

  28. Liberati A, Himel HN, Chalmers TC (1986) A quality assessment of randomized control trials of primary treatment of breast cancer. J Clin Oncol 4:942–951

    Google Scholar 

  29. Edwards AG, Rollnick S (1997) Outcome studies of brief alcohol intervention in general practice: the problem of lost subjects. Addiction 12:1699–1704

    Article  Google Scholar 

  30. Brown C, Schulberg HC (1995) The efficacy of psychosocial treatments in primary care. A review of randomized clinical trials. Gen Hosp Psychiatry 17:414–424

    Article  Google Scholar 

  31. Moncrieff J, Drummond DC (1997) New drug treatments for alcohol problems: a critical appraisal. Addiction 92:939–947

    Article  Google Scholar 

  32. Scott JE, Dixon LB (1995) Psychological interventions for schizophrenia. Schizophr Bull 21:621–630

    Google Scholar 

  33. May WW (1998) Findings from Project MATCH: fact or artifact? Behav Health Management 18:38–39

    Google Scholar 

  34. Fortmann SP, Killen JD (1994) Who shall quit? Comparison of volunteer and population-based recruitment in two minimalcontact smoking cessation studies. Am J Epidemiol 140:39–51

    Google Scholar 

  35. Rapaport MH, Frevert T, Babior S, Zisook S, Judd LL (1995) A comparison of demographic variables, symptom profiles, and measurements of functioning in symptomatic volunteers and an outpatient clinical population. Psychopharmacol Bull 31:111–114

    Google Scholar 

  36. Drummond DC (1997) Alcohol interventions: do the best things come in small packages? Addiction 92:375–379

    Article  CAS  PubMed  Google Scholar 

  37. Essock SM, Hargreaves WA, Covell NH, Goethe J (1996) Clozapine’s effectiveness for patients in state hospitals: results from a randomized trial. Psychopharmacol Bull 32(4):683–697

    Google Scholar 

  38. Zarin DA, Pincus HA, West JC, McIntyre JS (1997) Practice-based research in psychiatry. Am J Psychiatry 154:1199–1208

    Google Scholar 

  39. American Psychiatric Association (1994) Practice guideline for the treatment of patients with bipolar disorder.Am J Psychiatry 151(Suppl):1–36

    Google Scholar 

  40. American Psychiatric Association (1997) Practice guideline for the treatment of patients with schizophrenia. Am J Psychiatry 154(Suppl):1–63

    Google Scholar 

  41. Zarin DA, Seigle L, Pincus HA, MyIntyre JS (1997) Evidence-based practice guidelines. Psychopharmacol Bull 33:641–646

    Google Scholar 

  42. Bowden CL, Brugger AM, Swann AC, Calabrese JR, Janicak PG, Petty F, Dilsaver SC, Davis JM, Rush AJ, Small JG, et al. (1994) Efficacy of divalproex vs lithium and placebo in the treatment of mania. JAMA 271:918–924

    Article  Google Scholar 

  43. Pope HG Jr, McElroy SL, Keck PE Jr, Hudson JI (1991) Valproate in the treatment of acute mania: a placebo controlled study. Arch Gen Psychiatry 48:62–68

    Google Scholar 

  44. Marder SR, Meibach RC (1994) Risperidone in the treatment of schizophrenia. Am J Psychiatry 151:825–835

    CAS  PubMed  Google Scholar 

  45. Zarin DA, Pincus HA, Peterson BD, West JC, Suarez AP, Marcus SC, McIntyre JS (1998) Characterizing psychiatry with findings from the 1996 National Survey of Psychiatric Practice. Am J Psychiatry 155:397–404

    Google Scholar 

  46. West JC, Zarin DA, Peterson BD, Pincus HA (1998) Assessing the feasibility of recruiting a randomly selected sample of psychiatrists to participate in a national practice-based research network. Soc Psychiatry Psychiatr Epidemiol 33:620–623

    Article  Google Scholar 

  47. Pincus HA, Zarin DA, Tanielian TL, Johnson JL, West JC, Pettit AR, Marcus SC, Kessler RC, McIntyre JS (1999) Psychiatric patients and treatments in 1997: findings from the American Psychiatric Practice Research Network. Arch Gen Psychiatry 56(5):441–449

    Article  Google Scholar 

  48. USP DI (1998) Volume I: Drug Information for the Health Care Professional. 18th edition. Rockville, MD: The United States Pharmacopeial Convention, Inc

  49. American Psychiatric Association (1994) Diagnostic and Statistical Manual of Mental Disorders, 4th ed. Washington, D. C.: American Psychiatric Association

  50. Shah BV, Barnwell BG, Bieler GS (1997) SUDAAN User’s Manual, Release 7.5. Research Triangle Park, NC: Research Triangle Institute

  51. Olfson M, Klerman GL, Pincus HA (1993) The roles of psychiatrists in organized outpatient mental health settings. Am J Psychiatry 150:625–631

    Google Scholar 

  52. Lehman AF, Thompson JW, Dixon LB, Scott JE (1995) Schizophrenia: treatment outcomes research—editor’s introduction. Schizophr Bull 21:561–566

    Google Scholar 

  53. Lebowitz BD,Rudorfer MV (1998) Treatment research at the millennium: from efficacy to effectiveness (editorial). J Clin Psychopharmacol 18:1

    Article  Google Scholar 

  54. Stahl SM (1999) Antipsychotic polypharmacy, part 1: Therapeutic option of dirty little secret? J Clin Psychiatry 60:425–426

    CAS  PubMed  Google Scholar 

  55. Kingsbury SJ, Yi D, Simpson GM (2001) Rational and irrational polypharmacy. Psych Services 52:1033–1036

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. American Psychiatric Institute for Research and Education Practice Research Network, 1000 Wilson Blvd, Ste 1825, Arlington (VA), 22209, USA

    Joyce C. West Ph. D., M. P. P.

  2. Technology Assessment Program Agency for Healthcare Research and Quality (AHRQ), Rockville (MD), USA

    Deborah A. Zarin M. D.

  3. Mathematica Policy Research, Inc., Washington (D.C.), USA

    Julia L. Young M. A.

Authors
  1. Deborah A. Zarin M. D.
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Julia L. Young M. A.
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Joyce C. West Ph. D., M. P. P.
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

About this article

Cite this article

Zarin, D.A., Young, J.L. & West, J.C. Challenges to evidence-based medicine:. Soc Psychiatry Psychiatr Epidemiol 40, 27–35 (2005). https://doi.org/10.1007/s00127-005-0838-9

Download citation

  • Accepted:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00127-005-0838-9

Key words

Search

Navigation