Patients included in randomised controlled trials do not represent those seen in clinical practice: focus on antimicrobial agents

doi:10.1016/j.ijantimicag.2010.03.020

International Journal of Antimicrobial Agents

Volume 36, Issue 1, July 2010, Pages 1-13

https://doi.org/10.1016/j.ijantimicag.2010.03.020 Get rights and content

Abstract

Clinicians rely on the findings of randomised controlled trials (RCTs) to formulate clinical decisions regarding individual patients. We examined whether patients included in RCTs focusing on antimicrobial agents are representative of those encountered in real-life clinical situations. PubMed was searched for RCTs referring to the field of infectious diseases. Data regarding the exclusion criteria of the identified RCTs were extracted and critically evaluated. In total, 30 trials (17 referring to respiratory tract, 5 to skin and soft-tissue, 4 to intra-abdominal, 2 to gynaecological and 2 to bloodstream infections) were included in the study. All retrieved RCTs reported extensive exclusion criteria. After comparing in a qualitative manner (based on our clinical experience) the eligible patient population in the identified RCTs with the respective population that would be encountered in general practice, it was observed that the abovementioned patient populations differ considerably. In conclusion, RCTs in the field of infectious diseases use extensive and stringent exclusion criteria, a fact that may lead to considerable difference between the patient populations of RCTs and those viewed in clinical practice. The application of the findings of RCTs to the care of individual patients should be performed cautiously.

Introduction

Clinicians are confronted with a wide variety of clinical questions and make decisions that affect individual patients. In this clinical decision-making process, evidence-based medicine may play a significant role by collecting and evaluating the best available evidence. Randomised controlled trials (RCTs), and meta-analyses of RCTs, are considered to provide evidence of the highest grade [1], [2]; thus, their findings, which are often easily accessed [3], [4], are implemented to answer clinically relevant questions [5].

The study populations of RCTs consist of individuals who must meet specific criteria, predefined by the researchers. In an attempt to increase homogeneity of the study population, researchers tend to use strict inclusion criteria. Thus, a proportion of screened patients do not fulfil the criteria for entry into RCTs; often this proportion of excluded patients seems not to be negligible. For instance, in a recent RCT evaluating the usefulness of silver-coated endotracheal tubes in preventing ventilator-associated pneumonia, 84% of the initially screened patients were excluded from the primary efficacy analysis [6]. In addition, the use of strict inclusion criteria may also contribute to the explanation of several paradoxical findings of RCTs. For example, in a meta-analysis conducted by our research team [7], all included RCTs evaluating the short versus long antimicrobial treatment of children with acute bacterial meningitis reported no death due to this well-known fatal infection (namely bacterial meningitis), a fact that presumably raises concerns regarding the external validity of the above RCTs [7].

Taking all the above into consideration, this study was performed to examine, by reviewing the exclusion criteria of RCTs, whether the patients included in RCTs are representative of those seeking medical care in real-life clinical situations.

Section snippets

Data sources

Potentially eligible articles were identified through search of the PubMed database. An article was considered eligible for inclusion in this study if it was a RCT that enrolled individuals with infections. Inclusion and exclusion criteria used in each RCT should have been clearly stated in the text. The first 30 articles that met all the above criteria and for which a full-text was obtained were included in this study. No lower time limit was applied to the search.

Data extraction and evaluation

The following information was extracted from each of the included RCTs: first author and year of publication; type of infection; characteristics of the groups to which the included patients were assigned; and inclusion and exclusion criteria as presented in the text by the authors of each study. Based on our clinical experience, the patient population that would be regarded as eligible for inclusion in each RCT was compared, in a qualitative manner, with the patient population with the same

Discussion

The main finding of this study is that the incorporation of extensive and stringent exclusion criteria of RCTs may lead to the enrolment of patients who are considerably different from those encountered in clinical practice. Thus, the conclusions generated from RCTs may not apply to a considerable proportion of patients viewed in real-life clinical situations.

Although RCTs and meta-analyses are considered to represent the top in the hierarchy of evidence-based medicine [38], the applicability

Conclusions

Patients included in RCTs appear to differ substantially from those seen in clinical practice. This fact may limit the applicability of the evidence derived from RCTs to real life. Thus, given that evidence-based medicine remains a useful tool in the decision process, clinicians should consider the limitations of the patient population involved in each clinical trial they are referring to in order to make treatment decisions as applied to individual patients encountered in every-day clinical

What is already known on this topic?

1.
Clinicians rely on the findings of RCTs to formulate clinical decisions regarding individual patients.
2.
In an attempt to increase homogeneity of study populations, researchers tend to use strict inclusion criteria.

What this study adds?

Incorporation of extensive and strict exclusion criteria of RCTs may lead to enrolment of patients that are considerably different from those seen in clinical practice.

Funding: No funding sources.

Competing interest: None declared.

Ethical approval: Not required.

References (57)

I.I. Siempos et al.
Meta-analyses on the prevention and treatment of respiratory tract infections
Infect Dis Clin North Am
(2009)
M.B. Conde et al.
Moxifloxacin versus ethambutol in the initial treatment of tuberculosis: a double-blind, randomised, controlled phase II trial
Lancet
(2009)
R.H. Dworkin et al.
A randomized, placebo-controlled trial of oxycodone and of gabapentin for acute pain in herpes zoster
Pain
(2009)
C. Lucasti et al.
Efficacy and tolerability of IV doripenem versus meropenem in adults with complicated intra-abdominal infection: a phase III, prospective, multicenter, randomized, double-blind, noninferiority study
Clin Ther
(2008)
L.D. Sher et al.
A multicenter, randomized, investigator-blinded study of 5- and 10-day gatifloxacin versus 10-day amoxicillin/clavulanate in patients with acute bacterial sinusitis
Clin Ther
(2002)
A.R. Feinstein et al.
Problems in the ‘evidence’ of ‘evidence-based medicine’
Am J Med
(1997)
M.E. Falagas et al.
Arrhythmias associated with fluoroquinolone therapy
Int J Antimicrob Agents
(2007)
F.B. Garfield et al.
Postmarketing studies: benefits and risks
Value Health
(1999)
A.X. Garg et al.
Estimating the prevalence of renal insufficiency in seniors requiring long-term care
Kidney Int
(2004)
T.P. Ryan et al.
Chronic kidney disease prevalence and rate of diagnosis
Am J Med
(2007)

Cited by (27)

Are we enrolling the right patients? A scoping review of external validity and generalizability of clinical trials in bloodstream infections
2023, Clinical Microbiology and Infection
Having a representative population in randomized clinical trials (RCTs) improves external validity and generalizability of trial results. There are limited data examining differences between RCT-enrolled and real-world populations in bloodstream infections (BSI).
We conducted a scoping review aiming to review studies assessing generalizability of BSI RCT populations, to identify sub-groups that have been systematically under-represented and to explore approaches to improve external validity of future RCTs.
MEDLINE, Embase, and Cochrane Library databases were searched for terms related to external validity or generalizability, BSI, and clinical trials in papers published up to 1 August 2023. Studies comparing enrolled versus nonenrolled patients, or papers discussing external validity or generalizability in the context of BSI RCTs were included.
Sixteen papers were included in the final review. Five compared RCT-enrolled and nonenrolled participants from the same source population. There were significant differences between the two groups in all studies, with nonenrolled patients having a greater comorbidity burden and consistently worse outcomes including mortality. We identified several barriers to improving generalizability of RCT populations and outlined potential approaches to reduce these barriers, such as alternative/simplified consent processes, streamlining eligibility criteria and follow-up procedures, quota-based sampling techniques, and ensuring diversity in site and study team selection.
Study cohorts in BSI RCTs are not representative of the general BSI patient population. As we increasingly adopt large pragmatic trials in infectious diseases, it is important to recognize the importance of maximizing generalizability to ensure that our research findings are of direct relevance to our patients.
Inadequate reporting of participants eligible for randomized controlled trials – A systematic review and meta-analysis
2021, Journal of Clinical Epidemiology
Citation Excerpt :
Restrictive inclusion and exclusion criteria may affect trial recruitment rates and thus threaten and limit external validity of RCTs. Studies have shown that trial participants were different from those who were eligible but not recruited in terms of baseline characteristics, ability to provide informed consent, and outcomes [12-15]. Adequate and transparent reporting of CONSORT flow diagram in trials allows proper assessment of the applicability and relevance of trial findings in clinical practice.
to characterize randomized controlled trials (RCTs) that did not report the overall number of participants assessed for eligibility and to identify factors associated with higher enrollment rates.
Systematic review and meta-analysis of RCTs in several pre-defined fields in internal medicine. We randomly extracted 360 articles that were published in 2017. Trials that reported numbers of assessed for eligibility patients were compared with those who did not. Recruitment rates were calculated in order to investigate whether they were associated with trial characteristics.
A total of 360 RCTs were included. Only 2-thirds of the trials (242/360) reported the number of patients assessed for eligibility. Trials reporting eligibility data had better methodology, reported on the tested hypothesis, included a placebo arm, evaluated soft outcomes, published their findings in higher impact journals and recruited a higher number of randomized patients than those who did not. Recruitment rates in 225 (62.5%) trials enabling their calculation, were significantly higher in trials sponsored by industry, conducted in multiple centers and countries, including inpatients, tested non-inferiority hypothesis, included a placebo arm, and evaluated surrogate outcomes.
Reporting of participant eligibility continues to be scarce. Inadequate reporting was associated with poor methodological characteristics in trials.
Rapidly Improving ARDS in Therapeutic Randomized Controlled Trials
2019, Chest
Citation Excerpt :
However, a sensitivity analysis was performed after excluding those patients, and we found similar results with our main analysis (e-Table 8). Finally, as it is well documented in the literature, there is a difference between patients enrolled in randomized controlled trials and those whom clinicians are obliged to treat.39 All patients included in our analysis had been enrolled in randomized controlled trials, which had strict exclusion criteria,14-19 and may not be generalizable to all patients with ARDS.
Observational studies suggest that some patients meeting criteria for ARDS no longer fulfill the oxygenation criterion early in the course of their illness. This subphenotype of rapidly improving ARDS has not been well characterized. We attempted to assess the prevalence, characteristics, and outcomes of rapidly improving ARDS and to identify which variables are useful to predict it.
A secondary analysis was performed of patient level data from six ARDS Network randomized controlled trials. We defined rapidly improving ARDS, contrasted with ARDS > 1 day, as extubation or a Pao₂ to Fio₂ ratio (Pao₂:Fio₂) > 300 on the first study day following enrollment.
The prevalence of rapidly improving ARDS was 10.5% (458 of 4,361 patients) and increased over time. Of the 1,909 patients enrolled in the three most recently published trials, 197 (10.3%) were extubated on the first study day, and 265 (13.9%) in total had rapidly improving ARDS. Patients with rapidly improving ARDS had lower baseline severity of illness and lower 60-day mortality (10.2% vs 26.3%; P < .0001) than ARDS > 1 day. Pao₂:Fio₂ at screening, change in Pao₂:Fio₂ from screening to enrollment, use of vasopressor agents, Fio₂ at enrollment, and serum bilirubin levels were useful predictive variables.
Rapidly improving ARDS, mostly defined by early extubation, is an increasingly prevalent and distinct subphenotype, associated with better outcomes than ARDS > 1 day. Enrollment of patients with rapidly improving ARDS may negatively affect the prognostic enrichment and contribute to the failure of therapeutic trials.
The generalizability of bronchiectasis randomized controlled trials: A multicentre cohort study
2016, Respiratory Medicine
Citation Excerpt :
RCT's often have strict inclusion and exclusion criteria required to limit the risk of adverse effects and to increase the likelihood of obtaining a positive outcome by controlling potential confounders. The result of this, in respiratory diseases such as COPD and asthma and in other conditions, has been that populations of patients enrolled into clinical trials are often not representative of the general population [10–12]. For example, Herland estimates that only 5.4% of asthma patients and 17% of COPD patients in Scandinavia were eligible for typical RCT's [12].
Randomized controlled trials (RCTs) for bronchiectasis have experienced difficulties with recruitment and in reaching their efficacy end-points. To estimate the generalizability of such studies we applied the eligibility criteria for major RCTs in bronchiectasis to 6 representative observational European Bronchiectasis cohorts.
Inclusion and exclusion criteria from 10 major RCTs were applied in each cohort. Demographics and outcomes were compared between patients eligible and ineligible for RCTs.
1672 patients were included. On average 33.0% were eligible for macrolide trials, 15.0% were eligible for inhaled antibiotic trials, 15.9% for the DNAse study and 47.7% were eligible for a study of dry powder mannitol. Within these groups, some trials were highly selective with only 1–9% of patients eligible. Eligible patients were generally more severe with higher mortality during follow-up (mean 17.2 vs 9.0% for macrolide studies, 19.2%% vs 10.7% for inhaled antibiotic studies), and a higher frequency of exacerbations than ineligible patients. As up to 93% of patients were ineligible for studies, however, numerically more deaths and exacerbations occurred in ineligible patient across studies (mean 56% of deaths occurred in ineligible patients across all studies).
Our data suggest that patients enrolled in RCT's in bronchiectasis are only partially representative of patients in clinical practice. The majority of mortality and morbidity in bronchiectasis occurs in patients ineligible for many current trials.
Review of meta-analyses of vancomycin compared with new treatments for Gram-positive skin and soft-tissue infections: Are we any clearer?
2015, International Journal of Antimicrobial Agents
Citation Excerpt :
This review has specific limitations that need to be considered in the interpretation of findings. Most of the studies included in the MAs were conducted before the implementation of the FDA guidance for the design and conduct of clinical trials in ABSSIs published in 2013 that proposed significant changes such as the development of objective, quantifiable and reproducible assessments of clinical response replacing the subjective investigator assessment [44,45]. Most of the RCTs included in the MAs were sponsored by the pharmaceutical companies that produce newer antimicrobials, and studies with the largest weight on results were the pivotal trials.
Vancomycin has been considered the standard of care for treatment of Gram-positive skin and soft-tissue infections (SSTIs). Its value has been questioned over the last decade owing to well acknowledged limitations in efficacy and tolerability and the emergence of newer meticillin-resistant Staphylococcus aureus (MRSA)-active antibacterial agents. However, no single agent has shown better results versus vancomycin in SSTI trials. The aim of this review was to identify and summarise data from meta-analyses (MAs) for the treatment of Gram-positive and MRSA SSTIs. A systematic search identified 21 published MAs examining the use of newer antibiotics and vancomycin in SSTIs. In terms of clinical and microbiological efficacy, linezolid (in Gram-positive and MRSA SSTIs) and telavancin (in MRSA SSTIs) were shown to be more effective than vancomycin. The safety of newer antimicrobials in general was comparable with vancomycin, except for telavancin, which was associated with more severe adverse events (AEs), and tigecycline owing to an all-cause mortality imbalance observed in all infections but not confirmed in SSTIs. Specific AEs were related to the use of newer agents, such as nephrotoxicity for telavancin, creatine phosphokinase elevations for daptomycin, and thrombocytopenia with linezolid. Some evidence suggests that daptomycin could be associated with reduced treatment duration, and linezolid with reduced length of intravenous treatment and hospital length of stay compared with vancomycin. Considering the limitations of this type of research and the comparative efficacy results demonstrated in head-to-head randomised controlled trials, data are still not sufficient to support the widespread use of new agents over vancomycin.
Guidelines for the treatment of pneumonia and urinary tract infections: Evaluation of methodological quality using the appraisal of guidelines, research and evaluation II instrument
2013, Clinical Microbiology and Infection
Citation Excerpt :
Evidence is often lacking for critically ill patients, who are seldom included in clinical trials. Thus, the patient population included in randomized controlled trials (RCTs) on bacterial infections might not represent the patient population seen in clinical practice [1]. Conceptual trials examining questions such as ‘Should we treat this infection?
Reliance on evidence-based medicine requires high methodological standards from guideline developers. We sought to determine the methodological quality of guidelines on pneumonia and urinary tract infections (UTIs). We included guidelines published by national or international committees in the last 10 years providing recommendations for antibiotic type or duration. We applied the Appraisal of Guidelines for Research and Evaluation II checklist, adding under each item the specific focus relevant to bacterial infections, addressing antibiotic resistance and local epidemiology. Three assessors scored each guideline independently. Mean aggregated scores, converted to percentage per domain, are presented. We included 13 guidelines on the treatment of pneumonia and seven guidelines for the treatment of UTI. ‘Scope and purpose’ scored 69.4% for pneumonia and 71.4% for UTI. Guidelines were downgraded for lack of an epidemiological overview relevant to intended users. ‘Stakeholder involvement’ scored 39.5% for pneumonia and 44.5% UTI, with the major fault being lack of patient consultation. ‘Rigour of development’ scored 42.8% for pneumonia and 56.9% for UTI. Commonly, the search process lacked precision, no risk of bias assessment was performed, outcomes in primary studies were not critically assessed or used to direct recommendations, and there was no formal methodology for formulating recommendations. ‘Clarity of presentation’ scored highest: 67.7% for pneumonia and 68.5% for UTI. ‘Applicability’ of the guidelines in antibiotic stewardship programmes was usually not addressed: 16.9% and 25.4%, respectively. ‘Editorial independence’ scored 30.6% for pneumonia and 55.6% for UTI. Formal examination of guidelines in infectious diseases showed worrying findings related to core methodology and potential bias caused by competing interests.

View all citing articles on Scopus

View full text

ReviewPatients included in randomised controlled trials do not represent those seen in clinical practice: focus on antimicrobial agents

Abstract

Introduction

Section snippets

Data sources

Data extraction and evaluation

Discussion

Conclusions

What is already known on this topic?

What this study adds?

Infect Dis Clin North Am

Lancet

Pain

Clin Ther

Clin Ther

Am J Med

Int J Antimicrob Agents

Value Health

Kidney Int

Am J Med

Am J Gastroenterol

Control Clin Trials

Randomized, controlled trials, observational studies, and the hierarchy of research designs

N Engl J Med

Evidence based medicine: an approach to clinical problem-solving

BMJ

Open-access world wide web resources on sepsis

Clin Infect Dis

World Wide Web resources on control of nosocomial infections

Crit Care

Silver-coated endotracheal tubes and incidence of ventilator-associated pneumonia: the NASCENT randomized trial

JAMA

Short versus long duration of antibiotic therapy for bacterial meningitis: a meta-analysis of randomised controlled trials in children

Arch Dis Child

The effect of high dose inhaled corticosteroids on wheeze in infants after respiratory syncytial virus infection: randomised double blind placebo controlled trial

BMJ

Long-term follow-up trial of oral rifampin–cotrimoxazole combination versus intravenous cloxacillin in treatment of chronic staphylococcal osteomyelitis

Antimicrob Agents Chemother

Epinephrine and dexamethasone in children with bronchiolitis

N Engl J Med

Community-based outpatient parenteral antimicrobial therapy (CoPAT) for Staphylococcus aureus bacteraemia with or without infective endocarditis: analysis of the randomized trial comparing daptomycin with standard therapy

J Antimicrob Chemother

The efficacy and safety of daptomycin vs. vancomycin for the treatment of cellulitis and erysipelas

Int J Clin Pract

Randomized, multicentre, comparative clinical evaluation of cefuroxime–sulbactam versus amoxicillin–clavulanic acid therapy in the treatment of lower respiratory tract infections

J Int Med Res

Clinical efficacy and time to symptom resolution of 5-day telithromycin versus 10-day amoxicillin–clavulanate in the treatment of acute bacterial sinusitis

Curr Med Res Opin

Improved treatment of vulvovaginal candidiasis with fluconazole plus probiotic Lactobacillus rhamnosus GR-1 and Lactobacillus reuteri RC-14

Lett Appl Microbiol

A randomized comparative study of levofloxacin versus amoxicillin/clavulanate for treatment of infants and young children with recurrent or persistent acute otitis media

Pediatr Infect Dis J

Micafungin versus caspofungin for treatment of candidemia and other forms of invasive candidiasis

Clin Infect Dis

Review
Patients included in randomised controlled trials do not represent those seen in clinical practice: focus on antimicrobial agents