Identification, collection, and reporting of harms among non-industry-sponsored randomized clinical trials of pharmacologic interventions in the critically ill population: a systematic review

The present study is a systematic review of harms reporting in critical care trials of pharmacologic interventions. The manuscript adheres to standard systematic review conventions; however, the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidance does not directly apply given that the present study does not focus on specific interventions, but rather on harms reporting [4].

Randomized, non-industry-sponsored human clinical trials of pharmacologic interventions in adult critically ill populations published between January 1, 2015, and December 31, 2018, in high-impact medical journals were included. Only high-impact journals were selected to allow an efficient assessment of those trials perhaps more rigorously vetted by peers, and those with the greatest reach into clinical practice. Industry studies were excluded as they are more commonly focused on obtaining regulatory approval, often requiring extensive safety monitoring. Industry studies may also have more resources for safety monitoring, which is not representative of what is feasible for most academic investigators. Studies were included if they compared one specific pharmacologic therapy (either type, dosage, frequency, or duration) against another pharmacologic therapy or placebo. Only studies in which at least 25 patients per study arm were enrolled were considered. Critically ill patients were defined as patients evaluated in the emergency department or ICU setting who carried a high probability of imminent or life-threatening clinical deterioration as determined by the physician reviewers. High-impact medical journals included were (1) New England Journal of Medicine [NEJM], (2) Lancet, (3) Journal of the American Medical Association [JAMA], (4) British Medical Journal [BMJ], (5) The American Journal of Respiratory and Critical Care Medicine [AJRCCM], (6) Critical Care Medicine [CCM], (7) Intensive Care Medicine [ICM], and (8) Chest. These journals were selected by a consensus of the authors who are all critical care physicians and/or researchers.

The MEDLINE electronic bibliographic database was used to identify studies potentially meeting inclusion criteria. The search term can be found in the Supplementary Materials and focuses on critical care RCTs published in the above journals during the target time frame. In addition, each journal’s archives from the included years were manually reviewed to identify any RCTs not captured by the search strategy.

For each included trial, we attempted to obtain the clinical trial protocol. To this end, we searched clinical trial registries (e.g., ClinicalTrials.​gov) and published protocols in peer-reviewed journals. If the protocol could not be obtained through these methods, the author/study group was contacted for additional information. If no protocol could be obtained, this is noted in the “Results” section.

A predefined standardized data abstraction form was used to extract data as pertinent to the research question. AEs were defined as any adverse outcomes potentially related to the active intervention reported, including any safety outcomes described by the investigators. Included studies were reviewed independently by two reviewers and disagreements with respect to data abstraction or harms reporting resolved via discussion between those reviewers. A third reviewer was available to adjudicate any substantial disagreement.

As has been done in prior studies when assessing adherence to CONSORT harms recommendations, a set of items derived from the 10 items in the 2004 CONSORT harms checklist was used. This derived set of items was based on previously used checklist criteria and modified by the authors to better reflect trials in the critical care setting [5, 6]. This step was taken to better operationalize the checklist and ensure homogeneity in recording, given the open-ended nature of the original checklist items. Item 9 on the CONSORT checklist was excluded as subgroup analyses for AEs are rare. A liberal approach was taken to the assessment of each criterion, and partial completion was generally considered satisfactory. The published manuscript, including any supplemental material, was reviewed. Information present in data supplements was considered equal to that in the manuscript. The operational checklist used for this study can be found in Table 1 and ultimately included 18 items [5‐7].

In cases where at least 2 RCTs examined the same or very similar intervention in a critically ill population, expected AE definitions and frequencies were directly compared.

Adherence to CONSORT recommendations was assessed by determining the total number of criteria met in each trial. Two-way comparisons were made using Wilcoxon rank-sum tests or Fisher’s exact tests as appropriate. All statistical tests were two-sided and an alpha < 0.05 considered statistically significant.

A total of 689 abstracts were identified via the search term and an additional 18 abstracts were identified by hand review of the included journals, for a total of 707 abstracts reviewed. Of these, the two abstract reviewers highlighted 53 trials requiring full-text review (ƙ = 0.85). After review of these studies in full, 40 studies were ultimately included for full data abstraction. There were no substantial disagreements requiring adjudication by a third reviewer. A total of 40 trials were ultimately included in the analysis. Please see Fig. 1 flow diagram.

The included trials represent 28,636 randomized patients with a median of 293 (IQR 100–546) patients per trial. Most studies were blinded (n = 34 [85%]) and most were multicenter (n = 34 [85%]). A total of 31 (78%) studies had a placebo or usual care control arm. The most common disease states studied were general critical care (35%) and sepsis (25%). JAMA contributed the most studies to the review, 12 (30%). Selected details of the included articles can be found in Supplemental Table 1.

Overall, the median number of CONSORT items met was 12 (9–14) of 18. The highest number of items met was 17 and the lowest number was 1. The most commonly met items were “the efficacy and safety analyses were performed using the same populations” (n = 36, 90%), “AE results reported separately for each treatment arm” (39, 98%), and “article provides both number of AEs and number of patients with AEs” (37, 93%). Most trials failed to meet items, “article mentions how AE severity grading was performed’ (14, 25%), ‘article lists addressed AEs and provides definitions for each” (19, 48%), “article reports the number of deaths due to AEs in each arm” (18, 45%), “article presents severity grading of AEs” (17, 43%). Please see Table 1 for additional details.

There was no difference in median CONSORT checklist items met comparing multicenter to single-center trials (12 [9–14] vs. 12 [7–14], p = 0.79) or blinded to non-blinded trials (12 [9–14] vs. 11 [6–11], p = 0.22).

AE reporting was restricted to just serious AEs in 12 (30%) of studies. Eight (20%) studies reported only pre-specified safety outcomes and no studies reported only AEs in which there was a significant difference in AEs between arms.

There were a number of trials examining the same or similar interventions. Three (8%) of the trials examined haloperidol for delirium, four (10%) examined corticosteroids for infection (pneumonia, sepsis, and septic shock), and four (10.0%) examined dexmedetomidine. Among trials of the same or similar intervention, there was substantial variability in the choice, definitions, and rates of safety outcomes and adverse events. Additional details can be found in Tables 2, 3, and 4.

Clinical trial protocols were obtained for 30 (75%) trials. Of these, 25 (80%) protocols provided some definitions for expected AEs. These definitions varied substantially in the level of details provided. Fifteen (50%) protocols provided general definitions for what constitutes a “serious” AE. Six (20%) protocols provided guidance for specific AE severity grading. Three (10%) protocols referenced a validated dictionary (e.g., the Medical Dictionary for Regulated Activities) for AE definition. Sixteen (53%) protocols included guidance for AE attribution to study drugs. In most cases, this guidance was that the site investigator should use the best judgment. In one trial AEs were adjudicated by an independent board.