A systematic review of the quality of reporting in published smoking cessation trials for pregnant women: an explanation for the evidence-practice gap?

Effective translation of evidence into clinical practice should follow a logical pathway. First, evidence should be generated through the conduct of well-designed and methodologically rigorous randomised controlled trials, which are considered the gold standard in evaluating the comparative effectiveness of healthcare interventions [1]. Replication of effective interventions should then occur to provide verification of findings, allow for the extension and generalization of results, and ensure that adoption of findings will be effective, efficient, and improve care [2]. The findings of well-designed trials can then be combined in meta-analysis to increase the precision of estimations of intervention effect [3]. Clear, specific, and actionable clinical practice guidelines can then be generated for to guide care, and recommendations communicated and disseminated through training and accreditation processes [4]. This process of evidence generation necessitates that randomised controlled trials, on which other information is based, provide detailed, specific, and easily interpretable information.

In the early 1990’s, a number of reviews identified systemic failures to fully report the methodological characteristics of intervention studies [2],[5]–[7]. As a result, in 1996 the Consolidated Standards of Reporting Trials (CONSORT) guidelines were developed to ensure uniform reporting of the methodological aspects of randomised controlled trials, reduce bias in estimates of intervention effectiveness, and improve internal and external validity [8]. Use of CONSORT criteria has been shown to improve accuracy and quality of reporting [9]–[12], and is now a pre-requisite for publication in many journals. A number of guidelines aimed at improving the quality of reporting of observational studies [4] and systematic reviews [7],[13],[14] have since also been published and adopted [15]. However, less attention has been paid to ensuring detailed and specific information is provided about the content of interventions and precisely how they are delivered [16]. The adequacy of description of intervention content and implementation is not currently captured by these tools. CONSORT guidelines ask researchers to report the “precise details of the treatments intended for each group, and when and where they were actually administered”, however more specific details including timing of treatment, content of intervention materials, and the intervals between delivery of intervention components that are needed to guide clinical practice are not captured by CONSORT or other methodological quality rating tools including those produced by Cochrane [17].

Several recent studies have examined the extent of accurate and complete reporting of essential elements of interventions across different fields of healthcare [16]–[20]. In one analysis by Glasziou et al. [17], only 50% of intervention studies deemed important for clinical practice selected for abstraction in the journal Evidence Based Medicine were deemed replicable. This was despite studies examined being published in high-impact journals including The New England Journal of Medicine, The Lancet, and The British Medical Journal. Duff et al. [18] examined reporting of therapeutic details required for clinical application of the findings of oncology drug trials, finding that only 11% of trials reported all aspects deemed to be essential for replication and clinical translation. Currow et al. [19] examined reporting of data items deemed essential for interpretation of study generalizability in specialist palliative care research, finding that adequate data was reported only 28% of the time. Most recently, Schroter et al. [16] found that 33% of drug treatments and 73% of non-drug treatments published in the British Medical Journal over one year were considered non-replicable.

Smoking is the most significant modifiable cause of adverse pregnancy outcomes. Babies born to mothers who smoke during pregnancy have 50% greater risk of perinatal death [21], 60% greater risk of preterm birth [21], and weigh on average 200 g less than babies born to non-smoking mothers [22]. Smoking cessation during pregnancy reduces negative pregnancy outcomes, including miscarriage, foetal growth retardation, preterm delivery [23], and perinatal mortality [24]. Despite evidence for the effectiveness of smoking cessation interventions for pregnant women [25], the provision of best-evidence care continues to be suboptimal. A recent review [26] found that just over half of all healthcare providers are likely to ask pregnant women if they smoke and provide advice to quit, while less than one-half actually assist women to quit by providing specific intervention or referral. Failure to appropriately intervene with pregnant women who smoke could be the result of a number of factors, including healthcare providers not being aware of or agreeing with the evidence for effective interventions, and perceiving that the results of trials do not apply to their patient group or are too difficult or too expensive to implement [26]. Another explanation may be that poor reporting of data in clinical trials has inhibited the implementation of best-practice care in clinical practice.

The information reported in randomised controlled trials should be sufficient to allow readers to assess to whom the trial result can reasonably be applied, and how to replicate the intervention. Several studies have developed checklists for assessment of study external validity and other information deemed important in reporting of randomised controlled trials [27]–[30]. These criteria typically include reporting of the setting of the study; recruitment method; participant information, treatment, or intervention delivered (duration, intensity, dose, delivery format, timing, compliance); outcome assessment; and follow-up. Smoking cessation interventions typically incorporate substantial behavioural components that are difficult to both describe and reproduce. To provide appropriate clinical care, clinicians also need clear advice about: the precise behavioural strategies that should be used, what skill individuals providing behavioural support should have, how intense the intervention should be, whether to use nicotine replacement therapy, how to deal with women who do not want to quit, and those who relapse. No studies have examined the adequacy of reporting of this specific information for smoking cessation trials with pregnant women.

To examine whether twenty elements of interventions and their implementation deemed essential for replication, implementation, and aggregation of studies in meta-analysis are reported in sufficient detail in smoking cessation trials with pregnant women. Criteria were developed based on previously published checklists developed to assess the external validity and applicability of intervention trials, and critical assessment of the information necessary to effectively provide smoking cessation in line with clinical practice guidelines.

Medline, The Cochrane Library, and PsycINFO databases were searched for relevant studies using MeSH terms and subject headings (see Table 1). Searches were limited to studies published between 2001 and 2012. The reference lists of previous reviews of relevant literature and the reference lists of retrieved articles were also searched to identify any additional relevant studies. The review protocol is available from the authors.

Studies were include if they: evaluated a smoking cessation intervention targeted at pregnant women; had smoking cessation or abstinence as an outcome measure; used a design endorsed by the Effective Practice and Organisation of Care (EPOC) group [31] (randomised controlled trials; controlled clinical trials; controlled before and after studies that provided adjustment for confounders; and interrupted time series studies were included); and were published in an English language peer-reviewed journal. Studies were excluded if: the intervention was targeted at a pregnant or postpartum woman’s partner or family member; the intervention was targeted at changing healthcare provider behaviour only; or the intervention examined relapse prevention only without examining cessation or abstinence.

The titles and abstracts of identified studies were assessed for relevance by one reviewer and rejected on initial screening if the reviewer could determine that the study did not meet inclusion criteria. The remaining studies were retained and assessed against eligibility criteria by two reviewers (JB and AH). A backward and forward reference list search was conducted using Web of Science for all included studies to determine whether additional details of any studies were described in other publications reporting the same study.

A concise checklist of elements deemed essential to the reporting of intervention content and implementation was developed based on previously published checklists developed to assess the external validity and applicability of intervention trials [32]–[34]; and critical assessment by the authors of the information required to effectively provide smoking cessation care from prenatal into the postnatal period based on recommendations contained with clinical practice guidelines. A standard data extraction checklist was developed, then pilot tested by two authors (JB and AH) by independently assessing five studies using the checklist, then meeting to refine and resolve any disagreements. The data extraction checklist is described in Table 2.

Studies meeting eligibility criteria were coded by two individuals. Discrepancies were resolved by a third. Each study was coded as to whether the study reported the information (yes or no). As not all criteria were relevant to all studies (for example, some studies did not use nicotine replacement therapy), the proportion of all relevant elements reported for each study was then calculated. A score of 100% indicated complete reporting of all applicable elements.

The initial search yielded 319 citations, and five additional citations were identified through reference list searches (see Figure 1). After duplicates were removed, 258 unique articles were assessed against eligibility criteria of which, 30 studies [35]–[64] met criteria for inclusion. These studies described 45 intervention arms. One additional publication [65] reporting the same study methodology of an included study [49] was identified through backwards and forwards referencing searching. This manuscript was referred to when coding the primary manuscript, however only the primary publication reporting the cessation or abstinence outcome is cited in the results.

Twenty individually randomised controlled trials [35],[37],[38],[41],[43],[45],[46],[48]–[51],[56]–[64], six cluster randomised controlled trials [40],[42],[52]–[55], and four non-randomised controlled trials (which used quasi-randomisation methods) [36],[39],[44],[47] were identified. The majority of studies (n = 20) were conducted in the United States [35]–[38],[41],[43],[45]–[47],[50],[51],[54],[56]–[59],[61]–[64], with five studies conducted in the United Kingdom [39],[42],[49],[53],[60], two in the Netherlands [40],[44], and one study each conducted in Poland [55], Canada [48], and New Zealand [52]. The majority of studies used behavioural counselling approaches. Five studies used nicotine replacement therapy [44],[48],[51],[56],[63].

Table 3 outlines the quality of reporting of the 45 interventions arms against the twenty criteria. The mode of delivery of the intervention was reported in 100% of intervention arms. Other basic details of the study including the person who delivered the intervention (96%), the characteristics of the sample (80%), the setting where the intervention was delivered (80%), and the frequency of contact between healthcare providers and participants (73%) were well described. All eight intervention arms (100%) that used a financial incentive intervention described the type, amount, frequency, and conditions under which the incentives were provided. Five of the six intervention arms (83%) that used nicotine replacement therapy adequately described the type, strength, frequency, and duration of use.

Table 4 summarises the proportion of intervention elements reported for each intervention arm. No intervention arms reported 100% of all relevant criteria. One intervention arm reported 94% of relevant criteria. Thirty-six percent of intervention arms reported less than 50% of relevant criteria. Four intervention arms reported less than 30% of relevant criteria.

A number of studies have made recommendations about ways to improve reporting in scientific studies. One suggested approach is the creation of checklists that dictate the specific elements of interventions that should be reported [32]. As suggested by Glasziou et al. [17], it may be necessary to tailor such checklists to different types of interventions. Several attempts have been made to do this. For example, in an attempt to improve standards in reporting of acupuncture trials, Standards for Reporting Interventions in Controlled Trials of Acupuncture (STRICTA) [66] have been developed as an extension to the CONSORT criteria. The STRICTA guidelines have since been adopted by a number of journals publishing in acupuncture research. There has been a similar call for standards of reporting to be developed for drug administration in oncology [18] and surgery [67] in order to ensure consistency in the details reported.

There are clear advantages to adopting a standard of reporting for smoking cessation trials. Smoking cessation interventions usually incorporate behavioural strategies, which by their nature are often multi-factorial and can be heavily dependent on the skills and precise behaviours of the person delivering the intervention. If we are to identify the most effective, successful, and appropriate smoking cessation strategies for pregnant women, we need to understand and be able to replicate the specific components of these trials. Michie [68] and Abraham [69] have written extensively about this issue and suggested a taxonomy of behaviour change techniques may be useful in standardising the reporting of components of interventions. An examination of the use of such behaviour change techniques in published smoking cessation trials with pregnant women was recently conducted, with eleven techniques identified as effective [70]. However, the authors noted that the descriptions of interventions were not detailed enough to allow analysis of the effectiveness of individual techniques across studies, highlighting the critical need for improvement in reporting for advancing the evidence base.

Strict length limitations imposed by peer-reviewed journals may also limit the reporting of intervention detail. Electronic methods of peer-review publication allow for supplements or appendices providing detailed information to be published online alongside traditional journal articles [17], and could be better used by authors to make available study protocols as a supplement to the peer reviewed paper, or linking the publication with the published trial registration [68]. While this recommendation is frequently made in many papers examining quality of reporting [71],[72], it is not yet standard practice. Some journals are however starting to require additional information be provided. For example, the journal Addiction requires studies reporting behavioural interventions to include very detailed descriptions of the full study protocol or intervention manual to be published as supplementary material to the peer-reviewed manuscript.

The findings of this review should be considered in light of several limitations. First, the checklist of essential elements used may not capture all of the elements relevant for clinical translation. However, this checklist was developed based on previous studies in other areas [69],[73], and could be used as a starting point to the development of a more comprehensive smoking cessation specific checklist. Second, it is possible that some eligible relevant articles were not identified by our search strategy and therefore not included in this review. Finally, we only coded the presence or absence of reporting of key information, not the quality or accuracy of the information presented.