Stepped wedge cluster randomized controlled trial designs: a review of reporting quality and design features

The randomized controlled trial (RCT) has long been considered the ‘gold standard’ for determining the effectiveness of an experimental intervention compared with a contemporary control [1]. Consequently, much research has been conducted to determine how to best design, analyze, and report such trials [1‐4]. Since the 1980s, a subset of possible RCT designs has been given substantial attention; the cluster randomized controlled trial (CRCT) design [5]. As defined by Hayes and Moulton in 2009 [6], in a CRCT, ‘groups [are] randomly allocated to treatment arms. These groups are referred to as clusters […] Examples of clusters include schools, communities, factories, hospitals, or medical practices, but there are many other possible choices.’ In particular, such designs have been employed when it is inappropriate or impossible to use individual-level randomization, or when the group effects are of primary interest.

One possible CRCT design is the stepped wedge (SW); first proposed by Cook and Campbell [7], and first utilized in the Gambia Hepatitis Study [8]. In the SW-CRCT approach, the intervention is introduced over a number of time periods, with the order in which each cluster begins receiving the intervention determined at random. In the classical form of the design, all clusters begin in the control condition, and all receive the intervention by the completion of the trial, with all clusters actively taking part in the trial at each time period. Moreover, once a cluster switches to the intervention, it does not switch back to the control. However, whilst this latter feature remains common to all proposed SW-CRCT designs, recently numerous extensions have been proposed that, for example, allow some clusters to begin in the intervention, end in the control condition, or only actively take part in certain time periods [9].

In addition, there are several variations of the SW-CRCT according to the chosen method of data acquisition. For example, some trials may involve cross-sectional examination of a population, potentially accruing measurements from an entirely different set of subjects in each time period [10]. At the other end of the scale, they could involve gathering repeated measurements from a single cohort [11].

Indeed, much has been written in recent years about SW-CRCTs; from papers recommending how to design and analyze these trials, to reports on completed SW trials. Methodology [10‐14] and software [15] now exists to determine required sample sizes and the optimal timings of the steps in a stepped wedge design have been ascertained [16, 17], whilst, as discussed, extensions to the standard design to allow for multiple levels of clustering or ‘incomplete block’ designs have also been presented [9]. Numerous papers have discussed when the design is more efficient, or preferential on alternate grounds, such as ethical or logistical grounds, to the parallel group CRCT [10, 18‐27], whilst there have also been recommendations on how to analyze and report trials employing the SW-CRCT design [28]. Overviews of some of the benefits and limitations of SW-CRCTs can be found in the studies of de Hoop et al. [29], Hargreaves et al. [30], and Prost et al. [31].

However, little research has been conducted to determine the reporting quality of SW-CRCTs. The first two systematic reviews assessing SW-CRCTs [32, 33] were performed when few trials had used the design. Thus, they were unable to come to any strong conclusions about the quality of reporting; however, they did suggest that there was much room for improvement. Two more recently completed reviews [34] did give some consideration to the standard of reporting of completed SW-CRCTs. However, they only assessed a small selection of reporting quality indicators, primarily focusing on the employed analysis methods, and on trial reports published from 2010 to 2014. Similarly, Martin et al. [35] did assess reporting standards, but concentrated on nine indicators relating to sample size calculations. Finally, a 2016 review focused on the available statistical methodology for designing and analyzing SW-CRCTs that have been used to date [36]. Consequently, identification of any substantial reporting failures, unrelated to sample size calculation or trial analysis, remains an important step to be undertaken. Specifically, little is known about how well trials have been introduced, the results presented, or the trials discussed thereafter. In this work, we address this issue by conducting a literature review on SW-CRCTs with no limits on the calendar time, with 43 indicators of the quality of reporting considered, drawn from across the CONSORT statement for CRCTs [37, 38], allowing us to assess broadly the standard of reporting of conducted SW-CRCTs. In addition, we are able to provide a detailed analysis of numerous design features of SW-CRCTs.

A protocol of the literature review is provided as Additional file 1. A completed PRISMA checklist [39] is available as Additional file 2 and a flow diagram for the review appears in Fig. 1.

The SW-CRCT design has received substantial attention over the last few years. In this work, we have addressed some of the issues associated with it through a literature review. Principally, we sought to discern the reporting quality of completed SW-CRCTs. To this end, we presented a review of 39 completed studies, assessing their reporting quality through 43 chosen indicators. Previously conducted reviews on SW-CRCTs have assessed reporting quality on at most 15 completed studies and 23 indicators at once. Therefore, we here provided analysis based on a far larger number of studies and indicators than previous reviews.

Our analysis found that much work remains to be done to improve quality. Sound reporting is key to assessing the validity of a study; however, the median performance across all considered criteria was 66.7%. Moreover, whilst the type of randomization used was well described (37/39, 92.3%), the method of random allocation (23/39, 59.0%), the allocation concealment mechanism (12/29, 30.8%) and the person who implemented the randomization (15/39, 38.5%) was not. Moreover, 25.6% (10/39) of studies did not describe their designs as randomized in the title or abstract.

Some 15.4% (6/39) of studies successfully detailed any harm associated with their intervention. In many instances this may well be because there was no harm, but it is important to make this clear, particularly as many of the interventions involve drugs for which side-effects could well be expected.

In line with a 2016 review [35], 38.5% (15/39) of studies did not provide any form of justification for their sample size, and 43.6% (17/39) mentioned the intracluster correlation or coefficient of variation. Moreover, 43.6% (17/39) and 41.0% (16/39) of the studies did not detail planned type-I and type-II error rates for their design. Thus, again there is great room for improvement in the reporting of the specification of SW-CRCTs.

Nonetheless, from Fig. 4 it does appear somewhat that the standard of reporting of SW-CRCTs has improved since 2010. This improvement may be because more has now been published about the design, or could perhaps be a result of the lag time for journals adopting the new criteria contained within the CONSORT extension to CRCTs [37, 38].

Whilst the median performance across the criteria listed on the CONSORT extension to CRCTs was 80.8%, it is more alarming that the mean performance was 70.8%. This, however, should perhaps not surprise us, as it is not a problem restricted solely to SW-CRCTs. Indeed several recent reviews have identified that the reporting of CRCTs in several areas has been poor [48‐51]. In particular, one 2011 review identified that the quality of reporting of only 5 of 14 considered criteria had improved following the publication of the CONSORT extension to CRCTs [51]. However, this may have been the result of a lag of uptake of the new criteria. Regardless, further efforts are needed to improve reporting quality for not only SW-CRCTs, but all CRCTs. Nonetheless, there are several features specific to SW-CRCTs not contained within this CONSORT extension, and therefore further guidance is needed on publication of SW-CRCTs. Fortunately, therefore, a CONSORT extension to SW-CRCTs is now under development [52]. In this extension, it appears warranted to encourage greater detail on the timing of each step, to ensure that it is improved in the future. It will also, in light of recent extensions to the classical SW-CRCT, be important to define clearly what constitutes a SW-CRCT, and therefore which trials would need to adhere to such guidance. Finally, 33.3% (13/39) of trials completed thus far did not include a diagram of their design. This is strongly encouraged by the CONSORT extension to CRCTs. However, it may be wise for it to be listed as a specific point on an extension to SW-CRCTs, perhaps even with certain required features, as it is clearly a very powerful means of detailing a SW-CRCT design.

In addition, we sought to ascertain the general design features of all SW-CRCTs conducted to date. To this end, we presented a review of features from an additional 84 study protocols, registrations and conference reports, along with those from the aforementioned 39 completed studies.

We observe that 91% of the identified and included records were from 2010 onwards. Thus, it appears that the design is increasing in popularity, perhaps reflecting an increasing acknowledgement of its usefulness, or simply that more is now known about the design.

It is clear that the design has been used in an extremely broad range of research areas, in a wide range of countries. Moreover, the actual characteristics of the design have varied substantially as well. Therefore, it seems that the design has proven useful to a large selection of study scenarios, its benefits seemingly coming from reasons other than one specific trial scenario. Indeed, our work highlighted once again some of the advantages of the SW-CRCT design that have frequently been cited.

We identified that the most common reason for use was that trial organizers wished to give the intervention to all clusters eventually. Additionally, a large number cited logistical and practical constraints that made the staggered implementation involved in a SW-CRCT favourable. A small number of studies cited that the design was preferred for efficiency or power related reasons. It is, of course, essential to remember that this is not universally true of the SW-CRCT design. Moreover, some studies cited that the design would allow them to fine-tune the intervention over time, but it is important to note that in SW-CRCTs one should not set out to change the intervention across time periods as this would imply that the intervention effect would change during the trial. Overall, however, it seems that there are a large number of reasons for which the design has been preferred, contributing to the broad array of research domains in which it has been utilized.

There are also several disadvantages associated with the SW-CRCT design [27, 29, 31]. Specifically, if the chosen SW-CRCT design is one that requires data collection at each point when a new cluster receives the experimental intervention, this could, in some circumstances, see the cost of data collection become considerable. Given that the design is also not guaranteed to require a smaller sample size than a parallel group CRCT, it may also require a substantially longer trial duration. Consequently, as discussed recently, researchers must remain careful when deciding whether to employ the design [29]. This seems particularly true in light of our findings that 52.0% of completed studies did not find a significant effect on any primary outcome measure for which a minimum clinically important difference had been specified. Of course, it could be that these studies were simply underpowered, owing to misspecification of the variance parameters. Extension of the classical SW-CRCT design to incorporate sample size re-estimation could therefore be an important advance. Alternatively, to guard against over-enthusiastic use of the design, interim analyses to stop the trial early for futility may also be a useful design extension for trial organizers to consider. This approach, however, would, of course, not always be applicable; for example if the intervention were part of some wider planned roll out, or if the possibility that some clusters might not receive the intervention was deemed unacceptable.

We identified that regardless of evidence of recent improvements, the quality of reporting of SW-CRCTs has been low. There is therefore a need for further guidance on how to report such trials, and the CONSORT extension to SW-CRCTs, currently under development, certainly seems warranted. Through this, precisely which information should be included when presenting the results of a SW-CRCT, and what constitutes a SW-CRCT, should be clarified, such that future SW-CRCTs may be better reported.