Survey of professional views on sharing interim results by the Data Safety Monitoring Board (DSMB): what to share, with whom and why

We designed our survey to have 14 questions, many of which have parts to them (see Additional file 1 for all survey questions). Questions 1 to 6 solicit responses enabling a better understanding of what type of interim results or other types of interim information should be shared at interim, with whom and why. These first six questions had advanced branching such that latter parts of a particular question would appear depending on how the respondent answered an earlier part of the question (see Additional file 1). The definitions of the interim results were provided on the survey pages. We prioritized these questions first because they were vital to understanding what should be shared or not.

Questions 7 to 14 were demographic questions asking respondents about the roles they had in relation to trials, the number of trials they were involved with, their main profession by training, professional roles they have taken on, the kind of environment(s) they usually work in, the number of trials they were involved with that had some form of private industry sponsorship and the number of trials they have been involved with that had a DSMB.

We used fluidsurveys.​com to disseminate and collect responses. The software used to analyse the results was integrated software within fluidsurvey.​com [12], WINPEPI Version 11.65 [13] and Microsoft Excel® 2010 [14]. Responses were collected anonymously. We report results in aggregate by count and percentages, indicating how many respondents chose a particular option and by mean where applicable with 95% confidence intervals (95% CIs). We summarized all written responses to questions qualitatively and quantitatively where applicable. For the questions related to whether the ICombinedER, IControlER, ACP or UCP should be shared, reasons for why an interim result measure should or should not be shared were assessed for emergent themes in relation to trial research. The description and the text given by respondents were first collated and then each response was read carefully. With iterative reading of the responses, similar reasons were grouped together. When no more groups of similar reasons existed, the groups that were there were then assessed for emerging overarching themes that were drawn from the reasons/responses within each group. This study was approved by the Hamilton Integrated Research Ethics Board (HiREB) approval [15].

Table 2 summarizes the main respondent demographics. The largest proportion of responses (42.0%) was from statisticians (156 out of 203 people responded to this question) and at least 53.6% of respondents were involved in at least one trial (203 responded to this question). About 50% (50.4%) of respondents were involved in at least one trial with DSMB monitoring (197 responded to this question) and the largest proportion of respondents (33.2%) usually work at a university or academic institution (202 responded to this question). Percentages are based on the total number of respondents to the survey (n = 371).

Table 3 summarizes the main results regarding sharing the ICombinedER, IControlER, ACP and UCP, respectively.

About half of the respondents to the question about sharing other kinds of information at interim by the DSMB with non-DSMB members reported that no other information should be shared (109/210; 51.9% [95% CI, 45.0% to 58.8%]) while 101 out of 210 (48.1% [95% CI, 41.2% to 55.0%]) respondents said yes, that other information should be shared. Table 4 summarizes all the responses. The top three responses for those that said “yes” to share other information at interim by the DSMB with non-DSMB were information about trial conduct (67; 31.9% [95% CI, 25.7% to 38.6%]), a safety issue or concern (50; 23.8% [95% CI, 18.3% to 30.1%]) and DSMB interim trial recommendations (21; 10.0% (95% CI, 6.3% to 14.9%)). The mean usefulness to share these three types of interim information were 9.16 (95% CI, 8.89 to 9.42), 9.35 (95% CI, 9.02 to 9.69) and 9.52 (95% CI, 9.08 to 9.96), respectively. Additionally, the medians for the usefulness to share these three types of interim information were 10 (IQR, 8–10), 10 (IQR, 9–10) and 10 (IQR, 10-10) [10], respectively. With whom to share varied. For information about trial conduct, it was indicated that this should be shared with the sponsor, SC, investigators or any relevant party. For information about a safety issue or concern, it was reported that this should be shared with the sponsor, SC, investigators or the Ethics Committee. For DSMB interim trial recommendation, it was indicated that this should be shared with the sponsor or the SC.

About half (104/207; 50.2% [95% CI, 43.3 to 57.2%]) of the respondents to this question about sharing interim information as indicated by the DSMB charters they encountered reported “yes,” that they were involved in a trial where sharing such information was explicitly stated in the DSMB charter. Table 5 summarizes all the responses. For those that said “yes” to the first part of this question, they were then asked which of the following pieces of interim information (ICombinedER, IControlER, ACP and UCP) should be shared during the interim of a trial, with whom and under what circumstance the sharing would happen according to the DSMB charters they encountered as well as any other additional information. The greatest proportion of respondents (55/207, 26.6% [95% CI, 20.7 to 33.1%]) reported that the ICombinedER would be shared with various parties if the overall rate was much lower than hypothesized, if there was a need to adjust the sample size or re-assess the trial’s power, if benchmarks were not met, if there was a safety or ethical issue, or if there was a recommendation from the DSMB to stop the trial because of futility or efficacy. Various respondents reported that the ICombinedER would be shared with the sponsor, investigator, funder, SC, regulator or another relevant party when deemed appropriate.

Our results empirically show that the ICombinedER is the only interim result measure where the majority of respondents think that the DSMB for a randomized controlled trial (RCT) should share with non-DSMB members. The majority of respondents indicate that it could be shared specifically with the SC. However, they did not give the ICombinedER a very high score on usefulness, only a moderate score of 6.97. Their reasoning generally for sharing this measure is that it does not tell you anything about relative effects between compared groups in a trial so it does not do any harm in terms of unmasking interim results and keeps investigators informed about the trial’s progress. They do indicate though that guesses can be made and that the ICombinedER should not be shared if the IControlER is known as having both can be unmasking of relative effects. The reason we think this measure was rated in the moderate range (6.97) in terms of usefulness is that many needed changes or decisions about the trial based on the ICombinedER can be suggested by the DSMB from their own interim review without the need to necessarily share the ICombinedER. The minority of respondents who said you do not need to share the ICombinedER indicated that the DSMB can recommend the needed changes or adaptations to the trial to the SC or sponsor without having to release the ICombinedER to them. The usefulness in sharing this measure is questionable if the DSMB is entrusted to guide the SC to make needed changes and decisions based on the DSMB’s review of the interim data. Even though the majority of respondents indicated that the ICombinedER should be shared, we do not recommend sharing it. Part of the reason for not sharing was indicated by respondents in that guesses can be made about comparative effects between treatment groups at interim. There is evidence to suggest in the academic literature [8] that the ICombinedER is a flawed measure to share and rely on as it can be compatible with any of three types of interim results: 1) one group (e.g., drug X) is performing better than another (e.g., placebo), 2) one group is doing worse than another or 3) both groups are performing the same. For instance, in this scenario question based survey [8], respondents correctly pointed out that having been given the ICombinedER of 0.34 or 34% for mortality in a hypothetical interim trial scenario could mean a 25% relative risk reduction, 25% relative risk increase or about a 2% relative risk reduction (where both groups are performing about the same). This flaw in sharing this measure is also dangerous as non-DSMB members could make speculations about comparative effects between treatment groups at interim that could consciously or subconsciously alter their behavior towards the trial, introducing bias.

Our results also empirically show that IControlER, the ACP and UCP are measures where the majority of respondents think that the DSMB for an RCT should not share with any non-DSMB member. Their reasoning generally for not sharing the IControlER and the ACP is that it can be unmasking of interim results, hence jeopardizing the integrity of the trial and potentially introducing bias. The IControlER can be directly unmasking because in many cases, the SC or the sponsor has access to pooled data, and being given the IControlER would allow them to back calculate the event rate for the intervention group. The ACP is very informative of the presence or absence of a relative treatment effect and hence it is partially unmasking to those responsible for the trial’s conduct. It was indicated that everyone involved in the trial should remain unaware of such interim results so that they can carry on enrolling, treating and following up with patients without being influenced by speculations and knowledge that could cause the introduction of trial bias. As for the UCP, comments against sharing correctly pointed out there is the potential for misunderstanding this result measure as it is simply computed under the original alternative hypothesis, not the current interim group event rates. Releasing this information could result in speculation of relative effects between groups, most often incorrect, possibly influencing behavior at all levels of study conduct. We think that the majority of respondents are correct when they say that these latter three measures should not be shared with any non-DSMB members because of the potential to introduce bias in the trial from having such information. We believe the ACP to be informative of relative group effects and the UPC to be a confusing measure that is misunderstood and possibly misinterpreted as an ACP as suggested by evidence [8]. If these types of interim results are to be shared by the DSMB, they should be shared with the SC at times when trial futility is in question or there is a major safety concern and such situations should be pre-specified in the protocol or DSMB charter. Otherwise, it seems best to let the DSMB be stewards of the trial. Respondents to our survey realize that there is a lot of risk to the integrity of the trial when sharing the latter three measures with non-DSMB members.

Beyond these four interim results, respondents indicated sharing other types of information that is typical of what is usually shared in practice. This included information on trial conduct, a safety issue or concern, DSMB interim recommendations, overall patient baseline characteristics and important information from outside the trial; all of which is very useful information that helps those responsible for the study to protect trial integrity and safety. This type of information also scored high by respondents on its usefulness for sharing at interim by the DSMB because such information provides the SC and the sponsor information needed to ensure good trial conduct without needing to unmask any group comparative results on the outcomes of interest. Sharing this type of information, when and for what reason, should be determined and agreed upon by the SC and the DSMB a priori and stated within the trial protocol and DSMB charter.

We also found out that about half of the respondents were involved in a trial where sharing interim results was explicitly stated in the DSMB charter and with whom that information should be shared during the trial’s interim. It is reassuring that there has been consideration given by trialists, before the commencement of a trial, about the possible need to share certain types of interim results with non-DSMB members and that such a need to share is explicitly stated a priori. However, this is not enough. We recommend that all trials should consider situations when there may be a need for the DSMB to share certain types of interim information and with whom. It needs to be explicit in DSMB charters how those situations that may entail sharing interim results will be handled to minimize trial bias.

This study is unique in that it empirically evaluates and focuses on whether four main forms of interim results should be shared, with whom, the usefulness of sharing that result measure, and why it should be shared, by soliciting the views of those involved in trials. A scenario-based survey published in 2017 asked trial experts how they interpreted the ICombinedER, ACP and UCP when shared in a hypothetical trial scenario. They concluded from their results that knowledge of these three interim measures should not be shared by DSMB with non-DSMB members at interim as they may mislead or unmask interim results, potentially introducing trial bias [8]. This previous survey corroborates our findings in that the majority of trial experts who responded to our survey also think the ACP and UPC should not be shared with non-DSMB members. However, the majority of respondents from our survey thought that the ICombinedER should be shared because it is not directly unmasking of the event rate per group and keeps investigators informed about the trial’s progress. They also indicated in this survey that guesses can be made about the effects between treatment group with this information; thus, caution and protocol pre-specification should be exercised when sharing this kind of information.

Six other surveys found, dating from 1999 to 2011, did not specifically focus on the issue of the DSMB sharing interim results, and were very limited in regards to the amount of detail they collected regarding what should be shared by the DSMB, with whom and why. These surveys globally looked at data monitoring practices and so each one does not provide a complete picture of the issue of the DSMB sharing interim results with non-DSMB members even when assessed as a group of articles. In general, we did find that three of the six surveys found [16‐18] (one qualitatively and two quantitatively reported) support the view that interim results should not be shared by the DSMB with at least one type of non-DSMB member. Two surveys [17, 19] (one qualitatively and one quantitatively reported) showed support for the view that interim results should be shared by the DSMB with at least one type of non-DSMB member. One survey [17] (quantitatively reported) supports that interim results should be not be shared except in particular circumstances.

In regards to limitations of our study, we had a very low response rate despite best efforts to solicit responses. We do not have any way of knowing how our non-respondents were different from our respondents. We do know from our demographic information that the largest proportion of our respondents were statisticians and about half were involved in at least one trial which reassured us that many of our respondents were most likely familiar with calculated interim result measures in a trial. Another limitation of our survey is that there was a lot of missing data. Though we received 371 responses, 202 were complete responses, meaning they filled all the questions to our survey and 169 were partial or incomplete responses, meaning that questions were skipped and left blank. In many cases, especially with our demographic information, we had 40% or more missing information from respondents. Information regarding how respondents viewed sharing the four interim result measures had less missing data, most likely because these were questions situated at the beginning of the survey. A potential reason for the amount of missing data may partially be that some people who are involved in trials may not be a part of generating or reviewing interim results, or regularly interacting with DSMBs or SCs and were thus less likely to be familiar with interim result measures. Nevertheless, it was important to include those interested or involved in trials as part of the sampling frame to capture the community’s understanding of which interim result measures should or should not be shared. On the contrary, it is possible that those that have experience being on, or interacting with, DSMBs are more acquainted with interim result measures and were thus more likely to answer the survey questions asked at the beginning that were related to these measures.

Another limitation is the possibility that an individual who is a member of both SCT and ISCB may have filled out the survey twice. We made a respondent’s anonymity and privacy a top priority and did not collect identifiable information that would allow us to crosscheck who filled out the survey from both societies. We also had to have a generic link to the survey because the survey was sent on our behalf by both SCT and ISCB. Thus, we could not provide a special and identifiable link to each unique respondent. However, if an individual was a member of both societies, it is possible that they remembered filling out the survey and would not elect to fill it out again as the same survey was used. The survey title page would have been recognizable before clicking the next button to officially start the survey. Additionally, it is important to note that most of our respondents, as indicated in Table 2, were statisticians and methodologists. In the future, it may be important to ensure a more balanced group of respondents to any survey related to this topic and make an additional effort to target non-statisticians/methodologists about these interim result measures. Their representation and interpretation on sharing these measures are equally important to understanding what interim result measures should be shared by the DSMB.

Trials are susceptible to bias and it is important to have a protocol with safeguards in place to prevent the introduction of biases that could alter trial results away from the true effect size, especially in phase III trials used to generate definitive results on efficacy and safety endpoints and provide evidence for practice and regulatory approval. In cases where there may be a request from non-DSMB members to have interim results shared with them by the trial’s DSMB, we do not recommend sharing the ICombinedER, IControlER, ACP or the UCP. Though there may be solid a priori plans in place in the trial protocol or DSMB charter to share the ICombinedER, as this measure is not directly unmasking of interim results on its own as the majority of respondents indicate, we think it lends non-DSMB members to make speculations about group rates, especially if there is a good inkling of the control rate in the academic literature. As mentioned before, the ICombinedER can also be interpreted to mean any one of three relative effects between groups making it a flawed measure to share and also lends non-DSMB members to mistakenly speculate on group rates. The results of this survey suggest that respondents from the trial community are not aware of this flaw with sharing the ICombinedER and may need to be educated on this issue. We should keep in mind that the DSMB needs to be trusted stewards of the trial and should be using discretion if there comes a time when sharing any of these four measures is needed. If such information had to be shared with a particular non-DSMB member, safeguards should be in place to prevent other non-DSMB members directly responsible for the operation or conduct of the trial, or those participating in the trial in some way, from knowing such interim result measures.