Patient enrollment and logistical problems top the list of difficulties in clinical research: a cross-sectional survey

We defined medical researchers as principal investigators (PIs) of study protocols approved by the REC of the Geneva University Hospitals, Switzerland between January 1, 2001 and December 31, 2005. The Geneva University Hospitals is a single 1800-bed public teaching hospital, covering the entire range of medical disciplines, with eight sites across the Geneva region. About 1200 research protocols are currently signaled as ongoing by the REC.

We conducted a cross-sectional survey among PIs of the selected protocols. We sent them a self-administered online questionnaire between November and December 2010, followed by two reminders. We asked PIs to answer one questionnaire for each separate protocol approved by the REC, identified by its title. The questionnaire had been pre-tested previously among of 12 medical researchers who were not part of the study. We asked PIs to report if the study was completed (defined as recruitment, data collection and analysis completed, whatever the study is published or not) or not (ongoing or abandoned) and whether they had received training in quantitative methods (none, continuing education at the University level, or bachelor/master/PhD in quantitative methods) at the time of the submission to the REC. If the study was completed, we asked PIs to state if they encountered any of 14 specific difficulties in the course of research (Fig. 1) on the following scale of difficulty: none, minor, medium, great, very great, or not applicable. Based on a review of the literature, a group of experts including methodologists (AGA, GH, SC, TA, TVP) and biostatisticians (DSC and Dr. Christophe Combescure, see acknowledgements) identified these 14 difficulties in conducting clinical research. The results from the pre-test with medical researchers suggested the difficulties were clearly described. “Other difficulties” could be reported in a fifteenth item as free text. We did not invite PIs to rate their difficulties with ongoing studies, as we wanted to include only hindsight evaluations of experience. We also asked PIs whether the results of the study were published or “in press”, to provide the reference in an open field, and also whether they had papers in submission or in preparation; for the latter studies, we updated the database by identifying relevant papers published up to November 2013 (PubMed search).

We identified characteristics of study protocols using information recorded in the archives of the REC. We abstracted data on study design (interventional, observational), first decision of the REC (positive, positive with recommendations, positive with written modifications required, positive with conditions and reassessment, negative, and not considered), public vs. private sponsorship of the study (i.e. hospital or university vs. industry or private foundation), link with the industry (none, industry with indication of support, industry without indication of support), planned sample size (later dichotomized as up to 100 vs. larger), reporting of sample size calculation (yes, no), number of centers (single versus multi center), clinical specialty (oncology, neurology, infectious disease, rheumatology, psychiatry, cardiology, general, other specialties, combination of two specialties), and presence of a statistician on study team (yes, no). The category “Other specialties” included all other specialties distributed across the medical services of the Geneva University Hospitals (65 medical services), or studies with no clear clinical specialty. We classified the type of research conducted into the following seven groups: study of health or illness (i.e., organs, tissues, cells, receptors, etc.), interventional or treatment, diagnosis, prognosis, public health or health economics or opinions or perceptions, study methods, and other. We also dichotomized this variable in study protocols informing clinical questions (i.e., interventional/treatment, diagnostic and prognostic studies) versus other types of research (i.e., studies of health/illness, public health, study methods, and other).

In the survey among PIs, each of the 14 items of self-reported difficulties was dichotomized into medium, great or very great difficulty (1) versus no or minor difficulty (0). When PIs answered “not applicable”, the value “0” (no difficulty) was assigned, because we considered not applicable difficulties as an absence of difficulty. The fifteenth item for “other difficulty(ies)” was followed by an open field for description. Some free-text responses from this item were similar to the other 14 difficulties already defined. One co-author (SC) recoded the text responses into the pre-defined difficulties and another co-author (TP) checked the recoding. Eight free-text responses were re-classified and 41 remained in the “other” category.

Dichotomized self-reported difficulties were used to calculate sums of difficulties for each study and then to compare mean numbers of difficulties across study characteristics.

We defined three dependent variables. The first was a score summing all difficulties across the whole research process. We then classified the 14 difficulties into dimensions using factor analysis with varimax rotation (for a complete description, see Additional file 1: Appendix 1). The analysis suggested two dimensions of seven difficulties each: a dimension related to “scientific difficulties” (literature review, research question, study design, data management, data analysis, manuscript writing, publication in a peer-reviewed journal) and a dimension related to “logistical difficulties” (approval by the REC, getting funding, getting approval by safety agencies, patient enrollment, data collection, collaboration with caregivers for patient enrollment, recruiting competent staff). Dimension scores based on the sum of corresponding items were the second and third dependent variables. Both had adequate internal consistency, with a Cronbach alpha coefficient of 0.83 for scientific difficulties and 0.73 for logistical difficulties.

The analysis of difficulties encountered by researchers was a secondary objective of this study. The primary objective of this study was concerned with publication of clinical studies (manuscript in preparation) and sample size was determined with regard to the primary objective. However, a sample size of 400 protocols yielded the following precision (exact 95 % confidence interval) in the estimation of prevalence for a given difficulty: 5.0 % (3.1 to 7.6 %), 10.0% (7.2 to 13.4 %), 20 % (16.2 to 24.3 %), 30% (25.5 to 34.8 %), and 40 % (35.2 to 45.0 %). A comparison of means of 2 groups of 200 observations each would allow for the detection of a small effect size (0.28) with an alpha of 0.05 and a power of 80 %.

Among the 996 study protocols, 588 questionnaires were returned by the PIs (participation rate 59.0 %). Among these 588 studies, 391 (66.5 %) were completed at the time of the survey, 96 (16.3 %) were still ongoing and 101 (17.2 %) had been abandoned.

More than three of four completed protocols were sponsored by a public institution, about two thirds were published, a majority was single center studies, and about half were observational studies (Table 1). The median of the sample sizes was 100 (mean 591, interquartile range 260, range 1 to 25,000) and a majority of study protocols did not report sample size calculation. Study protocols reporting a sample size calculation had more often a sample size above the median compared to study protocols that did not (60.9 % versus 30.4 %, p-value <0.001). Almost three protocols out of four did not have a statistician in the study team. Most PIs reported training in quantitative methods – courses or continuing education or a University degree –at the time of the protocol submission to the REC.

For 408 study protocols the questionnaire was not returned by the PI. Information on these 408 studies was collected using their study protocol stored in the archives of the REC. More than seven of ten studies had no statistician in the study team, around two third had public funding, and the majority had no link with the industry, were single center studies, were unpublished and had an interventional design (Additional file 1: Appendix 2). The median sample size was 100 (mean 973, interquartile range 321, range 3 to 136,000) and a majority of study protocols did not justify sample size. Compared with included studies, studies with no questionnaire had higher proportions of interventional studies, private sponsorship, link with the industry and were more frequently unpublished.

Levels of difficulties are reported in Fig. 1: patient enrollment and data collection were the most frequently reported. The mean number of difficulties was 2.8 (standard deviation (SD) 2.8, min 0, max 12); 26.4 % of PIs reported no difficulty and 18.7 % reported one difficulty only (Additional file 1: Appendix 3). The mean number of scientific difficulties was 1.1 (SD 1.7, min 0, max 7) and the mean number of logistical difficulties was 1.6 (SD 1.6, min 0, max 7).

In univariable analysis, study protocols sponsored by a public institution (hospital or University) and single center studies encountered more difficulties overall and more scientific difficulties in particular (Table 2). Compared to interventional designs, observational designs encountered more scientific difficulties, but fewer logistical difficulties. The presence of a statistician in the study team was associated with fewer scientific difficulties. Study protocols informing clinical questions (i.e., interventional, diagnostic or prognostic studies) were associated with more logistical difficulties compared to other types of research (studies of health/illness, public health, study methods, etc.). The same applied to the first REC decision: study protocols whose first REC decision either required an additional assessment or was negative (including non-consideration) reported more logistical difficulties compared to study protocols that received a positive first decision.

In multivariable analysis, study design, first decision of the REC and the presence of a statistician on the study team were not associated with a higher number of difficulties (Additional file 1: Appendix 5) while several risk factors remained significantly associated. In a parsimonious model (Table 3), publicly sponsored and single center studies were associated with more difficulties overall and more scientific difficulties. Studies informing clinical questions were associated with higher number of difficulties overall and with more logistical difficulties.

The three study characteristics from the above multivariate model (private vs. public, multi vs. single center and “clinical” vs. other) were cross-tabulated with specific difficulties (Table 4). Compared to publicly sponsored studies, studies initiated by a private sponsor experienced fewer difficulties with the research question, study design, obtaining funding, data management, data analysis and interpretation, manuscript writing, and publication in a peer-reviewed journal. Similar results were observed when multi center studies were compared to single-center studies. Publicly sponsored studies and single center studies experienced fewer difficulties concerning the approval by safety agencies, however more difficulty with getting approval from the REC.

Furthermore, studies informing clinical questions reported more difficulties with approval by safety agencies or with patient enrollment. Non-clinical studies reported more difficulties with the formulation of the research question.

This study has several limitations. First, it was carried out at a single Swiss university hospital and we do not know to what extent the findings apply to other settings. Second, we only surveyed PIs of completed clinical studies, which may have decreased the apparent prevalence of major difficulties, because abandoned clinical studies [25, 29] may have experienced more difficulties. Third, PIs’ answers may have been affected by various recall biases [38]. Social desirability may also have influenced responses, considering that the investigators in charge of the present study belong to the same institution as the surveyed PIs. Fourth, the scale of difficulties (and the two subscales, logistic and scientific difficulties) was built for the purpose of this study, based on literature review and expert consensus. We have no evidence of their validity beyond face validity, though their reliability was acceptable (Cronbach alpha between 0.73 and 0.83). Moreover, among the 14 items composing the scale, four items (getting funding, data collection, recruitment of competent staff to conduct the study, and data management) shared variance on both subscales (see Additional file 1: Appendix 1). Fifth, the analysis of studies with no questionnaire suggested some evidence for non-response bias: undocumented studies were more frequently interventional studies, with industry sponsorship, and unpublished.