Article Text
PDF
Abstract
Objectives Patient-reported outcome measures (PROMs) collected in clinical trials should be administered in a standardised way across sites and routinely screened for avoidable missing data in order to maximise data quality/minimise risk of bias. Recent qualitative findings, however, have raised concerns about the consistency of PROM administration in UK trials. The purpose of this study was to determine the generalisability of these findings across the wider community of trial personnel.
Design Online cross-sectional survey.
Setting Participants were recruited from 55 UK Clinical Research Collaboration Registered Clinical Trials Units and 19 Comprehensive Local Research Networks.
Participants Research nurses, data managers/coordinators, trial managers and chief/principal investigators involved in clinical trials collecting PROMs.
Analysis We undertook descriptive analyses of the quantitative data and directed thematic analysis of free-text comments. Factors associated with the management of missing PRO data were explored using logistic regression.
Results Survey data from 767 respondents supported the generalisability of qualitative study findings, suggesting inconsistencies in PROM administration with regard to: the level of assistance given to trial participants; the timing of PROM completion in relation to the clinical consultation; and the management of missing data. Having ≥10 years experience in a research role was significantly associated with the appropriate management of missing PROM data (OR 2.26 (95% CI 1.06 to 4.82), p=0.035). There was a consensus that more PROM guidance was needed in future trials and agreement between professional groups about the necessary components.
Conclusions There are inconsistencies in the way PROMs are administered by trial staff. Such inconsistencies may reduce the quality of data and have the potential to introduce bias. There is a need for improved guidance in future trials that support trial personnel in conducting optimal PROM data collection to inform patient care.
- Survey
- patient-reported outcomes
- PROs
- patient-reported outcome measures
- PROMs
This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: http://creativecommons.org/licenses/by/4.0/
Statistics from Altmetric.com
Strengths and limitations of this study
This study is the first to survey the opinions of researchers and trial personnel regarding the administration of patient-reported outcome measures in UK clinical trials.
The large research nurse sample size should allow generalisation of the results in this population. Owing to their much smaller sample sizes, caution should be exercised when generalising the results from the remaining subgroups.
Respondents were self-selecting and may include those with more knowledge regarding patient-reported outcomes (PROs); this should be taken into account when interpreting the results of the study.
As the survey was anonymised, it was not possible to link staff together on a particular study. Thus, further work is needed to definitively establish whether the PRO administration variability seen in this survey may be present in a single trial.
Introduction
Patient-reported outcomes (PROs) are commonly measured in clinical trials in order to evaluate the effectiveness of medical interventions from the point of view of patients.1 PRO trial data are collected using validated questionnaires, known as patient-reported outcome measures (PROMs). PROs trial results can inform the healthcare decisions made by patients and their clinicians, support licensing claims for new medicines and influence the development of health policy, including decisions about cost-effectiveness.2–4 In view of their importance, there is a need to ensure rigorous PRO data collection.
Unfortunately, missing PRO data can be a problem in clinical trials. In a 2008 review of 285 randomised controlled trials (RCTs) collecting PROs, Fielding et al5 found that 18% reported between 11% and 20% missing data and a further 18% reported >20% missing data. Such data are generally considered ‘not missing at random’, rather they may be missing from those participants with the poorest outcomes, rendering it non-ignorable.6 Retrospective PRO data capture is frequently not possible; therefore, missing data of this type can result in bias if such participants are concentrated in a particular trial arm.6 ,7
Trials should therefore be designed to ensure that PROMs are administered in a standardised way across trial sites and are routinely screened for avoidable missing data in order to maximise data quality and reduce the risk of systematic bias.6 ,8–10 PROM administration guidance should, therefore, be included in the trial protocol and in site start-up training, and may also be incorporated into supporting trial documentation such as standard operating procedures (SOPs).7 ,11 ,12 Recent qualitative evidence, however, has raised concerns about the conduct of PRO measurement in UK trials.13 The study, conducted by the authors, outlined three main findings.13 First, there were reported inconsistencies in the way in which PROMs were administered that could adversely affect the quality of PRO trial data and potentially bias results. Second, there was a reported lack of PRO-specific protocol content, training and education available to trial staff. Third, data collection staff reported being intermittently exposed to PROM data that caused them to become concerned for the well-being of a trial participant (also known as a ‘PRO alert’,14 box 1) and, in the absence of trial level guidance, reported providing off-protocol cointerventions. Some of these interventions appeared to risk biasing the results of the trial.
Definitions
Patient-reported outcome (PRO)—‘… any report of the status of a patient's health condition that comes directly from the patient, without interpretation of the patient's response by a clinician or anyone else’4
Patient-reported outcome measure (PROM)—A validated paper-based or electronic psychometric questionnaire used to collect PRO data
PRO alert—The exposure of data collection staff to PRO data displaying ‘concerning levels of psychological distress or physical symptoms that may require an immediate response’14
The aim of this study was to determine the extent to which our qualitative findings were generalisable to the wider community of trial staff using a large-scale cross-sectional national survey of UK-based trial personnel. Survey respondents’ experiences of PRO alerts and their management are presented in a separate publication.15 In this paper, we present the results of the survey specific to PROM administration, with the following objectives:
To investigate reported inconsistencies in PROM administration in trials.
To investigate a reported lack of PRO-specific trial protocol content and training.
To determine what PRO-specific trial protocol content and training respondents would like to see in future trials.
Methods
Survey design
An online survey (see online supplementary file 1) was developed by investigators with PRO and ethics expertise, and the content informed by the results of our qualitative study.13 The survey contained questions on: (1) demographics, (2) the participants’ experiences of PROM administration with reference to the most recent trial in which they had been involved, (3) the provision of PRO-specific guidance within the trial and (4) PRO guidance/training they would like to see in future trials. Most questions contained space for free-text comments, to allow respondents to expand on their answers. The survey was pilot-tested in a purposive sample of n=9 research nurses to ensure the content was clearly understood and to establish the feasibility of the distribution/collection methods. Additional free-text comment boxes were added to the survey instruments following pilot feedback. No other changes to the survey questions were necessary.
supplementary file
Survey sample
The study recruitment methods have been reported elsewhere.15 In brief, an anonymised online exploratory cross-sectional national survey of UK research nurses, data managers/coordinators, trial managers and chief and principal investigators (CPIs) involved in clinical trials using either a primary or secondary PRO was conducted in 2013/2014. The survey was anonymised to encourage respondents to freely discuss potentially controversial aspects surrounding PRO assessment/PRO alerts and to maximise responses. An email containing information about the study, and a link to the online survey, was distributed via 55 UK Clinical Research Collaboration Registered Clinical Trials Units (CRC-RCTUs) and 19 Comprehensive Local Research Networks (CLRNs). Eligible individuals were invited to click on the link and complete the survey.
Analysis
Descriptive quantitative analysis was undertaken for each respondent group. Frequency distributions were used to describe respondent characteristics and survey responses. All analysis was conducted using SPSS (V.21, IBM). An exploratory prespecified logistic regression analysis was also undertaken to investigate which factors were associated with the appropriate management of missing PRO data by data collection staff, as this can represent an important potential source of bias. Existing literature recommends routine checking of completed PROMs and subsequent ‘chasing’ of missing data.16 Thus, the dependent variable in the model was the appropriate management of missing data, defined as: ‘whether the completed PROM was checked for missing data and participants were subsequently asked to complete missing items/questionnaires’. The independent variables were: the role of the data collector (ie, ‘research nurse’ or ‘data manager/coordinator’); their length of experience in the research role; whether PRO-specific information was reportedly present in the trial protocol; and whether PRO-specific information was reportedly included in trial training. A minimum of 60 responses were required to satisfy the sample size requirement for this regression analysis (15 per covariate).17 Significance was set at p<0.05.
DK undertook directed content analysis of the free-text comments responses, using the data from the qualitative study13 to develop the initial research questions and coding framework.18 Additional codes were developed as the analysis was conducted and the framework was modified accordingly.18 JI formally reviewed all coding to enhance trustworthiness, and any disagreements about coding were discussed and resolved.
Results
A total of 767 individuals responded to the online survey (table 1). The respondents’ most recent experience of a trial collecting PROMs was predominantly in the secondary care setting, with trials ranging across clinical specialities (most commonly oncology). Trials appeared to use a number of different PROMs, of which the most common were the five-dimension European Quality of Life instrument (EQ-5D), Hospital Anxiety and Depression scale (HADS), the Short-Form Health Survey 12-item (SF-12) and 36-item (SF-36) questionnaires, the European Organisation for Research and Treatment of Cancer Core Quality of Life Questionnaire (EORTC QLQ-C30) and the Health Assessment Questionnaire (HAQ). The survey results are presented in table 2 and summarised below. Qualitative themes and illustrative respondent quotations are also presented. A response rate calculation is not appropriate for the study as ours was a non-probability sample.19 Moreover, neither the UK CRC-CTUs nor the NIHR CLRNs held data regarding the number of staff involved in trials with a primary or secondary PRO, so there was no way to determine a denominator. However, in discussion with the CRC-CTU coordinating centre and with CLRN contacts, we estimate the total number of researchers receiving our survey invite, including those individuals ineligible for the study, was ∼1800; our sample represented 43% of this number.
Characteristics of respondents
Questionnaire responses
supplementary file
Inconsistencies in PROM administration
Participant assistance
Respondents were asked what level of assistance they had given to participants completing PROMs during their most recent relevant trial. 66.2% of research nurse respondents reported giving no assistance. The remainder gave assistance in a variety of different ways. Of these, 37.9% reported helping participants to understand the questions, 36.9% reported reading the PROM questions out to the participants and 23.0% reported being given the answers by participants then filling in the questionnaire on their behalf.
Timing of PROM completion
There were varying responses with regard to the timing of questionnaire completion. 47.9% of nurses reported that the timing with which they administered the PROM varied (ie, it was ‘sometimes before’ and ‘sometimes after’ the clinical consultation) during their most recent trial. 9.3% reported routinely administering the PROM after the consultation. Only 18.2% reported routinely administering PROMs prior to the participant's clinical consultation, inline with suggested guidelines.20 ,21
Management of missing PRO data
77.3% of research nurses and 72.0% of data managers reported routinely checking completed PROM questionnaires for missing data in their most recent trial. However, of the total sample, only 49.9% research nurses and 15.4% of data managers checked for missing data and subsequently attempted to follow-up participants to complete the missing items. 27.6% of research nurses and 76.0% of data mangers reported checking PROM question responses for scoring errors (eg, where two options were selected instead of one), which are often logged as missing data. Of the total sample, just 21.2% research nurses and 9.8% of data managers reported checking for scoring errors and subsequently attempting to follow-up participants in order to correct the errors.
Determinants of differences in the management of missing data
The findings of the exploratory logistic regression analysis are available in online supplementary file 3. In the final model, only ‘10 years or more experience in the research role’ was significant (p=0.035). The odds of individuals with such experience routinely checking and chasing missing PROM data were 2.26 (95% CI 1.06 to 4.82) times higher than those with less experience. There were no significant associations between the dependent variable and ‘the role of the data collector’ (p=0.45); ‘whether PRO-specific information was reportedly provided in the trial protocol’ (p=0.94); or ‘whether PRO-specific information was reportedly included in trial training’ (p=0.64).
supplementary file
Current PRO-specific protocol content and training
Survey respondents were questioned about the PRO-specific protocol content and training delivered in their most recent trial. Results are presented first from members of the trial management team (CPIs and trial managers), and then from frontline data collection staff (research nurses and data managers/coordinators).
Protocol content and training provision
82.9% of CPIs and 70.9% of trial managers reported giving instructions to trial staff on how to administer the PROM questionnaire in their most recent relevant trial. Respondents were asked what particular information was provided to data collection staff. 93.1% of CPIs and 86.6% of trial managers reported providing information about the purpose and/or importance of PROM data to the trial and 75.9% and 52.4%, respectively, reported giving information surrounding the relevance and reasoning behind individual PROM questions. One hundred per cent of respondents in both groups reported providing information on when to administer the PROM questionnaire. Over three-quarters of CPIs reported providing information on when to administer the PROM during a clinic (78.6%); how much assistance to give the participant during questionnaire completion (86.2%); and how to check for, and deal with, missing PRO data (82.8%). The proportion of trial managers who reported providing this information was uniformly lower: 67.9%, 63.4% and 58.5%, respectively. 41.4% of CPIs and 28.0% of trial managers reported providing information on what action should be taken if participants had written additional information on their PROM, or had attached further information (such as letter to the trial team) to their PROM.
Protocol content and training available to frontline staff
92.2% of research nurses and 50.0% of data managers/coordinators reported that the trial protocol had included some form of PRO-specific information, with 87.7% and 76.9%, respectively, reporting that the content was adequate for their needs. 32.7% of research nurses and 39.1% of data managers/coordinators reported receiving trial training that incorporated PRO guidance. 94.4% of research nurses and 88.9% of data managers/coordinators who reported receiving PRO training felt it was adequate for their needs. 60.5% of research nurses reported they had received an explanation of why the PROM was being collected in the trial and 87.7% felt confident they could explain this to participants. 30.3% of research nurses reported receiving an explanation regarding the relevance and reasoning behind individual PROM questions and 59.9% felt confident they could explain this to their participants.
Free-text comments relating to trial protocol content and training
There were 40 free-text comments in this section, with most appearing to suggest that, while nurses felt PRO-specific information was generally present within trial protocols, it could be limited in depth:Usually there is reference to the fact that the questionnaires are to be completed. No other information is provided or instructions on use, administration etc. (Research nurse)
A number of comments suggested that nursing staff had received little in the way of PRO training:I was not given any training on PROM basically just been told if patients consent for the study they fill this document. (Research nurse)
Some research nurses suggested PROM training was particularly lacking for staff joining the trial at a later stage:Our centre was invited to take part in the study quite late on so missed the initial set up that other centres had. Whilst there was some verbal communication regarding how to deliver the questionnaires much of it was down to previous experience/personal communication skills…. (Research nurse)
I took over the study partway through and received minimal instruction relating to the questionnaires. Anything additional I learn en route. (Research nurse)
Some reported that PRO trial training was inconsistently delivered across trials:I have been taught how to use it many times but not for this study. However, to ensure consistency I believe we should be trained on this for each study. (Research nurse)
A number of comments implied that the impact of a lack of guidance may be minimal. These nurses reported either relying on either previous experience of PRO assessment in trials, or knowledge gained via attendance at previous training courses, or an independent search for the information they required:I have used QOL questionnaires a fair amount so felt confident using the provided tools without needing training. (Research nurse)
No specific training given by the study centre for this study, but I have completed training on many of the QoL [PROMs] previously. (Research nurse)
The trial training said that the questionnaires had to be done by the patient, but did not give any reasons why. I did my own reading to find out why this was the case. (Research nurse)
Conversely, one respondent comment suggested a lack of PROM guidance resulted in an impaired ability to explain aspects surrounding PROM assessment to trial participants:I can roughly explain to participants why this information is required, but would prefer to have a better understanding myself to be able to fully explain this to study participants. (Research nurse)
and one respondent comment suggested it led to more queries to the trial team:The trial coordinator had to be contacted quite often for clarification as subject asked questions that was not covered in the training session. (Research nurse)
Future PRO-specific protocol content and training
Survey respondents were asked whether they felt more PRO guidance was needed in future trials. While 85.1% of research nurses and 78.6% of data managers/coordinators ‘strongly agreed’ or ‘agreed’, there should be more PRO guidance provided in future trials with PRO end points, in contrast, only 58.2% of trial managers and 56.5% of CPIs felt the same.
After considering a list of possibilities suggested by the findings of our qualitative study, respondents were also asked which particular PRO-specific items of information they felt were needed and where should they be provided: in the trial protocol, in trial training or in supporting trial documentation such as SOPs. In general, all groups supported the inclusion of the majority of proposed PRO guidance within trial training and/or a SOP, but there was less support for including items within the protocol. In order to highlight where there was agreement on the necessary components of PRO guidance, items that were selected by more than 50% of respondents in a professional group are presented in table 3.
Future patient-reported outcome guidance provision
Free-text comments regarding future PRO guidance
There were 22 free-text comments in this section. These most commonly suggested that trial protocols should signpost sources of PRO information, rather than include the information themselves:I think that mention of some things within the Protocol could be quite short e.g. ‘how to deal with difficult situations will be covered in trial training and in SOP xxx date yyy’. (Research nurse)
Some comments indicated PRO training should be conducted outside of the trial:some of this could be generic to many trials so may be able to train as ‘general training’ via R&D depts rather than trial specific training via CTUs. (Research nurse)
Two research nurses each suggested one additional element of PRO guidance, but the optimal location was not specified:whether the questionnaire should always be answered by the individual or whether it can be used by family/friends on the patient's behalf. (Research nurse)
how to answer when a question is ambiguous or the [information] given does not fit in with the suggested answer. (Research nurse)
Finally, one research nurse highlighted the importance of including PROM guidance in the participant information:You haven't asked about putting this into the [participant] info sheets which is very important. The patients need to know exactly what is required of them…. (Research nurse)
Discussion
Principal findings
The survey findings support the generalisability of our previous qualitative evidence,13 suggesting there are inconsistencies in the way PROMs are administered by trial personnel, with regard to: the level of assistance given to participants during PROM completion; the timing with which PROMs are completed in relation to the clinical consultation; and the way missing PRO data are monitored and acted on.
This variability in PRO administration practice is problematic on two fronts. Where it exists between trials, it may lessen the confidence with which different PRO trial results may be compared by key stakeholders, including: patients, clinicians, regulatory authorities and policymakers. Where variability is present in a single trial, however, it raises a number of concerns. First, marked differences reported in the level of assistance given to trial participants during PRO assessment may result in measurement variability within the study, reducing the quality of the trial data. In addition, increased assistance given to some participants could lead to response bias.22 Second, the practice of administering PROM questionnaires after a clinical consultation may lead to PRO data contamination, as, if a participant receives bad news or undergoes an invasive procedure, this may colour their questionnaire responses.20 Our data suggest that the timing of questionnaire delivery may not be consistent between individual trial staff; therefore, it may not be possible to compensate in the analyses for this potential confounder. Third, variation seen in the management of missing PRO data risks introducing bias, as data are more likely to be missing from those participants in a trial with the poorest outcomes,6–8 ,10 who may be concentrated in a particular arm of the trial, for example, if one intervention in the study results in greater levels of side effects or toxicity.23 In our sample, over one-fifth of research nurses and data manager respondents reported that they did not check completed PROMs for missing data. In addition, only 50% of research nurses and 15% of data managers/coordinators who did report checking, subsequently followed up with participants to ensure the missing items or questionnaires were completed. It is concerning that a sizeable proportion of staff did not routinely check for missing PRO data; however, the low rates of follow-up across all data collection personnel are equally worrying: there is little point in monitoring missing data if nothing is performed to rectify the situation. These findings suggest that a formal procedure needs to be in place for monitoring and responding to missing PRO items/questionnaires in trials. Communicating this procedure to all frontline staff may help prevent the relatively high rates of missing PRO data seen in some studies.5 Failure to standardise PRO assessment methods and minimise avoidable missing data reduces data quality, misuses valuable participant time and research resources, risks the introduction of bias and ultimately devalues these important patient-centred outcomes, undermining their usefulness in informing policy, regulatory decisions and patient care, resulting in ‘research waste’.6 ,24
Our previous qualitative data suggested a lack of PRO-specific guidance in the trial protocol and in the form of trial training.13 The survey findings revealed mixed opinions in this area. Over two-thirds of trial management respondents reported giving instructions to data collection staff on how to administer the PROM questionnaire. Research nurses concurred that PRO information was commonly present in the protocol, whereas only half of data managers agreed. The research nurse reports are, however, at odds with other available evidence. A recent review of the PRO content of trial protocols25 found that instructions on the PRO rationale/hypothesis, data collection methods, training and management were frequently absent from the protocols, even when the PRO was the primary outcome. Research nurse free-text comments in the survey appeared to align with these findings, suggesting that PRO protocol content could often consist of little more than a statement outlining that a PROM would be collected.
There was, however, some agreement between staff groups regarding levels of PRO training. Less than one-third of research nurses and less than two-fifths of data managers reported receiving trial training that incorporated PRO guidance. Moreover, some staff reported PRO-specific training provision was inconsistent across trials and that training was not generally accessible for staff entering a trial once it was underway.
Some research nurses in our survey questioned the usefulness of PRO-specific trial guidance, appearing to rely on their experience and judgement instead. Also, our regression model indicated that staff with greater experience (≥10 years) tended to report dealing more appropriately with missing PRO data, presumably because they had been involved in a greater number of relevant trials, but that trial protocol content or training were not significant predictors. It is possible, however, that the protocol content and training received by the respondents did not contain adequate information on the management of missing data. A review of PRO protocol content25 reported that under half of the n=75 included protocols detailed plans to minimise levels of avoidable missing PRO data.
When asked about future trials, an overwhelming majority of research nurse and data manager respondents reported that more PRO information should be provided in trials. It would appear, however, that respondents did not feel additional PRO-specific content in protocols was useful and that other mediums of information transfer, including trial training and SOPs, might be more appropriate. Views differed regarding the exact PRO information that should be provided to each professional group, indicating that each had different needs. This should be taken into account during the design of future trials collecting PROMs.
The findings of this study suggest there is a need to develop general PRO guidance for trialists, which highlight the key issues that should be considered by researchers involved in trial design and implementation. The study team are currently working to produce such guidelines in partnership with the International Society for Quality of Life Research ‘Best Practices for PROs in Randomised Clinical Trials’ taskforce.
Strengths and limitations of the study
This study is the first to survey the opinions of researchers and trial personnel regarding the administration of PROs in trials. Our large research nurse sample size should allow reasonably confident generalisation of the results in this population. Owing to their much smaller sample sizes, caution should be exercised when generalising the results from the remaining subgroups. Respondents were self-selecting and may include those with more knowledge regarding PROs; this should be taken into account when interpreting the results of the study. As the survey was anonymised, it was not possible to link staff together within a particular study or study centre. We were therefore unable to determine to what extent the broader population of UK study centres was represented in the survey. In addition, further work is needed to definitively establish whether the PRO administration variability seen in this survey may be present in a single trial.
Conclusions
The findings of this large-scale survey of clinical trial staff suggest there are inconsistencies in the way PROMs are administered by trial staff, which may reduce the quality of PRO data and have the potential to introduce bias. There was general agreement among respondents that the provision of PRO guidance in future trials should be improved, with the majority of information included in trial training and SOPs, and signposted in the trial protocol.
References
Footnotes
Contributors The study was conceived by DK, JI, HD and MC. DK conducted the survey and completed the analysis, with input and supervision from MC, HD and JI. DK prepared the first draft of the manuscript. JI, HD and MC all provided edits and critiqued the manuscript for intellectual content.
Funding The study was supported by a National Institute for Health Research (NIHR) School for Primary Care Research (SPCR) studentship grant. No other funding sources reported.
Disclaimer The funders had no role in study design, data collection and analysis, decision to publish or preparation of the manuscript.
Competing interests None declared.
Ethics approval The study was approved by the West Midlands Research Ethics Committee in April 2012 (ref no 12/wm/0068).
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement No additional data are available.