Challenges of one-year longitudinal follow-up of a prospective, observational cohort study using an anonymised database: recommendations for trainee research collaboratives
verfasst von:
Dmitri Nepogodiev, STARSurg Collaborative
Trainee research collaboratives (TRCs) have pioneered high quality, prospective ‘snap-shot’ surgical cohort studies in the UK. Outcomes After Kidney injury in Surgery (OAKS) was the first TRC cohort study to attempt to collect one-year follow-up data. The aims of this study were to evaluate one-year follow-up and data completion rates, and to identify factors associated with improved follow-up rates.
Methods
In this multicentre study, patients undergoing major gastrointestinal surgery were prospectively identified and followed up at one-year following surgery for six clinical outcomes. The primary outcome for this report was the follow-up rate for mortality at 1 year. The secondary outcome was the data completeness rate in those patients who were followed-up. An electronic survey was disseminated to investigators to identify strategies associated with improved follow-up.
Results
Of the 173 centres that collected baseline data, 126 centres registered to participate in one-year follow-up. Overall 62.3% (3482/5585) of patients were followed-up at 1 year; in centres registered to collect one-year outcomes, the follow-up rate was 82.6% (3482/4213). There were no differences in sex, comorbidity, operative urgency, or 7-day postoperative AKI rate between patients who were lost to follow-up and those who were successfully followed-up. In centres registered to collect one-year follow-up outcomes, overall data completeness was 83.1%, with 57.9% (73/126) of centres having ≥95% data completeness. Factors associated with increased likelihood of achieving ≥95% data completeness were total number of patients to be followed-up (77.4% in centres with < 15 patients, 59.0% with 15–29 patients, 51.4% with 30–59 patients, and 36.8% with > 60 patients, p = 0.030), and central versus local storage of patient identifiers (72.5% vs 48.0%, respectively, p = 0.006).
Conclusions
TRC methodology can be used to follow-up patients identified in prospective cohort studies at one-year. Follow-up rates are maximized by central storage of patient identifiers.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Abkürzungen
AKI
Acute Kidney Injury
ASA
American Society of Anaesthesiologists
CHI
Community Health Index
OAKS
Outcomes After Kidney injury in Surgery
RCRI
Revised Cardiac Risk Index
REDCap
Research Electronic Data Capture
STARSurg
Student Audit and Research in Surgery
TRCs
Trainee research collaboratives
Background
Trainee research collaboratives (TRCs) have pioneered methods of rapidly delivering high quality, prospective, cross-sectional ‘snapshots’ of surgical practice and outcomes [1, 2]. TRC studies are led by frontline clinicians and students, without the need for significant additional infrastructural resources or funding. They capture data over short time periods across multiple centres, collating large datasets [3‐5] that can be used to generate hypotheses for future randomised trials and identify targets for national quality improvement [6‐10].
TRC studies are delivered by student and postgraduate trainees who rotate between hospitals at least once every 12 months, which would create discontinuity within local teams if studies were run over protracted periods of time. Consequently, most surgical TRC studies follow patients up to the point of discharge or to postoperative day 30; no published observational studies from TRCs have undertaken outcome assessment beyond 6 months [6‐10]. In planning longer term follow-up, a particular challenge is ensuring safe local storage of patient identifiers so that patients can be followed-up at one-year even if the original study collaborators at that site have rotated to continue their training at another centre.
Anzeige
Outcomes After Kidney injury in Surgery (OAKS) was the first TRC cohort study to attempt to collect to one-year follow-up data. The aim of this study was to evaluate one-year follow-up and data completion rates, and to identify factors associated with improved follow-up rates.
Methods
Student Audit and Research in Surgery (STARSurg)
Student Audit and Research in Surgery (STARSurg) is the UK’s national medical student research collaborative. It is coordinated by a team of medical students and postgraduate trainees. The collaborative model and the educational benefits to participating students have been described previously [11, 12]. STARSurg studies are delivered by ‘mini-teams’ at each centre consisting of consultant surgeons, junior doctors and medical students.
Outcomes After Kidney injury in Surgery
Outcomes After Kidney injury in Surgery (OAKS) [13] is a multicentre study which prospectively identified patients in the UK and Republic of Ireland undergoing major gastrointestinal or liver resection, or reversal of ileostomy or colostomy from 23 September 2015 to 18 November 2015. In the United Kingdom, the South-East Scotland Research Ethics Service (reference: NR/1506AB4) confirmed that ethical review was not required, as this observational study only collected routine, non- patient identifiable data. Individual participating UK centres were responsible for registering the study locally as either clinical audit or service evaluation. In the Republic of Ireland, participating centres were responsible for securing research ethics approval locally, as required by institutional regulations. The 30-day outcomes from the OAKS study have been reported previously [14, 15].
Data was collected on the Research Electronic Data Capture (REDCap) system, an online platform for secure web-based data collection. The REDCap platform was developed in 2004 at Vanderbilt University, which is a secure data collection tool meeting the Health Insurance Portability and Accountability Act (HIPAA) compliance standards. Patients’ hospital or NHS identification numbers and linked study-specific identification numbers were stored in accordance with local Caldicott Guardian approvals; either centrally on the REDCap system or within an encrypted spreadsheet held securely on the local hospital computer network by a member of the data collection team (a local investigator, supervising consultant, or audit officer).
Anzeige
In the period November 2016 to May 2017 the STARSurg network collected one-year outcomes for the patients identified in the initial prospective patient enrolment phase of OAKS. Patients were excluded from one-year follow-up if they had died within 30 days of index surgery, as there would be no additional data to collect from these patients since the 30-day follow-up that had already been completed previously. At centres that had participated in initial OAKS data collection, new mini-teams were recruited to complete one-year follow-up. The clinical endpoints collected at one-year were (1) mortality at 1-year, (2) myocardial infarction or cerebrovascular accident at 1-year, (3) total combined hospital length of stay up to 1-year postoperatively, (4) the most recent available serum creatinine value up to 1-year, (5) nephrology review at 1-year, and (6) dialysis at 1-year. These clinical endpoints based on a review of the literature on postoperative AKI [16‐19]. In this observational study, clinic follow-up visits and blood tests were arranged by clinical teams according to their normal practice. No additional follow-up visits or blood tests were arranged for this study. Follow-up was considered to have been achieved if patients’ records had been successfully reviewed, even if no creatinine tests had been completed by the clinical team during the follow-up period.
Centres were considered to have registered for collection of one-year follow-up if a data collection mini-team was established at the site, institutional approval was granted for collected of follow-up data, and at least one patient was followed-up at the site.
Outcome measures
The primary outcome for this report was the mortality follow-up rate for mortality. This was the proportion of patients for whom the primary endpoint (mortality) was followed-up at 1-year. The secondary outcome was the data completeness rate in centres that registered to collect one-year follow-up. The data completion rate was the proportion of patients with complete data for all six clinical endpoints.
Investigator feedback survey
Following locking of the OAKS database, an electronic survey was disseminated to all investigators who had participated in one-year follow-up (Additional file 1: Table S1). This assessed investigators’ experience of one-year follow-up data collection. 5-point Likert scales were used to assess investigators’ experience of the following (from 1 = very difficult, to 5 = very easy): identifying a supervising consultant; registering the audit; linking patient hospital identifier to the study-specific identifier; collecting data using local hospital computer systems, or paper records. For analysis, scores of 4 to 5 out of 5 were categorised as “Positive”, and scores of 1 to 3 out of 5 were categorised as “Negative” responses to create a dichotomous variable.
Statistical analysis
The baseline characteristics of patients lost to follow-up were compared to those patients who were successfully followed-up. Continuous variables were expressed as mean with standard deviation, or median with interquartile range. Continuous variables were analysed using t-test or Mann-Whitney test, where appropriate. Categorical variables were expressed as percentages and analysed using Chi-squared test, or with Fisher’s exact modification if expected cell counts were less than five. For all analyses, a p-value of < 0.05 was considered as statistically significant. Data analysis was undertaken using R Foundation Statistical Software (R 3.2.1, R Foundation for Statistical Computing, Vienna, Austria).
Results
Centre registration
Of 173 centres that had collected baseline data in the initial phase of OAKS, 126 centres registered to participate in one-year follow-up. Of the 47 centres that did not register, 35 were unable to obtain patient ID link sheets, and 12 were not granted audit and/or Caldicott Guardian approval prior to the data collection deadline (Fig. 1). Centres in Scotland, Ireland, and England all achieved similar levels of registration to participate in one-year follow-up (88.9% vs 78.6% vs 72.5% respectively, Table 2), however there were significantly fewer centres in Wales (40.0%). Centres at which a junior doctor was engaged in the process were more likely to register to enter data (80.6% vs 46.2%, p < 0.001). Centres which had stored patient hospital identifiers on the central REDCap system during the initial data collection phase had a significantly higher participation rate in one-year follow-up (83.6% vs 67.0%, p = 0.019). However, prior centre participation in STARSurg projects preceding OAKS did not affect the likelihood of centres registering to collect one-year follow-up (74.8% vs 61.5%, p = 0.160).
×
Follow-up rates
The initial data collection phase of OAKS captured 5745 patients, of whom 5585 remained alive at 30-days following their index procedure (Fig. 1) and eligible for one-year follow-up. Overall 62.3% (3482/5585) of patients were followed-up at 1 year. Of the 2103 patients lost to follow-up, 65.2% (1372) were from the 47 centres that did not register to participate in one-year follow-up. In registered centres, the follow-up rate was 82.6% (3482/4213).
Characteristics of patients followed-up at one-year
There were no significant differences in age, American Society of Anaesthesiologists (ASA) grade, Revised Cardiac Risk Index (RCRI), urgency of surgery and contamination between patients with and without follow-up at one-year (Table 1). However, patients who were followed-up had significantly higher rates of open surgery compared to patients that were not followed-up (61.5% vs 52.0%, p < 0.001). There were no significant difference in rates of AKI (12.7% vs 10.7%, p = 0.060) between patients with and without one-year follow-up.
Table 1
Characteristics of patients with one-year follow-up completed
No follow-up
(n = 2104)
Follow-up
(n = 3481)
p-value
Age (years)
Mean (SD)
62.8 (15.9)
62.6 (16)
0.689
Gender
Female
1561 (44.8)
967 (46.0)
0.417
Male
1920 (55.2)
1137 (54.0)
ASA Grade
ASA I-II
2176 (67.0)
1312 (68.5)
0.279
ASA III-V
1071 (33.0)
604 (31.5)
Operative urgency
Elective
2752 (79.1)
1687 (80.2)
0.314
Emergency
729 (20.9)
417 (19.8)
RCRI score
< 3
3328 (95.7)
2014 (96.0)
0.491
≥3
151 (4.3)
83 (4.0)
Operative approach
Laparoscopic
1339 (38.5)
1006 (48.0)
< 0.001
Open/ laparoscopic converted to open
2136 (61.5)
1090 (52.0)
Operative contamination
Clean-contaminated
3234 (93.2)
1928 (91.9)
0.065
Contaminated
235 (6.8)
170 (8.1)
AKI within 7 days of index surgery
No AKI
2979 (85.6)
1837 (87.3)
0.060
AKI
443 (12.7)
225 (10.7)
Missing
59 (1.7)
42 (2.0)
AKI Acute Kidney Injury, ASA American Society of Anaesthesiologists, RCRI Revised Cardiac Risk Index
Anzeige
Data completeness
Centre characteristics associated with ≥95% completeness are presented in Table 2. In centres registered to collect 1-year follow-up outcomes, overall data completeness was 83.1%. Of the 126 of centres that participated, 57.9% (n = 73) had ≥95% data completeness. Scotland had significantly more centres with ≥95% data completeness (100.0%) compared to England, Ireland, and Wales (55.8% vs 36.4% vs 0%, p < 0.001). The more patients a centre had to follow-up, the less likely it was to achieve ≥95% data completeness (< 15: 77.4% vs 15–29: 59.0% vs 30–59: 51.4 vs > 60: 36.8%). Centres storing patient identifiers on the central REDCap system had significantly higher rates of ≥95% data completeness than those storing identifiers locally (72.5% vs 48.0%, respectively, p < 0.001).
Table 2
OAKS centre characteristics, centre activity and data completeness at one-year postoperatively
Centre active in OAKS
Centre ≥95% completeness in OAKS
Active (n = 126)
Inactive (n = 47)
p-value
Yes (n = 73)
No (n = 53)
p-value
UK Countries
England
95 (72.5)
36 (27.5)
0.045
53 (55.8)
42 (44.2)
< 0.001
Ireland
11 (78.6)
3 (21.4)
4 (36.4)
7 (63.6)
Scotland
16 (88.9)
2 (11.1)
16 (100.0)
0 (0.0)
Wales
4 (40.0)
6 (60.0)
0 (0.0)
4 (100.0)
Total number of patients to be followed up
< 15 patients
31 (70.5)
13 (29.5)
0.491
24 (77.4)
7 (22.6)
0.030
15–29 patients
39 (72.2)
15 (27.8)
23 (59.0)
16 (41.0)
30–59 patients
37 (69.8)
16 (30.2)
19 (51.4)
18 (48.6)
> 60 patients
19 (86.4)
3 (13.6)
7 (36.8)
12 (63.2)
Percentage of patients with complete follow-up
Mean (SD)
–
–
–
28.3 (21.6)
27.9 (21.9)
0.877
Junior doctor present in OAKS mini-teama
Yes
108 (80.6)
26 (19.4)
< 0.001
63 (58.3)
45 (41.7)
0.825
No
18 (46.2)
21 (53.8)
10 (55.6)
8 (44.4)
Central storage of patient hospital identifiers
Yes
51 (83.6)
10 (16.4)
0.019
37 (72.5)
14 (27.5)
0.006
No
75 (67.0)
37 (33.0)
36 (48.0)
39 (52.0)
Survey respondents
Yes
125 (84.5)
23 (15.5)
< 0.001
72 (57.6)
53 (42.4)
0.392
No
1 (4.0)
24 (96.0)
1 (100.0)
0 (0.0)
aJunior doctors are present in each mini-team over a data collection period
Investigator feedback survey
Survey responses were received from 285 students and junior doctors, a 78% (285/365) response rate. At least one response was received from 86% (148/173) of centres that participated in initial data collection in 2015. Of centres that returned the survey, 59 (40.0%) had 100% data completion of one-year follow-up, 72 (48.6%) had ≥95% data completion and 23 (15.5%) did not register to submit one-year follow-up data. Table 3 summarises respondent characteristics and experience of OAKS-2 by data completeness. Only collaborators with positive experience of linking patient ID were more likely to achieve > 95% data completeness (71.6% vs 37.6%, p < 0.001). There was no association between perceived difficulty with registering audit, data collection, and identifying a supervising consultant, and > 95% data completeness. Following this study, a summary of recommendations for future multi-centre collaborative studies with longitudinal follow-up were developed and presented in Table 4.
Table 3
OAKS collaborator survey responses, centre activity and data completeness at one-year postoperatively
Respondent at a centre with ≥95% completeness
Yes (n = 141)
No (n = 111)
p-value
Stage of Training
Junior Doctor
47 (56.6)
36 (43.4)
0.929
Later Year Student
52 (56.5)
40 (43.5)
Early Year Student
41 (53.9)
35 (46.1)
Previous participation in initial phase of OAKS data collection
Yes
66 (56.4)
51 (43.6)
0.850
No
74 (55.2)
60 (44.8)
Prior experience with audit
Yes
65 (60.2)
43 (39.8)
0.222
No
75 (52.4)
68 (47.6)
Rating of experience identifying consultant
Positive (4–5)
91 (58.7)
64 (41.3)
0.235
Not Positive (< 4)
49 (51.0)
47 (49.0)
Rating of experience registering audit a
Positive (4–5)
67 (56.8)
51 (43.2)
0.763
Not Positive (< 4)
73 (54.9)
60 (45.1)
Rating of experience linking Patient ID a
Positive (4–5)
96 (71.6)
38 (28.4)
< 0.001
Not Positive (< 4)
44 (37.6)
73 (62.4)
Rating of experience collecting data a
Positive (4–5)
106 (59.2)
73 (40.8)
0.104
Not Positive (< 4)
34 (47.9)
37 (52.1)
aRated on a self-reported Likert scale between 1 (very difficult) and 5 (very easy)
Table 4
Summary of recommendations for future multi-centre collaborative studies with longitudinal follow-up
Number
Recommendation to improve completeness of longitudinal follow-up
1. Study Design Recommendations
1.1
Linked patient identifiers should be kept in a central repository (for example a REDCap system) if Caldicott Guardian approval is given to minimise loss to follow-up
2. Study Delivery Recommendations
2.1
Included at least one team member with previous experience in trainee research collaborative projects in each data collection team, where possible.
2.2
Having junior doctors paired to students in data collection teams improve centre participation and data completeness rates. Where this is not possible, at least one senior medical student with previous collaborative audit experience should be part of the data collection team.
2.3
A network of regional leads are useful to monitor local progress and feedback to steering committee
2.4
Tracking regional variation in performance through the study and targeting specific efforts to improve follow-up and data completeness in these areas may increase data quality and maximise efficiency.
2.5
In high-volume centres where achieving high data completeness may be burdensome, consider permitting involvement additional team members to provide support.
Discussion
OAKS was the first TRC prospective cohort study to attempt to complete longitudinal one-year follow-up. This report demonstrates that most centres were able to collect one-year follow-up data with high levels of data completeness. Although the overall follow-up rate was only 62%, there was no evidence of systematic bias in patients being followed-up. Factors associated with increased likelihood of achieving > 95% data completeness were lower numbers of patients to be followed-up, and central storage of patient hospital identifiers. As TRCs have now been set up across Europe [20, 21], validating this methodology will have broad international benefits.
Most studies that complete longitudinal follow-up of prospectively identified patients [22] require patient consent, ethical approval, and significant research infrastructural funding. Even in well resourced, funded trials loss to follow-up of up to 15% is expected and built in to sample size calculations [23]. In the UK, National Research Ethics Service regulations permitted collection of one-year outcomes to be completed as clinical audit, without the need for research ethics approval. Without ethical approval it was not possible to collect identifiable data centrally. This report demonstrates that satisfactory follow-up is achievable within this regulatory framework, and without dedicated funding.
Anzeige
The most commonly reported barrier to achieving one-year follow-up was inability to identify linked patient records. Methods for maintaining linkage between hospital identifiers and study-specific identifiers were either holding hospital identifiers directly on the REDCap system or holding a cross-reference of hospital and study-specific identifiers on hospital computer systems by audit offices or consultants. Collaborating investigators found it easier to complete data collection when approved hospital identifiers were stored on the REDCap system. In Scotland, where national approval was gained for Community Health Index (CHI) identifiers to be stored on REDCap, data completion rates were higher. Therefore, future studies should seek local or national Caldicott guardian approval to store approved hospital identifiers on REDCap.
Loss to follow-up presents a major risk to the internal validity of a study as it leaves a specific population where outcomes remain unassessed, which may differ between groups. In the OAKS study, there were no significant differences in the patient-level demographics, operative indications or ASA grades between the group that underwent one-year follow-up and those that did not. The AKI and mortality rates at 30-days postoperative follow-up also were not significantly different between the groups that did and did not achieve one-year follow-up data.
A significant limitation to the method of follow-up in this study was its restriction to the hospital where the index surgery was performed. A small number of patients may choose to move their care to another centre, and some patients may be readmitted to a different hospital. Consequently, when patients were followed-up at the index hospital, there may have been no record of readmissions, treatments, and blood tests that took place at other hospitals. In addition, since no specific clinic visits were arranged for this study, if clinical teams did not arrange any postoperative clinic visits, or patients did not attend arranged visits, it is possible that the hospital records that were reviewed as source data for this study may not have been fully accurate.
The evaluation of this study’s methodology was limited by the broadness of the barriers explored in the investigator survey. A qualitative approach with detailed interviews with investigators may have been more likely to identify specific difficulties that precluded follow-up from being completed. Incorporating such a qualitative component to future studies may improve follow-up by identifying more solutions [24].
Anzeige
Conclusion
The OAKS study has demonstrated that prospective TRC cohort studies can successfully complete one-year longitudinal follow-up, with acceptable data completeness rates. Future studies may maximise follow-up rates by optimising procedures for storage of patient identifiers, embedding collaborators with previous experience of TRC studies within data collection team, and tracking regional variation in performance throughout the study.
Ethics approval was sought from the South-East Scotland Research Ethics Service. Since OAKS was an audit limited to data obtained as part of usual care, NHS ethical review was not required. Patient consent for this study was not applicable.
Consent for publication
Not applicable.
Competing interests
The authors declare that they have no competing interests.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Challenges of one-year longitudinal follow-up of a prospective, observational cohort study using an anonymised database: recommendations for trainee research collaboratives