Self-reported and routinely collected electronic healthcare resource-use data for trial-based economic evaluations: the current state of play in England and considerations for the future

Traditionally, trial-based economic evaluations were largely reliant on self-reported methods for collecting resource-use information about the care resources people consumed. However, since the National Programme for IT, which ran from 2002 until 2013, prompted UK health and social care services to record person-level care-related data using Electronic Health Record (EHR) systems, there has been a keen interest in how to utilise these routinely collected data. These systems are usually designed by independent contractors, specifically for the administrative needs of the service while conforming to national specifications (e.g. general practitioner systems of choice [GP SoC] specifications [5]), and could provide a rich source of data (both resource use and clinical outcomes) for analysis in clinical trials rather than requesting similar data from the patient directly (e.g. how often the patient saw their GP or number of days spent as an inpatient). NHS Digital and other providers currently offer a variety of electronic databases for research and commissioning purposes. However, consenting trial participants may not have their data stored in these databases, data from any large database usually has both monetary and logistical costs, and the database may not contain appropriate Personal Identifiable Data (PID e.g. NHS numbers) to link data with participants. Owing to the complexities and cost of accessing data from routine sources, self-reported questionnaires are still commonly used to collect data for the purposes of research studies. Different types of resources may be collected using these two different methodologies; for example, hospital stays might be collected using routine data while informal carer arrangements, such as how much time a son cares for his elderly mother, would typically be collected by questionnaire because these data are probably not routinely collected. Some studies have used patient-report [4, 6, 7] and some have used a combination of patient-report data and collecting data from databases to provide complementary (i.e. each data collection method provides different information for the trial) or even substitutive (i.e. each data method provides similar information for cross-checking or validity assessment of the data obtained) information [8‐10]. Even when administrative datasets are available, there are conflicting reports of the validity of routinely collected data compared with patient-reported data even when the data collected is perceived to be based on the same type of resource-use [11]. Therefore, neither data collection method can currently be considered the ‘gold standard’ for the purpose of informing trial-based analyses; rather, it is up the researcher to decide on which method to use dependent on the circumstance (e.g. trial design, setting, aims and objectives) and the data of interest for the trial.

Many trials employ data-collection methods based on patient recall [4], in part because a questionnaire is relatively cheap. Use of questionnaires allows the researcher to control data collection to a large degree, as it does not rely on third parties granting access; the value of this element of control should not be underestimated in trial situations with significant time pressures. Researchers can also tailor questionnaires to request specific information needed for analysis. This is particularly important when an economic analysis from the societal perspective is planned, as data on patients’ productivity, travel methods, and informal care or childcare requirements, for example, can only come from the patient.

However, relying on patient recall has some significant disadvantages. Patients tend not to be able to recall their resource use accurately [12] and this gets worse as the time period over which they must remember increases (e.g. retrospectively remembering the care they have received over 3 or 6 month periods); if the inaccuracy is systematically different between trial arms, it can constitute a recall bias, leading to biased estimates. Data entry clerks can introduce mistakes when capturing data electronically from paper-based questionnaires, either by simple typographical error or by reinterpreting non-standard responses in an arbitrary fashion. Completing questionnaires is time consuming for patients; researchers (and ethics committees) prefer to minimise the research burden placed on patients and primary investigators are often keen to minimise the amount of space taken up by resource-use questions. Although adjusting for baseline differences in resource-use is recommended as part of statistical analyses associated with trials [13], collecting data by patient recall at baseline involves an additional burden on patients, and is typically not undertaken. As a result of many of these issues, resource-use questionnaires tend to suffer from missing data [14].

In terms of the practical development and administration of questionnaires, there is a significant amount of research time wastage. These instruments are rarely validated [4], and even more rarely re-validated when used with alterations or in a different context. Reporting also tends to be poor [7]. As a partial solution to some of these problems, the Database of Instruments for Resource-Use Measurement (DIRUM) was created as a repository for instruments based on patient recall (www.​dirum.​org). It is a useful source for researchers wanting to find an instrument that may be reusable in their own work and the database is recommended for use by the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) taskforce on good research practices [15].

Standardisation of patient-reported outcome measurement in economic evaluation has been accepted as a principle for some time, with NICE recommending the use of the EuroQol five-dimension (EQ-5D) instrument [16, 17] as a measure of preference-weighted health status [2] to enable cross-comparison of outcomes across trials. Standardised instruments for patient-reported cost measurement have also been developed, with the Client Service Receipt Inventory (CSRI) the most commonly used and validated example [18]. However, the CSRI was originally developed to be administered as an interview, which is a costly means of gathering data and impractical in many trials. It has also been adapted many times, with over 200 versions believed to have been used [19], and was created specifically for use with patients with mental health conditions. It is therefore neither fully standardised nor universally applicable. The Annotated Patient Cost Questionnaire (APCQ) [20] was a more recent attempt to standardise cost measurement, and has invoked much interest since being deposited on DIRUM. However, it requires some effort to use effectively and, although it appears to perform well in validation studies [21], has not yet been widely adopted. Other standardised resource-use questionnaires have been developed, including one in Dutch [22], one for cancer trials [23], and one for patients with dementia [24], but a standardised (and well validated) instrument that is relevant to all trials and could be used ‘off the shelf’ by researchers in the UK is lacking. While this ideal may not be fully achievable, a recent Delphi survey based on the opinions of health economists with experience of UK-based trials found that it was possible to identify a list of 10 essential items that should be included in such a standardised instrument [25], with additional modules identified as important in some cases.

Within this section we explore the use of routinely collected data; from raw data extracted straight from the service up to the level of linked datasets across multiple services, and the evolution of more efficient trial designs utilising these data.

Raw data extraction can include anything from direct system data extraction and anonymization to recording data from computer screens into data extraction forms. Of 63 studies that used patient-reported methods, 43 supplemented these data with routine sources such as GP records, hospital notes, and hospital databases [4]. A previous study by M Franklin, V Berdunov, et al. [26] extracted raw data from a range of services, including hospitals’ Patient Administration Systems (PAS), primary care practice EHR software systems (including SystmOne [27], EMIS [28], and Vision [29]), as well as from the electronic systems of intermediate care, mental health trust, ambulance, and social care services; these methods were then used to obtain data for two subsequent RCTs [30, 31]. It is worth noting that when obtaining primary care data and other external service data (e.g. community care), SystmOne proved particularly useful within these studies because remote access was possible due to the central database and interoperability of this system – information pertaining to patients’ care from any SystmOne system can be held centrally and then accessed providing a smart card with appropriate permissions from the practice has been gained (and the person and service have consented to certain data access) [26, 32, 33]. However, the permissions and time required to obtain data from all these disaggregated services and their electronic systems makes obtaining such data labour intensive and so not viable for many studies.

Primary care data have been particularly disaggregated among a number of software systems (e.g. EMIS, SystmOne, Vision) [34] and clinical coding systems (e.g. Clinical Terms Version [CTV3] and Read Version 2) which has caused issues for researchers. MIQUEST was an example of a stand-alone query language within GP practices that could be used to extract certain data parameters; however, the output from such a query language limited its use for economic analysis despite its benefits for patient identification based on characterising patients using Read codes (such as if patients with a particular condition need to be identified for a study, for example) [35‐38]. A current move to the Systematized Nomenclature of Medicine - Clinical Terms (SNOMED CT) as a single clinical terminology within primary care [39] and across other care settings [40] could deal with some cross-system data extraction issues, but this will need to be examined in future research; it should also be noted that due to the implementation of SNOMED CT, NHS Digital announced that they would not be undertaking any future development work on MIQUEST (as it was not compatible with SNOMED CT) and advised that organisations plan to transition away from MIQUEST by April 2018 [41, 42]. Attempts to enable better formats for extracted information from primary care software systems, such as the Apollo software system [10, 43, 44], could enable trial-based analysis and should be assessed in future research.

Raw data collection often requires aid from data processors or those with knowledge of health informatics at the service, or researchers with appropriate knowledge, training and permission to access and process the data. This imposes a practical restriction on researchers using the systems as they must liaise with third parties.

NHS Digital provides a variety of electronic datasets (see Table 1 for examples), offering access through their Data Access and Request Service (DARS) dependent on a five-stage process (application, approval, access, audit, and deletion). NHS Digital have appropriate permissions to handle PID and therefore accessing data for consenting patients is in theory feasible assuming information governance (IG) and data-sharing agreements can be arranged; this may not be possible with other datasets (see section titled "Other large observational datasets: Primary care").

NHS Digital’s Hospital Episode Statistics (HES) dataset has become popular for the purpose of analysis within England. HES provides a large amount of aggregated or person-level hospital care data and unified codes, such as: International Classification of Diseases version 10 (ICD-10); Office of Population and Surveys Censuses (OPCS) Classification of Interventions and Procedures version 4 (OPCS-4) codes; and Healthcare Resource Group (HRG) codes which can be linked with reference cost data [45, 46]. However, note that HES only has HRG codes until 2012; researchers will need to use HRG software toolkits to derive HRGs for the latest HRG version 4+ (HRG-4+) [47, 48] and earlier versions (e.g. HRG-4 and HRG v3.5) [48] from codes within HES.

In addition to HES, NHS Digital provides a variety of electronic datasets including:

Not all the data provided in these datasets are useful for the purpose of clinical or economic evaluations, but they are useful examples for indicating the amount of electronic data available. Descriptions of these databases and links to their data dictionaries are provided in Table 1. It is also worth noting that various mental health datasets have or may be amalgamated into the Mental Health Services Data Set (MHSDS) [55] (see also Table 1).

NHS Digital also provides datasets for commissioners, such as the current Clinical Commissioning Groups (CCGs: clinician-led bodies responsible for commissioning healthcare services within a local area). The data flows via the Data Services for Commissioners Regional Offices (DSCROs), who provide data to a CCG for their geographical area of interest and responsibility only. The CCG may also have ‘local data’ flows such as from acute, primary, mental health, and social care services to supplement NHS Digital data. Partnerships between commissioners and researchers can benefit both parties and patients through the research objectives and outcomes that could be achieved by sharing data. However, these datasets are governed by legislation and, often, NHS Digital policies; the CCG are bound by the terms of their data sharing agreement with NHS Digital.

For primary care data, a variety of datasets exist (which obtain their data from specific GP software systems) including: Clinical Practice Research Datalink (CPRD; traditionally obtains its data from the Vision system, although has reportedly started obtaining data from practices with the EMIS system and is piloting to obtain data from SystmOne) [56, 57]; The Health Improvement Network (THIN; Vision) [58]; ResearchOne (SystmOne) [59]; and QResearch (EMIS) [60].

However, these primary care databases do not allow access to PID (e.g. NHS number) and therefore are not suitable for existing trial-based evaluations with consenting patients. Note that although PID cannot be accessed through these databases, the database providers may be able to extract data from primary care systems for consenting patients (for example, ResearchOne explicitly offer such a service on their website); there is also still potential to develop more efficient study designs using databases such as these (e.g. CPRD) while maintaining the anonymised nature of the data (e.g. no PID; see section titled "Efficient study designs using large observational datasets"). It is worth noting that L McDonnell, B Delaney and F Sullivan [61] have compiled a more comprehensive list of UK primary care datasets that may be of interest.

An issue with electronic datasets is that they may only adequately record data for patients who use that particular service. Previous studies have already described the need for linked data when assessing the burden of injury [62] and for myocardial infarction [63]. This aspect has also been raised as a potential issue when interpreting information about external care services (e.g. hospital and community services) recorded within primary care [64, 65]. There have been attempts at linking datasets, for example: (i) Hospital Episode Statistics (HES) has data linkages with the Office for National Statistics (ONS) mortality data (including causes of death), PROMs (patient reported outcome measures), diagnostic imaging, and MHMDS datasets; (ii) a subset (approximately 75%) of consenting CPRD English practice data can be linked with HES, ONS (mortality data), area-based deprivation data, and Cancer Registry UK data [64].

Other examples of linked datasets described in Table 1 include the (i) NorthWest EHealth linked database and (ii) Clinical research using LInked Bespoke studies and Electronic health Records (CALIBER) dataset. For the latter, the practical issues with using linked data have been described by M Asaria, K Grasic and S Walker [66]. Linking datasets is important to obtain the best possible data for the services of interest and is an aspect for consideration if the proposed analysis requires a more holistic approach rather than focussing on a single service.

There is currently a move to enable RCTs to be carried out in large databases of routinely collected data while retaining the anonymised nature of the data – these have been described as ‘efficient’ [65] and ‘pragmatic’ [67] study designs. Using data from existing patient groups within these large observational datasets has been suggested as a way to enable patients to be entered into RCTs more quickly than traditional study designs [67]; however, without patient consent, these data must be kept anonymised for research purposes. CPRD have designed methods for interventional research to take place within the confines of their anonymised dataset by enabling practices to recruit patients and then randomise them to a treatment before data are then entered into the EHR records and anonymised as normal [68]; an example of a cluster RCT and economic evaluation within CPRD is the PLEASANT trial [65, 69, 70]. Another example of an efficient study design using similar methods includes the Salford Lung Study [44] using the NorthWest EHealth linked database. Although these trial designs are not currently the norm, they represent a methodology which may enable trials and the accompanying analyses to utilise routinely collected electronic data; however, such trials are constrained to rely only on the data routinely collected within the associated large database, meaning it may not be viable for all trials dependent on the relevant trial outcomes and associated data requirements.