Background
The mitochondrial fatty acid oxidation disorder, medium-chain acyl-CoA dehydrogenase (MCAD) deficiency, is one of the most common inherited metabolic diseases, with an estimated birth prevalence as high as 1 in 12,000 in Canada [
1,
2]. The MCAD enzyme is involved in the breakdown of medium-chain fatty acids, [
3] which is required for sustaining euglycemia after the depletion of glycogen stores, for example, during high energy activities, when fasting, or when unwell with fever or vomiting [
4,
5]. Deficiency of this enzyme markedly increases the risk of life-threatening manifestations during such periods of catabolic stress, including metabolic decompensations characterized by hypoketotic hypoglycemia, lethargy, and/or seizures [
6,
7]. Treatment typically involves the avoidance of prolonged fasting, medical monitoring during periods of illness, and the provision of rapidly available carbohydrates [
8]. Longer-term preventive interventions, such as carnitine supplementation, are used in some children with MCAD deficiency, although evidence regarding their benefits and harms is lacking [
9‐
11]. Newborn screening has transformed outcomes for children with MCAD Deficiency by allowing early diagnosis and presymptomatic treatment to prevent mortality (1, 2, 30, 31). Newborn screening panels in Canada vary from province to province; most provinces began screening for MCAD deficiency in the early 2000s (range 2001–2012).
To improve care and long-term outcomes for children with MCAD deficiency, rigorous approaches to evaluation of treatments are needed, informed by reliable, sustainable, and longitudinal measurement of clinically meaningful and patient-centred outcomes [
12,
13]. A core outcome set (COS) is a small group of priority outcomes agreed upon by stakeholders interested in a specific health condition with the goal of encouraging the standardized measurement and reporting of endpoints measured during clinical trials for that condition [
14,
15]. The development and implementation of COSs can support the synthesis of evidence and the comparison of findings across clinical trials where appropriate. These outcomes can also be collected as part of a high-quality disease registry to establish robust observational data over time and to facilitate registry-based randomized trials, where a trial is implemented in a registry platform that incorporates rigorous outcome measurement [
16]. There is a particular need for multi-centre and international collaboration in rare disease settings, given the small number of patients in any single centre. A COS can facilitate such collaboration in rare disease research as part of the harmonization of data on long-term outcomes and treatment effectiveness in small populations, thereby increasing the robustness of data pooling and thus improving the quality of evidence.
We recently developed a COS for children with MCAD deficiency as part of the Core Outcome Measures in Effectiveness Trials (COMET) initiative (
www.comet-initiative.org), [
17] relying on: (i) a systematic review of prior studies of MCAD deficiency to derive a potential list of relevant outcomes; [
18] and (ii) a multi-stakeholder consensus approach (Delphi survey and workshop) involving patients and families, clinicians, and policymakers [
19]. The final COS comprised eight core outcomes for children up to age 12 years diagnosed with MCAD deficiency, four of which could be ascertained from a child’s metabolic chart and therefore were identified as being of primary interest for the present study: emergency department use, fasting times, metabolic decompensation, and death [
19]. Some outcomes, notably emergency department use, may alternatively be measured using population-wide healthcare administrative data, as demonstrated in a previous Ontario-based study from our group [
20]. However, these administrative records often lack the detailed clinical information needed to reliably measure outcomes such as fasting and episodes of metabolic decompensation.
To facilitate prospective collection of these clinical outcomes and thereby support observational registries and clinical trials for children with MCAD deficiency, there is a need to establish the feasibility and sustainability of measuring these outcomes in routine clinical settings, and for outcomes other than death, the opportunity for ascertainment on a long-term and regular basis [
21]. To assess the quality of existing metabolic chart core outcomes data and their future suitability for prospective measurement during metabolic clinic visits, we used existing cohort data to investigate the frequency of clinic visits and quality of metabolic chart data for selected outcomes.
Materials and methods
Data source and eligibility
The Canadian Inherited Metabolic Diseases Research Network (CIMDRN) established a consent-based cohort of nearly 800 children across Canada diagnosed with one of 31 Inherited Metabolic Diseases (IMD), including MCAD deficiency, [
22] and included collection of clinical data from the metabolic charts for enrolled children, from birth up to a maximum of 11 years of age. The cohort was developed as a platform to support research that broadly seeks to understand health care and outcomes in this pediatric population.
Children were eligible for the CIMDRN cohort if they were born between January 1, 2006 and December 31, 2015, and received care for a confirmed diagnosis of MCAD deficiency at one of the 13 participating treatment centres between birth and March 31, 2017. Research staff at participating centres retrospectively abstracted data from electronic and/or paper charts depending on the type of metabolic chart in use at the participating centre at the time of each visit. At baseline, abstracted data pertained to participant and family characteristics, medical history, source of ascertainment, and diagnostic tests completed. For each visit to the metabolic clinic after diagnosis, the results of follow-up tests, disease-specific outcomes, treatment, and acute and chronic diagnoses were abstracted. Data fields were selected to support anticipated research questions related to health care and outcomes and to capture information likely to be present in existing metabolic charts. All data were entered as open- and closed-ended responses in a series of study-specific, web-based data collection forms developed with extensive input from metabolic clinicians across Canada who were members of CIMDRN. The data collection forms were submitted to and stored on a central study database in Research Electronic Data Capture (REDCap), [
23,
24] a secure, web-based software platform hosted at the Children’s Hospital of Eastern Ontario Research Institute; these forms can be obtained by contacting the corresponding author). In addition to the careful design of intuitive data collection tools and regular communication with treatment centre research staff, in order to maintain data quality, the data were subject to a detailed verification process by staff at the CIMDRN central office, including a review of each participant’s full dataset and periodic monitoring of summary measures [
22]. A unique, study-specific patient identifier was assigned to each participant in lieu of names and other identifying information to uphold patient confidentiality. Ethics approval for the protocol outlining cohort enrollment, clinical data collection, and analysis was granted by the Children’s Hospital of Eastern Ontario Research Ethics Board, the Ottawa Health Science Network Research Ethics Board, and the research ethics board at each participating centre.
For the present study, we conducted an analysis of these previously abstracted metabolic chart data from enrolled children with MCAD deficiency. Children were excluded if they had no recorded clinic visits after initial enrollment, for example, due to complete absence of data entry or death prior to their initial clinic visit. Children were followed until the study end date of March 31, 2017 unless they were deceased or discharged from a participating metabolic centre during the study period (e.g., due to relocation to a centre not participating in the cohort study). Children who moved to a participating centre from a non-participating centre during the study period were followed from the date of their first recorded clinic visit with the participating centre.
For each participant, data were abstracted from charts and entered into REDCap chronologically, starting from birth or the youngest age of a first recorded clinic visit at a participating centre. If data entry for a participant ended before March 31, 2017 and we were unable to confirm a death or discharge from the clinic, we used all available data for that participant in the analysis. An exception was when calculating visit frequencies (rates per child per year) that involved summing follow-up time; for these analyses, we considered children with incomplete data to be lost to follow-up at the end of the oldest age group to which they were known to be followed for the complete period.
Analysis
We described the demographic characteristics (e.g., year of birth, sex, consenting treatment centre) and baseline clinical characteristics (e.g., ascertainment method and neonatal complications) using frequencies and percentages. We calculated confidence intervals for incidence rates using the exact Poisson distribution or the normal approximation to the Poisson distribution as appropriate. Confidence intervals for means were expressed using the standard normal distribution. All cell counts representing fewer than five children were suppressed as “ < 5” to reduce the risk of identifying participants, in accordance with research ethics requirements. SAS software® version 9.4 (SAS Institute Inc., Cary, North Carolina, USA) was used for all statistical analyses.
Data collection intervals among participants varied, depending on each child’s schedule of visits to the metabolic clinic. We considered the frequency of visits as an indicator of the frequency of opportunity for outcome measurement. To determine the potential future feasibility of collecting core outcomes prospectively by relying on existing clinical encounters, we reported the frequency of visits to the metabolic clinic and telehealth encounters over time, expressed as rates per child per year, calculated as the total number of visits divided by total person-time of follow-up. Visit rates are presented by child age using 6-month intervals in the first year of life and 2-year intervals thereafter. They exclude visits occurring prior to two months of age in order to focus on visits occurring after a complete diagnosis is typically established, and when core outcome measurement may be most relevant. We explored variation in the frequency of clinic visits among participating centres treating five or more children by presenting results separately for each centre.
To evaluate the quality of data for each of the outcomes of interest, we explored the four core outcomes of interest and their components, guided by Kahn et al.’s framework [
25] covering three data quality concepts: completeness, conformance, and plausibility. To address completeness, we examined the extent to which individual components of a particular outcome were captured in abstracted data. To address conformance, we measured the extent to which data were entered in the proposed format and whether there existed variation in how outcomes were measured, abstracted, or recorded across sites. To address plausibility, we examined whether aggregate measures reflected a reasonable measurement or health trajectory over time for a child with MCAD deficiency, expressed as rates or summary measures of core outcomes across age groups as appropriate. For one of the core outcomes of interest, metabolic decompensation, there exists no widely accepted clinical definition. Thus, to ascertain whether metabolic chart-abstracted data may be used to measure this outcome for children with MCAD deficiency, we first identified acute clinical manifestations commonly associated with metabolic decompensation (e.g., hypoglycemia, seizures). Next, two pediatric metabolic physicians (PC, MTG) made independent inferences about whether each event identified by abstracted data constituted a true metabolic decompensation, judging each event as a yes (decompensation) or no (not a decompensation). The physicians relied solely on the abstracted data items to make this judgement. We measured agreement between the two raters using Cohen’s kappa coefficient.
Discussion
This study sought to evaluate the quality of existing metabolic chart data and its future suitability for measuring select core outcomes from a COS for MCAD deficiency: emergency department use, fasting time, metabolic decompensation, and death. We found a decreasing rate of follow-up visits to the metabolic clinic by age, consistent with evidence suggesting that the highest risk periods for developing complications occur in younger age groups [
31]. Based on the frequency of follow-up visits to the metabolic clinic, an opportunity exists to measure these clinical core outcomes in the metabolic clinic setting at least once per year regardless of patient age. Although other COS studies have defined specific time points for measurement of core outcomes according to disease-specific milestones and timelines, [
32] there is no published benchmark for the frequency of collection of core outcomes in children. For MCAD deficiency, the minimum frequency of visits required to capture meaningful change in an outcome may depend on the outcome itself, its measurement scale’s responsiveness to change, and, in studies evaluating management strategies, the specific treatment being evaluated as it relates to the expected length of time until a change in outcome can be expected. Therefore, alternative strategies to collect clinical core outcomes from children who visit the metabolic clinic less frequently may be required; improvements to the provision of virtual ambulatory health care and to shared electronic health records during the SARS-COv2 pandemic may present opportunities for such alternatives.
We believe that rates of emergency department visits were likely to have been underestimated when relying on metabolic chart data alone, but otherwise followed anticipated trends by age. We found the highest rate of emergency department visits between 6 and 12 months of age, which is consistent with previous literature and corresponds with the highest risk period for experiencing metabolic decompensations [
33]. It is well known that the accuracy of medical records data depend on the type of data collected; demographic, outcome, and discharge information collected as part of standardized sections of the chart have been found to have the highest accuracy [
34]. Data quality may be further maximized when relying on outcomes, such as emergency department visits, that rely in large part on a binary response, such as the presence or absence of a health event. However,
post-hoc follow-up investigation with research staff at each participating centre suggested that information captured within the metabolic chart pertaining to emergency department visits often corresponded to visits that had occurred at same health care institution’s emergency department. Emergency department visits occurring at other centres, such as at local community hospitals, may not have been captured unless the information had been parent-reported, forwarded by the emergency care team, or otherwise requested by the metabolic team. There was an improved comparability of rates of inpatient hospitalizations with gold standard measures relative to rates of emergency department visits [
20]. This suggests that emergency department visits warranting more critical and longer-term care may be more rigorously documented in the metabolic chart. Collectively then, when relying on metabolic chart data, we hypothesize that whether an emergency department visit that occurs is captured depends on the nature and severity of the visit and its relevance to a child’s disease as well as whether the visit occurred at the same institution (which in turn seems likely to vary according to both disease severity and geographic proximity of the child’s residence to the hospital with which the outpatient metabolic clinic is associated). The fact that emergency visits occurring at community hospitals may be missed from metabolic charts and that these visits may be distinct in nature from those that are captured suggests a need for additional pro-active measures to systematically capture emergency care. Strategies may include supplementing metabolic chart data with patient- or parent-reported forms and linkable comprehensive health care administrative data.
Measuring prescribed maximum fasting times from metabolic chart data that had been documented for clinical rather than research purposes was problematic. Although similar to emergency department visits in that the quality of reporting of this outcome relied on the presence or absence of a fasting prescription recorded in the metabolic chart, [
34] it is possible that updates were often the product of informal discussions between a clinician and a patient or their family member/caregiver that are not documented. We found that some centres provided families/caregivers with a clinic-specific protocol describing standard recommended fasting times according to age periods. In such cases, only exceptions to this algorithm may have been recorded in the metabolic chart. Consequently, recommended fasting times were updated highly infrequently in patients’ charts, particularly at certain centres. Therefore, consistent reporting of fasting times for prospective research will require the use of clinic report forms completed at each metabolic clinic visit that appropriately capture prescribed diets and recommended fasting times.
In this study, the agreement among metabolic physicians was considered moderate to substantial in terms of characterizing abstracted data items as episodes of metabolic decompensation. The nature of the disagreements in classification between the two observers suggested that each had a differential threshold for an event to be considered a metabolic decompensation. For example, hypoglycemia was a common clinical characteristic among events identified as decompensations by both raters, but invariably reflects a late symptom of metabolic decompensation [
33]. A factor rendering it difficult to measure occurrence of metabolic decompensations consistently is that emergency care for MCAD deficiency is often focused on mitigating risk and thus it can be difficult to distinguish a “near miss” (i.e., decompensation avoided due to an intervention) from a full blown decompensation. It was also difficult to temporally associate clinical manifestations contributing to a single event based on metabolic chart data. Certain features occurring alone may not always constitute a true decompensation and may require clinical expertise and judgement (e.g., based on the age of child, the perceived severity of disease, temporal association with other indicators, “susceptibility” from prior events, and other comorbidities). In general, diagnostic information has been found to have lower accuracy compared to other data, which can introduce systematic errors in the absence of standardized definitions and non-adherence to those definitions [
34]. Thus, this outcome requires a clear operational definition meaningful to a large number of clinicians treating children with MCAD deficiency. The refinement and application of practice guidelines to include the diagnosis of metabolic decompensation events and for acute management of patients with MCAD Deficiency could also both benefit patient care and also the usability of chart abstracted data for research and quality improvement.
Metabolic centres with both paper-based charts and electronic health records participated in the initial cohort study, with some having referenced both formats of the chart concurrently and others having transitioned to their electronic system during the latter period of the clinic visit eligibility. It was not possible to formally examine whether the quality of metabolic chart data differed based on the format of the chart. Anecdotally, however, we anticipate that differences between centres in the quality of data and reporting are not dependent on the format of the chart due to the inconsistent way that clinical data are currently captured even in electronic forms. Standardization of clinical forms and field definitions (whether paper or electronic) across clinics to the extent possible will likely be required.
The data quality issues we identified currently limit our ability to leverage real-world clinical data for children with MCAD deficiency in research and quality assurance initiatives to inform and improve care. These findings indicate a need for investment in platforms and infrastructure to support high quality routine outcome monitoring in clinical settings to produce usable data that can be synthesized and compared across care centres. Such platforms will facilitate evidence generation to ultimately improve care and outcomes for children with MCAD deficiency, for example through high-quality prospective multi-centre registries that can provide long-term natural history data and support pragmatic trials.
Limitations
We capitalized on the availability of existing cohort data abstracted from metabolic charts to evaluate reporting of core outcomes for pediatric MCAD deficiency. However, because the CIMDRN cohort study was not specifically designed to collect our core outcome set, it is possible that we underestimated the extent to which the information required to document core outcomes was present in patients’ metabolic charts compared with the variables that we could extract from the database. Specifically, a health event or variable that was indicative of a particular outcome may be inadequate or unavailable in the longitudinal database simply because the definitions and measurement criteria for that outcome were selected a posteriori relative to the cohort data collection. A further limitation to our study is the current scarcity of data available for older children. Included data on older age groups were populated mainly by children with longer follow-up and born in earlier years by nature of the data collection period relative to the years of birth included in this study. Finally, clinicians relied solely on abstracted data items to evaluate the incidence of metabolic decompensations and did not have access to the child’s medical history, which may have influenced their clinical judgement and their ability to correctly identify decompensation episodes.
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.