The development process followed steps detailed by Streiner and Norman [
14] with the instrument items designed to measure a structure, a process, or an outcome according to Donabedian’s model of quality health care [
15]. Research Ethics Board (REB) approval was obtained from the University of Toronto (#23107) and at each recruitment site.
Phase 1: literature search
Potential indicators of high quality end-of-life care were identified through a search of Medline and Cumulative Index to Nursing & Allied Health Literature (CINAHL) databases in March 2008. Search terms from 1) type of care (e.g., palliative care, bereavement care, or end-of-life care), 2) care assessments (e.g., quality of health care, needs assessment, outcome assessment, program evaluation), and 3) population (e.g., parents, mothers, fathers, family) were mapped to MeSH headings and combined. Results were limited to papers published in the preceding 10 years (1997 or later), in English, and related to children (0 – 18 years). Research papers that detailed parents’ perceptions of palliative, end-of-life, or bereavement care provided to the family by health professionals were included. Studies were not specifically assessed for scientific rigor because it was important to include the full depth and breadth of possible indicators at this early stage of instrument development [
14].
Phase 2: focus groups
Focus groups were planned, conducted, and analyzed according to procedures outlined in “The Focus Group Kit” [
16]. Focus group discussions were audio recorded and transcribed verbatim. Bereaved parents received a $40 token of appreciation for participation. Parents were recruited through advertisements in newsletters and web sites associated with local bereavement organizations and support groups, as well as by word of mouth. Parents indicated interest in the study by telephone or email and were included if 1) they were the biological, step, or adoptive parent of the child who died, 2) their child who died was aged 19 years or less, and 3) the death had occurred at least one year previously. Parents who did not speak English or whose children were stillborn were excluded. Initially, only parents of children who died in hospital settings were sought. However, the child of about half the parents who responded to the advertisements had died at home. As these parents had extensive experience with hospital care prior to the death, the inclusion criteria were expanded to include them.
Parents were initially asked a broad question about what they thought was important to the provision of high quality end-of-life care for children and their families. Parents were then asked to confirm, challenge, or add to the list of indicators identified in the Phase 1 literature search. Content analysis of the transcripts was completed following each group [
17]. Domains and indicators of quality pediatric end-of-life care initially identified through the literature review were revised and refined based on the focus groups.
Phase 3: item development and refinement
Items used in previous research [
8,
9] were revised and new items developed to assess the domains and associated indicators identified in Phases 1 and 2. Items reflected structures, processes, and outcomes of care [
15]; however, most assessed processes and were worded to ask about care provided to the child and family or specifically to the parent. Most items had five response options on an adjectival scale ranging from “never” to “always” to reflect the frequency of occurrence of a particular aspect of care. Outcomes were assessed through ratings of satisfaction with care, overall quality, and whether the child had a ‘peaceful’ death and a ‘good’ death.
Health professionals with clinical or research expertise in pediatric end-of-life care assessed the content validity of the instrument. These judges were initially contacted through email, then sent a package with the consent form, instructions on how to assess the instrument, and the instrument. Judges rated each item on a four-point scale: one = not representative of quality care, two = needs major revisions to be representative of quality care, three = needs minor revisions to be representative of quality care, and four = representative of quality care [
18]. A Content Validity Index (CVI) was computed for each item by calculating the proportion of judges who rated the item as a three or four [
19]. Items receiving a rating less than 0.80 were revised. The CVI for the entire instrument was calculated by averaging the CVIs for all items [
19]. An acceptable index is greater than or equal to 0.80 [
18]. Judges also suggested wording changes to improve clarity, helped develop subscales by indicating which domain the item was measuring, and assessed if the instrument as a whole was a comprehensive measure of quality end-of-life care for children.
Parents who took part in the Phase 2 focus groups and gave permission to be contacted as well as parents who responded to Phase 2 advertisements after completion of the focus groups were invited to participate in Phase 3. In individual cognitive testing over a two-hour time period, parents read and re-phrased each item and/or talked through how s/he chose the item response [
20]. Parents’ comments and possible changes to the instrument were incorporated from one interview into the next to see which words or questions appealed to the most parents. To assess face and content validity, parents were asked if items related to quality end-of-life care and if all aspects were included [
14]. Items were revised, deleted, or added as directed by comments from health professionals and parents; parents’ comments were given greater weighting when there were discrepancies.
Phase 4: psychometric testing
With assistance from a local investigator at each site, Phase 4 participants were recruited through 10 children’s hospitals and hospices across Canada. Although both mothers and fathers participated in focus groups and initial instrument testing, only mothers were recruited for Phase 4 as we recognized there may be differences between mothers’ and fathers’ needs and how well these needs are met. Mothers were invited to take part in the study if 1) they were the biological, step, or adoptive parent of the deceased child, 2) the child died between 6 months and 36 months prior to survey mailing, 3) the child died in hospital/hospice after minimum 24-hour admission, and 4) the child was aged 19 years or younger at time of death. Mothers were excluded if, 1) the infant died within 48 hours of birth, 2) the child died in a hospice and received only terminal care in the hospice (for example if the child was transferred from hospital to the hospice primarily for withdrawal of ventilator support), 3) they requested to have no further contact from the hospital/hospice (such as for bereavement follow-up support) 4) either parent had been implicated in the death of the child, or 5) they could not read English.
Recruitment procedures varied somewhat across sites due to local REB requirements. At all sites, contact information for eligible mothers was obtained through review of records. A healthcare professional known to the family mailed a letter with a brief description of the study to the mother. At seven sites, mothers returned an enclosed card if they did not wish to take part in the study (opt-out). If the card was not received at the originating site within three weeks, the survey package was mailed. At the other three sites, mothers returned a card indicating they wished to participate in the study (opt-in) before the survey package was mailed.
The survey package included information needed to make an informed decision about participation, the instrument itself, and an addressed, stamped return envelope. Return of the completed survey signified consent. A reminder letter was mailed through one site resulting in return of one additional survey. Reminders were not sent to non-responders at any other site due to logistical issues or REB concerns. Mothers provided contact information if they were willing to repeat the instrument in two weeks to facilitate assessment of test-retest reliability.
The instrument was designed and tested as five unique scales comprising a larger index, rather than as one single measure [
21,
22]. Psychometric testing included exploratory factor analysis (EFA) to assess validity, plus internal consistency and test-retest to assess reliability. A correlation matrix of items in each subscale was generated. Items with correlations less than 0.30 or greater than 0.90 were considered for deletion [
23]. The Measure of Sampling Adequacy (MSA) was calculated for each item using an anti-image correlation matrix and items less than 0.70 were considered for deletion [
23]. The Kaiser-Meyer-Olkin MSA was calculated as a summary for each subscale. When the summary value (recalculated when items were deleted) was greater than 0.70, analysis continued with an assessment of factor structure [
23]. EFA was conducted on each subscale using principal axis factoring and eigenvalues greater than one to generate factors. Only one factor for each subscale was expected, but when more than one factor was identified a varimax rotation was used to improve interpretation [
23,
24]. Items with factor loadings less than 0.30 were considered for deletion [
23]. The theoretical foundation of the instrument developed in Phase 1 and 2, the study team’s clinical and research experience with this population, and parents’ comments in Phase 3 about each item were considered in final decisions on items to retain or delete. Cronbach’s alpha was calculated for remaining items in each domain with a score between 0.70 and 0.90 indicating good internal consistency [
25].
A subsample of participants completed the instrument twice. An intra-class correlation coefficient (ICC) was calculated for each revised subscale to compare scores using a two-way random effects model with a single measure and absolute agreement of scores [
14]. A value greater than or equal to 0.75 was evidence of test-retest reliability [
26].