Global core indicators for measuring WHO’s paediatric quality-of-care standards in health facilities: development and expert consensus

Figure 1 shows the structure of WHO’s paediatric QSs. The 8 QSs describe what should be provided to achieve high-quality health care for children and young adolescents across the 8 quality domains (QDs). The 40 corresponding quality statements (see Additional File 1) are designed to drive continuous improvement to achieve positive care outcomes and a positive experience of care. Five-hundred and twenty (520) quality measures (QMs) were identified from the 8 domains as a means to measure whether specific aspects of quality are provided or achieved.

The structure and content of the MNH QSs and Paediatric QSs were adopted in the maternal, newborn, and child health (MNCH) QoC monitoring framework as the organizing framework for developing core QoC indicators [11]. The MNCH QoC monitoring framework recognized the different QoC measurement needs by stakeholders across different levels of the health system and proposes three QoC measurement components, including a) core (common) indicators: which is a small set of prioritized input, process, outcome, and impact indicators for use by all stakeholders at every level of the health system to track and compare process across and within regions and countries; b) Quality improvement (QI) indicator catalogue: which consists of a flexible menu of indicators to support QI facility and subnational levels led—respectively—by facility-based QI teams and district or regional health authorities to support improving and sustaining QoC in health facilities; and c) implementation milestones which help track the progress of country-specific or subnational implementation activities to develop and maintain quality improvement programmes. However, the latter are not directly linked to QSs but rather to national QoC implementation roadmaps, policy and strategies.

Core indicators and the QI indicator catalogue are directly linked to and measure the QSs. The focus of this article is core indicators, and we describe how they were selected from the QI indicator catalogue to maintain the linkages with the QSs.

The proposed core indicators were developed using a deductive approach. This approach is typically built upon a specific conceptual framework and links the indicator to clinical and/or patient-centered inputs, processes, and outcomes [12]. The proposed core indicators were therefore linked to the specific QSs and QMs. To derive these indicators, we followed nine complementary steps (Fig. 2) which allowed for an iterative process of indicator development. These steps included a rapid literature review, development of a methodology, the application of a systematic algorithm for indicator systematization and prioritization, and multiple iterative expert consultations.

A gold standard methodology or approach for QoC indicator development is yet to be defined [13]. Therefore, we first conducted a rapid review of the published and grey literature to scope available methodological approaches for developing core QoC indicators. The most-commonly used approaches identified were: reviewing existing health information systems to identify and adopt or adapt already available QoC indicators [14], using guideline-based approaches to systematically align QoC indicators to specific clinical and non-clinical guidelines [15], deriving QoC indicators from the available evidence base (e.g., indicators that have been validated in the literature as good measures of quality), combining expert consensus with evidence review, or a combination of multiple approaches [13, 16]. Using a peer-review process, we analysed and synthesized the strengths and weaknesses of available approaches, their relevance to LMIC settings and used the results to develop a technical protocol that outlined our methodology for selecting core indicators.

Following the development of the methodology document in step 1, WHO convened a global expert consultative meeting (Step 2a) to review and reach consensus on the proposed methodology and process for developing the quality indicators. Participants included 24 independent technical experts and 10 experts from WHO, USAID, and UNICEF that represented a range of expertise in paediatric QoC programming and measurement, paediatric research, health informatics, paediatric care, and child rights. During the consultation, experts were placed in small heterogenous groups based on their expertise to discuss and make recommendations on the methodology. Each group’s recommendations were presented in a plenary session for discussion and consensus building, and the resulting recommendations were presented in a separate consultative process (Step 2b) to the Child Health Accountability Tracking (CHAT) Technical Advisory Group [17]. The recommendations from the initial expert meeting and CHAT were all used to revise the methodology before implementation (Step 3).

The expert panel recommended the use of indicator development criteria and principles (described later) that recognize the complexity of measuring quality of clinical care and patient or caregiver-reported experience of care. These principles guided the generation of the most critical input, process, output, outcome, and impact indicators that can be used as “signals of quality”, thereby eliminating the need for measuring every aspect of service provision and related outcomes. These principles were applied to the 520 QMs during the indicator development process using a criterion- and score-based indicator prioritization algorithm which we built in an interactive Microsoft Excel spreadsheet (see Additional File 2). Initially, the algorithm allowed for the prioritization of a non-prescriptive menu of QMs linked to the QS. Based on this set, the core indicators were selected, identifying those which were most aligned with the QSs. The indicator prioritization algorithm is illustrated in Fig. 3.

As shown in Fig. 3, the QI indicator catalogue was generated from the 520 QMs. All QMs were first categorised into 3 measurement domains and 11 measurement subdomains as shown in Table 1. The objective of classifying QMs by measurement domains and measurement subdomains was to identify potential measurement areas to which distinct prioritization criteria should be applied. For example, the prioritization criterion for measurement domain 1 (adherence to evidence-based practices) aimed to identify areas for which there is strong evidence that links the care process to a desired health outcome. The same criterion, however, could not be applied to the QMs for child- and family-centered practices (measurement domain 3) which required consideration of patients’ rights, involvement in care, and elimination of harmful practices.

Two categories of prioritization criteria were used: a) Criteria applied to all QMs to prioritize valid, relevant, actionable, and feasible QMs; and b) Additional criteria applied to the resultant QMs in different measurement domains or subdomains to select QI indicators for the catalogue. As described in Table 2, these criteria were built in a standardized Microsoft Excel template, which facilitated a systematic and sequential approach to selecting specific QMs based on a scoring algorithm (Additional File 2) developed using a Microsoft Excel visual basic application. The final set included only QMs that met the minimum cut-off score for both prioritization steps. The final QMs were assigned non-prescriptive numerators and denominators to constitute the QI indicator catalogue which formed the basis for prioritizing core indicators.

The following criteria were applied to all indicators in the QI indicator catalogue to prioritize a smaller set of core indicators: usefulness, impact, and international comparability (see Table 2 for detailed rationale of each criterion). These criteria helped prioritise indicators for measuring QoC of paediatric services that are: 1) useful for guiding decisions around resource allocation and programming and especially at national and global levels; 2) sensitive to detecting change in QoC interventions at the service delivery level for priority paediatric conditions, and 3) aligned to the extent possible with standardized and validated global child health care indicators to enable comparisons.

These criteria and associated scores were applied to the QI indicator catalogue by a measurement expert (TC) who drafted a small set of core indicators for further review by two other experts (WW & MM). Details of the scoring mechanisms and cut-off scores are provided in Table 2.

During Step 5, the draft set of core indicators generated from step 4 were reviewed internally by a small group of 3 experts from WHO and University Research Co (TC, WW, MM) to determine the extent to which they fulfilled the selection criteria. Once the review was completed, the draft core indicators went through a global expert consultation process (Step 6) to generate the feedback on the content, clarity, definition, prioritization and other elements of the indicator metadata. These included program officials responsible for child health in various WHO country offices and ministries of health across three WHO regional offices which participated (Regional office for Africa, Regional Office for South-East Asia, and the Regional Office for Europe); individual child health QoC programming and measurement experts from academic and research institutions; WHO’s implementing partners at country level; global technical working groups; as well as technical focal points from other UN and multilateral agencies. All the resulting recommendations from the review were synthesized and revised by a small panel of experts (TC, WW, MM), as part of Step 7, and integrated into the draft core indicators.

The WHO expert team made final decisions on the core indicator selection based on the following guiding principles derived from the literature and expert recommendations:

During step 8 the small expert group (TC, WW, MM) developed a comprehensive metadata and indicator dictionary for each recommended core indicator. Finally, (Step 9) these indicators and metadata were prepared for eventual field testing across different settings.

We make recommendations for core indicators that are mapped to the paediatric QSs for health facilities. Almost all QSs are represented by at least one core indicator. However, certain QSs have greater representation than others because they are more complex in terms of the number of quality dimensions to be considered and their relative importance in influencing health care outcomes. This approach resulted in prioritization of more provision and experience of care indicators, compared to indicators that measure availability of inputs. While important, inputs alone do not guarantee that the service is provided correctly and consistently.

Our goal was to develop a parsimonious set of robust indicators that provide high level insights into the quality of paediatric care in health facilities, as opposed to a comprehensive suite of indicators that provide detailed measurement of the quality of paediatric care. Less emphasis was thus placed on overall feasibility of measurement in the short term, especially in LMICs, mainly because of two reasons.

Firstly, QoC measurement is a relatively new area of measurement in most LMICs. Many countries in such settings are yet to institutionalise QoC measurement within their mainstream health information systems [18]. Developing paediatric QoC indicators with strong emphasis on feasibility of measurement would have yielded an even more imbalanced set of indicators, from the perspective of their ability to measure the QSs from which they were derived, and the indicator metadata that defines critical attributes for QoC indicators. We therefore recognize that not all the recommended core indicators can be measured across all settings in the short-term. Globally, the ability of countries and institutions to measure, collect, report and use data for most health care indicators in general varies greatly across countries, including in high income settings [19‐21]. Our argument is therefore that the audience for these indicators—which includes countries and partners working in the paediatric QoC programming and QI space – should, in the short term, select and use the recommended indicators for which data are available in their health information systems. Users can then continue to improve their data systems to accommodate indicators that are not currently reported in standardized medical documentation, or are more complex to measure, or for which more data elements are required to collect the full set of core indicators. We also note that even in high-income settings, introducing new indicators in hospital settings may come with administrative challenges. The existing health information systems may require some changes such as the adaptation of patient administration forms and facility registers, and an increase in the administrative burden of data collection, collation and reporting [22].

Secondly, the recommended set is a response to urgent programmatic needs on a global scale in paediatric QoC implementation and measurement. Field-testing of indicators takes time and resources and there are methodological weakness of assessing the feasibility and sustainability of measuring and monitoring newly-developed indicators outside research contexts [23, 24]. Field-testing of these indicators is the next step that WHO and its partners will lead in multiple countries.

The implementation of expert recommendations on the methodology to develop the core indicators was not without challenges. The health status and risk factors for illness for young infants, infants, children under 5, and young adolescents differ and change over time. This created some challenges in prioritizing core indicators that are reflective of all age categories. The age categories where quality measurement is most crucial were thus prioritized, which introduced some imbalances. Furthermore, the indicators recommended for measuring experience of care will likely require the collection of views of children and/or their caregivers, as appropriate, and the content of the information provided would depend on several individual and contextual factors, including the age of the child and local cultural norms regarding care, age of majority, and parental rights. For example, meaningful participation and input from children can vary based on age and condition. Similarly, child health rights, especially young adolescents, may vary from one country to another which would render global reporting a challenge. In some cases, the experience of the caregiver and the child may be different and future work will be needed to understand how to measure these both together and separately.

A systematic review of methods used to develop quality indicators showed that patient participation during development of quality of care indicators remains uncommon and there is no standardized methodology on how best to involve patients and communities in the process [27]. To address this limitation, organizations and experts working on paediatric experience of care and patient care advocacy and child rights were included in development of experience of care standards and the indicator prioritization processes. Furthermore, during the field-based evaluation of the proposed indicators, WHO will seek to involve children and their caregivers to test the smaller set of indicators that focus on respectful and experience of care in various settings.

Some clinical indicators are compound metrics which require computation using several data elements, in some instances from different data sources. This was inevitable given the focus on robust measurement of quality as opposed to a convenient approach. While compound indicators provide high level, comprehensive understanding of the QoC at the national and global levels, these indicators need further disaggregation or additional catalogue indicators at the subnational and facility levels to identify and address the root-causes of poor quality services.

Furthermore, we acknowledge that the indicator prioritization algorithm we used was complex, and in some places, the algorithm involved the use of subjective criteria due to lack of scientific evidence to guide the process. For example, the criteria used to prioritize some input measures involved weights which are arguably subjective.

Similarly, because of the goal of keeping the list of core indicators short, there are few areas (e.g., referral systems) that are not included in the core list but could be part of a future research agenda. Despite all these challenges, however, the iterative peer-review process with various expert groups added value to the robustness of the proposed suite of indicators. Furthermore, the criteria used to prioritize the indicators are evidence-based and presented an opportunity to select and define indicators that are core for paediatric QoC measurement in health facilities where the QSs are supposed to be implemented.

Lastly, the excel tool which we developed to operationalize the indicator prioritization methodology, allows the user to add additional indicators, change assigned scores and prioritization weights, and designate cut-off scores. After this, the tool automatically selects the core and catalogue indicators. These features, along with the ability to systematize and sort the measures by various categories, are particularly important in the context of constantly changing evidence and emerging local or global priorities.

The proposed suite of indicators and their metadata are part of WHO’s living normative guidance on MNCH QoC measurement. They are not a final product and may be refined or revised based on the findings from future research on their validity in various contexts, especially in LMICs. An important next step will therefore be a large-scale, multicounty study to determine how these indicators can best be measured in different contexts. Immediate utilization of these indicators in practice and on a global scale will require a tiered approach that considers varying country contexts. Countries will need to decide which indicators they can measure and report on in the short-term, and which ones require further health information system adaptation in the medium to long terms.