Assessing the performance of maternity care in Europe: a critical exploration of tools and indicators

A dominant focus on the measurement of intrapartum care was identified. Most indicators were clinically focused, while the postnatal care of the mother was found to be the area least measured. This finding underlines existing concern about the limited monitoring of the quality of postpartum care [49-51]. It must also be recognised that despite the comprehensive measurement of technical interventions identified, many of these may have been unnecessary [16].

Table 3 reveals the predominance of CR as an overall indicator; this indicator was present in all of the tools devised to monitor and compare maternity care, and its comprehensive application highlights the perceived value of measuring CR in comparing performance both within and across hospitals. Some discussion emerged in the literature reviewed regarding the application of this indicator, however. One source highlighted the importance of measuring the CR rates of particular institutions and comparing these with the ‘normal’ range of elective and emergency caesareans; hospitals found to be above the normal range might need to review pre-labour obstetric practices, for example. Approaches were identified in some sources whereby CRs were disaggregated in relation to a core indicator of total CR, such as rates of caesarean section before/after onset of labour [52] and proportions of elective and emergency caesarean section [53] providing more meaningful information about the performance of the system, including decisions associated with the use of caesarean section and their outcomes. Since the initial review was conducted a new approach has emerged from the World Health Organisation (WHO) to measure CR at hospital level, based on Robson classification [54].

The importance of nuanced understanding of the particular contexts in which interventions take place, and in particular the various sub-groups of users to whom care is offered, was similarly highlighted. It was argued, for example, that knowledge of the case mix in different settings is essential to the full understanding of any indicators [3]. Only two of the seven tools identified attempted to directly address this problem, however. In one tool a homogenous group of ‘standard primaparae’ (a typology of low-risk women) was defined in order for indicators in relation to these users to be compared across sites; in another the examination of CR rates both before and during labour included a disaggregated sub-sample of low-risk women to enhance understanding of this measurement [53].

The non-inclusion of the specific, care pathway (process and outcome indicators) of low-risk women in most of the general tools identified is, indeed, striking. As noted above, it has been proposed that disaggregated indicator sets should be used to represent the most important risk and client groups and the most essential interventions (be these preventive, curative or caring) for these groups [15]. Low-risk women can be deemed both a key risk and client group for monitoring care, not least due to their numerical predominance on a national level, in addition to other high risk user groups which may inevitably require a greater level of technical intervention, including caesarean section. Tracking the care of low-risk women requires particular tailored indicators throughout pregnancy and intrapartum and post-partum, including an indicator for normal, physiological birth (non-intervention); such measurement was found to be lacking.

Birth requiring no instrumental intervention was only found to be readily measurable through the application of one of the seven general tools identified. Only through the combined analysis of two indicators presented in this tool (‘delivery was vaginal, not caesarean section’ and ‘delivery occurred without instruments’) would it be possible to define the occurrence of non-instrumental vaginal births [55]. In terms of overall indicators, however, vaginal birth (no instruments) was common (see Table 3).

Moreover, it emerged that apart from this measurement, few of the total 388 indicators identified were found to be appraising optimal or positive outcomes more generally (excluding satisfaction, which is examined separately, below). Six of the seven tools were found to capture certain positive outcomes, but with a particular focus on the condition of the perineum (degree of tears/whether intact) and breastfeeding (early initiation/ continuation/support).

It has been argued that one of the criteria for selecting quality indicators in terms of importance and relevance is whether they clarify consensus on the objectives of a system/organisation. The overwhelming focus on technical, clinical interventions amongst the indicators and tools identified suggests that either the objectives of the system/organisation are to provide technical intervention, or current quality indicators are failing to clarify consensus regarding the overall philosophy and objectives of care; key elements of the system are not yet being captured comprehensively.

A focus on adverse outcomes in maternal care appraisal, and the lack of appropriate measurements for monitoring non-intervention during pregnancy and birth in low-risk cases and in the absence of complications has been highlighted in previous reviews [6, 7, 12]. Similarly, the current review identified an emphasis on technical aspects of care rather than a consideration of the systematic measurement of different and varying care processes which may contribute to the mode of birth. This is despite the fact that it has been recognised for some time that the complex inter-relationship between separate elements of care are crucial to outcome [56, 57]. Moreover, the links between care processes and outcomes related to physiological birth and any mix of indicators related to optimal outcomes was also found to be lacking. Findings of the current review therefore confirm the need identified by previous authors [11, 12] for the measurement of factors that contribute to well-being and positive health outcomes, but not only in the area of intrapartum care but across the spectrum of maternity care, including postpartum processes.

Feasibility and the need to focus on the most measurable data as opposed to what might be the most effective/interesting information was an issue which emerged from the literature [41, 58-60], raising the question as to whether normal birth is not being measured due to the difficulties involved in generating consensus and effective tools and indicators for monitoring this mode of birth. At the same time, Chappell et al. [59] made a persuasive case for the development of ‘aspirational’ indicators related to normal or unassisted birth through wider consultation and representation, challenging the argument that certain indicators are problematic to collect. Indeed, it has been argued that the indicators which are made available are closely tied to the objectives and philosophy of the organisation in question, not merely attributable to the availability of data.

Both the professional context and the national or local maternal healthcare models from which care measurement strategies emerge inevitably condition the focus of maternity care appraisal. The construction of indicators is known to depend upon who designs and manages the measurement process, their understanding of the philosophy and objectives of the care system in question, what the aims of this exercise are, and what decisions will be taken as a consequence of the findings [18]. While maternity care is usually provided by a range of professionals from different disciplines whose philosophy of maternity care may range from pathology to normality, and whose definition/s of quality service provision and approaches to measuring this may differ accordingly, current measures are predominantly clinical.

The possible impact of differing perceptions regarding overall philosophy of care and associated assessment of services is reflected in the variability of the tools and indicators reviewed, which exposes the challenge of developing a systematic, transferrable approach of appraisal across different sites. For example, it is difficult to make meaning or practical use of indicators if they are ‘stand-alone’, without a clearly identified target value or indicator direction identified, yet this is a subjective issue. In one of the selected references where a tool for monitoring quality of care was presented, the stated unwanted direction of rate change for epidural analgesia use amongst women who delivered vaginally was ‘decrease’ [41] viewing the rise of interventions as favourable is a debatable objective. This problem is exacerbated by the fact that, as noted in several of the sources scrutinised, there is no consensus about what constitutes good, or optimal care, and therefore no agreed criteria against which progress should be measured in relation to these categories [43].

In some of the literature reviewed it was described how targets and thresholds are a complicated aspect of performance monitoring, requiring national and international development and should ideally reflect universally accepted standards [19]. The issue of consensus is central here. At a local level, the Delphi approach is understood by some authors as a popular, successful method for defining a set of indicators (tools) as it enables individuals in various locations and with different areas of expertise to be included anonymously, often without a physical meeting, which prevents the views of a minority from dominating the group [25]. However, as already noted, the current review found the involvement of non-clinicians in the delineation of tools through the Delphi method to be rare, and it is recommended that the development of data sets includes the involvement of clinically-based health care professionals, maternity health care researchers as well as users of maternal health services [4].

Maternal morbidity and maternal and perinatal mortality rates were validated indicators frequently included in the tools reviewed. The limitations of using such rare events as indicators in developed country settings have been observed, and it has been argued that these measurements lack sensitivity for assessing obstetrical care and, particularly in the case of mortality, such events can be uncontrollable and uncertain, regardless of health intervention [11]. Alternatively, it has also been argued that rare and significant events (adverse outcomes) can provide an important starting point for in-depth studies aimed at understanding key issues relating to the care system [3]. The focus of current measurements of maternity care on rare, adverse events yet the neglect of “normality” and optimal outcomes is, nonetheless, a marked contrast.

Satisfaction is a complex element of maternity care to measure. Studies have shown that in some cases women were satisfied with care even if this does not meet their previous expectations, for example [61]. Two discrete areas of satisfaction emerged from the literature included in this review: one relating to women’s perceptions of the care they received and the other associated with the structure of services, such as the care pathway during the course of pregnancy (Table 4). However, the full findings of the review process related to satisfaction will be analysed in a separate paper, as certain publications emerged from the original literature searches which did not explicitly fit the requirements of the present review (and hence were not included for detailed analysis) but nonetheless provided important, alternative methods for assessing the impact and outcomes of care provision as expressed by women. One tool not included in this current review, for instance, explored women’s perceptions of outcomes and quality of life over an extended period following birth [62], an approach which has important implications for care provision.

We considered a tool to be transferable when the tool containing indicators was well defined and could be replicable in other settings, in other words the same item could be measured in the same way in another (indeed any) setting. Using transferable tools or indicators may help for comparing different settings or organisations and could contribute to the identification of areas requiring improvement. Many of the indicators and tools identified can readily be used at different levels of service provision, to measure the activity of individuals or teams of clinicians, at maternity unit or birth centre level, or at the level of hospital site. They could equally be applied and aggregated, including at national or inter-country level, to provide comparison. While non-European countries were excluded from the review as our aim was to examine tools being used to measure maternity care in European countries, much may be learned from adapting and piloting tools across different contexts. For example, our review elicited a tool, the Optimality Index, which was elaborated in the Netherlands [63], and adapted elsewhere, including the US, and subsequently transferred from there to the UK, for example. This tool provides an innovative method of focusing on the positive side of each indicator, instead of the negative side, combining results into an overall index. It should be noted, however, that all of the tools identified depend on standard administrative procedures regarding registration data and are only as effective as the administration system in operation. In some settings ‘deviant’ registration tendencies have been identified whereby certain interventions that are routinely performed are not registered (e.g. artificial amniotomy, the use of certain medication to stimulate the normal process of labour) or no data are registered in a situation when no technical procedure/intervention has been performed [64].