Combining patient, clinical and system perspectives in assessing performance in healthcare: an integrated measurement framework

The first set of frameworks conceptualise performance to be relating inputs, activities, outputs and outcomes. These models look at flows of production [37] and logic models [38, 39]. These models often build on the structures, processes and outcomes of healthcare categorisation proposed by Donabedian [40] where structure describes the settings where care is delivered and the physical, human and financial resources required. Process refers to patient and practitioner activities involved in giving and receiving care and outcome describes the effects of care in terms of changes in health status, patient’s knowledge and/or behaviour; and patient satisfaction.

The second set of frameworks are more theoretically based and conceptualise performance in terms of functions or roles within systems. These models draw on Parson’s theory of social action [41] where performance is seen as achievements in the functions of adaptation, goal attainment, production and values maintenance [42, 43]. In this paradigm, the alignment and balance of these key functions is the primary concern, rather than the actual relationships between inputs and outputs.. It defines adaptation as the ability to secure resources, shape structure, systems and processes according to community needs. Goal attainment is here defined as achievement of targets relating to population health and equity. The production function relates to the quantity and quality of services. Finally, the value maintenance function refers to how systems maintain their capacity and continually develop and evolve.

The third set of frameworks assess performance in terms of societal goal achievement. These models are conceptually agnostic and depend on the definition and codification of a set of values, standards or objectives against which performance is to be judged. These models are grounded in the organisational literature on scientific management, goal setting [44] and management by objectives [45], and consider socially determined goals as core to performance and the strategic orientation required to assess performance. Within this paradigm, assessment of performance involves evaluating the extent to which goals are realised or achieved.

Building upon the conceptual antecedents in existing performance frameworks described above, our framework integrates all three of these approaches (See Table 1 and Additional file 2). This new framework builds on a logic model base – but moves away from simplistic ‘counts’ of needs, resources, activities or outcomes - recognising that increases or decreases in any of these do not necessarily correspond to an improvement or deterioration in performance. It also incorporates functional aspects of healthcare systems with derived constructs such as accessibility and effectiveness and safety, and encapsulates goals of the healthcare system such as equity and impact. Crucially, it acknowledges complexity and dynamism and considers performance using a perspective that relates measurable elements to each other (e.g. patients’ needs and expectations with activity) allowing for more meaningful judgements to be made about performance in context and across different time horizons.

The integrated framework proposes five measurable constructs (patients’ needs and expectations; healthcare resources and structures; receipt and experience of healthcare services; healthcare processes, functions and context; and healthcare outcomes) which are generally used to populate logic model approaches. These five elements are the aspects of healthcare performance that can be directly measured through quantitative data collection systems or approaches.

Building on these measurable constructs and encapsulating a functional approach, the framework identifies 10 derived constructs of performance (coverage; accessibility; appropriateness; safety; effectiveness; productivity; efficiency; adaptability; sustainability; resilience). The framework also incorporates two overarching derived constructs that relate to goal achievement (population health impact; equity) – recognising their importance in many healthcare systems. Equity is an overarching construct that relates to the population distribution of other constructs, such as access, appropriateness and effectiveness. Similarly, impact is an overarching construct determined by the cumulative contribution of all other constructs. While these are the 12 key constructs of performance, their measurement requires the combination of the previous five measurable constructs as they cannot be measured directly (e.g. to derive efficiency measures, quantification of resources expended is assessed in relation to quantification of outcomes achieved.) Figure 2 illustrates the relationship between measurable and derived constructs of performance.

Table 2 summarises the inclusion of constructs in selected frameworks from the literature. From this table we can see very few frameworks are theoretically based but those that are tend to be more comprehensive in construct inclusion.

The notion of a measurement system has been present in the broader management literature for 50 years [48]. Much of this work clearly differentiates measurement that is for system or performance management purposes from measurement that is for benchmarking and improvement purposes [49]. In health, this is a key distinction with Ministries or Departments of Health often focused on performance management while agencies mandated to secure quality improvement and clinical innovation are more focused on identifying areas of variation in the delivery of healthcare to patients and ways to address them.

The proposed framework can support more comprehensive assessment and also provide transparency about decisions regarding which aspects of performance are measured and reported. Its principal purpose is measurement – although it has clear relevance for quality improvement and for policy.

Healthcare system performance reporting efforts to date have generally featured a preponderance of the simple measurable elements (e.g. utilisation) and a lack of measurement of derived constructs (e.g. organisational functioning). In populating frameworks with metrics, indicators and data, most systems have used a pragmatic approach, focusing on readily available measures and aspects that are relevant to specific policies and contexts, but remain mute about important aspects of performance. In some cases, this has led to sets of disparate indicators that are unable to provide a comprehensive picture of performance. There is a growing recognition that performance measurement efforts should move beyond opportunistic and piecemeal approaches to indicator selection and towards deliberative filling of information gaps [50].

Conceptual clarity does not mean that a handful of measures will do. Over-reliance on a very small number of metrics infers a strong correlation of performance across all constructs, whether measured or not. However, this has not been supported empirically and a number of studies report only weak correlations between different metrics [51‐53]. In other words, performance in one construct of healthcare is not necessarily informative about performance in others [54].

Overall, we found that most performance frameworks lack a theoretical basis. The lack of theory has perhaps led to a proliferation of measurement scorecards and frameworks that are based on empiricism. This empirical base has meant what is measurable features in many performance framework efforts. More theory or conceptual clarity may mean more parsimony – we don’t need every permutation of measurement to get an understanding of the measurement balance between aspects of performance – what we need is a well-constructed conceptually sound model that can be used for a range of purposes - measurement, quality improvement and policy – and in a range of contexts.

Some frameworks currently in widespread use provide only a partial picture of performance – albeit a critically important one. For example the Institute for Health Improvement’s hugely influential “triple aim” [32] encapsulates elements of appropriateness (experience of care), efficiency (per capita cost) and impact (population health) but it provides a partial view of performance – and while its focus attracts attention with its simplicity, it can be considered to be somewhat reductive – overlooking essential elements of performance, such as accessibility or equity. Simple models can resonate but the trade-off is a loss of sensitivity to complexity and measurement. Paradoxically – more conceptually grounded frameworks may appear piecemeal – because they may have empty categories – our ability to quantify with data may not be advanced sufficiently to fill all the conceptual categories. It does however allow for future proofing and guide data collection and analysis efforts.

Similarly, there is a clear tension between trying to summarise whole health system performance in a ‘single number’ measure and juxtaposing fragmentary metrics that inform about parts of the system [1]. Trying to integrate measures of performance into a single score is likely to prove to be a meaningless task. Embracing the complexity of performance with a framework that enables a clearer assessment and understanding of which data can inform the constructs of performance to be assessed, and how they relate to each other, is more productive than trying to oversimplify performance. For example, understanding trends in the needs of population allows better understanding of why coverage may be decreasing; and reflects on adaptability. In assessing overarching constructs such as equity – sophisticated performance measurement approaches are able to reveal how disparity may be explained by another construct, such as coverage (e.g. unwarranted variation in receipt of surgery by socioeconomic status, as a result of resource allocation that does not match needs for surgery).

The framework provides a clear set of principles for measuring performance. While it is represented by a simple visualisation, it encapsulates considerable complexity and requires substantial effort to be populated with a comprehensive set of indicators. This breadth means that the framework may not be best suited to target efforts or focus a system on key current problems. The metrics highlighted in the paper are quantitative in nature however, ‘soft intelligence’ or experiential evidence is increasingly considered to be an important additional source of performance information - capturing perspectives of actors and sensitivities to context [55]. Because we have distilled the variety of terms used in many frameworks, policy makers and clinicians might be unsettled not to see some classic constructs such as ‘quality’ that have previously been promoted. The review adopted neither a systematic review nor a scoping review method. This was because its purpose was to collect the range and distribution of concepts that have been used to measure healthcare performance and their theoretical basis and the more conceptually grounded approach of Jabareen better aligned with that purpose. Having said that however, the search phase was extensive and seminal reports and papers have been used in a snowball approach to access other key references and ensure an exhaustive review. Finally, because of the salience of performance measurement mostly in high income countries, the framework produced is mostly applicable to these settings and its transposition to other healthcare settings remains to be assessed.