Applying the balanced scorecard to local public health performance measurement: deliberations and decisions

The BSC provides a framework for reporting about performance. It also has the potential to be a management tool that aligns strategic direction with internal processes and instils understanding of and engagement in continuous quality improvement. To embrace these latter intentions an inclusive, participatory approach to indicator selection and development was chosen. A BSC panel was struck, comprised of three interested staff members, (i.e. a Director, manager and front-line staff person) from each of the 5 Divisions within the public health branch (i.e. Child and Family Health, Health Protection, Healthy Lifestyles, Infectious Disease Control, Dental and Nutrition) along with representatives from supporting services (e.g. community development, business services, epidemiology, emergency response). Borrowing from an earlier precedent, [10] the panel participated in a facilitated Delphi exercise to discuss and develop indicators to populate the four quadrants.

The BSC potentially has multiple audiences: the local Board of Health and the provincial Ministry who share fiscal responsibility, the staff who plan and deliver programs, the general public who receive public health services and senior administration who set strategic direction. Our primary intent was to demonstrate accountability to our elected Board of Health and to provide them with information to inform decisions about mandate and resource allocation. As a result our audience for this initial BSC was the local Board of Health. The final version of the BSC was widely disseminated to staff, senior administrators, the provincial Ministry of Health and Long Term Care, federal and provincial agencies of public health and peer health units.

Some of the initial discussion among the panel members focused on the level of performance measurement. A main question that arose was 'Should the scorecard report performance at the overall health unit or at the division -specific (i.e. programmatic) level?' The original intent of the project was to develop an efficient scorecard for the health unit by populating a "dashboard" with a few key indicators aligned either with strategic priorities or particular themes, such as health inequity. However, we soon discovered that the participatory process involving representatives from all divisions and from three levels of staff invited and supported the development of a much more comprehensive and larger number of indicators that mostly described performance at the program-specific level. It became evident that the various program areas were not familiar with the activities in other areas and different programs began to compete for representation of their activities in the scorecard. Consequently, in order to build trust among different program areas and to garner sustained buy-in for the process and product, we decided to make the first scorecard comprehensive and include a large number of indicators, anticipating that over time and in future iterations the number and meaningfulness of indicators would be refined according to their relevance, utility, reliability and alignment with strategic priorities.

Over a period of two years, the team was able to operationalize the BSC, develop performance measures and raise awareness about continuous quality improvement among the health unit staff. The BSC implementation team consisted of an epidemiologist and a continuous quality improvement co-ordinator, who both dedicated 0.2 FTE over the course of the project. The office of the Medical Officer of Health championed the exercise. A consultant familiar with the BSC [10] was hired to initiate the process. Orientation sessions were held with the staff to educate them about the meaning of the BSC and about the health status of the residents within the catchment area for the health unit. The consultant led a series of four half-day facilitated exercises with the panel to rank and select indicators for the health determinants and status, community engagement and integration and responsiveness quadrants. To develop indicators about key activities, and the level of service activities and dedicated resources, the BSC team facilitated discussion with each program area. The panel and program areas submitted their selected indicators and provided text to describe their key activities. The result was a 75 page report containing 46 tables of indicators posted on the York Region website. [11].

According to the ICES document [8], this quadrant contains measures traditionally found in community health status reports, such as trends in disease mortality/morbidity and health behaviours. The primary purpose is to identify the need for public health services. Ultimately this quadrant should contain three types of measures: determinants of population health, population health status measures and population health intervention impact measures. In Ontario, a lot of these data are routinely collected through provincial and federal surveys and made available to health units. Since the number of indicators is potentially very large, the panel was asked to select key indicators that described the burden of illness and the key demographic and social features that characterize York Region as unique. To assist the panel in choosing indicators for this quadrant the following guiding questions were posed: Whom does the York Region Public Health Branch serve? What are the health needs of the residents? Where does the burden of illness lie?

According to the ICES document [8], the primary purpose of this quadrant is to report the amount of resources and services delivered in the health unit so that comparisons may be made across health units. The types of indicators that should be reported include indicators of financial resources in terms of per capita and total costs, the number of services delivered and the number of staff allocated to different services. To assist in indicator development for this quadrant the following guiding questions were posed: What are the key activities of the health unit? Who is the target population for these activities? Where is the bulk of our resources going?

According to the ICES document [8], the views and involvement of the community who utilize local public health services and are impacted by public health policies need to be included as part of a public health performance report. Understanding the views of the population a program serves is a fundamental component of accountability and can improve the way services are delivered. Client satisfaction surveys are one of the traditional ways to invite input and feedback. Ideally community engagement should go well beyond client satisfactions surveys and encompass community and partner involvement in program planning, evaluation and service delivery (Table 2). Questions to guide the development of indicators for this quadrant included: How is the health unit engaging the community? How does the health unit ensure community input into public health planning and service delivery?

This fourth quadrant relates to the structural capacity of public health to keep it well integrated into the healthcare system as well as the capacity to continually transform services in response to evolving needs, issues and evidence (Table 2). This is achieved through the development of partnerships with local health service providers and community agencies. These partnerships have a mandate that impacts health determinants, through a commitment to research and academic pursuits, and through a corporate emphasis on continuing professional development and quality improvement. Questions to guide indicator development in this quadrant included: How are prevention, promotion and protection services integrated into the local healthcare system? How does the health unit identify and respond to emerging issues and evidence? How are continuing professional development and competency ensured?

Criteria for indicator development were established (See additional file 2 for criteria for indicator selection) and a data dictionary built to record a definition for each indicator. Responsibility for data collection and verification was distributed throughout the various program areas and involved the frontline staff who directly delivered the programs. Each program had a BSC champion, usually the same individual who participated in the Delphi panel, who worked with an epidemiologist to collect and verify the data.

There was a lot of initial deliberation about whether the BSC should attempt to link the quadrants to "tell a story". A hypothetical example would be exploring the relationship between the rate of motor vehicle collisions and high alcohol use in the health unit (health status and determinants quadrant). Currently the health unit dedicates 0.1 FTE to this issue through the substance abuse and injury prevention team (resources and services quadrant). There may be effective engagement between the health unit and the school board in promoting responsible drinking. The health unit may not have developed strategic alliances with the police (community engagement quadrant) and there is emerging evidence that reducing the legal alcohol limit for driving and promoting designated drivers programs is associated with a lower motor vehicle collision rate (integration and responsiveness quadrant). Based on this assessment, a decision to enhance resources in this program area may be made together with the establishment of a partnership with the police department to advocate for a change in legislation.

As this example illustrates, it is fairly easy to tie the quadrants together at the program level, but this is more difficult to accomplish at the overall health unit level. The above example does not touch on the numerous activities in the other mandated areas (Table 1). To attempt to weave stories around each activity for the purposes of reporting would result in information overload for the BSC report. Consequently our first BSC report was comprised of tables of indicators with little text to provide narration or interpretation. We decided to rely on the reader to interpret and draw conclusions based on the performance indicators, particularly when the health unit's performance was poor. However, program areas were encouraged to begin linking the quadrants as they moved forward with 2008 program planning and implementation on the basis of their reported 2007 performance.

Our initial report contains baseline measures that present a snapshot of the health unit's performance. Indicators are more meaningful when comparators are used. There are many options for comparators. One of the best options is to compare with oneself over time and look for trends in improvement. We intend to do this in future reports by comparing the baseline data with subsequent years. Other possible sources of comparison might be peer health units, as constructed by Statistics Canada [12], or provincial averages. Peer groups are better comparators than provincial averages since these health units share similar socio-economic characteristics that in part determine health status. However, since York Region is one of the first health units to develop indicators under the four quadrants suggested by ICES, comparable indicators from peer health units are not presently available. In order to encourage uptake of this framework among our peers, York Region invited members of neighbouring health units to participate as observers in the Delphi panel process.

The Board of Health received the BSC report at Committee and Council and requested a report back in six months explaining the action taken by the health unit in light of its performance. To assist with this, the BSC team facilitated focus groups with the various program areas and offered guidance questions such as: Which of these performance measures prompt you to change the way you do business? Each Division has been asked to identify three measures and to explain their impact on program planning and delivery. After reviewing the 46 tables, the indicators highlighted by program areas for response are those that most resonate with them. Through this selection process we will further identify common themes and use them to establish and inform strategic direction and future programmatic priorities. The BSC process is iterative and plans to refine and improve the selected indicators and our performance is ongoing.

After completing our first BSC, we conducted a process evaluation with the staff who had been involved with performance measure development. In general the staff experienced the development of the scorecard as an informative and valuable exercise, but questioned the utility and relevance of some of the indicators as aids in program specific decision making. They recommended that a process be established to revisit and refine indicators and that more interpretation of indicators be provided in the text of the report. These finding are quite similar to the results reported by the Toronto-based University Health Network in the early phases of implementing their BSC [7]. Overall the University Health Network staff felt the University Health Network was moving in the right direction with its BSC but improvements were needed to make it more responsive and relevant to the University Health Network staff.

York Region Public Health Branch is in the process of producing our second report card based on the original indicators. As we revisit them, it is clear that some are not reliable or relevant and require refinement or removal. The program areas have been tasked with identifying variances from baseline and offering explanations, which will be provided in our second scorecard and presented to our governing Board.