The design of an evaluation framework for diabetes self-management education and support programs delivered nationally

A participatory approach was adopted to achieve the aims of the research, with two key areas of focus, including the development of: (1) nationally standardized outcomes and indicators; program categories; objectives and measurement tools; and evaluation processes; and (2) quality standards and assessment procedures. The approach and areas of focus are depicted in Fig. 1 and described in greater detail below.

A National Evaluation Team (NET) was established to lead the development and ongoing implementation of the National Evaluation Framework. The team was comprised of research and evaluation experts employed by Diabetes WA, with funding support from the NDSS. The Team met regularly with representatives from Diabetes Australia, who provided project governance and oversight.

An Expert Reference Group (ERG) was established to guide the design, development, and implementation of the Framework. The group included representatives from Diabetes Australia, the Australian Centre for Behavioural Research in Diabetes, The Australian Diabetes Educators Association, Deakin University, Charles Darwin University, a consumer representative, and Agents.

The NET and ERG consulted widely with the NDSS National Services Group, a diverse group of health professionals working with people with diabetes (e.g., Dietitians, Diabetes Educators, and Health Service Managers) representing Agents. The range of health professionals providing diabetes services in Australia is diverse, both organizationally and geographically. Therefore, understanding the viewpoints of a broad range of experts was integral in shaping the design and implementation of the Framework.

Research evidence guided the identification of outcomes, indicators, and objectives for diabetes programs and services that were most likely to lead to favorable outcomes. To support nationally consistent program delivery and evaluation and ensure the best opportunity to improve the outcomes of people with diabetes, outcomes and indicators were adopted from the national consensus position statement previously developed on behalf of Diabetes Australia [20]. Informed by an international evidence-base and extensive consultation, the position statement outlines three key goals for diabetes education, including: (1) optimal adjustment to living with diabetes, (2) optimal health outcomes, and (3) optimal cost effectiveness. Goals 2 and 3 recognize the potential impact of diabetes education on physical health outcomes and optimal cost effectiveness. However, the statement acknowledges that it is challenging to directly attribute these to diabetes education, and thus, attention should primarily focus on Goal 1. Components or ‘indicators’ of diabetes education identified as relating to the goal of optimal adjustment to living with diabetes include: (a) knowledge and understanding, (b) self-management (i.e., diabetes self-care skills and behaviors), (c) self-determination and (d) psychological adjustment.

The identification of outcomes and indicators for DSMES allowed for the categorization of existing diabetes programs based on the indicators targeted within those programs. This required the identification of group-based programs currently delivered throughout Australia, and the collection of information related to those programs. In order to ascertain what programs were being delivered nationally, Agents from each state and territory logged information about the programs they offered, including the population targeted (e.g., people with type 1 diabetes), targeted outcomes, and duration of each program, in a national register held by Diabetes Australia. Members of the NET, in collaboration with state and territory Agent representatives, the National Services Group, and Diabetes Australia then compared the outcomes of each program included in the register with those specified in the National Evaluation Framework (i.e., each program was assessed as to whether it targeted knowledge and understanding, self-management, self-determination, and/or psychological adjustment). In instances where it was not clear whether a program outcome matched one or more of the outcomes specified in the Framework, further clarification was sought from Agent representatives to better understand the nature of that program. Together, the group categorized the programs that had been presented by state and territory Agents of the NDSS according to the number and types of outcomes targeted in the National Evaluation Framework.

The categorization of programs enabled the specification of a three-tiered system of evaluation, allowing for differential program evaluation, dependent on the nature of the programs ascribed to each tier. More comprehensive evaluation processes were ascribed to resource-intensive, higher-cost programs predicted to demonstrate the greatest behavioral impact (Tier 3). Less intensive evaluation processes were ascribed to moderately resource intensive programs predicted to demonstrate a more modest degree of behavioral impact (Tier 2); and the least intensive evaluation processes were ascribed to the lowest cost, most basic programs (Tier 1). Program objectives were then defined for each evaluation tier, and appropriate measurement tools selected to assess outcomes against those objectives. Potential participant burden was weighed against the benefits of intensive program evaluation. Moreover, pragmatic assessment of program objectives had to be applied across the broad range of programs being delivered.

The selection of measurement instruments was guided by a recent review of psychometric tools for diabetes education services [21]. The review investigated instruments to measure commonly targeted outcomes of diabetes education. Instruments were assessed on suitability, validity, reliability, feasibility, and sensitivity. Just three of the 37 tools evaluated met all five criteria. Several other instruments were deemed suitable as they met all but one of the criteria. Based on these findings, potential instruments to measure targeted objectives were selected for consideration by the ERG. Specific instruments were then nominated for inclusion in the National Evaluation Framework. Estimated time required to complete measures ranges from approximately 5-10 min for the most basic level of assessment, to around 20 min for the most complex level of assessment. A combination of online and paper-based methods of dissemination were utilized. States and territories used standardized spreadsheet to collate data for each program before sending it to the NET. Assessments at baseline and post-participation were collected by program educators. For follow-up assessments, measures were mailed to participants and returned in postage-paid envelopes.

A set of quality standards was developed to elucidate processes of care and to ensure that NDSS programs were of high-quality and contained components to elicit key outcomes associated with optimal diabetes self-management. DSMES can have a significant positive effect on health [22]. For example, DSMES are effective in reducing blood glucose levels and improving psychosocial outcomes in people with type 2 diabetes [23‐25] and type 1 diabetes [24]. Such programs have also been assessed as cost effective in empowering people with diabetes to self-manage their condition and mitigate risks of complications, with an incremental cost-effectiveness ratio of USD $5047 per additional quality-adjusted life year, when compared to usual care [26]. Therefore, it was important that the quality standards supported the delivery of programs that were structured and tailored to support the needs of the individual.

The content of the quality standards was informed by existing national and international standards and guidelines [27‐30], state and federal government policy for primary care, chronic conditions, and diabetes [31, 32], and the National Safety and Quality Health Service Standards (NSQHS [33];). Collectively, these standards and guidelines recommend that DSMES should be person-centered (i.e., responsive to the unique needs of the individual), and provided at the time of diagnosis and throughout the person’s journey with diabetes. It is also recommended that DSMES should be accessible, culturally appropriate, and provide appropriate information and education for all people with diabetes, their families, and carers. The importance of strategies promoting active learning, goal setting, and supported decision making is also recognized. Programs should have a written curriculum, standardized facilitator training, and a quality development pathway to ensure fidelity and facilitate quality assurance.

Existing topic specific and comprehensive NDSS DSMES were assessed in accordance with the newly developed standards, to ensure consistent quality across all states and territories. In September 2016, representatives of each Agent assessed their existing DSMES utilizing a user-friendly self-assessment tool, developed to guide the application of the quality standards into practice. The tool is presented in Additional file 1. Agent representatives were engaged as Health Services Managers or Program Managers in their respective organizations. Agent representatives attended a one and a half day training workshop facilitated by the NET. Attendees were familiarized with the quality standards and self-assessment tools, and then had the opportunity to apply the tool to one of their organization’s programs, under the guidance of the workshop facilitators.

To mitigate risk of self-report bias, the evaluations of DSMES conducted by each of the Agents were independently reviewed by two members of the NET. The reviewers then met to discuss the outcomes of the independent assessments and whether individual programs met quality standards. Discrepancies between reviewers were identified and resolved through mutual agreement. Feedback was then provided to the NDSS Agents, including areas for quality improvement. Behavior change outcomes were not expected from tier 1 basic education programs; thus, no formal review of these programs was undertaken. However, the standards provide a general guide for the provision of basic education programs through the NDSS.

As the primary purpose of this work was to improve the quality of service delivered by an organization, the work was exempt from formal requirements for ethical approval and consent in accordance with the Australian Government’s National Health and Medical Research Council policy on ethical considerations in quality assurance and evaluation activities [34].

The approach taken to develop the Framework was collaborative, systematic, and informed by evidence. The establishment and funding of a NET was critical to the development and implementation of the Framework and represents a clear strength of this work. Furthermore, the continued resourcing of a team of specialized evaluation experts ensures ongoing implementation and refinement of the Framework, thereby allowing for continued critical assessment and quality improvement of Australian Government-funded NDSS programs.

The development of the Framework involved extensive consultation with stakeholders to ensure acceptability among NDSS administrators and Agents. The credibility of the Framework and the process was enhanced by the contributions of the ERG who contributed extensive experience in diabetes care. The establishment and implementation of common objectives and quality standards supports nationally consistent delivery of high-quality programs targeting factors known to be associated with improved outcomes among people with diabetes by service providers across all states and territories of Australia. Moreover, the application of standardized evaluation processes utilizing evidence-based instruments for the assessment of outcomes ensures accountability and transparency in service delivery and expenditure of public funds.

Despite the strengths of this work, there are limitations that should be acknowledged. Firstly, greater input from Australians with diabetes could have allowed for the identification of additional factors to be considered when developing the Framework. Secondly, the benefits of rigorous and detailed program evaluation need to be balanced against the availability of resources, accessibility of information, and the need to ensure people who take part in the programs are not overburdened. Based on the available evidence, it is anticipated that the provision of DSMES that improve knowledge and understanding, self-management, self-determination, and psychosocial adjustment will lead to improved clinical outcomes and reduced economic burden at the individual and societal level [20]. However, direct evaluation of these outcomes is not feasible given resources, readily available information, and the interests of NDSS consumers. Moreover, the objectives, quality standards, and measurement instruments were developed or selected based on their suitability to the general population of people with diabetes and may not be suitable to all sub-populations (e.g., culturally and linguistically diverse, Aboriginal and Torres Strait Islanders). The development and refinement of national evaluation tools suitable for application within these populations remains a priority.