Background
Evaluation: methodological challenges and opportunities
Political context
A framework for assessing policy success and failure to aid in the selection of policy levers
Evaluative Measure | Evidence of Success | Evidence of Failure |
---|---|---|
Original objectives | Objectives achieved | Objectives not achieved |
Target group impact | Perceived positive impact | Perceived negative impact |
Results (i.e. outcomes) | Problem improvement | Problem worsening |
Significance | Important to act | Failing to act |
Source of support/opposition | Key groups support | Key groups oppose |
Jurisdictional comparisons | Leading or best practice | Someone else is doing better elsewhere |
Balance sheet | Benefits outweigh costs | Costs outweigh benefits |
Level of innovation | New changes | Old response |
Normative stance | Right thing to do | Wrong thing to do |
Aims of this study
Methods
Case study and documents
Analysis
Results
Evaluation scope and approach over time
First Plan | Second Plan | COAG Plan | |
---|---|---|---|
Intended Aim |
O; I-Focus on reform approach rather than outcomes
N- Appropriateness of initiatives (normative stance) |
O; I; R- Focus on reform approach as well as outcomes
N- Appropriateness of initiatives (normative stance) |
O- Focus on outcomes
R- Performance against 12 population indicators |
Actual Focus |
O-Approach
R- Problem improvement
TG- Consumer perspectives |
O; I- Approach
R- Problem improvement
TG- Consumer perspectives |
O- Realisation of original objectives
R- Problem improvement |
Data Sources | • Repurposing of available national datasets • In-depth case studies • National stakeholder survey and consultations • International expert commentary | • Repurposing of available national datasets • National stakeholder survey and consultations • International expert commentary • Review of mental health by Mental Health Reference Group | • Repurposing of existing and new national datasets to report on 12 population indicators • Jurisdictional data (budget allocations and funding commitments) |
First plan
Second plan
COAG plan
Evaluation of policy levers over time
First National Mental Health Plan | ||||
---|---|---|---|---|
Policy Lever | Policy Objective | Proposal(s) | Success | Failure |
Organisation |
RCN
Involve consumers/carers in policy review and formulation | Formalise the inclusion of consumers and carers within working committees |
O- Met to some degree
R- Improvement in formalised participation |
O- Met for only half of public sector organisations Not translated to private sector
R- Participation but not leading to intended ‘good’ outcomes in terms of respect
TG- Public/political dissatisfaction |
SS
Mainstream mental health service management
| Merge mental health into mainstream health management |
O- Substantively met
R- Mainstream management arrangements adopted across all jurisdictions | ||
SS
Shift acute beds to general hospitals | Shift psychiatric beds from stand-alone facilities to general hospitals |
O- Substantively met
R- Decrease in use of hospital-based services Funding shifted to community service sector |
R- Resource reallocation and service availability is variable across jurisdictions Little adoption of population-based funding model to facilitate resource transfer
TG- Community and public dissatisfaction High reports of areas of unmet need | |
SS
Improve access to community crisis services
| Increase ambulatory workforce |
O- Met
R- Significant increase in ambulatory workforce | ||
SS
Improve coordination of care across service providers | Introduce case management system |
O- Partially met – system introduced |
R- Under-utilisation of case managers service Little measurable improvement in continuity of care
TG- Community and public dissatisfaction | |
Regulation |
HR&CA
Reduce discrimination and stigmatisation of mental health consumers | Review anti-discrimination legislation |
O- Substantively met
R- Improvement in anti-discrimination legislation. |
TG- Public/political dissatisfaction |
HR&CA
Adhere to UN Resolution 9B and Mental Health Statement of Rights and Responsibilities
| Review consumer rights and responsibilities as per State/Territory and Federal legislation |
O- Substantively met (or in progress)
R- Improvement in State/Territory and Federal legislation | ||
R&SA
Simplification of cross-border treatment
| Identify and remove cross-border anomalies in diagnosis and treatment |
O- Not met
R- No change in cross-border anomalies
TG- Low impact | ||
SQ&E
Improve service quality and standards | Introduce nationally consistent standards for mental health care |
O- Met
R- Standards adopted across all jurisdictions Quality assurance programs introduced in some jurisdictions |
O- Considerable ongoing development work required to see Standards fully accepted and implemented across all jurisdictions | |
SQ&E
Introduce independent evaluation body | Introduce an independent evaluation steering committee |
O- Met
R- Independent evaluation steering committee and National Mental Health Commission established | ||
SQ&E
Ongoing accountability and evaluation | Publish progress within annual Mental Health Reports Develop a National Mental Health Information strategy and minimum data set |
O- Substantively met (at least for inpatient services)
R- Accountability standards used as an example for other public policy |
O- Not met for community based services (no minimum data set)
R- No qualitative measure of ‘accountability’ No outcome measures yet recorded to evaluate intervention effect Routine assessment established in very few mental health centres | |
SS
Improve coordination of care across sectors | Review of interagency protocols |
O- Substantively met |
R- Under-utilisation of case managers service Little measurable improvement in continuity of care Not translated to local service level
TG- Community and public dissatisfaction | |
Finance |
R&SA
Increase mental health budget | Increase recurrent mental health spending for Federal and State/Territory Governments |
O- Substantively met
R- Funding increases observed |
R- Variable increase in funding across jurisdictions |
R&SA
Increase community-based and general hospital funding
| Increase community-based and general hospital funding for mental health |
O- Substantively met
R- Funds shifted to community service sector Significant increase in non-institutional spending | ||
R&SA
Modify funding allocations for mental health
| Review Medicare Agreements |
O- Substantively met
R- Agreements more clearly outline bilateral funding arrangements | ||
SQ&E
Ensure fiscal accountability for mental health spending | Create a separate budget for mental health |
O- Met |
R- Funding continues to be allocated on historical basis Mental health sector-specific outcome-based funding tools remain underdeveloped and under-utilised | |
Community Education |
HR&CA
Improve mental health literacy (general public) | National Community Awareness Program |
O- Partially met
R- National community awareness program implemented |
O- No substantial benefit achieved
R- No measurable change in attitudes
TG- Public dissatisfaction Approach not appropriate for minority groups No opportunity for local groups to coordinate promotional activity with the national campaign |
Second National Mental Health Plan | ||||
---|---|---|---|---|
Policy Lever | Policy Objective | Proposal(s) | Success | Failure |
Organisation |
RCN
Formalise consumer/carer consultation | Increase public and private sector organisations with formal consumer/carer consultation |
O- Substantively met |
R- Low level of genuine involvement or consultation
TG- Public/consumer dissatisfaction (perceived change as insufficient) |
R&SA
Increase early intervention services for youth | Provide specialist centres for youth early intervention, including assessment and treatment |
O- Partially met
R- Specialist centres and early intervention services increased |
R- Not all groups’ needs met
TG- Not all groups’ needs met | |
R&SA
Improve service provision for special needs populations
| Develop new specialised service models |
O- Not met
R- Interventions underdeveloped Needs of CALD population not met Lack of service integration
TG- Consumers not satisfied with level of change achieved/scale of impact | ||
R&SA
Improve mental health curricula for Indigenous health workforce | Improve mental health curricula for Indigenous health workforce |
O- Partially met
R- Mental health curricula and culturally appropriate service models developed |
O- Not met
R- Shortage of health workers Ongoing need to improve links with mainstream services | |
SS
Shift acute beds to general hospitals
| Relocate beds in stand-alone facilities to general hospitals |
O- Substantively met
R- Beds from stand-alone facilities relocated to general hospitals | ||
SS
Improve coordination of care across service providers | Formal protocols and agreements to support continuity of care |
O- Partially met – system introduced |
R- Under-utilisation of case managers service Little measurable improvement in continuity of care
TG- Community and public dissatisfaction | |
Regulation |
SQ&E
Increased accountability for reform outcomes | Develop and apply new outcome measures |
O- Partially met |
R- Mental health workforce remain reluctant to participate in routine outcome monitoring
TG- Consumers dissatisfied with progress |
SQ&E
Improve service quality and standards | National Standards for Mental Health Services |
O- Partially met |
R- Mental health workforce remain reluctant to participate in routine outcome monitoring
TG- Consumers dissatisfied with progress | |
SS
Improve coordination of care across sectors | Review interagency protocols to support continuity of care |
O- Partially met |
R- Lack of accountability Inadequate progress | |
Finance |
R&SA
Increase Federal and State/Territory expenditure on mental health | Review allocations under the general health budget for Federal and State/Territory Governments |
O- Partially met |
R- Size of sector did not increase Variable spending across jurisdictions |
R&SA
Increase the size of community-based service sector | Grow 24-h staffed community based residential services Increase government spending on community services |
O- Partially met |
R- Low increase in services compared with closure of hospital services
TG- Consumers report dissatisfaction with improvements | |
Community Education |
HR&CA
Reduce discrimination and stigmatisation of mental health consumers | Review media portrayal of mental illness |
O- Partially met
R- Review conducted Strategies implemented
I- Use of innovative approach to engage general public in monitoring |
R- No outcome measures collected to compare with baseline |
HR&CA
Improve mental health literacy (health workers) | Provide mental health training to frontline workers |
O- Partially met
R- Education materials developed Strategies implemented |
R- Consumers and carers continue to experience stigmatisation and discrimination (by professionals both inside and outside health sector) | |
HR&CA
Improve mental health literacy (general public) | Frontline service providers to distribute mental health brochures to patients Educate school children about mental health |
O- Partially met
O- Information material availability and distribution improved Community telephone survey suggested that people consider mental health to be a serious problem and to be more prevalent than in previous decades |
O- Little or no substantial improvement in health literacy
R- Materials do not suit all groups (i.e. exclude CALD and other minority groups) | |
Payment |
SS
Ensure better links between primary and secondary providers
| Introduce new funding models Review existing MBS items |
O- Substantively met | |
R&SA
Improve service provision in rural/remote areas | Introduce new specialised funding models |
O- Partially met
R- Increase in dedicated service programs Improvement in recognition of special needs of this group |
O- Most of NMHS objectives tailored to metropolitan areas
R- Service gaps and workforce shortages in rural and regional areas remain |
COAG National Action Plan | ||||
---|---|---|---|---|
Policy Lever | Policy Objective | Proposal(s) | Success | Failure |
Organisation |
R&SA
Improve youth mental health services
| Review and consolidate existing youth mentoring and early intervention programs into a single program Initiate new youth early intervention projects |
O- Substantively met | |
R&SA
Offer increased support for carers and families of people with mental illness
| Introduce a new Family Mental Health Support Service Increase mental health respite services for carers |
O- Substantively met
R- Increase in respite places Increase in education and formal support services | ||
R&SA
Greater employment and day-to-day living support for the mentally ill
Indicators 9 and 10
| Increase places in day-to-day living support programs and Personal Support program Increase number of Personal Helpers and Mentors Introduce Disability Employment Services group |
O- Substantively met Disability Employment Services Group introduced
R- Increase in funding and service provision |
R- Employment rates remain low among the target group | |
R&SA
Increase mental health workforce
| Increase the number of supported places in university mental health degrees, particularly to Indigenous students Specific funding targeted towards increasing Indigenous health workforce |
O- Substantively met
R- Increase in supported places and scholarships for formal training | ||
SS
Improve and integrate drug and alcohol services within broader mental health services
| Provide additional funding to drug and alcohol service providers |
O- Substantively met
R- Funding increased, grants awarded to NGOs | ||
R&SA
Increase service coverage in rural/remote areas
Indicator 5
| Introduce a 24-h 7 day mental health telephone service Increase web-based mental health resources |
O- Substantively met Flexible service delivery modes introduced (telephone, online services) | ||
SS
Improve coordination of care
Indicator 7
| Introduce step-up and step-down community facilities Utilise community coordinators |
O- Partially met
R- Principles and implementation guidelines developed |
R- Variable progress in jurisdictions Lack of consistent approach or outcome Lack of accountability | |
Regulation |
SQ&E
Increase consultation between State/Territory and Federal Governments
| Establish COAG Mental Health Groups in each jurisdiction |
O- Substantively met | |
SQ&E
Increased accountability for reform outcomes
| Publish official progress reports annually |
O- Substantively met | ||
Payment |
R&SA
Improve service provision in rural/remote areas
| Use of flexible funding models to improve access to allied and nursing mental health services in rural and regional areas |
O- Substantively met | |
R&SA
Increase health workforce in rural and remote areas, particularly mental health nurses
Indicator 5
| Introduce Mental Health Nurse Incentive Program Introduce flexible employment schemes for rural and regional areas |
O- Substantively met Incentives and flexible employment schemes introduced | ||
SS
Improve links between primary and secondary providers
Indicator 5
| Introduction of new MBS items to support referral between health practitioners |
O- Substantively met | ||
Community Education | R&SA
Review mental health content in tertiary health degrees
| Review mental health content in tertiary health degrees |
O- Substantively met
R- Final project reports identify an increased focus on mental health in both theoretical and clinical subjects |