Background
Methods
Design
Sampling strategy
SME discipline (a priori minimum sample size) | ||||
---|---|---|---|---|
Health policy/strategy or advocacy (n ≥5) | Clinical practice and/or clinical/health services research (n ≥6) | Health service or program delivery (n ≥5) | Consumers (inclusive of carers) (n ≥5) | |
Inclusion Criteria | ● At least one SME per state of WA, VIC, NSW ● At least one SME reflecting each of the sub-categories below ● Experience at a seniora level in musculoskeletal and/or chronic disease (that implicitly includes musculoskeletal health) health policy or advocacy for at least 1 year ● Awareness of Australian musculoskeletal MoCs as defined by Briggs et al. [7] | ● At least one SME per state of WA, VIC, NSW ● At least one SME reflecting each of the sub-categories below ● Experience in delivery of clinical care for people with musculoskeletal conditions at a senior practitioner levelb for at least 5 years; or at least 5 years experience in undertaking clinical and/or health services research in musculoskeletal health at a senior levelb with evidence of peer-reviewed publication(s) in the area ● Awareness of Australian musculoskeletal MoCs as defined by Briggs et al. [7] | ● At least one SME per state of WA, VIC, NSW ● At least one SME reflecting each of the sub-categories below ● Experience in health service or program delivery, coordination, management or funding related to musculoskeletal healthcare and/or chronic disease (that implicitly includes musculoskeletal health) for at least 1 year at a senior levelc
● Awareness of Australian musculoskeletal MoCs as defined by Briggs et al. [7] | ● At least one SME per state of WA, VIC, NSW ● Person lives with a chronic musculoskeletal condition (at least 5 years) or cares for someone with a chronic musculoskeletal condition (at least 5 years) ● Awareness of Australian musculoskeletal MoCs as defined by Briggs et al. [7] |
Sub-categories | ● State or federal government health policy, strategy or MoC development ● State or federal government policy, strategy or MoC evaluation ● State or federal government health workforce policy or strategy ● National-level advocacy for musculoskeletal healthcare | Clinical disciples represented ● Physiotherapy ● Rheumatology ● General practice ● Endocrinology ● Pain or rehabilitation medicine ● Community pharmacy | ● Private health insurance industry ● Health service management or coordination at tertiary setting ● Health service management or coordination at primary care setting | ● At least one female ● At least one male |
Ethics, consent and permissions
Development of interview schedule
Data collection
Data analysis
Results
SME discipline category | Current profession | N (%)a
| Mean (SD) years in current professional role |
---|---|---|---|
[range] | |||
Health policy/strategy or advocacy | Health policy and/or program development for chronic disease | 14 (51.9) | 12.6 (6.5) |
[4–30] | |||
Public health system funding | 4 (14.8) | 6.3 (2.9) | |
[4–10] | |||
Advocacy and/or consumer representation for musculoskeletal health | 4 (14.8) | 14 (12.4) | |
[4–32] | |||
Health workforce policy/strategy | 8 (29.6) | 12.1 (6.6) | |
[3–25] | |||
Clinical practice and/or clinical/health services research | Clinical practice in musculoskeletal healthcare (currently active) | 8 (29.6) | 26.3 (4.7) |
[20–32] | |||
General practice | 2 (7.4) | ||
Endocrinology | 1 (3.7) | ||
Rheumatology | 2 (7.4) | ||
Pain/rehabilitation medicine | 1 (3.7) | ||
Community pharmacy | 1 (3.7) | ||
Physiotherapy | 1 (3.7) | ||
Clinical practice in musculoskeletal healthcare (currently inactive) | |||
General practice | 1 (3.7) | ||
Physiotherapy | 3 (11.1) | ||
Clinical and/or health services research in musculoskeletal healthcare | 4 (14.8) | 21 (7.2) | |
[15–29] | |||
Tertiary education of healthcare professionals | 7 (25.9) | 15.7 (8.3) | |
[7–30] | |||
Health service or program delivery | Health service delivery, coordination or management related to chronic diseases | 10 (37.0) | 14 (8.0) |
[5–30] | |||
Private health insurance | 2 (7.4) | 1.5 (0.7) | |
[1-2] | |||
Consumer | Consumer | 5 (18.5) | 31.8 (14.6) |
[15–50] | |||
Other | Other (health economics; primary care system change and capacity building; development and evaluation of healthcare models | 3 (11.1) | 20.0 (5.0) |
[15–25] |
Aspects of musculoskeletal MoCs that are important to stakeholders’ work
“…we should all be evidence-based practitioners and we should be all trying to implement evidence into what we do, and so I see models of care and evaluation of Models of Care as an integral component of that” (SME 8)
MoC as platform for change
-
the development and consultation processes
-
a compelling case for change
-
a clear outline of objectives, core elements and anticipated outcomes, along with performance indicators to evaluate outcomes
-
a description of how services and resources would be delivered.
“That the end product can demonstrate that its had the input from the right professions… it’s very much having a representative group…you know, multi-disciplinary, as well as having the people with influence, the colleges, the association, that kind of stuff. And involving them from the beginning. So, not just bringing them to review something at the end. It’s actually they’re involved in drafting and developing.” (SME 7)
“Well I think the bad thing is, as a community pharmacist, a Model of Care is really not important to my work at all - it should be, but it’s not. And I think that’s because as a community pharmacist I had absolutely no awareness of [its] existence…” (SME 15)
“I would be interested in…the argument for the evidence in which the Models of Care are based and how that fits in the service delivery models both in terms of an economic perspective and also an integrated sort of model.” (SME 2)
“It’s got to be safe, so the safety elements have to be considered strongly. It must either improve or at least maintain the quality of care that’s already likely one would hope improve, but certainly not in any way diminish. It has to ensure that the effectiveness of care is going to be improved or again at least maintained…It should improve the patient journey and the patient experience. It should increase the efficiency from a service or system perspective.” (SME 5)
“One of the most important things is having a document which is actually going to be understandable to a person outside of that special [musculoskeletal] area…that specific managers understand it as well as clinicians [and] consumers.” (SME 13)
“And they’re [General Practitioners] not interested in pages of documentation, they’re not interested in the eighty page Model of Care document; they’re interested in two pages… Give it to me simply; have you got an assessment for me with my package that I can use, and how do I do it, and who do I refer to? Keep it really simple with a link to the more detailed document.” (SME 22)
“… as a clinician I think the most important things are that it [MoC] sets down…the way in which care will be delivered for people with musculoskeletal conditions…so that there’s hopefully a clear pathway with services available and accessible that need to be delivered in line with the model of care.” (SME 14)
“I think probably the key point is access…consumers need to be able to access these opportunities easily with the minimal cost and reduced waiting times, and to be able to have a service that gives them as much multidisciplinary care as possible.” (SME 24)
MoC providing practical guidance to implement and evaluate services
“I’d like to see whether a pilot study has been run in a small number of sites of different, maybe, locations and environments to make sure that the Model of Care is both effective, but is also generalizable.” (SME 12)
“…particularly when you see a preliminary evaluation– if there’s pilot studies done and we’ve, you know, had that advantage in New South Wales obviously to do some pilot studies and then present the results from the pilot studies, then I think that’s really important because they [stakeholders] say “Oh, I want that too…””(SME 20)
“I think in advance it’s also helpful to make sure that the evaluation that you’d like to do at the end is thought about…so what on a day do you want to capture and systems that are required for that, and the person burden, but also resource burden in developing and implementing that…” (SME 10)
“…there’s a step between the Models of Care and then what it is that you plan to implement. The work we did…was trying to look at how you turn the Model of Care into discreet programs. I think there needs to be a lot more work of that nature. The real lesson…is that those processes then need rigorous project methodology and do need very structured implementation targets.” (SME 19)
Perceptions of standardised evaluation frameworks to judge ‘readiness’ and ‘success’ of musculoskeletal MoCs
“I think an “evaluability assessment” or “readiness assessment”, whatever it’s called, is really, really important… [there have been] so many evaluations where you’ve gone in to do it and not only has it not been ready to implement, the enablers aren’t in place to support a new Model of Care, staff perceptions are quite obtuse, but there might not even be a baseline” (SME 4)
Evaluating the readiness of a MoC
“…because it is difficult to know [when a MoC is ready] because so many parts of the system and such a wide variety of things need to align that it’s difficult to know when all of that stuff is aligning up…so some sort of framework or structured way or a model that could take you through…and give you the confidence that it’s now ready would be very valuable.” (SME 1)
“… it comes down to the whole principles of change management and whether you’ve got a receptive audience. Do you have fertile ground in which to plant this thing?” (SME 19)
“So I think it’s really important to make sure you’ve got everything, all the ingredients right, the context is right, that the people involved are ready and engaged with it, ready to deliver it and embrace it before you actually implement it and evaluate it.” (SME 8)
“…in some ways I think there can be a tendency to boil everything down to a checklist, and that concerns me as well.” (SME 4)
“…it’s got to be seen as a value add exercise and not something that’s just kind of ticking a box to do a function that you know may or may not be useful.”(SME 14)
Evaluating success of an implemented MoC
“…if you want to make statements or judgments about the effectiveness of a Model of Care, which presumably at some point you would want to because the whole – one of the purposes behind a Model of Care is to improve…and you can’t make judgments about effectiveness if you don’t know whether it’s been implemented appropriately in the first instance…” (SME 8)
“I guess we need to measure what we do to know whether we’re making a difference…I mean, we’re in a place of increasing need and demand, and we have finite resources, so…measuring what we do is important to make sure that it’s delivering value, so value in terms of quality and cost”. (SME 7)
“I think measuring people’s commitment to the implementation and to the Model of Care is very telling so that’s really very important…how they’ve been engaged and enabled to change whatever it is that they have needed to change. That can be rich I think in terms of what you might do the next time, both good and bad…so what we might not do, but also but what you might want to replicate.” (SME 2)
“…evaluation is obviously also critically important for consumer satisfaction and potential need for modifying the Model of Care, assessing the consistency of uptake and obviously the acceptability to the stakeholders” (SME 9)
“…if there were a particular framework that could be used, then there is some measure of comparison rather than, well we evaluated this [MoC] implementation by this process but another evaluation process was used for [another] Model of Care…” (SME 3)
Challenges associated with evaluating musculoskeletal MoCs
Identifying, defining and measuring indicators of readiness and success
“…Models of Care are pretty high level documents and as such they often lack that specificity or that operational level of detail and specificity of activities that allow for betterment, so it can be very difficult to know where to target your measures and determine what to measure. And then, yeah, there’s often a lack of agreement I think about how to measure, is something ready to be measured…” (SME 1)
“ one [challenge] that I’ve also eluded to is the comorbidity - the fact that our patients are wonderfully diverse and that’s often our biggest dilemma is trying to decide whether somebody who’s got so many comorbidities is still worthwhile trying to include in it [MoC] or whether you know they are so different in all other respects and there only going to be a confounder.” [SME 11]
Impacts of systems or context changes on prospective evaluations
“… but the Model of Care is probably only one of the things that accounts for improved or changed outcomes in the patient population. Because almost invariably there are other things happening at the same time, some of them planned and some of them unplanned. So the evaluation framework around the Model of Care is very important but I think we have to also be a bit pragmatic and remind ourselves that sometimes when there is a change in outcomes, whether it’s good or bad, there is probably a number of factors impacting upon it and that’s always the dilemma…” (SME 11)
Other challenges
Key theme | Summary description | Illustrative quote |
---|---|---|
System constraints | Implementation of a MoC into an existing system may be unfeasible due to constraints within the current system. For example, some of the aspects of the MoC might need system enablers in place (e.g. new IT infrastructure), so implementation and subsequent evaluation cannot proceed successfully until system changes are completed. Additionally system design constraints, such as the split health funding models between the Australian Commonwealth and State/Territory governments, also presents as a significant barrier to evaluation across settings. | “So I think they’re all external constraints and it’s around the purchasing plan. So this is the amount of activity you will do and you know, this is the dollars that are attached to that because it’s worth you know, X number of dollars to – episode of care or service event and then it depends very much on the types of service models that the area health services or the local health network are wanting to implement. So depending on what the service models are, what the funding sources are, what the purchasing plans say, it’s really hard to do a pre and a post evaluation…”(SME 18) |
Cost of evaluation | It was emphasised that evaluations can be resource intensive, depending on the study design, governance and data collection arrangements. SMEs indicated that external funding and partnership with Universities are ideally needed to assist with the collection of data (particularly an issue in the primary healthcare sector). |
“One of the biggest issues we have in evaluating in general practice is that they don’t get paid for this sort of work…I actually get quite frustrated that it always ends up coming down to a dollar figure, but general practice isn’t paid to stop it’s work and to write an evaluation or to do a survey…one of the things that worked very well…they incentivised general practice to participate pre and post. It’s not a lot but to ensure that their nurse will be able to collect the information…” (SME 22) |
Ensuring adequate involvement of stakeholders within the evaluation process | SMEs emphasised the need to ensure adequate involvement of stakeholders within evaluation processes in order to obtain a comprehensive understanding of issues relating to implementation and outcomes. Challenges in engaging stakeholders in evaluations included: | "I think quite often people jump to a solution and think they know the answer…we actually firstly need to have all the right people in the room, and when I say the right people, I don’t just mean the best clinicians, I also mean management of front line and I mean people who have a state wide role in funding and planning and some consumers.” (SME 16) |
● Ensuring all relevant stakeholders are involved, given diversity and complexity of healthcare settings relevant to musculoskeletal health, particularly in the private community setting. | ||
● Getting stakeholders to understand the need to build evaluation into the entire process of a MoC; i.e. from inception to implementation. | ||
● Achieving a cohesive understanding of terminology relating to MoCs across diverse stakeholders in different sectors of the care continuum. |
“…but national consistency and quality in a country like Australia, not just the [geographical] vastness, but the different jurisdictions, is always going to be problematic…” (SME 6)