The evaluation
To conduct a service evaluation, data were collected from a variety of sources. Quantitative patient outcome data were collected using pre- and post- treatment patient questionnaires, using consecutive sampling. Patient experiences and opinions of the service were obtained using a post-treatment service survey, collecting predominantly qualitative data. Interviews with healthcare professionals involved in the service collected qualitative data regarding their views of the service. These mixed methodologies are recommended for this type of evaluation [
31]. Ethics approval for the evaluation was obtained from the University of Westminster Ethics Committee. Informed written consent was collected from all study participants.
Patient Questionnaires
Participants were provided with their pre-treatment questionnaire by reception staff when they booked their first appointment, and their post-treatment questionnaire by their acupuncturist/osteopath at the end of their final session. Participants who did not attend their final session had their questionnaire posted to them by the researcher (AC). All participants were able to ask a researcher for help completing the questionnaires, enabling participants with low literacy or whose first language was not English to be included in the evaluation. Participant demographics (including age, gender and ethnicity) and previous CAM use were collected by the pre-treatment questionnaire. Patient questionnaire measures included:
MSK pain, which was measured using the Bournemouth questionnaire (BQ) core items [
32]. The BQ was developed specifically for patients with MSK pain and has been shown to be reliable, valid and responsive to clinical change [e.g. [
32]]. The BQ incorporates dimensions of the biopsychosocial model for MSK pain including levels of pain, interference with everyday tasks and social activities, anxiety, depression, the extent to which work affects their condition and coping ability. It comprises seven items scored from 0 to 10 which can then be summed to provide a total score ranging from 0 to 70. Higher scores indicate increased MSK problems.
Quality of Life (QoL), which was measured using the EuroQol-5D (EQ-5D) [
33] a widely used, generic measure of health-related quality of life. It is quick and easy to complete and has been shown to be valid and reliable [
34,
35]. The first part comprises five items (measuring mobility, self-care, usual activities, pain and anxiety/depression) which are graded on three levels according to severity. Using the established algorithms for the UK [
36], these items were translated directly into index scores, ranging from -0.59 (worst possible health state) to 1 (best possible state). The second part is a visual analogue scale (VAS) measuring overall health, anchored 0 (worst possible health state) to 100 (best possible health state).
Participants were further asked if they were using analgesics, and about areas where they experienced pain and work status. They were also asked to rate their general health and well-being, and physical activity levels on a five and six point Likert scale respectively.
Service Survey
Participants who completed their post-treatment questionnaire were also asked to complete a service survey. According to patient preference, the service survey was available to complete online or by hand. Both versions of the survey were identical. The survey comprised a combination of open-ended questions with space for participants to write answers, as well as "yes" / "no" closed response questions aimed at ascertaining participants' opinions and experiences of the service including: perceived benefits, satisfaction, problems, suggestions for improvement, continuing provision of the service, treatment by staff and future use of acupuncture/osteopathy.
Healthcare professional and CAM practitioner interviews
All healthcare professionals involved in the service (all seven GPs and the administration manager at the practice, and the three CAM practitioners) were invited to participate in an interview. Semi-structured interviews aimed to elicit participants' views on the service were conducted approximately five months into the service by AC. While questions and topics were on the interview schedule, there was flexibility to follow up issues raised by the interviewee. Topics included benefits of the service, problems encountered, helpfulness to patients, ease of incorporation and improvements to the service. Interviews lasted between 10 and 20 minutes. Ten of the interviews were recorded; one was documented using note taking at the request of the participant.
Patient Outcomes
Comparisons between pre- and post-treatment for the primary outcome measure, the BQ, revealed a highly statistically significant improvement in MSK problems, including BQ total score (p < 0.0001) and all seven subscales: pain (p < 0.0001), interference with daily activities (p < 0.0001), interference with social routine (p < 0.0001), anxiety (p < 0.0001), depression (p < 0.0001), effect of work on pain (p < 0.0001), and coping with pain (p < 0.0001), see Table
1. Applying the threshold of 0.5 for effect size, 52.9%, 95%CI [42.3%, 61.7%] of participants experienced a clinically significant reduction in their MSK pain.
Table 1
BQ total and sub-scales scores pre and post-treatment
BQ total score (range 0-70 ↑ = worse) | 38.5 | (25.0-50.2) | 23.0 | (10.0-40.0) | 5.77 | < 0.0001 |
BQ subscales (range 1-10 ↑ = worse) | | | | |
Pain | 7.0 | (5.0-8.0) | 4.0 | (2.0-7.0) | -6.25 | <0.0001 |
Interference with activities | 6.0 | (3.0-8.0) | 3.0 | (1.0-7.0) | -5.23 | <0.0001 |
Interference with social | 5.0 | (2.0-8.0) | 2.5 | (0.0-6.0) | -4.79 | <0.0001 |
Anxiety | 6.0 | (3.0-8.0) | 4.0 | (1.0-7.0) | -4.57 | <0.0001 |
Depression | 4.0 | (0.0-7.0) | 3.0 | (0.0-6.0) | -3.04 | <0.0001 |
Effect of work | 5.0 | (2.0-8.0) | 3.0 | (1.0-7.0) | -3.61 | <0.0001 |
Coping | 5.0 | (3.0-7.0) | 3.0 | (1.0-6.0) | -5.47 | <0.0001 |
Comparisons between other study variables pre- and post-treatment revealed a statistically significant improvement in health-related QoL (EQ-5D index) (p < 0.0001), however, there was only a trend towards an improvement on the EQ-5D VAS (p = 0.064). A statistically significant reduction in analgesic use (82.8% to 68.7%, p < 0.003) was also found but there was no significant change for physical activity (p = 0.307) or general health and well-being (p = 0.541), see Table
2. There were inadequate numbers of participants in categories to conduct statistical analysis regarding current work status.
Table 2
Study variable scores pre and post-treatment
EQ-5D - index (range -.59-1 ↑ = better) | .440 | (.137-.727) | .621 | (.533-.796) | -4.82 | < 0.0001 |
EQ-5D - VAS (range 0-100 ↑ = better) | 70.0 | (54.5-80.0) | 70.0 | (50.0-85.0) | -1.85 | 0.064 |
Physical activity (range 1-5 ↑ = better) | 3.0 | (2.0-4.0) | 3.0 | (2.0-4.0) | -1.02 | 0.307 |
Well-being and general health (range 1-6↑ = better) | 3.0 | (2.0-4.0) | 3.0 | (2.0-4.0) | -6.11 | 0.541 |
The service survey provided qualitative data supporting outcome benefits of the treatment to patients. In tune with the quantitative data, many patients reported that they valued the improvements in their MSK problem as a result of treatment. Patients reported decreased pain, and improved mobility including joint mobility. Some patients felt these improvements helped them to get on better with their daily lives.
"The treatment was really efficient. Since then I haven't had any problems with my back." P102
In addition, some patients reported improvements in other physical health conditions, for example decreased headaches, menstruation pain and improved energy levels. Other patients felt they had experienced improvements in their psychological well-being. Some patients described finding treatment relaxing and enjoyable, others experienced a reduction in their depression and anxiety, or felt more able to cope with their lives.
"It was surprisingly effective for many ailments. I had acupuncture and it helped with not only back pain but also illness reduction and depression." P84
Some patients felt better able to self-manage their condition. They learnt from practitioners a better understanding of their MSK problem, including what had caused it, what exacerbated it, and which exercises, stretches and changes to undertake to manage their condition better and prevent relapse.
"I was able to find out and understand more about what was wrong and learn new techniques to help me deal with the problem in my knees." P7
Acceptability of the service
Service survey data showed that patient satisfaction with the service was extremely high. More than 9 in 10 participants reported that they were satisfied with the way they had been treated by staff in relation to the service, and 96.7% believed that the surgery should continue to provide the service in the future. Themes emerging from the qualitative analysis revealed the aspects of the service patients valued. Patients appreciated having the service at their GP practice, it was a convenient location and a familiar environment. They trusted a service provided through their GP practice, and felt reassured that their GP would know details about their osteopathy/acupuncture treatment. Patients also described finding the service straightforward (especially in terms of booking appointments), they appreciated the short waiting time for appointments and the efficient time-keeping of the service. In addition, some participants welcomed being offered a CAM therapy in the first place. They liked using these approaches compared to medication, as well as the additional time and attention given to the pain problem. Some patients liked the alternative (e.g. Chinese) explanatory model for health given by their practitioner, and the way in which treatment sought to get to the "root" of their problem. Other participants were just grateful that they had been offered something new to try to help with their MSK problem.
"Treatment at the practice would be in a familiar place and my doctor would be informed sooner than going to hospital and waiting. It's a good system, because you are being treated within your doctor's practice and communication should be more efficient. Local, to save you travelling to different hospitals." P43
"Osteopathy is an essential treatment as it treats the condition causing the pain. This is much better than taking painkillers." P25
A number of participants mentioned the positive qualities of their CAM practitioner. They valued the relationship they had formed with them, their professionalism and caring nature, and being provided with an explanation of the treatment they were receiving.
"The osteopath was very professional, pleasant and easy to talk to in regard to my problem. It was the first time I had been referred to an osteopath before and he was understanding and made me feel relaxed when being treated." P82
Ninety-one percent of participants said that they would use osteopathy/acupuncture again at their GP surgery, predominantly because they felt it had the potential to help MSK problems. This figure fell to only 30.8% who would use it privately, this was principally because of the cost of treatment, but also for the aforementioned reasons (e.g. convenience).
"Acupuncture and osteopathy are very good for people who suffer from pain, but in private it's very expensive. Myself I cannot afford to pay for it privately." P10
One quarter of participants said they had experienced some problems with the service. The analysis showed that the majority of these issues were related to the popularity of the service. For example, as the service became full, some participants had to wait for their first appointment, or for longer between appointments. Some participants wanted more appointment availability and flexibility (such as outside of working hours); others wanted to receive more and/or longer sessions. In addition, a small number of participants said they would like to receive more assistance from the reception, and some would have preferred a female practitioner.
"I had difficulty booking a time that would fit into my work schedule. Plus I couldn't book weekly appointments and I feel this was important for treatment. More flexible appointment times [needed]." P3
Interviews with healthcare professionals involved in the service also revealed high levels of satisfaction with the service among staff. In terms of service provision by the practice, all practitioners reported that the service had been incorporated well; overall the referral process had been simple and straightforward and the service ran smoothly. In relation to patient benefit, it was felt that the service was helpful for the practice's patients in terms of reducing their pain, increasing their flexibility and movement, improving general well-being, providing an explanation for their pain, and helping them to understand and manage their condition. In addition, GPs particularly valued having the service on site, this meant they were aware that their patients were having CAM treatment and were able to access details of patient appointments on the practice's computerised system and communicate with CAM practitioners easily. GPs also welcomed the relatively short waiting time for appointments and having an extra referral option.
"Very good, very prompt and the patients love it, you can't ask for more." GP4
"From referral to seeing patient, to appointment to getting feedback, I think it's worked very smoothly, there's been no logistical problem. It's very easy to do." GP3
Despite the favourable opinions of the service some problems emerged. Firstly, the popularity of the service needed to be managed. Interestingly, the service reached capacity very quickly. This high demand sometimes resulted in CAM practitioners being unable to treat their patients on a weekly basis, and there had been a period where GPs had been unable to refer to osteopathy. In addition, patients' expectations needed to be managed in terms of the total number of sessions they could receive. Secondly, CAM practitioners felt they could have benefited from more feedback regarding service provision from GPs and other members of staff.
"If you see them [GPs] in the corridor they're very nice and say 'hi', but they're rushed off their feet, so there's virtually no time to have any interaction." Practitioner1
Thirdly, there were issues regarding high numbers of referrals of chronic patients to the clinic. CAM practitioners were happy to try and treat any patient, but had to alter their expectations regarding the kind of success that was likely to be achieved with some patients.
"Some [patients] have been very long-term and difficult, but I guess that's just the demographic you're just going to see here. From my point of view you just have to accept that and get on with it." Practitioner2
"... we're so desperate to get some of these heart sink patients to be seen by somebody. And part of it is the therapy and part of it is the time they're spent with. And perhaps those types of patients were not quite so appropriate, but on the other hand you can't just pick up the easiest patients." GP1
Finally, there had initially been some practical problems regarding appropriate room space and equipment availability (e.g. couches) for practitioners. These issues had taken time to resolve and practitioners felt they should have been organised prior to their arrival.
In summary, data suggest that successful provision of osteopathy and acupuncture services for MSK pain within General Practice is achievable. However, some issues with provision will arise and need to be managed in order to provide as efficient a service as possible.