Perceived barriers to monitoring service quality
The first overarching theme that emerged from the focus groups is “perceived barriers to monitoring service quality”. This theme refers to barriers to the collection and use of data for monitoring service quality. All participants noted that they put considerable effort into collecting patient data. While most participants felt that the data they collect are potentially useful, they reported several challenges to monitoring service quality. These challenges are described below.
Outdated methods of collecting data
Participants thought they collected valuable information from patients that could be used for monitoring service quality. However, they reported that their use of pen-and-paper modes of data capture and storage limited their ability to retrieve and use these data to assess service quality. While this issue was raised mainly by participants from publicly-funded non-profit treatment centres, some participants from private treatment centres also talked about the need to move away from hand-written notes and files to electronic information systems. Limited access to computers was cited as the main obstacle to migrating to electronic information systems. Only a handful of participants had access to computers and data entry software at their facility, as reflected in the following statements:
“Do you know what difference it would make if all our staff had access to computers, if all of us could enter our valuable client information on a PC (personal computer) and retrieve that information with a click of a button?” [FG2; female social worker]
“If only all our social workers had computers, or even just access to one computer only used by the social workers and psychologists, that would be the ideal…we can do more (data collection) if given the resources (computers and data entry software)”. [FG1; male service manager]
Participants ascribed many of the challenges they had in monitoring service quality to their overreliance on pen-and-paper data collection methods. One prominent example of this was the inconsistencies in the type and quality of data collected, largely because hand-written notes allowed for variation in terms of what content is recorded. All participants noted that the type of information recorded and the level of detail with which information is recorded varies by counsellor. Participants also reported that paper records are often not updated in a timely manner, there is often duplication of information recorded, there are difficulties in retrieving information from these files, and that files are sometimes misplaced. As one participant reflected:
“So we are really looking forward to a computerised system instead of the paper and pen so that things are captured and whoever is allowed access can have access to it immediately. The file won’t get held up somewhere or you lose it or just struggle to keep it together”. [FG1; female social worker]
The paper monster
All participants commented that they were overwhelmed by the amount of paperwork associated with current data collection practices. Participants were frustrated that facilities used very little of the information they collected and that there was duplication of information collected because of the use of multiple forms with overlapping content.
“We collect data from admission to discharge, which is important, but it often gets out of hand and I think the patients get tired of completing the same information over and over…and don’t you think it is unethical to bother them with forms and information that never even get used…” [FG1; male psychologist]
Several participants noted that completing this paperwork required a significant amount of time that could have been reserved for service delivery. This is reflected by the following comment:
“If you look at all the reports you have to write, you get to the stage where you write, write, write…., and in the end it’s about the client and not about writing. And now you are focused so on writing the reports, you are not getting to the client”. [FG2; female social worker]
Some participants reported that their facilities tried to reduce the amount of time clinical staff spent on paperwork through task-shifting data collection responsibilities to administrative staff. According to these participants, this was problematic as having multiple people responsible for data collection often diminished the quality of data collected. As one participant noted:
“We have so many people including data in the file: from the clerk, the administrator, the social workers, the psychologist…Sometimes data that should be collected by the social worker is collected by an admin person only to save time, and this creates a problem with missing information and accountability because you don’t know who does what”. [FG3; female social worker]
Overburdened staff with little time to collect data
Most participants reported that clinical staff in outpatient treatment settings have very high patient caseloads and are required to perform multiple functions. These overburdened personnel have little time to complete multiple and lengthy pen-and-paper forms. Almost all participants thought that expecting these overburdened staff to complete additional data collection forms would compromise data quality. They noted that overburdened personnel are less likely to collect reliable information. As one respondent reflected:
“There is so much our social workers have to do, so many assessments to complete, group notes, urine tests, and the list goes on…we don’t have someone dedicated to doing admissions and case histories and reports, so we really need to be careful about adding to their burden”. [FG2; female service manager]
Lack of skills to analyse and interpret data
Participants acknowledged that the forms they currently use to monitor service quality are potentially flawed. They recognised that poorly designed evaluation forms could lead to bias, limiting the usefulness of these findings for performance improvement efforts. However, they felt that they did not have the skills to improve these forms:
“We have our own evaluation forms, and patients often give us good reviews. We seldom have someone telling us that services are not good, but then we have so many patients not completing treatment. I think our evaluation forms need improvement and of course there is bias involved, but I do not know how to deal with that, we just do the best we can”. [FG 3; female social worker]
In addition, treatment service managers thought they lacked the skills to analyse and interpret their facility data which prevented them from using these data for service improvement efforts.
“One can often look at data and you don’t really know what it means”. [FG1; male service manager]
“We have very limited capacity, you know…to analyse data”. [FG2; female service manager]
“We can’t analyse it, if we want something complex answered we give it (the data) to someone else to do for us”. [FG3; male service manager]
On reflection, treatment personnel thought that they would be able to analyse and interpret their data if given the appropriate training, mentoring and support.
Between-facility variation in types of data collected
All participants identified considerable between-facility variation in the types of data collected and data collection formats used. Participants noted that most facilities had developed their own data collection forms, with facilities not collecting data on a shared set of indicators. Several participants raised questions about whether all treatment centres “are capturing the same kind of data”. Variation in the types of data collected and the manner in which data are recorded makes it difficult to compare facility performance on key indicators.
Recommendations for improving the collection and use of service quality data
The second overarching theme was “recommendations for improving the collection and use of service quality data”. This theme refers to participants’ perceptions of possible solutions to these barriers. Participants provided several recommendations for how to address these perceived barriers to the collection and use of service quality data. Participants highlighted the adoption of contemporary data collection methods, the use of psychometrically validated measures, and the collection of service quality data using a shared set of common indicators as critical to enabling the monitoring of treatment quality.
Adoption of contemporary data collection methods
Participants emphasised the need to shift towards contemporary methods for data collection, storage and retrieval. Electronic health information systems (EHIS) were seen as a means to facilitate service monitoring, improve data quality and enhance the likelihood of data-driven quality improvement initiatives. This is reflected in the following statements:
“I worked in Ireland and they had that (electronic) system, and they did not have this paper trail there”. [FG2; female social worker]
“An electronic system would be very good. I mean like it would help standardisation of information taking”. [FG1; female addiction counsellor]
Participants also noted the potential benefits of having a central electronic patient information system in which people’s health, social welfare, criminal justice, and education records are contained within a single electronic system and accessible to any service provider on provision of the patient’s official identification number and the provider’s secure access code. Several participants believed that this would provide access to a wealth of information about the patient’s health and social status and limit the need for extensive history taking. According to participants this would streamline the data collection process:
“It kind of makes me think of the system they have in the UK…. the nice thing for me in the UK was that you have immediate access to a lot of different information about a client… Now at least if you have a system like that you know what’s happening with the client, you know where they are and that kind of stuff. You would be getting regular reports from teachers, police, different NGOs. So they would be sharing standardised information which would go into a central database. Anybody new who starts with the case would immediately get a history of everything that is going on”. [FG1; male social worker]
Nevertheless, some participants were reluctant to use a centralised electronic patient record system due to data safety and confidentiality concerns. Specifically, they were worried about their patients being identified as drug users and how this might affect their access to other services. They also voiced concerns about who would have access to these records:
“There’s a lot of shame around addiction and a lot of the stuff is illegal. You’ll get little 13 year old boys coming and not sure if they’ve got a problem, parents bring them; you don’t want to put someone like that on some kind of list that might stick with them/him.” [FG1; female psychologist]
Furthermore, the vast majority of participants noted that the successful implementation of an EHIS would require various resource and skills deficits within treatment facilities to be addressed such as limited access to electronic equipment, low levels of computer literacy, and poor analytic skills. This is illustrated below:
“My staff are doing the best they can, given the circumstances…but I would say that our annual reports for example is not as detailed and correct as the treatment centres that have these facilities (computers and staff trained in data analysis)”. [FG2; female service manager]
“It’s not just the initial outlay; it’s also maintaining it, the IT support”. [FG2; female service manager]
“Even the new social workers who are newly qualified, that come out of some of the universities, are still not very comfortable using computers”. [FG3; male service manager]
Although many participants expressed a sense of pride in their evaluation forms, they also noted that these forms have not been rigorously tested and validated. All participants expressed the “need for tried and tested tools” that would generate reliable data for service improvement. Providers thought that using psychometrically validated measures of service quality would improve their confidence in the data generated. As one provider noted:
“I designed our evaluation forms myself, and it’s a very detailed form. We have been using this form for about a year now, and one of our students will use the tool as her master’s project, you know doing the cognitive testing and analysis. So as yet we are not sure whether it works…” [FG1; male psychologist]
Collecting service quality data on a shared set of indicators
Participants suggested that forms used to monitor service quality be standardised across facilities in order to allow for the collection of service quality data on a shared set of indicators. Participants believed that this would improve the quality of data collected and would allow the performance of treatment facilities to be compared on a common set of quality indicators. As participants reflected:
“If you want to determine the quality of the service that you give maybe that (data) can be collected in a standardised way”. [FG1; female social worker]
“There is an overlap across facilities; you find similar forms with similar information. We should be able to standardise, it’s difficult but we should”. [FG2; female social worker].
Willingness to participate in a performance measurement system for substance abuse treatment
All participants were supportive of an initiative to develop and implement a performance measurement system for South African substance abuse treatment services. Participants noted that the initiative could address many of the shortcomings of current data collection methods. Participants reflected that the success of this initiative would rest on the active participation of treatment providers in the design, testing and implementation of the system. They cautioned against imposing a performance measurement system on service providers without consultation:
“There are times when we don’t know what is important and what is of lesser importance, and we need help determining this. And I can tell you now already that people will not be happy to change what they are doing completely, unless what you are offering is better than our current efforts…the best way for you to know what will work at our centres is to involve us in the decision making else you will be met with a lot of resistance”. [FG3; male service manager]