Using patient comments from a standardised experience survey to investigate their perceptions and prioritise improvement actions: a thematic and syntactic analysis

Patient Experience (PE) has been recognised as a useful measure to help evaluate and improve healthcare systems [1, 2]. PE measurement and evaluation programmes derived from standardised tools and specifically designed for healthcare institutions have been nationally implemented in several countries [3, 4]. However, patient associations raised two main limitations regarding these programmes [5, 6]. First, it is not certain that the data and evidence generated really reflect the experience of the patients as the approach privileges the evaluation of service quality rather than focusing on all the aspects of PE. Secondly, these programmes do not appear to be effective drivers of change within healthcare institutions. The most common barrier to the use of the results from local or standardised surveys is the difficulty to engage staff as the quantitative results are perceived as not sufficiently specific and are hard to translate into effective interventions [7, 8]. To improve healthcare practices and quality of care, healthcare institutions are increasingly using patient feedback as a tool to inform strategies for improving care, both at the hospital governance, department, and staff levels [9, 10]. For example, a survey showed that more than 40% of outpatient healthcare providers use patient feedback for implementing measures to improve patient care [11]. In this context, we designed an innovative approach to analyse patient comments from the French national patient survey of an academic hospital federation. The hypothesis was that these data allow to reflect patient views and identify specific improvements and practices while at the same time valuing the use of a national patient experience survey. The objectives were to (i) not presuppose an a priori framework for patient experience (to be as close as possible to their experience), (ii) to encompass all experiences (by including all comments and keeping all the subjects mentioned), (iii) to propose a prioritisation of the subjects based on the most important issues for patients, and (iv) to identify, within the comments, actions and initiatives that could help staff and institutions improve their practice.

Over a year, nearly 80% of the respondents to the French PE survey left a comment after their hospitalisation in the HCL. The analysis of the surveys containing comments allowed to create a patient journey with a very high number of precise sub-themes. More than 15% of these were prioritised using the present methodology and are described in operational sheets to help institutions, departments, and staff conduct quality improvements regarding four key categories: relationship between patient and staff; environment; surgery and pain management; information and care coordination.

The results obtained herein reflect those found in the existing literature, particularly the fact that more than half of the themes mentioned in patient comments were not present in the survey’s closed-ended questions [16] and that patient narratives can improve healthcare quality beyond what standardised survey scores can accomplish [10, 17]. The most commented themes and sub-themes identified herein also confirm those reported in the literature, especially the importance of patient-staff interaction to improve patient journey [18], and the need to improve information in the aftercare [19].

The four key categories identified herein are in line with the primary dimensions of health service quality -interpersonal, technical, environment, and administrative quality-, as described by Dagger et al. (2007) [20]. The relationship between patient and staff is predominantly determined by the interpersonal quality of care, which reflects the ability of healthcare providers to establish and maintain relationships with patients. The environment in which care is delivered encompasses non-medical aspects of care such as accessibility, comfort, and the overall care experience. Surgery and pain management are primarily linked to the technical quality, which encompasses the clinical skills and expertise of healthcare providers. Dagger's model also includes an administrative dimension, which reflects the efficiency and effectiveness of the healthcare system as a whole. This dimension is closely related to the information and care coordination aspect of PE identified herein. The administrative dimension encompasses factors such as waiting times, errors in patient files, and internal processes, which are dependent on effective information and communication systems. Therefore, ensuring effective information and communication is necessary for achieving high levels of administrative quality, which in turn contributes to the overall quality of care provided to patients.

Some expectations reported by the patients are specific to the present analysis as they concern the specific institution commented on (for example, the orientation issues) or are specific to French hospitals (for example, the incomprehension of patients who find themselves in a shared room despite their request for a single one and despite the fact that they pay their insurance company to cover this particular service).This approach places the patient's view at the heart of the institutional strategy while combining it with action levers that are directed at the department and staff level. It also encourages the engagement of patients by bringing patient comments to light. Communication of the results and the actions defined should also increase the overall engagement of patients in the process. Identifying such actions at different levels and on different time scales allows to rapidly implement targeted actions at the department and staff level while developing long-term institutional strategies, thus enabling to keep a balance between the improvements needed and the daily functioning of an academic hospital federation. For example, a first step consisted in valuing staff for their work by sharing the practices commented on by patients and the good quantitative and qualitative feedbacks. To this end, the modelling of the patient journey represented a good pedagogical tool when sharing the results to the staff. Another type of rapid action concerns practical improvements (e.g., orientation issues, information before admission, birth registration process) implemented by support teams, thus avoiding the burdening of front-line professionals. In a second and longer step, improvements such as the need to improve aftercare information, became part of the institutional strategy which aimed to engage staff at different moments of the patient journey (e.g.: after care, during preparation for discharge, follow-up after discharge, etc.). Some of the improvements needed (such as pain management and behaviours during visits) are not limited to the institutions included as such issues can be addressed during the initial training of the professionals (schools or universities). In this specific context of an academic hospital federation, the results will be shared with the actors involved in initial training with the aim to raise awareness on patients’ expectations as early as possible.

The methodology applied herein is particularly relevant to reflect PE and conduct changes at the staff, department, and institution level. For instance, another analysis conducted on the same material but using a different methodology (word and comment filtering and non-negative matrix factorisation algorithm), allowed to identify about twice as less themes compared to the present methodology [21]. That specific analysis, conducted by the French National Authority for Health (Haute Autorité de Santé, HAS), differed however, as it aimed to identify frequently raised themes and included more than two million comments. Another difference relates to the choice to analyse themes and sub-themes commonly, whether they are mentioned in the positive or negative question. This approach, which was applied herein, allows to measure how often a theme or a sub-theme is mentioned (positively or negatively), to determine the mean overall satisfaction regarding the theme/sub-theme, and to measure the proportion of positive and negative speech. This therefore allows to identify the reasons of dissatisfaction but also the practices valued by the patients on a same theme. Furthermore, when sharing the results with the staff, it contributes to conveying the results in a positive and engaging manner. The constraint of a joint approach, however, lies in the additional analytical time required, as it is necessary to differentiate positive, neutral, and negative speech. The present analysis also proposes to consider the syntax of the comments, allowing to measure speech engagement [14], to go beyond the mere recurrence of themes in order to prioritise improvement actions, and to highlight sub-themes that are rarely mentioned. In both approaches, the quantification of the identified themes allowed to transform qualitative material into measurable and comparable data that could allow comparisons in the long-term, given that the expression mode of patients remains stable over time. This expression mode depends on the case mix (age, sex, type of disease), the educational, and socio-economic level of patients [22, 23]. The selection of an analytical method must thus consider the number of respondents, the expected outcomes, the time constraints, and the technical challenges. A manual analysis, which can be time consuming, enables to overcome certain challenges met by natural language processing (NLP) as, according to Greaves et al., “machines struggle to read and understand comments accurately; software finds comments preceded by negatives difficult to interpret”⁹. Furthermore, the use of sarcasm and irony – a feature of British and French cultures- is hard to process by machines [9]. The proposed approach enables to analyse many patient comments, allowing a comparability of results between institutions and over time, which is difficult to obtain on small samples using methods such as qualitative interviews or focus groups.

The present study has certain limitations. The use of comments from open-ended questions that are placed after closed-ended questions may bias the content of the comments. However, one study tested the variation in narrative content and quantitative scores as a function of the placement of open-ended questions on the Consumer Assessment of Healthcare Providers and Systems (CAHPS) survey. The results showed that the relative placement of closed and open-ended questions had little impact on CAHPS narratives or scores [24]. The use of the e-Satis survey implies a bias in the selection of respondents, as it implies needing an e-mail address, answering the online survey, etc. This bias, however, is limited by the manual intervention which enables to identify specific sub-themes raised by only a few respondents and the use of patient characteristics that allows to target specific populations. An important limitation when analysing patient comments in healthcare setting is the time and financial cost that can be incurred. In the present research framework, which was part of a public health and management science thesis, no financial or time constraints were encountered. A budget and time estimate however, were simulated for the entire process, including researcher compensation, software usage, document production, and other expenses. A mean time of six weeks and a cost of 25,000 euros were estimated. The number of comments analysed is the most critical variable in determining costs. The present research protocol involved analysing all the comments over a specific period, but sampling techniques could be used to adjust analysis costs. Ultimately, the use of patient experience data from the years 2018–2019 may be subject to limitations in light of the COVID-19 pandemic. It is possible that patients' perceptions of healthcare quality and their overall experience of care may have changed as a result of the pandemic. For instance, concerns about infection control and access to care may have become more salient in patients' minds, potentially impacting their satisfaction regarding healthcare services. Similarly, changes to healthcare delivery models and resource allocation may have influenced patients' experiences of care in ways that are not captured in the data from 2018–2019. As such, it is important to interpret the results of the present study with caution and consider the potential impact of COVID-19 on patients’ perceptions of healthcare quality. Future research may need to incorporate more recent data to capture the evolving nature of patient experience in the context of the pandemic.

The next step in our research will consist in involving the staff to collect and analyse their perceptions regarding their interactions with patients in order to balance the present patient-centred results. This would allow to refine improvement actions to remain as close as possible to the staffs’ specific needs and engage staff in a dynamic process for improving their practice and patient care.

The analysis of patient comments obtained from a standardised survey allowed to characterise the patient journey using data that describes PE, further enabling a prioritisation of actions aiming to improve practice and quality of care at the institution, department, and staff level.