Background
Mental health (MH) indicators summarise data to reflect change in mental health services, their reach, and the populations served. Health information systems are a key building block in strengthening health systems, and indicators are described as key information system tools [
1]. Specifically, for mental health and well-being, indicators on suicide, and treatment of substance abuse have been included in the Sustainable Development Goals of Agenda 2030 [
2]
In 2015, the United States Agency for International Development, WHO and World Bank met during the Measurement and Accountability for Results in Health summit, and called for action in improving, and hence investing in, health facility and community information systems [
2,
3]. Globally there is also a clear need to strengthen routine data collection for mental health cases [
4,
5]. These systems are useful at different stages in planning and implementation of mental health care; that is, situational analysis, priority setting, option appraisal, programming, implementation and evaluation [
6].
Even though this robust system for routinely collecting MH data is recommended in the WHO Mental Health Action Plan of 2013-2030 [
7], few countries have a robust system for routinely collecting mental health data. Lower and middle-income countries (LMICs) in particular face a considerable challenge to strengthen information systems for mental health [
4]. Data from the most recent WHO Atlas survey [
5] suggested that mental health data are often lacking from most national routine health systems. There has been an ongoing measure to improve quality of information systems globally in the health sector [
8]. Routine data tends to be incomplete, inaccurate and are often focused on infectious and communicable diseases or maternal health. Countries are now utilizing information and communication technology to improve quality of information systems for mental health. Measures are taken to also place validation checks to increase culture of information to improve data quality [
9].
Mental health data in routine information systems in (LMICs) are considered too unreliable even to calculate essential indicators such as service delivery and system performance. A situational analysis of the status of the health management information systems for mental health in the countries where Emerald project was implemented concluded that countries face considerable challenges within the policy and governance system but also lack capacity in terms of health management information systems (HMIS) experts, infrastructure, supervision support affecting the quality of the mental health data collection, reporting and dissemination. [
4].
The current study which was also conducted within the Emerald project from 2014 to 2016, which focusses on strengthening mental health system outcomes in six LMICs including India [
10].
In India, for example, the most common method of mental health data collection is through treatment records or case sheets. A recent study in India noted that data on diagnosis in the information systems did not reflect the new ICD-10 system of disease classification [
11]. Upadhaya and colleagues also pointed out that in India and other countries mental health data collected through routine monitoring is inadequate and untimely to be of use by policy makers [
4].
Therefore, there is an identified need to update, develop and eventually integrate mental health data with the routine health information systems in India [
10].
Amongst indicators, the ones measuring coverage have been well documented to evaluate outcomes of mental health programmes in LMICs [
12]. Coverage can be studied on a spectrum, ranging from potential coverage, that is whether services are available for patients, to actual coverage, that is whether patients can use services effectively [
12].
Previously efforts to strengthen mental health information systems have been made in Ghana, South Africa and Uganda [
8,
9,
13], however most of them were generally not sustained. It is believed that lack of evidence base on what to measure and how to measure effectively has hindered scaling and sustaining the initiatives of integrating mental health care with community settings [
14].
Consideration of the implementation challenges during the design phase ensures sustainability of the new mental health indicators [
9]. Challenges included lack of policies and plans [
4], issues with the local capacity and the problems in the workflow mechanisms [
15] and insufficient health workforce motivated to collect and more often use the collected data [
14].
Countries can determine what to measure by defining their health priorities using priority setting exercises. In research, a priority setting exercise is seen as a social process involving theory, confronting practical obstacles and understanding context to allow decision makers to rate aspects of health services [
16]. In the area of health research priority setting has been used, for example, to plan for health care spending in Kenya [
16], to reach consensus on prioritising mental disorders in Nepal [
17], and to prioritise health conditions to achieve universal health coverage in LMICs [
8].
Drawing from these insights, this study aims to develop appropriate and feasible indicators measuring mental health service delivery and system performance through an inclusive process of stakeholder engagement for Sehore district of Madhya Pradesh state, India.
Results
Phase 1 Situational Analysis
The results from the situational analysis are summarised below in Table
1. The results are categorized as; governance with a focus on mental health and its data systems, human resources needed for data system management, mental health indicators in HMIS and components within routine HMIS such as data collection, reporting, analysis and dissemination.
Table 1
Results from situational analysis checklists
1 | Governance with a focus on mental health and its data systems |
a. | Existence of Mental Health policy and plan | Present |
b. | Provision of HMIS in Mental Health Policy | Yes, in the Mental Health Policy draft |
c. | General health policies which govern Health Management Information System/HMIS | Yes, in draft National Health policy |
d. | General health plans that govern HMIS | Yes, National Rural Health Mission |
e. | Standard Operating procedures for Mental Health | No |
f. | Initiatives to develop Mental Health Information Systems | No, except for its mention in the draft policy |
2 | Human Resources | |
a. | Minimum qualification to be an HMIS staff | Graduate in any discipline |
b. | HMIS expert qualification | BSc/MSc (Bachelors in Science/ Masters in Science) in Statistics |
c. | Number of HMIS specialists (at national level) | 20 |
d. | Number of HMIS trainers | Not available |
e. | Training manuals for HMIS | Present |
f. | Specialised courses in HMIS | No |
3 | Data Systems (MH indicators in HMIS) | |
a. | Mental Health indicators in national HMIS | Noa |
b. | Mental Health Out Patient Department attendance included in HMIS | Yes, at tertiary level in some states |
c. | Mental Health referrals recorded | No |
d. | Psychiatric inpatient bed occupancy rate | No |
e. | Mental health training data reflected | No |
f. | Average length of stay at hospital | No |
3.a | Components within routine HMIS | |
a. | Data collection | Paper and pencil below Primary Health Centre/PHC, electronic in PHC/CHCs and above |
b. | Data compilation | HMIS web portal |
c. | Data Analysis | Monthly |
d. | Frequency of data reporting to Ministry of Health (Tanzania Ministry of Health and Social Welfare) | Monthly, Quarterly and Annually |
e. | Data quality control mechanisms | Yes (Supervision, Audits) |
f. | Feedback mechanism to the lowest level | Yes, no implementation on ground |
g. | Dissemination of HMIS data | Yes, not involving data collection staff |
h. | Public access to HMIS report | Yes |
An HMIS operates in India running within the existing health programmes. However, there is no separate policy specific for HMIS, although various health policies such as the new National Health Policy 2017 [
23] and new mental health policy 2014 [
24] emphasise the importance of an integrated health information systems for routine monitoring. New National Health Policy includes a commitment to integrated information systems by developing linking systems into a common grid.
In terms of the human resources, there exists a staff limitation across levels for managing HMIS. HMIS staff are interdisciplinary and they often manage reporting for various health programmes. Training manual for HMIS exist and are widely used across levels. The general HMIS contains little to no information on mental health. However, some aspects of indicators such as suicide at tertiary hospital level are reported. It was reported that there is no HMIS personnel managing routine reporting for mental health either at the national or state level in India.
Phase 2 Prioritisation exercise
A total of 35 experts including mental health researchers (n=5), psychiatrists (n=8), psychologists (n=2), programme managers (n=14), and HMIS specialists (n=6) were invited to rate indicators for mental health service delivery and performance.
Round 1 of the prioritisation exercise generated 64 indicators against the four domains of needs, utilisation, quality and financial protection. After removing duplicates, a total number of 57 indicators remained. These were subsequently rated for significance, relevance and feasibility in Round 2. Mean priority score for these indicators ranged from 2.65 to 4.47, with higher values signifying greater agreement on the Likert scale.
This scoring resulted in a list of the most frequently endorsed 15 indicators, covering domains of measuring mental health treatment coverage, including needs, utilisation, quality and financial protection (mean priority scores ranging from 4.48 to 3.78) (see Table
2).
Table 2
Results of Prioritisation exercise
1 | Utilisation | Number of people with any mental disorder who received mental health treatment by specialist in a given clinic | 4.48b |
2 | Need | Number of people diagnosed with severe mental disorders | 4.43b |
3 | Need | Number of all people diagnosed with any mental disorder | 4.24b |
4 | Quality | Number of days in last one month that psychotropic medicines were out of stock | 4.19b |
5 | Quality | Number of persons taking psychotropic drugs | 4.14 |
6 | Quality | Number of trained mental health workers at inpatient and outpatient service | 4.14b |
7 | Quality | Rate of perceived stigma and discrimination among service users and caregivers | 4.05a |
8 | Needs | Rate of suicide deaths and attempts in a given clinic | 3.95b |
9 | Utilisation | Number of people with any mental disorder with moderate to severe dysfunction who received mental health treatment in a given clinic | 3.95b |
10 | Financial coverage | Number of people with mental disorders who have some kind of financial protection or insurance against the cost of mental health care treatment | 3.95 |
11 | Utilisation | Number of people with severe mental disorder who received mental health treatment in a given clinic | 3.90b |
12 | Utilisation | Number of people detected by community workers who came to a health care facility for treatment | 3.86a |
13 | Utilisation | Number of patients re-admitted to in-patient mental health care | 3.81b |
14 | Financial Coverage | Out of pocket expenditures for services as a proportion of household income or spending | 3.81 |
15 | Quality | Number of people who score above a validated cut-off score for any mental disorder on self-report checklist (based on national health survey) | 3.78a |
Indicators covering both service delivery and health system’s building blocks emerged in the final 15 set of indicators: namely, these included 3 indicators for need, 5 for utilisation, 5 for quality, and 2 for financial protection.
Phase 3 Consultative workshops
Themes critical for both the content and the context of indicator implementation emerged from the workshop notes. These mainly included: a final indicator list and reflections pertaining to decentralisation of mental health information systems, integrated mental and physical health routine systems, stakeholder involvement, and monitoring and evaluation of these indicators.
Respondents in the consultative workshop at the state level mostly consisted of medical officers who highlighted the need for a local monitoring system for mental health where they can understand the burden of mental health in their catchment area. The need to have additional personnel for data management at each sub-district hospital was brought up by many respondents. Some anticipated that mental health indicators should be integrated at national level in the health management information system to avoid duplication of work at the sub-district hospitals, which are often poorly capacitated in terms of health workforce. The process of involving local experts such as medical officers and health managers before finalising indicators was much appreciated by all the respondents. The role of the State Mental Health Society and the State Mental Health Programme staff were highlighted as crucial in the facilitation of implementation of the proposed indicators.
Experts proposed to reduce the number of indicators assessing quality of care. Whilst our study initially found four quality indicators amongst the 15 highest rated ones (measuring: status of psychotropic medicines in stock, actual number of people taking prescribed drugs, rate of perceived stigma by users, and trained mental health workforce), these were reduced to 2 indicators (measuring: trained staff status, and status of psychotropic drugs in stock) following the discussion during consultative workshops.
As a result of the consultative workshops it was concluded that out of the top 15 dashboard indicators produced via the prioritisation exercise, 9 were foreseen to be feasible and suitable for routine collection in the health care facilities without any immediate additional support. These reflected three indicators on need, two on quality, four on utilisation, to be included for future routine data collection (listed as ** in Table
2).
Discussion
This study marks one of the first efforts to develop a set of indicators for routine monitoring of mental health services for Sehore district of Madhya Pradesh, India. Using situational analysis, a prioritisation exercise and consultative workshops we identified nine indicators covering domains such as need for treatment, utilisation of care, quality of care and financial risk protection to measure a district’s health system performance for mental health.
These nine indicators were concluded to be immediately implementable at the primary care facilities in Sehore district of Madhya Pradesh, without any additional support. However, a functional district mental health programme and other research and implementation projects including Emerald arranged for a platform for delivering MH services at these primary care facilities. Implementation of these indicators will include training of health workers, managers and doctors involved in the mental health service delivery, procurement of registers for record keeping, and developing guides for indicator implementation (e.g. through adding a time bound component to indicators such as re-admission rates).
Notably, the routine information system for mental health in India is weak and national mental health surveys have been the major source of mortality and morbidity data [
25].
The need for strengthening routine data collection is even more pressing now as countries are moving towards an independent self-sustaining health model from an international agencies led/supported model.
The Indian government’s modest initiative to strengthen mental health services at primary care level through a district mental health programme has faced various implementation challenges. A lack of clinical skills to diagnose and treat mental disorders in a resource-poor? environment with limited mechanisms to track, refer and follow up patients have been documented. [
11]. This is coupled by marginal reporting of indicators to track performance which is predominantly capturing data from tertiary hospitals [
4].
Our study has filled these gaps in monitoring by producing context specific indicators for the primary care level where mental health services are also delivered. These feasible indicators cover both health service delivery aspects with indicators on coverage and health system aspects measuring length of stay, medicines in stock and bed occupancy rates. The latter are fundamental to monitoring services but are often missing.
Even though crucial to be reported, indicators on suicide rates and attempts, daily stock-out rates of medicines and rehospitalisation need a robust system in place which in this case is provided by a functional district mental health programme and research projects. Therefore, an ongoing implementation and evaluation of these indicators is needed to ensure sustainability of these indicators.
Substantial advancements in meeting information needs for mental health over the last decade have been reported in the literature. On one hand estimates from disease burden [
26] pushed for a need to report on mental health conditions in the countries, on the other hand initiatives by international organisations including the WHO Assessment Instrument for Mental Health Systems [
27], Mental Health Action Plan 2030 [
7], WHO Atlas 2017 [
5] and quality of mental health care indicators by the Organisation of Economic Co-Operation and Development [
28] made provisions to make reporting easier.
Overloaded HMIS in countries demand contextualised and feasible mental health indicators to be included in routine reporting. Similar to our study, realistic measures to develop and strengthen mental health indicators were reported in Nigeria [
8] and other LMICs [
14]. It has been argued that the absence of quality mental health care indicators is one of the reasons behind poor evidence on mental health performance [
29]. This study used a feasible and sustainable approach in developing indicators measuring mental health service delivery at facility-level. Similar measures can be adapted to include quality indicators in routine monitoring in comparable settings.
Measures to reduce the mental health treatment gap in LMICs involves scaling up mental health services [
30]. Even though such community scale up measures have been underway in India since the 1980s, strengthening information systems can augment the process of measuring effective coverage by estimating the outcome data for those treated [
12].
Limitations
The results of this study need to be interpreted in view of a number of limitations. While our approach of developing indicators may be valid across similar settings, the exact indicators recommended may not be generalisable to all other states of India and other LMICs. This study is also limited by its ability to draw conclusions form the situational analysis checklist, due to the scarcity of resources available in the literature based on which these conclusions were formulated. However, the use of interviews coupled with document review enabled us to respond to such gaps in the literature. The response rate in the priority setting exercise was low (22.8%), although the respondent retention rate from Round 1 to Round 2 was high (91%). Difficulty in getting experts to participate in the study might be due to the less familiar or less trustful electronic way of reaching out to the experts, a lack of interest/knowledge in the area, or the lack of time. However, an electronic way of contacting experts meant that more people could be contacted across a large geographical area, and this approach also provided experts flexibility in view of their busy schedules.
Consultative workshops as used in Phase 3 of this study limit the ability to remain impartial, due to the potential social desirability bias where some views may dominate over others (especially if driven by seniors in the team). Also, our focus has been development of mental health indicators for primary care facilities within the public sector. This study does not explore mental health indicator needs of private providers as this was beyond the scope of this work. Again, further research is needed to assess the implementation of these indicators over time to validate their sustainability in the public mental health systems. Evaluation of the implementation of these indicators is underway and will be reported in an upcoming publication.
Acknowledgements
The authors will like to acknowledge PHFI- Delhi and Sangath-Bhopal team for their contributions to the study protocol and components. We thank Psychiatric Research Trust- UK for their support. GT is supported by the National Institute for Health Research (NIHR) Collaboration for Leadership in Applied Health Research and Care South London at King’s College London NHS Foundation Trust. The views expressed are those of the author(s) and not necessarily those of the NHS, the NIHR or the Department of Health. GT acknowledges financial support from the Department of Health via the National Institute for Health Research (NIHR) Biomedical Research Centre and Dementia Unit awarded to South London and Maudsley NHS Foundation Trust in partnership with King’s College London and King’s College Hospital NHS Foundation Trust. GT is supported by the European Union Seventh Framework Programme (FP7/2007-2013) Emerald project. GT also receives support from the National Institute of Mental Health of the National Institutes of Health under award number R01MH100470 (Cobalt study). GT is also supported by the UK Medical Research Council in relation the Emilia (MR/S001255/1) and Indigo Partnership (MR/R023697/1) awards.