Routine use of DHIS2 data: a scoping review

In regard to health service planning and delivery, the use of information at different levels in the health system is vital, ranging from the influencing of policy to the programming of action to the ensuring of evidence-informed practices [1‐3]. There have been global calls to action, consortia, and frameworks to support information within the remit of health systems strengthening, for example: the Paris Declaration of 2005 and the establishment of Health Metrics Network the same year; World Health Organization’s Framework for Action; the focus on Strengthening Health Systems to Improve Outcomes in 2007; and the U.S. Global Health Initiative in 2011 [4]. The rationale for these commitments is that better quality data, that are both relevant and comprehensive, will increase use of these data in action and decision-making and ultimately improve health service delivery and health outcomes. However, neither ownership of, nor access to, good quality data guarantees actual use of these data [5‐7]. To ensure information use, relevant data need to be collected, processed and analysed in an accessible format [6]. This problem of underused data, and indeed the absence of data use entirely, is widespread and has been evident for decades [8‐11].

The existing literature reviews that examined data use have focused more on challenges faced rather than sharing of solutions and identifying ways to address these challenges. For example, Lemma et al. [12] in their 2020 scoping review of interventions that aimed to improve data quality and its use in routine health information systems in Low and Middle Income Countries (LMICs) classified challenges regarding data quality and its use in relation to staff, resources, or infrastructural factors. In a systematic literature review Wickremasinghe et al. [13] examined how district administrators and health managers in LMICs used health data to make decisions and found that there was a limited range of processes documented on the use of data for decision-making at district level.

A partial explanation for limited data use is that more emphasis has been placed on data collection in LMICs than on data-use itself, with evaluations of these systems focusing more on statistical data processes and data quality, and less on how data are assimilated into practice [14]. Other studies attribute limited data use to suboptimal quality of data generated by the routine health information systems, and to an absence of a culture of information-use [15‐17]. This suboptimal quality of data may be due to unintended mistakes or deliberate misreporting,¹ but other factors contributing to poor quality can include under-reporting or no reporting at all due to time pressures, lack of motivation, too many forms to complete and a lack of understanding of the importance of data [7]. Additionally, apart from poor data quality, there remains the possibility that no standardized process governing the usage of data exists [18, 19]. So, despite these reviews and studies exploring data use, there remains limited knowledge or understanding of how data are being used or which data and processes are involved.

Data use is not easy to define as both ‘data’ and ‘use’ can be conceptualised in many different ways. A Delphi study of information scientists by Zins (2007) yielded more than 40 different definitions of data, while Checkland and Holwell (1998) revealed 7 different definitions in Information Systems textbooks [quoted in 20]. Jones [20] describes a number of assumptions that are made around data that we should question: that all data are equal, that data represent a reality independent of themselves, that data exist independently of their use, that data form the foundation on which our understanding is built, and that data represent the world objectively. When we question these assumptions, we realise that data do not necessarily report reality, that data are not recorded in a vacuum but reflect a particular worldview, that data are interpreted and may be non-empirical, and that data may vary in their perceived value and quality. Once questioned in this manner we need to distinguish, as Jones [20] suggests, between “data in principle” (as they are recorded), and the “data in practice” (as they are used). In this review we are concerned with “data in practice”.

There are also different conceptualisations and hence definitions of data use. Manuals and reports on DHIS2 itself have referred to the information cycle as illustrative of the stages required before information gets used (collection; processing; analysis; presentation; interpretation and use). In this sense data use is defined as the last step in a process and fits the definition of use by Foreit et al.: “Decision makers and stakeholders explicitly consider information in one or more steps in the process of policy making, program planning and management, or service provision, even if the final decision or actions are not based on that information” ([21], p.5).

Similarly, Nutley interprets data use in decision-making “.. as the analysis, synthesis, interpretation, and review of data for data-informed decision-making processes, regardless of the source of data. ‘Data-informed decision making,’ then, refers to the proactive and interactive processes … that consider data during program monitoring, review, planning, and improvement; advocacy; and policy development and review” ([4], p.2). Nutley concludes that “… it is clear that data use goes beyond filling out data reporting forms at the various levels of a national health information system and the passive dissemination of reports and information products.” ([4], p.2). However, Nutley proceeds to extend this conceptualisation to the purpose of use.

Nutley [4] categorises data use in terms of data and information regularly demanded, analysed, synthesised, reviewed and used in: (i) program review and planning, (ii) advocacy and policy development, and (iii) decision-making processes. Nutley doesn’t define each of these categories but classifies all three as the long-term outcomes of the use of data.

In addition to the aspects of process and purpose, the Health Metrics Network framework [22] can be applied to illustrate variance in content, reflecting the diversity of uses and users and involving the wider community such as civil society. Data use in the Health Metrics Network framework’s definition involves varied levels of data granularity and a wide range of information products.

We return to this conceptualisation and definition of data use in subsequent discussion but data use in this review was explored in a way that transcends mere data collection, form filling and the passive production and dissemination of reports or products. We therefore examined documents that either covered the process, the purpose and/or the content governing the use data from DHIS2 – what Jones’ [20] would distinguish as the use of ‘data in practice’.

Because DHIS2 is a prominent Health Management Information Systems (HMIS) platform in LMICs, this study investigated the use of DHIS2 data in these countries. Typically, DHIS2 is used as the national health information system for: data management and analysis purposes, health program monitoring and evaluation, facility registries and service availability mapping, logistics management and various community-based services such as mobile tracking of pregnant mothers in rural areas. Alongside increased support and adoption of DHIS2, the strengthening of HMIS has been facilitated by both increased commitment and investment. The DHIS2 software platform has, over the last decade, witnessed tremendous adoption, and now supports routine health management for an estimated 2.4 billion people². It is used in over 70 countries and is by far the largest and most widespread dedicated health management software. So, the question is this: has the rapid and extensive scaling of DHIS2 been matched by a corresponding increase in the scaling of improved data use?

The Health Information Systems Programme (HISP) is a global action research network to support DHIS2 implementation, to facilitate local customisation and configuration, to offer in-country and regional training, and to promote DHIS2 as a global public good. HISP University of Oslo collaborates with a global network of HISP Groups in 17 countries in Asia, Africa, and the Americas. Walsham [7] notes that improved information necessitates an approach that combines the three elements of ‘software philosophy, educating people and changing institutions’ and cites the work on the HISP [23, 24] as a programme that addresses information systems from all three perspectives. However, despite this, ‘we still see only limited evidence as to how health information systems have contributed to improved health outcomes, and to advancing the state of the poor in developing countries.’ ([7], p.196). So, if DHIS2 is the largest global routine HMIS in LMICs and adopts a holistic socio-technical approach to development and implementation, and yet despite these data is still not being used for information, action taking and/or decision making then we are duty bound to explore why this is the case. However, there has been no rigorous review conducted on how DHIS2 data are being used despite the tremendous success recorded in scaling, implementation, and improvements in data quality and data access.

There are few reviews on data use and, more particularly, on data use in relation to DHIS2. In a review of the utilisation of DHIS2 data in decision-making (at the district, sub-district, and community levels in selected districts of the Brong Ahafo region in Ghana), Odei-Lartey et al. [25] explored the various facilities’ routine meetings in search of evidence of decision-making. Though they concluded that the use of DHIS2 data to inform decisions was suboptimal they also discovered that data were being used in regard to discussions about the DHIS2 platform itself, that findings from DHIS2 data informed action-oriented decisions in addition to actions taken to promote the usage of the DHIS2 platform. The 4 categories of action-oriented decisions were: i. performance recognition and role/responsibility revision, ii. shifting/mobilization of resources, iii. advocacy for more resources and iv. formation/revision of policies/strategies. A recent literature review of DHIS2 [26] explored the strengths and operational challenges in the technical and functional aspects of DHIS2 in 11 countries but did not focus on data use. Additionally, these reviews focus on peer-reviewed literature and thus exclude a large amount of grey literature such as conference papers and research theses in this area. Consequently, our review addresses this gap and focuses specifically on the documentation of routine use of the DHIS2 data for action and decision making.

Scoping reviews have been used widely ‘to identify knowledge gaps, scope a body of literature, clarify concepts or to investigate research conduct’ [27]. They are useful ‘when a body of literature has not yet been comprehensively reviewed or exhibits a complex or heterogeneous nature not amenable to a more precise systematic review of the evidence’ ([28], p141). Scoping reviews can also document research that informs and addresses practice [29]. A scoping review does not include aggregation and synthesis of data nor does it include an assessment of the quality of the documents included [27].

Thus, a scoping review suits our review consideration, namely, to map how routine DHIS2 data use has been documented. Our objectives were to review the literature (peer reviewed and grey) regarding DHIS2 data use, and to categorise key examples of use of DHIS2 data. This scoping review included a review of peer reviewed literature, key journals and conferences, and theses produced within the HISP programme. The primary research question uses the PCC method where the population group are users of DHIS2 data; the concept is DHIS2 the software, and the context is LMIC health systems. Therefore, the review question is: ‘How are DHIS2 data being used for action and decision making within LMIC health systems?’ Sub-questions explored to address the primary research question also included:

The following databases were searched for peer reviewed literature: Pubmed, EMBASE and Web of Science, as these are deemed the most relevant for literature related to the topic (see search strategies in Additional file 1: Appendix 1). The time frame for the search extended from the date of publication of the first article in a given database to March 20^th, 2021.

Due to language limitations, we included only English language articles. We hand searched (manually searched) reference lists of studies deemed to be highly relevant to the review question in order to identify other relevant studies. We sourced grey literature from the International Federation for Information Processing: Working Group 9.4 (IFIP 9.4) conferences (central as they were to HISP researchers’ ability to share their DHIS2 research), and from the Post Graduate (MSc and PhD) theses from the HISP in the Department of Informatics at the University of Oslo. A review of evaluations and assessments of DHIS2 internal to HISP was conducted as part of a separate study by the first author (EB) but this did not reveal any additional detailed examples of data use not previously included in other publicly available documents. These internal reports were not included as part of the scoping review, and consequently no ethical clearance was needed to conduct the review as all consulted material is publicly available.

Both authors (EB & JS) analysed the abstracts and full articles for review according to the inclusion or exclusion categories separately. Where there were conflicts, the authors met and resolved them. Colleagues from existing research and DHIS2 implementation groups within the department agreed to be included if a third opinion was needed, but most of the disagreements centred on ambiguity about the level of detail required for inclusion rather than whether or not articles met the inclusion/exclusion criteria. In these cases, the relevant articles were included in the full text review.

As noted, this review’s sole focus was DHIS2 (and previous versions of DHIS). Inclusion criteria demanded that research and conference articles were peer reviewed and described how the data from DHIS2 was being used for action / decision making OR that grey literature described how the data from DHIS2 was being used for action / decision making. Exclusion criteria included:

The five-stage approach of Arksey and O’Malley [29], progressed by Levac et al., [30] and culminating in the JBI Guidelines approach of Peters et al. [28] was followed. It included the following steps: definition and alignment of objective/s and question/s; development and alignment of inclusion criteria with the objective/s and question/s; description of the planned approach to evidence searching, selection, extraction, and charting; the final searching, selecting, charting, and summarising of evidence. The protocol was initially shared with the Heritage Project: Designing for Data Use (a research group with HISP at the University of Oslo) and its input was invited.

Duplicates were removed electronically in Covidence—a web-based software platform that supports all the steps of systematic literature reviews. Both authors independently screened titles and abstracts using Covidence for inclusion/exclusion. Disagreement between coders was resolved between team members, and even though, as mentioned, internal research groups were available for consultation this was not needed. For full article review both authors agreed on inclusion and exclusion independently and resolved any conflicts—again there was no need to bring in other groups as conflicts were easily resolved.

An extraction template was agreed upon and EB and JS extracted the full articles and grey material using this template. The data extraction template contained: author(s); year of publication; study title; journal/document source; study location; level of health system and health programme; study rationale; and description of use of data. Data was charted and exported from Covidence into Excel software. Standard descriptive information of included texts such as study site, year of publication, type of publication and health level and programme was conducted using this Excel spreadsheet. Study rationale and description of data from the charted data were subsequently categorised in relation to the focus of the study in terms of data use purpose, content, or process.

The findings from the scoping review were presented to the Paper Development Seminar Series at the Department of Informatics, University of Oslo and subsequently shared with other research groups and key individuals external to University of Oslo and their comments invited. This sharing of early drafts was for the purpose of validating the data that were included and providing an opportunity for colleagues to mention other articles or documents, especially grey literature, that we may have missed. The sharing also served to further discussion on what could be done to document use of DHIS2 data and the different conceptualisations of data use.

Both authors are currently part of HISP and by implication can be deemed ‘insiders’, but as noted in Byrne et al. [31] there are both advantages and disadvantages to this ‘insider’ versus ‘outsider’ status. A clear advantage in our case was the knowledge of the network, as well as the ability to identify who was involved in research and documentation of data use. The systematic approach of a scoping review coupled with the sharing of findings with key stakeholders have lent rigour to this review and contributed to a more collegiate interpretation of discovered data.

Findings are presented in two parts – a descriptive analysis of the texts included addressing sub-question 1, and categories of DHIS2 data use reported addressing sub-question 2.

Varying conceptualisations of data use are evident in all the documents reviewed. There is the medical focus on the clinical encounter in terms of tracking patients and managing cases (curative), the engineering perspective involving the manipulation of data into ‘usable’ formats (e.g., dashboards), and the public health perspective involving the use of data for disease prevention and health promotion. The latter category best fits the definition used in our scoping review, in terms of how data are routinely used to improve health care and service delivery at Primary Health Care levels. With this in mind, we concluded that many documents describing the use of data to generate charts and reports were not in fact providing examples of ‘data use in practice’ (unless there was a description of how those charts or reports were used, or a description of the process involved in their production).

It is also clear that many varying conceptualisations of the purpose, or type, of action expected from data are embedded in the HMIS. For example, Kelly et al. [55] question the more scientific ‘decisionistic’ focus on decision-making with an underlying ‘control at a distance’ ethos, i.e. using data in order to ‘control’ or manage performance of facilities, as opposed to processing data in order to provide occasions to hold ‘conversations that matter’. There is also recognition that evidence is socially and historically constructed—with different contexts different people will interpret evidence differently – a point made by Jones in his questioning of the assumptions underlying what is meant by data [20]. Related to this, in their review of design differences across their partnerships, Mutale et al. [56] conclude that different theories of change lead to different perceptions on what information is needed, on the manner in which that change is expected to take place and on who will be the users of that information. Madon et al. [57] also argue that there is a requirement to design and implement health information systems for local decision-making and accountability rather than reduce them to ‘mere reporting tools’. The view of HMIS as mechanisms for reporting is typical of centralistic attitudes to public sector management (see for example [58, 59]), and of expectations experienced in partnerships with international organisations. Each HMIS thus gives what Jones terms ‘a selective representation of the situation’ [20].

The shift over time over what constitutes evidence colours current debate on the use of data for action taking or decision making. Though clinical trials and other evidence for clinical decision making and delivery of health care are important sources of evidence, a more inclusive and sophisticated view of evidence has emerged (for example [60‐63]), with the term evidence- ‘informed’ practice now in more widespread use than evidence- ‘based’ practice [64, 65]. It is questionable whether or not the concept of ‘data use’ is best suited to what we have defined as ‘data use’ in th is review, whether or not we should promote evidence informed action-taking and decision-making and adopt a different language to describe this.

Another debate arose in the work of Asah et al. [32] over who is expected to make decisions. In their case of Cameroon Asah et al. [32] note that decisions are not expected to be made at the level below district and, consequently, facility level users of the system only have permission to input data. Thus, requests for information are made via the district office due, firstly, to lack of access and, secondly, to the expectation that data will not influence decision-making or action. Wickremasinghe et al. [13] note that when the data collectors and users are separate entities it is safe to conclude that the system has been designed for monitoring rather than decision-making. Similarly, many instances of DHIS2 focus on reporting data upwards and not on the creation of data for use at operational or facility level – what Madon et al. [57] refer to as data being used ‘as reporting tools’, the end goal being a fixation on quality-reporting rather than on local use. Data quality and reporting rates are much easier to measure, and consequently easier for donors and governments alike to monitor and include in their evaluations and reports – to paraphrase Robert Chambers what gets measured counts [66], and gets done.

The theory of change in relation to data collection, its processing and use, is often simplistic. Even though there is now increased recognition of a more holistic approach that embraces technical, behavioural, and environmental/organisational aspects of data use, the main focus in documented HIS interventions remains focused on challenges faced or on technical solutions. Hoxha et al. [67] systematically reviewed technical, behavioural and organisational/environmental challenges that hinder the use of routine health information systems (RHIS) data in LMICs and the strategies implemented to overcome these challenges. They concluded that “Additional research is needed to identify effective strategies for addressing the determinants of RHIS use, particularly given the disconnect identified between the type of challenge most commonly described in the literature and the type of challenge most commonly targeted for interventions.” Of the studies identified in their review, the number of articles describing challenges to the use of RHIS was double that of studies describing strategies to overcome them. Additionally, they discovered that even though technical challenges were the least commonly raised challenges in the literature, strategies that incorporated technical components were the most prevalent, many of which involved a focus on developing indicators, registers, and tools for the improvement of data use. On the other hand, only 13% of RHIS strategies address organisational or environmental challenges such as resource shortages, training, feedback, and management even though more than half of the studies described these as challenges. Their review included DHIS2 interventions. So, though it is acknowledged that technology cannot be the sole driver for improved use but can be used as a catalyst for change, there remains a disproportionate focus (or, at least, a documented focus) on the technical side of enabling data use. As Noir and Walsham remind us – Information and Communication Technologies in health often play a ‘mythical and ceremonial role’ and do not necessarily constitute a means to support local action and decision making [68].

Overall, though, we do not conclude that routine data is not being used nor that there is an absence of data culture at facility level. Dahal [69] presents an interesting case. It illustrates that data is being used at the operational level by healthcare workers on a routine basis, but that this is a manual system. The routine data is sent up manually in the system to be included in DHIS2, but this data is never reported or fed back so is not available at lower level for use. There are also many examples of charts being presented on walls or in notebooks which are used to track performance and cases but are not based on the data that have been entered into DHIS2. For instance, Damtew et al. [70] report the case of a community health worker in South Africa who drew a map of the areas where all the tuberculosis patients lived, so that the staff could go and follow-up if patients did not show up for treatment. Likewise, Health Extension Workers in Ethiopia use hand drawn maps to plan daily activities. Similar community level data collection is reported by Moyo [41] in Malawi. However, in all these cases these data were not entered into DHIS2. This could be identified as a shortcoming of this review (see below), that by focusing solely on DHIS2 data it excludes other parallel data use practises. More importantly, though, it raises the question as to why such data are not being used within DHIS2 when there is the functionality within DHIS2 to do so. As Jones [20] notes there are costs to data—costs of producing, storing, retrieving and using data, and we need to consider these when investigating use. Chrysantina et al. [71] offer one possible explanation to non-use: we often assume health staff, once trained, know how to use the various functions in DHIS2, but they find that data literacy is a neglected area in medical school training and in the Continuing Professional Development element of the DHIS2 training curriculum. Walsham [7] notes the relevancy of Gigler’s [72] work on the different capabilities required to use the internet to improve well-being in an under-resourced community in Bolivia. Besides having basic IT capabilities three groups of informational capabilities are needed – communication, information literacy and knowledge sharing. Asah [73] investigates the role of facility managers in empowering the staff with such informational capabilities. Returning to how we conceptualise data Jones [20] argues that we need to understand how data came to be (in terms of what is considered to be the phenomenon, what are considered to be the data about the phenomenon, what can be recorded, what gets chosen to be recorded and what actually gets recorded) as well as how are data used (what gets looked at, what gets found, what gets extracted, what gets understood and what actually gets used). Fundamentally he argues that data in practice is a culmination of a long series of steps and at each step there is the possibility of breakdown and alteration of the data. It is perhaps a combination of the solutions suggested by the above authors that we need to explore.

Walsham’s [7] conclusions (to his reflection on information for action) summarise our discussion well. He raises 4 points:

There are a number of limitations to this study. As mentioned earlier, the focus on DHIS2 may mean we have missed some examples of documented data use practice but given that DHIS2 is one of the largest routine HMIS the findings are likely to be applicable to other systems. The search keyword ‘DHIS’ may have resulted in missed articles or documents that do not specifically include the software platform name in the article, or may have excluded documents that have another name for their RHIS which is built on DHIS2. We tried to address this through the sharing of the draft scoping review article with forums involved in DHIS2 as mentioned, and to snowball from the reference lists of articles included. Additionally, one of the authors (JS) has worked with HISP for 20 years and, 15 years ago, the other author (EB) worked with HISP for a number of years, and has joined HISP as a guest researcher for the year. As interpretive researchers we acknowledge that this scoping review is conducted from a more internal perspective of HISP and that other perspectives and interpretations of the findings would also exist.

Overall, in response to our primary research question ‘How is DHIS2 data being used routinely for action and decision making within the health system?’ we can conclude that DHIS2 data is being used but there are few detailed descriptions of this use in peer review or grey literature. In one regard, this is surprising given the extent of the scale of implementation and use of DHIS2 and the many anecdotal stories we in HISP have of data use – it is in fact HISP’s raison d’être. On the other hand, the paucity of detailed descriptions of DHIS2 use is not surprising given that most of the literature on DHIS2 is written by HISP-affiliated people, who are increasingly occupied with making, implementing, and scaling software. A gap has grown steadily between use and technology development. An effect of the increased collaboration with international agencies is that HISP has become more and more focused on technology and software development rather than on an information system network as evidenced by the DHIS2 papers included in the review by Hoxha et al. [67] – a focus on software use rather than on data use.

Most commonly, we observe that DHIS2 data are being used primarily for planning and performance, for decision making and action taken. Whilst there are some rich descriptions in the examples provided, this richness could be enhanced if the use cases presented in these documents were made useful to other contexts. A three-pronged approach to improving data use is provided by Sahay et al. [74] based on considerable case material from India and Africa. They argue that we need to start focusing on conversations around data, supporting communities of practice as enablers of such conversations, and integrating IT solutions into the work processes and practices of the frontline healthcare worker. We would add to this that there is the need to document in more detail and share how data are being used. This may require research into why this documentation and dissemination is not currently being done. Further retrospective investigation into how data were created and who used the data could help make more explicit the philosophy behind the creation of HMIS, as well as the different conceptualisations of data, use and data use embedded in the HMIS.