Systemic factors of errors in the case identification process of the national routine health information system: A case study of Modified Field Health Services Information System in the Philippines

The item on "TB symptomatics with sputum exam" showed a difficulty index of 4.4% (5/113). For this item, 20.4% (23/113) of respondents chose at least all of the eligible cases and added suspected cases of tuberculosis regardless of the results of sputum examinations while the same number of respondents of another 20.4% (23/113) chose only suspected cases of tuberculosis regardless of the results of sputum examinations. Similarly, the item on "TB new sputum positive initiated treatment" showed a difficulty index of 3.5% (4/113). For this item, 37.2% (42/113) of respondents chose at least all of the eligible cases but added the non-eligible case with sputum negative result; 47.8% (54/113) chose some of the eligible cases but failed to choose all of them. The item on "Rabies: Animal bite cases seen" showed a difficulty index of 8.8% (10/113). For this item, 63.7% (72/113) of respondents chose all of the eligible cases but added patients with snake bites under this indicator. The item on "Pneumonia cases seen (0-59 months)" showed a difficulty index of 10.6% (12/113). For this item, 51.3% (58/113) chose at least the eligible case while 38.9% (44/113) added all severe pneumonia cases classified in the case definition of notifiable diseases [12], which were supposed to be excluded under this indicator. Another 15.0% (17/113) chose only the severe pneumonia cases. The item on "Family Planning: Current Users" had a difficulty index of 19.5% (22/113). However, 26.5% (30/113) of respondents answered the number of "Current Users of the reporting month" that was available when the number of "New Acceptor of the reporting month" was used instead of the number of "New Acceptor of the last reporting month" to add to the number of "Current Users of the reporting month" in the calculation. The item on "Fully Immunized Children (9-11 months)" showed a difficulty index of 12.4% (14/113). For this item, 45.1% (51/113) chose at least all of the eligible cases while adding a non-eligible case. For example, 37.2% (42/113) of respondents added children who received the measles vaccine after one year old who were supposed to be excluded.

The item on "Pregnant women with 3 or more prenatal visits" showed a difficulty index of 14.2% (16/113). For this item, 64.6% (73/113) of respondents chose at least the eligible case but added the case of a pregnant woman who had her first prenatal visit for the third trimester in the previous month. This same pregnant woman was eligible for the previous month although she had another visit in the third trimester in the current month. The item on "Infant given BCG" had a difficulty index of 24.8% (28/113). For this item, 29.2% (33/113) chose all of the eligible cases but added the non-eligible cases. Another 25.7% (29/113) chose at least an infant given BCG by the assigned facility but excluded those infants who did not reside in the assigned area of the health worker.

The item on "Pregnant women with TT2 plus" showed a difficulty index of 67.3% (76/113). However, 17.7% (20/113) of all respondents did not include pregnant women who received the second dose of tetanus toxoid (TT2). The item on "Family Planning: New Acceptors" showed a difficulty index of 46.9% (53/113). For this item, 52.2% (59/113) chose the eligible case but added clients who were supposed to be categorized into Current Users or Drop Outs. For example, 43.4% (49/113) added clients who changed their contraceptive methods and 17.7% (20/113) added clients who changed clinics without changing contraceptive methods. The item on "Severely underweight children (6-59 months)" showed a difficulty index of 48.7% (55/113). For this item, 34.5% (39/113) of respondents chose only either "Severely Underweight" or "Below Normal (Very low)," even though both criteria must be chosen to identify the case. In the malaria control program, the item on "Malaria: Confirmed Cases" showed a difficulty index of 57.5% (65/113). For this item, 17.7% (20/113) of respondents chose at least all of the eligible cases and added suspected cases of malaria regardless of the results of the blood smear examination.

Four of six indicators with low difficulty indices that led to non-significant differences in the level of understanding highlight characteristics of indicators that are unsupported by the current conditions in the health system. These indicators can be improved by first ensuring consistency between the definition of the indicator in the FHSIS and the current condition of health systems.

The definition of "TB symptomatics with sputum exam" seems to have been practically simplified by health workers according to the available function of generating data since most of the time health posts were not equipped to do the sputum microscopy. Although this item showed a low difficulty index, those who selected the correct choices may have established a feedback mechanism with their RHUs. Another similar example is "New sputum positive initiated treatment" for tuberculosis. This requires initial treatment but the treatment is given to a patient as soon as the patient is confirmed as tuberculosis at RHU, not at BHS. For both indicators, BHS cannot directly collect data for reporting. Indicators that require data to a facility that cannot directly generate the data was also reported in Chad [24]. On the contrary, the indicator of "Malaria: Confirmed cases" similarly requires blood smear examination at BHS. However it showed a relatively high difficulty index with a positive discrimination index. This can be explained by the availability of barangay malaria microscopists at health posts for blood smear examination in the province of Palawan. Unlike the other malaria endemic areas of the Philippines, Palawan has its own malaria control project called Kilusan Ligtas Malaria that trained and deployed malaria microscopists in the barangays (villages).

There seems to be a gap between the definition and practice of data collection for the indicator on "Rabies: Animal bite cases seen" because animal bite cases seen in RHU and BHS are often transferred to animal bite centers in hospitals and some RHUs, where a post exposure immunization is available and given if necessary. As a result, what health workers know is "animal bite cases transferred" when there was no feedback from animal bite centers. Also, ambiguous definition can be another reason because "others" in the definition of the indicator allows health workers to consider several interpretations. Even "snake bites" was considered as suspected cases of rabies. Since DOH intended "others" to mean animals with risks of transmission of the rabies virus, such as mammals or canines, an additional definition specifying the meaning of "others" would be another approach to make the definition more understandable.

"Pneumonia cases seen (0-59 months)" showed that severe pneumonia cases were considered eligible cases to report while the services accomplishment component of FHSIS excludes severe pneumonia cases from its definition of "Pneumonia cases seen (0-59 months)". This can be partly explained by the inconsistency between the definition in FHSIS and the case definition of notifiable diseases. Although the definition of this indicator in FHSIS covers children aged 0-59 months, there are only two categories for children less than 2 months - severe pneumonia and no pneumonia - according to the case definition of notifiable diseases. Confusions among health workers created by inconsistencies between definition of RHIS and its corresponding program were also known in other countries [25]. Another reason for the confusion can be a limited role of health workers because final diagnosis was often given by doctors while health workers have already counted them as pneumonia cases.

On the contrary, "Severely underweight children (6-59 months)" showed a relatively high difficulty index with a positive discrimination index although there were inconsistencies of two different definitions of weighing in FHSIS and the corresponding program. This indicator had the characteristics of covering cases in both daily health services and biannual special campaigns such as Operation Timbang [26]. In the 2004 update of the nutrition program, a new indicator was introduced for underweight cases reported from special campaigns of Operation Timbang [27], yet there was no update in the corresponding indicator in FHSIS in 2006. Both of these indicators used different criteria following a different standard of growth monitoring. Consequently, both of these criteria must be chosen to identify the eligible case in FHSIS. The health workers with the correct answer seem to have been informed of the situation through the Municipal Nutrition Office since the Nutrition Program gave an explicit instruction of difference between Operation Timbang and FHSIS in the implementation guideline [27].

Two indicators with low difficulty indices that led to significant differences in the level of understanding highlight characteristics of incomplete or ambiguous definitions. These indicators can be improved by clarifying their definitions first.

The item on "Pregnant women with 3 or more prenatal visits" showed that health workers may have at least two other interpretations for the pregnant woman who had several visits in different months in her 3rd trimester, such as reporting the same pregnant woman more than once or reporting the pregnant woman late by waiting until the woman completes the delivery. Even though DOH intended that the eligible pregnant woman be reported once during the pregnancy as soon as she met all criteria in the definition, an explanation of timing and number of reports was lacking in the definition. This would require an additional definition of the timing of report and a definition that the same pregnant woman should be reported only once during her pregnancy.

The item on "Infants given BCG" showed that the definition can allow health workers to consider several interpretations of eligible cases based on different target populations and eligible cases of FHSIS reporting. Since this indicator was added after implementation of the official guideline of FHSIS 1996, the department circular No. 289 s.2000 was distributed to health administrative offices and field health service units to inform them of its definition. Even though DOH intended the case to include infants who received BCG vaccine at field health service units regardless of infants' residence, such as within or outside the catchment area of the assigned field health service unit, there is no explanation of such specification in the definition. This situation allowed another interpretation of reporting infants who received BCG vaccine at the assigned field health service unit only when the infants resided within its catchment area. The original intention of FHSIS was to address the short term data needs of DOH staff managerial or supervisory functions in DOH facilities and in each of the program areas [26]. FHSIS was updated in 1996 to accommodate the situation under the devolution. Thus, Local Government Units were included as users of FHSIS for their management role of field health services unit [11]. These intentions led to a conclusion that the eligible population of all indicators in FHSIS was identified from a population who received health services at the assigned field health service unit. However, identification of eligible cases like an indicator of Fully Immunized Children (9-11 months) requires records of all given antigens even if some of them were not given at the assigned field health service unit. Furthermore, the number of the target population of each of required antigens and Fully Immunized Children (9-11 months) were projected based on total population from census. These practices seem to be a source of the confusion because they imply that eligible cases of reporting in FHSIS are identified within the target population of the responsible field health facility.

Two indicators with higher difficulty indices that result in a significant difference in the level of understanding among respondents highlight characteristics of indicators with complete definition yet are easily misunderstood by health workers. These indicators may need improved instructions so that all health workers will have the same level of understanding of the indicator. For example, a misunderstanding found in the item on "Pregnant women with TT2 plus" is explained by the understanding of the meaning of "TT2 plus"; "TT2 plus" could be interpreted as more than TT2. The item on "Family Planning: New Acceptors" indicated that most misunderstandings were explained by understanding the meaning of "New Acceptors". "New Acceptors" could suggest "new to methods" or "new to a clinic". Since definitions of these terms are clearly described in the official guideline of FHSIS, these items may show a need for further training. However, specification of terms can be considered further improvement of instructions, such as "TT2 to TT5" instead of "TT2 plus", and "New to program" instead of "New Acceptors". Differences in understanding of "New Acceptors" among health workers in RHU and those in BHS could be partly explained by the existing protocol of the Family Planning Program. Clients who are new to the program are recommended to visit RHU first for an examination, and are then followed up by BHS. Not all clients may visit RHU first but the frequency of meeting such clients will still be higher in RHU.

Even if the definition of the indicator in the FHSIS is consistent with the current condition of health systems, when an item shows a low difficulty index against a non-significant discrimination index, the indicator seems too difficult for health workers and the definition itself may require re-definition to simplify it.

For example, the definitions of the indicators for "Family Planning: Current Users" and "Fully Immunized Children (9-11 months)" were consistent with the current condition of health systems and clearly described in the manual of FHSIS. However, these indicators showed a relatively low difficulty index with a non-significant discrimination index. "Family Planning: Current Users" may be more difficult for health workers even though its definition is clearly described. To fully understand the indicator, health workers need to understand not only the term definitions but also a formula before its actual calculation unlike the other indicators which only require understanding of term definitions.

"Fully immunized children (9-11 months)" demonstrated the confusion among health workers between children who were given all antigens less than 1 year old (Fully Immunized Children) and those given all antigens regardless of being less than 1 year old.

If the instructions do not promote understanding of these indicators, the indicators themselves may require re-definition, such as asking for a report of the components of the formula or introducing a new indicator.

In addition to the systemic reasons described above, weakness in the current mechanism to achieve a full understanding of the FHSIS standard seems to influence the confusions among health workers. For example, the case definition "children given all antigens less than 1 year old" is easy to understand once there is a chance to learn about it. According to staff of the Provincial Health Office of Palawan, it is difficult to keep the right knowledge in their health system because of frequent staff changes and limited opportunities for training. Even though a training of trainers approach has been applied by the Provincial Health Office of Palawan to transfer knowledge and skills to municipalities, a reality of training under municipality level is still unknown. Better understandings of "Fully immunized children (9-11 months)" among health workers in the island municipality may be partly explained by the efficiency of training due to relatively smaller number of health workers. In such a situation under limited opportunities for training, the definition of indicators must be simple, and recording and reporting forms used for routine work need to contain instructions for data handling procedures [28].

Even if these 12 indicators were technically important for the management of local health systems, they are impractical because they risk inducing health workers to commit errors.

Figure 2 summarizes a link between behavior of a difficulty index, a discrimination index and possible countermeasures to be considered to achieve full understanding of definition of indicators. It also provides a guide to investigate potential causing factors of errors in case identification. For example, when one indicator shows non-discrimination against low difficulty index, in addition to investigations on ambiguous or incomplete aspect of the definition of indicators and those on needs for further training, investigations may be required for the consistency of the definition with the current condition of health system. Also when the consistency is assured and the definition was found to be complete, an investigation on difficulty among health workers of the indicator may be required. Such investigations are expected to provide the most probable and influencing factors of health workers' confusion. Furthermore, a characteristic of a discrimination index of not independent of the achievement level of the examinee population [29] implies that the combination of discrimination and difficulty indices can be used for tracking the efforts of continuous improvement in the system. For example, with keeping the original intention of the system, when the system design was modified and new instruction was given to health workers, and then health workers' level of understanding was increased, newly calculated discrimination index would tell again newly identified improvement opportunities. Also it would guide investigations into the most probable and influencing factors of health workers' confusions.

In low-information and low-communication technology (ICT) settings, manual and paper-based data-handling are still being used in RHIS, especially in local health systems. In such systems, most data-handling stages depend on human capacity. Even in computerized health information systems, the data entry step cannot be done without human effort. Therefore, control of human errors is one of the key issues to overcome in order to produce quality data in RHIS.

Errors are seen as consequences rather than causes, having their origins in "upstream" systemic factors [14]. The results of our study show that even in an integrated RHIS such as the FHSIS 1996 version in the Philippines, the quality of data for each indicator can be influenced by its characteristics in the case identification stage. More general factors influencing RHIS performance and their hypothetical relationships are described as three determinants, including technical, organizational and behavioral factors [1]. Negative influences of these factors need to be minimized, especially when data-handling of each indicator is done by human effort.

Continuous training is known as an effective method of improving the accuracy of data [30]. However, in low-resource settings, one limitation of a sustainable system operation is the small number of training opportunities [5]. This situation was applicable in our study site in the Philippines. If this is a limitation we could not avoid, the system needs to be designed as simple as possible for health workers while keeping its original purposes. Also, even if enough training opportunities exist, effective training cannot be designed when the standard of the health information system is neither coherent nor continually updated to the current situation of health systems [31]. When the system standard itself contains contradiction or imposes impossible tasks on health workers, we cannot expect that a continuous training approach alone will produce quality data.

In industry, it is well accepted that "quality is built in process [32]". "Process" (a set of interrelated or interacting activities which transform inputs into outputs [33]) is a unit of management that is designed, controlled and improved continuously to produce its intended output. Taking systemic factors into account, the case identification stage needs to be designed to produce its intended output. Further studies are needed of the systemic factors that may affect data quality and process designs that could control them.

Certain limitations of the study should be borne in mind. First, the findings from the questionnaire survey may not be entirely applicable to other areas in Palawan and the Philippines. Data were collected from health workers in municipalities in the mainland and nearby islands of Palawan. These municipalities are more readily accessible to the capital city of Palawan, thus they would be expected to have better access to a formal instruction of FHSIS. Second, the identified possible errors through focus group discussions and field visits, and related systemic factors may be a part of all existing errors and factors, since not all distracters of each indicator may be identified or reflected in the items. Although multiple-answers were applied for asking eligible cases in the questionnaire, difficulty and discrimination indices may change according to the standard of the RHIS and existing cases of distracters in the actual field. Third, the impacts of identified possible errors on quality of data remain unclear. Quality of data may not be influenced even when health workers misunderstood indicators if there was a place with no case of distracters used in each item and the respondent was assigned in such place. Fourth, program assignment, number of years of experience and experience of the specific formal trainings were asked in the questionnaire. However answers were not adequately available. Health workers had difficulty to distinguish the formal training and informal training. Also answers for the assigned program were omitted because the answers were obvious for health workers. In most of the cases, one midwife is assigned for at least one catchment area. Seven health workers refused to provide their year of experience. However, the length of experience did not appear to correlate with the health workers' total score. Coefficient of correlation of year of experience of the available 106 health workers and their total score was -0.063. Fifth, the study focuses on conformance of the current condition to system standard rather than fitness of the standard to use of data. In order to assess the RHIS design in relation with the needs of users, further investigation to gain a better understanding of use of data may be required based on the reality of practices in local health systems. Nevertheless, our results show that indicators can be characterized by how the health workers understand them.