Can we trust measures of healthcare utilization from household surveys?

Household surveys are frequently used to understand health seeking behaviour, particularly in developing countries where routine information systems may be unreliable for lack of investment and capacity. They are considered the standard tool for measuring healthcare utilization at the population level. A typical survey first asks household members whether they have been ill over a defined period and if so, whether they sought care and from whom. Recall periods vary between studies but standard practice is to extend the recall period to no more than one month in the case of outpatient care, and no more than a year in the case of inpatient care.

Measures of healthcare use from household surveys are widely used by the research community – to understand care seeking choices and to evaluate health programmes. Studies of the effect of health financing policies in developing countries rely heavily on such measures. Prominent examples in the recent literature include a randomized assessment of the Mexican universal health insurance programme [1] and various evaluations of the cooperative medical scheme in rural China [2, 3]. Utilization measures from household surveys also provide the basis for conducting benefit incidence analyses in health [4, 5].

To what extent can we trust measures of healthcare utilization from household surveys? There are several reasons to be cautious. First, individuals may struggle to recall episodes of illness and visits to healthcare providers. This failure to recall may vary systematically across population groups, thereby generating biases that have important implications for interpretation of the data. It is well documented, for example, that poorer individuals are less likely to report illness even though they experience greater morbidity [6]. A recent study in India experimentally assigned households to surveys with weekly and monthly recall periods, finding that the recall period had a large effect on reported morbidity, doctor visits and self-medication [7]. The impacts were particularly large for poorer households and those with a high burden of disease. However, when recall periods are shorter bias may be less of a problem. A study conducted in multiple developing countries showed that using a 7-day rather than a 2-day recall generated little bias in self-reports of illness [8]. Second, recall may vary according to the financing arrangements in place. If an individual has to pay out-of-pocket for care, he or she may be more likely to remember a doctor visit than someone who is covered by health insurance, particularly when it is costly. Such detection bias is why whenever feasible randomized controlled trials conceal assignment [9].

Much of the literature on problems with survey methods has addressed the issue of recall, but few studies have explored methods of data collection. In this paper we focus on the latter using data from rural Ghana. We first compare household survey measures of morbidity and healthcare utilization against our benchmark measure taken from a pictorial diary designed specifically for the study. We then take the novel step of comparing the results of a randomized trial of removing direct payments for healthcare using both sets of utilization measures. Our findings raise a number of concerns that have important methodological implications for our understanding of health reporting and the evaluation of policies on utilization of healthcare in resource-poor settings.

Table 1 provides descriptive statistics of the morbidity and utilization measures. According to the pictorial diary data, 41% of children had at least one episode of illness in the past month, compared with 46% in the household survey data. When the reference period is extended, the difference between the two sources of data becomes large. The pictorial diary data show that 93% of children had at least one episode of illness over the six month period of data collection. By contrast, the proportion of children with any illness in the past year based on the household survey data was 60% despite the fact that the reference period is twice as long. These data suggest there was severe under-reporting of illness in the household survey. At least one third of morbidity (33%) – and probably more if the reference periods were identical – was not captured by the household survey.

The proportion of children who sought healthcare in the past month is reasonably similar across the two sources of data in the case of both clinic visits (18% pictorial diary vs. 15% household survey) and informal care visits (16% pictorial diary vs. 21% household survey). When the reference period is lengthened to six months/ one year, the method of data collection appears important in the measurement of any clinic visit (63% pictorial diary vs. 22% household survey) and informal care visits (67% pictorial diary vs. 27% household survey).

Table 2 reports the effect of removing direct payments for healthcare across the different measures of utilization. The intervention had a positive effect on primary care visits according to the pictorial diary data (Table 2, Panel A). The unadjusted estimates indicate it increased the number of visits by 0.30 per year (unadjusted rate ratio of 12 percent) and the probability of any visit in the past six months by 3.7 percentage points (unadjusted rate ratio of 6 percent). According to the household survey data, the intervention had no statistically significant effect on clinic use (Table 2, Panel B). The point estimates are negative suggesting that, if anything, the intervention reduced primary care use. The point estimates for informal care utilization are negative irrespective of which data collection method was used. Only in the case of informal care visits from the pictorial diary is the estimate significant, with a negative effect of 0.27 per year (unadjusted rate ratio of 9 percent).

Table 3 shows the extent to which the method of data collection affects the estimated impact of the free care intervention on healthcare utilization by reporting the coefficient on the interaction between free care and the data collection method. With respect to clinic visits, the data collection method appears to matter. Depending on the reference period, the effect of free care on use of clinic visits is 3.6 percentage points to 5.4 percentage points smaller with the household data. In the case of informal care visits, there are no differences in the effect of free care with respect to the data collection method.

Using pictorial diary data as our benchmark, we addressed the question of whether household survey data on health seeking behaviour provide reliable measures of healthcare use. We found substantial under-reporting of morbidity and healthcare utilization when recall periods were extended beyond one month and qualitatively different conclusions as to the effect of removing user fees on primary care use. On a practical level the study demonstrated the feasibility of implementing pictorial diaries in a rural population where a sizeable proportion of respondents were illiterate. The caregivers did not appear to have any difficulty in using the pictorial diaries as a means of reporting health events.

The severe under-reporting of morbidity was consistent with a study in India examining how different recall periods affect reporting [7]. Since morbidity acts as the standard screening question prior to asking about health seeking behaviour in household surveys, this result inevitably implies substantial under-reporting of healthcare use. The findings were also consistent with those of a study in the US on alcohol consumption and sexual activity which found that participants more often underreported their sexual activities and condom use during retrospective studies as compared to when diaries were used [17].

Evidence on how the survey method alters the conclusions of the randomized trial of removing direct payments for healthcare is the most novel aspect of the paper. The change in the direction of effect between data sources is both intriguing and worrisome, given the ubiquitous use of household survey data in the evaluation of health policies. We may have viewed the findings with more scepticism had they been based on an observational study. But the randomized design and the fact that the treatment effects are negative for both of the two household survey measures of primary care utilization lend credibility to the findings.

We can only speculate as to the reason for this finding but one hypothesis is that the recall of families in the intervention group deteriorated precisely because they no longer had to pay for healthcare, making the event less salient and more easily forgotten, thereby dampening the true effect of the intervention^b. If so it would provide an example of how detection bias can emerge passively through the influence of an intervention on the recall of self-reported behaviours. Because the intervention did not affect the price of care at informal providers the salience and recollection of informal care seeking events should not have been affected by the intervention. Indeed our findings regarding the effect of the intervention on informal care visits were consistent across the two methods of data collection, providing further support to this hypothesis.

The study had a number of limitations that should be noted. First, although the contribution of the study is methodological, it is based on empirical findings from one district in Ghana. However, there is good reason to believe recall problems are universal. Extensive evidence from the US showing that around 20 percent of outpatient events are under-reported when the recall period extends beyond two to three days [25] has been confirmed by studies in developing countries [26, 27]. Second, owing to data limitations we were only able to construct one indicator of utilization that was standardised across the two data collection methods. A greater number of comparable measures of utilization would have strengthened the analysis. Finally, our method does not allow recall effects to be separated from data collection methods, although it seems likely the channel through which the data collection method altered the effect of free care is via recall.

The findings raise methodological concerns about measures of healthcare utilization from household surveys, particularly in the evaluation of health financing interventions. Further research is needed into the problems of bias when collecting utilization data in household surveys. Such data are widely used by health researchers to understand health seeking behaviour, in benefit incidence analysis and in the evaluation of interventions and policies. The implications of the study’s findings are therefore potentially far reaching. It may be the case that the trend towards using household diaries of income and expenditure should be extended to the collection of data on healthcare utilization.