Wanted: studies on mortality estimation methods for humanitarian emergencies, suggestions for future research

Simple random sampling, whereby n elements (e.g. households) are randomly selected from a list of the total elements, N, is the gold standard of representative sampling, but can only be performed if a list of all individual households with a unique identifier is available. When the area to be covered by the survey is large or the population is very unstable, as in a recently formed displacement camp, creating such a list may be extremely costly and time-consuming, thus defying one of the main purposes of surveys, which is to furnish rapid estimates. Systematic sampling, whereby each n th household is sampled, is the second best choice: it requires only knowledge of the total number of households, but is mainly feasible if households are laid out along clearly defined streets or in a clear pattern.

Simple and systematic random sampling designs are explained fully elsewhere [9] and, as their application for mortality estimation does not present specific challenges, they are not explored further here. Although they may be practicable only in some situations, investigators should always consider these two options first, especially if a sampling frame of households can readily be constructed. Analysis of a simple or systematic random sample is straightforward, robust and amenable to various statistical analyses, including post-stratification. These sampling designs also facilitate adequate sample size calculation, as they do not require any assumption about clustering of deaths (see below).

Cluster sampling should be used only if simple or systematic random sampling techniques are prohibited by cost, time or logistical constraints. In multi-stage cluster sampling, the entire sampling universe is firstly divided into well-defined areas such as districts or villages. The required numbers of primary sampling units (PSUs or clusters) are then selected from these areas using probability proportional to size (PPS), and occasionally using spatial (random co-ordinate) sampling. Most survey statisticians accept PPS cluster allocation as being a valid approach, although with some caveats, such as the accuracy of the population size data from which the sampling occurs. The final stage of sampling consists of selecting a given number of basic sampling units (in mortality surveys, these are usually households) within a cluster. We focus here on one approach for household selection that has led to much discussion.

In 1978, the World Health Organization's Expanded Programme on Immunization (WHO EPI) adopted the "30 by 7" two-stage cluster sample survey design for rapidly estimating vaccine coverage (VC) in children aged 12 to 23 months [10]. Manuals and step-by-step guidelines [11, 12] were subsequently developed and the method has since become widely used [13‐19]. Briefly, the WHO EPI "random walk" approach for household selection involves choosing a random direction from the centre of the community and then selection of one household at random along an imaginary line connecting the centre to the periphery. Subsequent households are then selected by proximity (the next nearest household) or visiting every N th closest household until the sample size for the cluster is achieved [12].

The EPI method has since been adapted for use in measuring nutritional status [20, 21], retrospective mortality [22, 23] and other variables [7, 24, 25]. For non-VC surveys, a sample size of 900 tends to be used (30 clusters of 30 households or children, depending on the variable being measured). Despite recommendations to improve and adapt second stage sampling for non-EPI purposes [17, 26, 27], over time it has become common practice to apply this survey method without questioning its appropriateness for the outcome being measured.

The main problem with the random walk approach followed by proximity sampling is that the probability of a household being selected is unknown, since the total number of households in the community is unknown. Preferred methods entail selecting a fixed number of households in a cluster so that the probability of selection can be calculated. Segmentation is one such method. The defined area to which the cluster has been allocated is further divided into segments of approximately equal size, at a level at which a complete listing of households in the segment can be made. One segment is then selected randomly [17], and all households within it may then be selected or, alternatively, the desired number of households to make up a cluster may be selected using simple or systematic sampling. However, segmentation is not always possible due to time, security and human resource constraints.

Other methods exist for selection of the first household in the cluster. For example, for sites with an existing map, a "sampling grid" can be superimposed and co-ordinates selected at random [28]. Similarly, GPS units can be used to establish the boundary of the cluster area, and select a random coordinate within it [29, 30]. For either of these methods, we recommend selecting subsequent households in the cluster by a different proximity rule – e.g. not next nearest, but every third or fifth household. This would result in greater geographical spread, thus decreasing clustering (see below) and favouring randomness.

Risk of death may be clustered, a phenomenon that is probably more pronounced in emergencies; particular villages may suffer disproportionate violent attacks, be different in terms of access to healthcare, or differ in certain epidemic-prone diseases like measles, which may spread from index cases to neighbours and close contacts [31, 32]. Thus, there is an increased probability that individuals within clusters will resemble each other (i.e. share a high or low risk of mortality), and differ considerably from individuals in other clusters. This probability increases as the number of clusters selected decreases, and as the sample size per cluster increases. High intra-cluster homogeneity decreases precision, since it results in larger standard errors and correspondingly wider confidence intervals.

There are two related measures of clustering within a cluster sample: the ICC and Deff. The ICC indicates the likelihood of individuals within the same cluster sharing the same outcome (an ICC of 1 means total clustering, and an ICC of zero means no clustering). The ICC depends on the variance within and between clusters, the actual frequency of the outcome being measured, the cluster size [33], and the number of clusters. Derived from the ICC and the cluster size, Deff has been defined as the ratio of the variance taking clustering into account to the variance assuming simple random sampling [34]. The Deff has been used more often than the ICC in mortality surveys since it provides an inflation factor for both calculating an adequate sample size before the survey (based on the expected Deff), and adjusting confidence intervals after the survey (based on the observed Deff). Figure 1 shows the variation in the 95% confidence interval and Deff for different numbers of clusters and varying numbers of individuals per cluster.

There is little published research from mortality studies on observed Deff and ICC to provide accessible guidance for non-statisticians in the context of humanitarian emergencies [35]. We recommend that, before any cluster survey, investigators perform a literature search for past reports providing Deff or ICC within the population to be studied, so as to inform sample size calculations. Henceforth, both statistics should be calculated after each survey, and presented in survey reports or publications so as to improve future survey designs.

Every stage in sampling (including any sub-segmentation required to home in on a sufficiently small enumeration area) can introduce a further level of clustering; e.g. clustering of deaths within the district, within villages in the district and within neighbourhoods of the village. Additional clustering may even occur within the households themselves; for example, a household in which a mother has died during childbirth is also more likely to experience infant deaths. Most data analyses only take into account variance at the PSU level (the variance accounting for differences between clusters), and ignore the household level. Although it is generally thought that once the variance at the cluster level is accounted for the variance at the household level is minimal, this has been poorly studied. We encourage investigators to assess this assumption in studies of mortality during humanitarian emergencies.

In any survey there will be some non-responders, due either to absenteeism or refusal to participate. The greater the proportion of non-responders, the higher the risk of selection bias, and the harder it becomes to interpret findings, since in most circumstances one cannot safely assume that non-responders will experience the same mortality risk as those who enter the sample. Investigators should thus attempt to minimise absenteeism and refusals. Surveys should be performed when people will be near their homes, and there should be a call-back procedure in place to deal with absenteeism; for example, survey investigators could return three times to check on absentee households. One way to pre-empt non-response is to inform community leaders and perform community sensitisation about the timing and objective of the survey, before data collection starts. However, this will result in many people staying at home in expectation of the survey, not realising that only a few households will be sampled: the benefits of this strategy need to be weighed against the harms of entire communities missing out on valuable work time.

Even the best surveys will have a number of non-responders. It may be better, therefore, to assume a certain proportion of absentees or refusals and to 'inflate' the survey sample size in advance accordingly, than to 'replace' non-responding households at the end of a cluster. For example, if the desired sample size is 900 (e.g. 30 clusters of 30 households) and we expect non-response from 10% of households, then the total sample size could be increased to 1000, or by about 4 households per cluster. 30 clusters of 34 households, or 1020 households, should give a final sample of 918 (i.e. 1020 - (0.1 × 1020)), slightly higher than that required for a sample size of 900 (in a 30 × 30 cluster survey). This technique is known as 'inflation'.

Points in favour of inflation are firstly that all households in a cluster will have an equal probability of being selected. Replacement violates this condition, which could become a concern if the enumeration area population is small relative to the intended cluster size. Secondly, the temptation of convenience sampling (i.e. going to households that seem available) may be reduced. Inflation does, however, have two problems. Firstly, there is a risk of not achieving the desired sample size if there are more non-responses than expected. Secondly, there may be a need to use weighting in the analysis to adjust for different cluster sizes occurring as a result of differing patterns of non-response. If non-response is associated with mortality, both replacement and inflation could result in selection bias because non-responders will not be represented in the sample. Inflation may be theoretically less biased than replacement (although there are no data to support this) since it does not result in over-sampling of non-responders. However, it could be argued that by giving greater weight to smaller clusters (i.e. those with high levels of non-response), these smaller clusters will be over-represented; this introduces a similar bias to that of replacement. In practice, the decision to perform weighting after inflation needs to take into account the reasons for non-response to avoid selection bias as a result of over-representation of non-responders. Accurate documentation of the reasons for each instance of non-response should be standard practice in every survey; this information will be key in interpreting findings, assessing the possibility of selection bias, considering the pros and cons of weighting, and could be important in performing sensitivity analyses which take likely mortality levels among non-responders into account.

In the past household census method all household members present at the beginning of the recall period are listed. The method includes questions about those who have been born since the start of the recall period and those who have out-migrated during the recall period. In-migrants are not usually included [39]. The denominator is calculated as the total number in the household at the start of the recall period, plus half of the newborns, minus half of the deaths and out-migrants, multiplied by the recall period.

In the current household census method, investigators list all household members at the end of the recall period, as well as all births and deaths during the recall period. In-migrants are included in this approach, but out-migrants who are not a part of the household at the end of the recall period are missed. The denominator is calculated as the total number in the household on the day of the survey, minus half of the births, plus half of the deaths, multiplied by the recall period.

In both the past and current census methods some exposure time is unaccounted for in the denominator. The hybrid method is a combination of these methods attempting to account for both in- and out-migration. It is described in the Standardised Monitoring and Assessment of Relief and Transitions (SMART) and WFP manuals [39, 38].

A modification to the prior birth history method has been put forward as an alternative tool for estimation of under-5 mortality [40]. Mothers are asked about all the children born in the last five years, and their fate during the recall period. Here the denominator can be calculated as the total number in the cohort of children under five who are alive at the start of the recall period, minus half of the deaths plus half of the births, multiplied by the recall period.

A disadvantage of this method is that orphans are missed, and mortality may thereby be underestimated. Underestimation may also be a problem in situations of very high maternal and child mortality [41, 42]. An advantage, however, is that both under 5 mortality during the recall period and that over a 5-year period can be computed. This potentially provides a local pre-crisis baseline.

Generally, we recommend recording individual rather than aggregate data. Although it may appear more time-consuming, listing each individual by sex, age, and date of death or in-/out-migration (as appropriate), is less complex for field teams and facilitates analysis.

If the approximate dates can reliably be established, each individual's person-time may be computed. This provides a more accurate denominator, especially when populations vary non-linearly during the recall period [43]. The related indicator of mortality would then be a real rate showing death occurrence per unit of person-time for a specific population size (e.g. number of deaths per 10 000 person-days). Whilst most populations are more or less static cohorts without much in- or out-migration, others undergo massive fragmentation and displacement during the recall period – for example, Hutu refugees who fled across Zaïre in 1994–1997, and ethnic Karen communities who either remained in Burma or fled to Thailand [44, 45].

The benefit of having a more accurate estimate by measuring person-time should be weighed against the difficulties involved if the cohort is dynamic. This could involve a more complex questionnaire; for example, when attempting to establish dates for births, deaths, and in- and out-migrations using a local events calendar. A local events calendar is needed if accurate dates are either unknown or cannot be approximated by household respondents.

Age-standardised rates are more informative than crude mortality rates, and can be computed using the WHO standard [46]. Whether or not mortality rates are age-standardised, it is important to illustrate the sampled population's age structure using age pyramids. A sex or age group with a clear deficit could indicate recent high mortality, high out-migration, over-representation of other age-groups and/or sex, high mortality several years before the survey, or changes in birth rate. Age pyramids are best constructed using 5-year age-groups [23]. Those under 5 can be shown on a separate age-pyramid by one-year categories to highlight recent trends among newborns.