Missing Millions and Measuring Development Progress

Summary

In developing countries, assessments of progress toward development goals are based increasingly on household surveys. These are inappropriate for obtaining information about the poorest. Typically, they omit by design: the homeless; those in institutions; and mobile, nomadic, or pastoralist populations. Moreover, in practice, household surveys typically under-represent: those in fragile, disjointed households; slum populations and areas posing security risks. Those six sub-groups constitute a large fraction of the “poorest of the poor”. We estimate that 250 million are missed worldwide from the sampling frames of such surveys and from many censuses and their omission may well lead to substantial biases.

Introduction

For several decades and in some countries for centuries, populations have been counted through national, usually decennial, censuses in which enumerators go to households. Inter-censal population estimates have usually depended on reliable birth and death registration systems. In the 2001 Census, the UK moved away from direct enumeration by asking people to self-report. In many developed countries, there are moves toward substituting administrative records such as municipal population registers and ad hoc or existing surveys for the once-in-a-decade census which is seen as cumbersome, rapidly out-of-date and encountering increasing difficulty in getting citizens to complete the census form. Comparing 2000 and 2010 for 40 European countries, Valente (2010) shows how the number using the traditional method has declined from 27 to 21 while the numbers using registers or a mixture of registers and total enumeration of sample surveys has increased from 9 to 18.

In other rich countries there is an increasing reliance on data linkage through, for example, linking the tax system with an Identity Card or Number that citizens are required to have by law. In most middle and low income countries, however, vital registration systems have never been fully functioning (Chan et al., 2010, Powell, 1981, Vlahov et al., 2011), and there has been a similar decline in donor interest in censuses and vital registration systems (Setel et al., 2007), as evidenced by the demise of the International Institute for Vital Registration and Statistics, and an increasing reliance on household surveys.

Many countries run national economic and social surveys to provide detailed information on consumer prices, income, and employment and other relevant data for planning. But the main sources are often internationally standardized surveys with reasonably large sample sizes (see Table 1); and, although now many of these surveys are funded at least in part by national governments, there is in fact very little variation in either content or methodology to respond to national circumstances.¹

There is the obvious “throwing the baby out with the bathwater” problem with this move away from censuses to relying on surveys because drawing a sample for a survey depends on having a sampling frame in the first place which is frequently based on the census. Clearly any problem with the census, if used as the sampling frame for a national survey, will lead to that sampling frame being biased. In addition, household surveys almost always have less complete coverage by design than censuses in ways we discuss later in this section. But there is—rather strangely—little recognition of the problems, which may be partly derived from reliance on an incomplete sampling frame and partly because of their design, in using household surveys to count or measure absolute numbers and the rates of income poverty or other forms of deprivation, especially for children who are the focus of many development goals such as the Millenium Development Goals (MDGs).

The issue is covered by Atkinson and Marlier (2010) but only briefly which is surprising given the focus of their book is on social inclusion. Mishra, Barrere, Hong & Khan (2008) claim to correct for bias in HIV sero-prevalence estimates from national household surveys including not only non-response in 14 countries but also non-household population groups in five countries. But their estimates of the non-household populations, which appear to be based solely on census reports, are very low and not consistent with the evidence.²

The remainder of this introduction provides illustrations of how censuses may themselves not always provide a complete sampling frame; how this may impact on assessments of levels of poverty; and introduces the added problems of using household surveys to measure poverty.

Population censuses have always faced problems of complete enumeration. Groups of adults have been excluded from censuses in some countries for political or practical reasons. Non-citizens, cultural minorities or marginalized groups and specific categories of prisoners or rebels have often been excluded for political reasons (Buettner and Garland, 2008). Although this is probably now less frequent and certainly becoming more transparent, there are still several examples: coverage of tribal groups in the recent 2011 census in India for example was disputed by the Refugee Review Tribunal Research and Information, 2008, COPTAM, 2011; Afghans in Iran are not counted (Abbasi-Shavazi and Sadeghi, 2011). Rebellious territories, isolated villages, or the number of Bangladeshi emigrants to India is disputed (Pempel, 2011).

People who object to or avoid government oversight have sometimes been excluded for practical reasons (Buettner and Garland, 2008). One population sub-group which is very often excluded from national censuses in developing countries is seasonal and temporary internal migrants or other highly mobile economic groups (Deshingkar, 2006). Pincus and Sender (2008), based on a detailed analysis of the fluidity of the labor market in Vietnam, show how both censuses and household surveys exclude most temporary migrants because they are based on official household lists which excluded those who had arrived less than 6 months previously. While a subsequent Law of Residence in 2007 (Refugee Research Tribunal, 2008) relaxed those rigid requirements, the 2009 Population and housing census (General Statistical Office, 2010, p. 31) says that they were not enumerated in the Census.

In addition, in many developing countries, the census enumerators are often police or other government officials who tend to use security based national identity cards or family registration cards to validate the citizenship status of those they are enumerating. Their incentive is to confirm their own registration work and to catch anyone who has escaped their net. This practice is widespread in Africa; but also has happened in Asia in, for example China (Di, 2011) and Indonesia (Dwinosumono, 2006), although the latter has tried to overcome the problem with better training and recruitment of non-officials as enumerators.

Therefore, the general problem that censuses are not themselves necessarily complete or accurate is well understood (see also Carr-Hill, 2009); a specific example is provided by Chandrasekhar (2005) who attributes the puzzling decline of eight million slum dwellers in India over the nineties to an underestimation of the number of people living in the urban slums. There is an emerging consensus as to what constitutes good census practice (see Appendix 1); and clear adoption of these UN guidelines would at least make interpretation and comparison easier. At the same time, the quality of censuses in developing countries has probably improved during 2000–10, with many more countries carrying out censuses and technological innovation in mapping, enumeration, and data capture (UN Statistical Division (UNSD), 2010).

The guidelines are clear in principle but there can still be problems in enumeration in practice for each of the concepts:

While Cinderella is a fairy tale, the exclusion of poor servants from the census count in rich households (even though they will usually be sharing some of the household food) especially in Asia is not, and their personal poverty is therefore missed; for different reasons,³

In developed countries, the young are highly mobile—usually male—are also difficult to count, especially when they live in collective households, but they are relatively well-off; in developing countries, the mobile are mainly nomads/pastoralists and rural–urban migrants, and they may well be among the poorest at least in income terms.

These will always be difficult to count, especially where there are disputes over nationality: for example over the stateless (“bidun jinsiya” meaning “without nationality”) in the Gulf States (Kohn, 2011, Refugees International, 2007); equally there are several millions internally displaced in many countries either as a result of civil war or because of environmental change (e.g. floods, nuclear accidents) have made their homes uninhabitable; and although counts tend to be compiled in more developed countries—for example in the Balkan States—this does not happen in Africa and Asia. A study of the nature and extent of homelessness in nine developing countries (Centre for Architectural Research and Development Overseas (CARDO), 2003) showed that most did not have any reliable data on the numbers of homeless people. Several did not have any official definition of homelessness to use in a census; but definition is important because “… most researchers agree on one fact: who we define as homeless determines how we count them”. (Peressini, Mcdonald, & Hulchanski, 2010, p. 1, chap. 8.3). In some countries, street sleepers are ignored for census purposes because they have no official house or address (CARDO, 2003).

There are several different types of institutions (care homes, (some) factory barracks, hospitals, the military, prisons, refugee camps, religious orders, and school dormitories) and there is still considerable variation over how some of the institutional population groups should be included in the population count. For example, there was no agreement in the 2010 US Census as to where military who are deployed overseas should be counted (whether at their originating barracks or allocated to their home States). Similarly, there are variations between the States of the US as to whether prisoners are counted where they are incarcerated or at their last home address; and, if the former, whether the Bureau relies on an administrative count or enumeration (Wagner, 2008). The problem is that their characteristics are often not fully reported and they are simply counted as special census blocks or special households.

Careful reporting of censuses as per the UN guidelines will acknowledge how well these groups have been enumerated and most categories—including the military and prisoners—are included in estimated census population counts of developed countries but not in the census reports of many developing countries.

In particular, in developing countries, according to the UN Population Division (Buettner and Garland, 2008), children are systematically under-counted. For example, the one-child-policy in China will have led to substantial under-reporting: although in the 2010 census, China attempted to count children properly by reducing fines for those who had violated the one-child-policy; and, for the first time, the Census will count those born in other countries (Cohn, 2010); in the US, children are over-represented in hard-to-count areas (O’Hare, 2009) and are therefore under-counted.

There is a huge literature on how to measure poverty on an international comparative basis⁴ spanning several decades (ILO, 1976, McGranahan et al., 1972, OECD, 1975) and several disciplines. But, in this context—the measurement of development progress –for better or for worse, a crude cash measure (US$1 or US$2 a day) has been adopted by most international organizations as the flagship measure, even though it makes little allowance for non-food needs which are mostly monetized in urban but not in rural areas (Mitlin & Sattherwaite, 2012). The argument in this paper however applies not only to that crude cash measure but also to many other dimensions of deprivation whether in terms of lack of education, ill-health, mortality, under-nutrition, poor sanitation, lack of clean drinking water, etc.

The census documents for several of the large-population developing countries have been examined for any commentary about difficulties or problems encountered; in fact, such internal commentary is rare and an extensive web search was carried out for other commentary. The sparse results of these efforts are included in Table 2. It is clear that many of the censuses have encountered severe difficulties in implementation; and that some either left out some groups by design or have been forced to omit certain areas or groups.

The problem with using household surveys to assess the absolute level of poverty or of any related characteristic is that, in contrast to the view of Munoz and Scott (2004), they are an inappropriate instrument for obtaining information about the poorest of the poor, especially in developing countries. This is because household surveys, with rare exceptions, typically omit by design:

• those not in households because they are homeless;

• those who are in institutions, including refugee camps; and

• mobile, nomadic or pastoralist populations.

In addition, in practice, because they are difficult to reach, household surveys will typically under-represent:

• those in fragile, disjointed, or multiple occupancy households (because of the difficulty of identifying them),

• those in urban slums (because of the difficulty of identifying and interviewing), and

• may omit certain areas of a country deemed to pose a security risk.

If one wanted an empirical—as distinct from a theoretical—definition of the “poorest of the poor”, the above collection of six population sub-groups could hardly be bettered.

A comprehensive search was carried out of the meta-documentation of the four main standardized household surveys—the Demographic and Health Surveys (DHS), the International Labour Office/Labour Force Surveys (LFS), the Living Standard Measurement Surveys (LSMS), and the Multiple Indicator Cluster Surveys (MICS)—and a sample of country surveys. None of the meta-documents, including those from the LSMS (Grosh and Glewwe, 1998, Scott et al., 2005) or the DHS (Vaessen, Thiam & Le, 2005, chap. XXII) which is the most professional and most concerned with quality, justifying its relatively small sample sizes specifically because of its attention to non-sampling errors, had anything to say about the coverage of the homeless, institutional populations, the mobile and/or any special arrangements to cover slum areas.⁵

Population censuses are, of course, themselves surveys of a kind, and, as we have illustrated above, have faced many of the same problems in the past; but a modern politically independent Census will intend to include the mobile (because they refer to those present in the household on a specific day or night), will cover those in institutions, will attempt to cover those in urban slums and in less secure areas exhaustively, will (if necessary) carry out special counts of the homeless, and will attempt to estimate the numbers of pastoralists, with varying degrees of success (Misra and Malhotra, 1982). In other words, a Census can potentially solve many of the problems of omitted populations; but this is not possible for household surveys.

The six groups identified above will also affect household surveys in developed countries, but to a lesser extent: there are fewer homeless; the geographically mobile tend to be the young and upwardly socially mobile rather than the income poor; there are fewer fragile households and urban slums are smaller. The only real issue in developed countries is the omission of the institutional population from the sampling frames of most surveys, which are usually household-based; and attempts to include them (National Centre for Social Research (NCSR), 2003) often leads to the use of proxy respondents which will usually be adequate for a census but not for a detailed household survey.

The extent to which household survey estimates can under-estimate poverty-related characteristics is illustrated for Vietnam, a country where the biennial household survey is considered to be one of the best designed and implemented (Pincus and Sender, 2008, p. 110); the problem would be much larger in other countries.⁶ In each socio-economic region, the comparison of the 2009 Census with the average scores for the 2008 and 2010 the Vietnam Health and Living Standards Survey (VHLSS) shows that the proportions reporting no qualifications are higher, the proportions reporting improved water are lower, and the proportions reporting agricultural, forestry, and fisheries are higher (and much higher in Red River Delta) except for the South East (see Table 3). The differences for agricultural, forestry, and fisheries might be related to the temporary residence problems described by Pincus and Sender (2008) but the different results for the other two characteristics are probably more simply related to the greater practical difficulties of carrying out sample surveys—as compared to censuses—in rural areas.

Nevertheless, although modern quality censuses recognize that they have to include these groups in the population counts, census officials, because of the difficulty of enumeration, even in developed countries, are often reduced (as we have documented) to making estimates of their size and location, so that the members of those groups are often not included in the available sampling frames for household surveys. This poses additional design problems for sample surveys; and, in developing countries, these marginalized groups may not be included at all, even in the estimated population counts. The important consequence of this lack of recognition of the additional problems with the design and implementation of household sample surveys, particularly although not exclusively in developing countries, is that no systematic attempt has been made to estimate the size and distribution of the population groups “missing” from the sampling frames of national household surveys, in addition to those who might be missing from the census. For obvious reasons, it is difficult to estimate numbers in these groups; the following sub-sections document what is known or has been estimated.