6 July 2001
Symposium on Global Review of 2000 Round of
Population and Housing Censuses:
Mid-Decade Assessment and Future Prospects
Department of Economic and Social Affairs
United Nations Secretariat
New York, 7-10 August 2001
David C. Whitford and Jeremiah P. Banda **
Post-Enumeration Surveys: Are They Worth It or Not?
The post-enumeration survey (PES) is a method for evaluating the results of a census. As censuses become more complicated, and as the results of censuses are used for more and more policy and planning purposes, it is important to examine the quality and limitations of census data and to understand the types and extent of inaccuracies that occur. Several methods are available to evaluate censuses, including demographic analysis, comparison of census results with data from other sources and matching census responses with responses from interviews conducted during a PES. In many developing countries, alternative sources of population data are not available, so the PES is the major tool for evaluating the census.
Basically, a PES is an independent survey that replicates a census. The survey results are compared with census results, permitting estimates to be made of coverage and content errors. Coverage errors refer to people missed in the census or erroneously included, whereas content errors evaluate response quality of selected questions. The PES allows census organizations to uncover deficiencies in the methodology of the census and make adjustments for future censuses. PES results can also be used to adjust census results, although this is as likely to be a political decision as a technical one.
1. Ideally, to ensure independence, the PES would be undertaken by staff who have not worked on the census. In practice, the PES generally uses the most qualified census workers available and ensures that they work in different enumeration areas (EAs) in the PES if they also worked in the census. Like all survey work, if the PES methodology is flawed—for example, if it uses poor sample design and incomplete frames—the results may not be reliable. The PES draws a sample of the population, which can be chosen in several stages. When the sample is established, addresses of all housing units are listed, and interviewing begins. The PES normally takes place near enough to the census to ensure that people remember who was in the household on census day. This is particularly essential in a country that takes a census on a de facto basis. The next steps are matching the census and PES data and reconciling discrepancies.
2. Results from a PES have been useful to many countries. In Zambia, for example, the PES found that age reporting was more accurate than anticipated, and it helped analysts to notice at an early stage the effects of the HIV/AIDS epidemic on age structure. The main objective of Cambodia’s PES was to provide national-level estimates of coverage and content errors in the census. The PES was conducted in March 1998, two weeks after the census. Mongolia conducted a limited PES to evaluate census coverage; lack of funds precluded a more elaborate survey. In some EAs, the census and the PES were not independent operations, as evidenced by high agreement between census and PES results. Namibia faced some operational problems in its PES, including confusion over boundaries of EAs; failure to pre-list housing units or pre-test questionnaires; and lack of reconciliation procedures. In the United States of America, use of laptop computers for the PES interviews and an automated software system for matching improved the speed and quality of the work.
3. Post-enumeration surveys are worth conducting if they are carefully planned and well implemented. The PES methodology is adaptable to many circumstances, and the fact that the PES is carried out immediately after the census means that overhead costs may be greatly reduced. For the PES to succeed, its planners should develop good area frames with well-defined EAs; design plausible probability samples; adopt efficient but realistic matching rules; attempt to maintain independence between the census and the PES; use the same definitions and concepts in both the census and the PES; use well-trained field staff; carry out pre-tests for the PES and reconciliation; allocate adequate funds for the PES; include relevant and useful items for matching purposes; and keep the PES as simple as possible and set objectives that are attainable.
4. Census taking is improving constantly throughout the world. As censuses improve and people get used to using information from them, censuses undergo more and more scrutiny. For instance, given one census, the science of demographic analysis can produce independent estimates of population size for comparison with the results of the following census. Differences between the demographic analysis predictions and the actual results of the new census are inevitable. These differences can take the form of unreasonable sex ratios or wild disparities in age cohorts.
5. We reiterate that as censuses improve, they are used more and examined more. A country’s last census may not have undergone much scrutiny, but its next one could be a bombshell waiting to explode. As time passes, the need for self-evaluation of census results inevitably increases.
6. It should also be noted that a population census is the most extensive and expensive data-collection exercise any country can undertake. With vast amounts of resources spent, there is usually tremendous pressure on census takers to ensure that census results are accurate. As a result of the massive nature of the census operation, it is inevitable that some inaccuracies arise from deficiencies, including errors of coverage and response. The major difference among countries is the extent of such errors. This, however, does not diminish the importance of the census as long as users understand the limitations of the data and the errors do not affect the major uses of the data (Cambodia, National Institute of Statistics, 1999).
7. It is against this background that a number of methods for evaluating censuses have been developed. Such methods include demographic analysis, comparison of census totals with figures from other sources and matching census returns with those from interviews selected on the basis of a probability sample in a post-enumeration survey (PES). This paper focuses on the PES as a method of evaluation. The paper discusses the purpose of PES’s; problems and constraints associated with the PES; design and methodological issues; country experiences; uses of PES’s and suggestions to improve PES programmes.
8. While a number of methods have been developed to evaluate census data, for many developing countries the PES seems to be the most ideal owing to paucity of appropriate data to facilitate the effective use of other methods. The lack or incompleteness of registration systems and absence of regular population and demographic surveys contributes to the lack of use or limited use of other methods of census evaluation. In general, a number of countries have relied primarily on PES methodology to evaluate the census undercount (Biemer et al., 2001).
9. Post-enumeration surveys are an accepted census self-evaluation tool. Typically, a PES is an independent survey that replicates a census. The survey and the census results are then compared (matched). The results of the comparison are used to measure the coverage and/or errors in content of the census. Estimates of net coverage, the number of people omitted in the census, the number erroneously enumerated and content error rates for specific questions are typical products of a PES.
10. Additionally, these estimates can be broken down further into their component parts. One can design the survey so that reliable estimates of undercount or overcount can be obtained for the entire census, for geographic areas of interest in the census, and for any of a host of demographic characteristics, such as age, race and sex, for which one might desire census coverage statistics.
11. The survey results also enable one to be able to uncover census methodologies or operations that, when implemented, produced less than desirable results. Suppose, for instance, that a high census omission rate was observed in rural areas. One might then use specific PES results to examine whether the rural errors were due to the omission of whole housing units. If so, this might well imply an incomplete census frame and cause one to re-examine the methodology for building an address list in rural areas.
12. PES results can be used to adjust census results. Using a carefully designed survey, under- or overcounts can be converted into adjustment factors and the census population increased or decreased accordingly by these factors. Later in this paper we will discuss post-stratification and the need to ensure homogeneity within each adjustment cell. It has been reported that in some African countries, PES results have been used to support or defend census results when the accuracy of the census is challenged (Onsembe, 1999).
13. In addition, censuses are used for many other purposes, such as updating population estimates; developing and updating sampling frames; correcting and updating population registers and the establishment and updating of key components of the Geographic Information System (GIS). These many uses suggest that there is a need to use an objective method for assessing coverage and content errors as a crucial step for concluding a census operation (Abu-Libdeh, 1999). Quality assurance alone, introduced at various stages of census operations, cannot ensure a complete evaluation of the qualitative and quantitative accuracy of census data (UN Economic Commission for Africa, 1975).
14. In summary, post-enumeration surveys have many good purposes. They basically inform users regarding the quality of the census data. As stated earlier, providing limitations of published census data increases the confidence of informed users in such data. On the other hand, there are distinct limitations and constraints in managing and implementing the evaluation survey.
15. Although a PES can be an important component of a census programme and can contribute to the process of building confidence in the census results, a poorly designed and executed survey can inflict considerable damage to census legitimacy. We list below some of the problems and constraints associated with PES’s:
· Planning and management of a PES, ideally, have to be undertaken by a staff that is separate from the census staff. This is not usually the case in many countries;
· The design of the survey—especially the matching step—is relatively complex. For example, in the United States planners continue to find design flaws in the matching system. However, as corporate experience grows, these flaws become more and more minor;
· The survey must be independent of the census. In the survey’s sample areas, census results must not be biased by the implementation of the PES;
· The PES interview itself is demanding. Usually it incorporates questions to determine if the respondent should “really” be counted at the residence in question. Also, the PES interview usually transpires after the census interview, at which point the respondent may feel overburdened and not be as forthcoming with accurate information;
· Some of the developing countries lack technical personnel with experience and skills in survey methodology in general and PES in particular;
· Past failures in some countries in conducting PES’s discourage such countries and others from conducting PES’s in the subsequent rounds;
· Some of the countries, such as the United States of America, which have conducted PES’s, have not used the results to adjust population census figures. In such cases questions have been raised about the rationale for conducting PES’s;
· In some countries, census planners feel it is enough to institute good-quality assurance procedures at various stages of census activities; therefore they see no need for a PES (UNSD/SADC, 2001).
16. Lastly, some of the countries are ambivalent about conducting a PES because a census is usually a grueling and taxing operation, which saps the energy of those involved. The general fatigue it generates may be sufficient to discourage the conduct of the survey. Additionally, by the time the census enumeration is completed there is usually a feeling of accomplishment among census planners. They may, therefore, not see the need for conducting a PES, which, after all, may just expose glaring discrepancies between census and PES results to the detriment of the reputation of the census or statistical organization.
17. Considering whether a post-enumeration survey is worth it or not leads immediately to some decisions that have to be made regarding the design of the survey. These decisions revolve around what goals one has for the survey and what answers best suit the individual situation in which the survey will be conducted.
18. We will assume in this paper that the goal of the PES interview is to establish carefully who lived in the subject housing unit on the day the census was officially taken. In the next step we match the results from the interview to appropriate census forms in a well-defined area around that subject housing unit.
19. The central facet of a PES is measurement of census omissions. PES methodology calls the sample used to measure omissions the P sample or population sample. Roughly, one interviews this sample and compares (matches) it to the census results.
The resulting tallies can be represented in a two-by-two table:
Out of census
Out of PES
is the estimate of the number of people counted in both the census and the survey,
is the estimate of the number of people counted in only the survey,
is the estimate of the number of people counted in only the census,
is the estimate of the number of people missed by both the census and the survey,
is the total estimate of the number of people counted in the survey,
is the estimate of the total number of people.
20. The dual-system estimation model assumes independence between inclusion in the census and in the PES. (We elaborate on how independence is implemented later in this paper.) The dual-system estimate (DSE) of the total population is given by
21. Simply stated, the DSE raises the census total by the ratio of the total estimate of the number of people in the PES divided by the estimate of the number that matched to the census.
22. In the section above on purposes of a PES, we discussed breaking down the DSE estimates by geographic areas and for any of a host of demographic characteristics, such as age, race and sex, for which one might desire census coverage statistics. If direct estimates are desired for any of these breakdowns, one might post-stratify the sample results into the categories desired. The objective of post-stratification is to include in each dual-system estimate people who have similar capture probabilities in the census.
23. Omissions are not the whole story in evaluating a census. Errors can be made in the census itself that affect the overall under- and overcount measurement:
· The census can contain duplicate or multiple enumerations;
· The census could have people or housing units ascribed to the wrong geographic location (and thus not matching the PES interview);
· People could be less than perfectly enumerated—that is, there could be insufficient information for matching to the PES interview;
· The census could have erroneously enumerated someone who should have been enumerated elsewhere or the enumerator could have made up a fictitious person.
24. So, if one is interested in quantifying these errors and their effect on census coverage, a sample of the census enumerations has to be checked to tally the number of times these types of errors, called erroneous enumerations, occurred. In PES parlance, this sample is called the E sample. The section below on estimation explains how the quantification of these errors is incorporated into the dual-system estimation formula.
25. A desirable option for the E sample is to draw it directly from the census for the sampled areas used in the P sample. This facilitates matching and helps ensure that the survey is balanced—that is, that one is searching for omissions in the exact same area where one is searching for erroneous enumerations. The area one searches for omissions and erroneous enumerations is called the search area.
26. For instance, the person might have lived elsewhere for the rest of the year. Some E‑sample units and the people in them will not match any of the PES interviews. They might have been missed in the P‑sample frame or truly erroneously enumerated. Since these people have not been asked the battery of questions to determine if a person should have actually been counted at the particular housing unit on census day, a follow-up operation is needed to determine if the unmatched people in the E‑sample unit were or were not erroneously enumerated—that is, whether one of the census errors listed above occurred or did not occur. More about this follow-up interview is presented in the section below on reconciliation.
27. Two other design decisions have to be made: What is the primary sampling unit for the survey? and What is the definition of cases to be included in the survey? This leads into our next topic, the sampling frame of the survey.
28. A popular choice for a sampling frame is to use an area sample for the coverage measurement survey. The primary sampling unit can be the block. Blocks are land areas surrounded by visible geographic features such as roads and streams. The frame, therefore, consists of creating a universe of blocks in the country and dividing those into sets of blocks (or clusters of blocks) that can be interviewed by a single interviewer within the allotted time.
29. Another option is to use a survey that is already in place that is being taken around the time of the census. This has the large advantage of using an existing organization to manage the PES. It also has several disadvantages:
· The existing survey may not be large enough, and supplementing it may be as complex as creating a specially designed survey;
· Procedures may have to be augmented with the result that the quality of the existing survey and the PES suffers;
· The ultimate sampling units may not lend themselves to being an efficient erroneous enumeration sample, where duplication, geocoding errors and so forth need to be easily discernible.
30. Above, we mentioned the option of using block clusters as a frame for the survey. One might want to design the sample in several stages by first choosing a group of these clusters and then optimizing the sample by subsampling. For instance, the housing unit totals from a previous census might be used to choose the initial sample of clusters; then, after the addresses of all of the housing units in the sample clusters are listed, the block clusters might be divided into small, medium and large sampling strata, where
· Small clusters might consist of 0-2 housing units;
· Medium clusters might consist of 3-79 housing units; and
· Large clusters might consist of 80 or more housing units.
31. The next step could be to subsample some of these clusters:
· Medium and large clusters might be subsampled whenever their actual counts of housing units differed significantly from what was expected (from the previous census);
· Probably, there are many small block clusters in rural areas. These can be subsampled to keep the fieldwork manageable;
· Again, to keep the fieldwork manageable, some within-cluster subsampling can be undertaken in very large clusters.
32. For the block clusters in which interviewers are to enumerate, interviewers need maps to find each subject housing unit and a listing of all the housing units in the block cluster. The listing operation is done independently of any census activity. Not only are the addresses of existing housing units listed, but inquiries are made at commercial structures and other structures to ensure that no people live in them. One option is to give listers blank maps upon which they put a numbered spot representing each living quarter or potential living quarters in the cluster.
33. Listing needs to be of high quality. It must ensure not only that the correct blocks are listed but also that a complete list is obtained within the cluster. So a quality assurance plan needs to be created to ensure correct listings—for instance, to ensure that commercial structures have, indeed, been checked to see if they contain residences.
34. The interview approach is currently the common method used in PES. The questionnaire asks about all people who resided at the sample address on census day and asks questions to ensure that the respondents should have been counted at this address. Subsequently it searches for them at that address and in the search area surrounding it.
35. Obviously, since people move, it is most efficient if the PES interview can occur as soon after census day as practical. Getting information about out-movers (those who move out of the sample address between census day and the PES interview) is usually difficult.
36. The training book, Evaluating Censuses of Population and Housing (U.S. Bureau of the Census, 1985), gives a set of basic questions for PES’s that are still valid today. Essentially they are:
· What are the names of all people living here on census day?
· What are their relationships to each other?
· What is each person’s age and sex?
· Is each person still residing here? If not what is his/her current address?
· What are the names and relationships of other people living here on census day?
37. The last two questions are examples of probing questions. To ensure that people didn’t live elsewhere on census day, the interview could include more probing “other residence” questions that fit the country’s existing situation.
38. Since this information is going to be matched to the census, it is obviously important that it be complete. Some countries have chosen to have the PES interview completed on a laptop computer which can check within its program to ensure complete data. Quality control measures need to be employed in any case.
39. When measuring a small fraction of the population, those not counted in the census, it is very difficult to deal with a high non-interview rate. To keep the non-interview rate low, a second and final phase of interviewing might be planned during which the best interviewers attempt to convert the remaining non-interviewed cases.
40. After data capture is completed for the PES interviews and the census data prepared for each PES cluster, the next step is to match the two. One approach is to accomplish this in two steps: computer matching, which “skims the cream” (that is, makes the easiest matches), followed by clerical matching of the remaining non-matches and possible matches as determined by the computer process.
41. Of course, matching can be completed manually. The process involves clerks first gathering materials to facilitate matching in a particular cluster. The materials include:
· Address lists from both the census and the PES;
· Census forms for the cluster;
· PES interview results for the cluster; and
· Maps for the cluster from the census and the PES.
42. Gathering materials for a cluster is indeed a cumbersome part of the matcher’s job. Regardless of how it is done, the basic process of matching remains the same: comparing people’s names and demographic characteristics between forms—the census form and the PES interview form. One design has clerical matching comprised of four basic steps:
· The clerk must first determine if he or she has sufficient information for matching—that is, whether the names and demographic information are complete enough to be able to discern a definite match. The clerk must determine this before looking at households from the other survey. It is a violation of independence to have the census or PES results from a household biasing a clerk’s ability to read, for instance, a sloppily written name;
· The clerk matches the P sample to the census throughout the cluster (or search area);
· The clerk then looks for duplicates within the E sample; and
· Lastly, one option is to have clerks do some surrounding block matching—that is, looking for matches in the blocks surrounding the cluster if and only if one balances this by looking for E‑sample correct enumerations with equal exactness.
43. We mentioned previously (in the section on the E sample) that many PES interviews ask a battery of questions to determine if a person should have actually been counted at the particular housing unit on census day. If whole households of E‑sample housing units have been not matched to anyone in the P sample, they would not have been asked the residence questions. A field follow-up operation is needed to determine if the unmatched people in the E‑sample unit were or were not erroneously enumerated.
44. Additionally, in a coverage measurement survey follow-up operation, P‑sample interviews with unmatched people that had been completed by proxy respondents can be followed up. Some research has indicated that PES interviews by proxy respondents need this additional attention to ensure accuracy.
45. After the follow-up operation, forms are received in the processing office and their final match status is coded—probably by the same people who did the earlier matching. This completes the operations for the PES, and we move on to estimating the under- or overcount.
a. Dual-system estimate incorporating an E sample
46. The section above on the P sample delineated the basic dual-system estimation formula for estimating the total population using the PES P‑sample results and the census results. The section on the E sample specified errors that could be made in census taking, called erroneous enumerations, which could also be taken into account in the total population estimates. These census errors included:
· Duplicate or multiple enumerations;
· Housing units ascribed to the wrong geographic location; and
· People with insufficient information for matching to the PES interview.
A dual-system estimate of the total population (U.S. Bureau of the Census, 1985) that included an E sample would subtract weighted totals of the errors above from the census count. The refined estimate would be:
= the total number of whole person imputations in the census,
= the weighted number of people erroneously enumerated in the census,
= the weighted estimate of duplicates,
= the weighted estimate of geographic errors,
= the weighted estimate of census people having insufficient information for matching,
= the weighted number of people estimated in the survey,
= the total number of people counted in the census, and
= the estimate of the total number of people.
b. Missing data
47. Inevitably in any survey, missing data are encountered. Interviewers, however tenacious, cannot obtain every answer to every question and, in fact, given time constraints in surveys, cannot interview every household. During estimation for coverage measurement surveys we must account for missing P-sample data and missing E-sample data.
48. Missing data can be separated into categories with different approaches taken for each. For instance, missing data can be divided into three types:
· Entire households that were unable to be interviewed in the P sample. With this type of missing data, planners can take the approach of redistributing the sampling weight assigned to each of these households to other households living in similar-type dwellings interviewed in the same block;
· Missing demographic characteristics data. These data are to be used in post-stratification (see next section). When they are missing, substitute data can be imputed in their place using a “hot deck” procedure. This procedure chooses data from a completed case that are very similar to the case with the missing data;
· Unresolved match status or residence status. In the P sample there are cases in which, even though they have undergone a reconciliation interview, match status and/or residence status cannot be resolved. Match status is whether a person matches a census enumeration or not, and residence status is whether or not a person actually should have been counted in the census at the subject residence. In the E sample similar missing data are encountered when there isn’t enough information to determine if someone is correctly enumerated in the census. Cases without match or residence status can be assigned a probability of matching and/or a probability of being a census day resident based on all the information collected about them and cases with similar characteristics (Cantwell et al., 2001).
49. The object of post-stratification is to include in each dual-system estimate people who have similar capture probabilities in the census. For instance, young people are usually more mobile than the elderly and so more difficult to count. To mix young and old in one dual-system estimate would lead to a bias in the estimate. However, to have separate post-strata for age groups and separate estimates for each and then to add the estimates across the age groups avoids this bias problem.
50. On the other hand, having too many post-strata such that each one does not receive a large enough sample will increase variance of the estimates. Post-stratification is a balancing act that has to be carefully thought out.
51. Some examples of coverage measurement survey post-stratification variables are, for instance: race, Hispanic origin, age, sex, tenure (whether one owned or rented his/her home), degree of urbanicity and type of enumeration area of the country and mail return rate of census forms.
52. Country examples, which are presented below, highlight recent national approaches to planning and executing of PES’s in selected countries. They are also aimed at illustrating that a PES can successfully be carried out, even for countries and areas that conducted censuses for the first time. It is, however, important for the PES to be included in the overall strategic plan of the census. If a PES is conducted half-heartedly, as an afterthought, is poorly planned and executed and has inadequate resources, the exercise is bound to fail. The country examples are illustrative and are used, in this paper, in a positive sense of bringing out issues and lessons learned.
53. A PES was conducted in March 1998, about two weeks after the census. This was essential because an accurate recall of persons in households on census night is critical for the evaluation of de facto census coverage. The main objective of the PES was to provide national-level estimates of coverage and content errors in the census.
a. Sample design
54. A sample of 100 enumeration areas (EAs) was selected for the PES from a population of about 24,918 EAs. The EAs in this case were clusters of households. The sampling frame used in the PES was the final list of EAs that were covered during the population census. All households in selected EAs were enumerated. The PES, like the census, excluded inmates of institutions such as hotels, hospitals and prisons. The homeless and transient population was also excluded. The following assumptions and requirements were set for the sample design: (1) a probability as opposed to a purposive sample was selected; (2) an area-based sample was selected to ensure that missed census households had a chance to be enumerated during the PES; (3) the presumed census undercoverage rate, for purposes of predicting reliability of the PES, was taken to be 3 per cent. Reliable estimates were expected at the national level only; however, separate estimates were calculated for the rural area and large regional groupings of provinces. No estimates were considered for the urban area.
55. The PES for Cambodia had three distinct operations, namely, listing and enumeration of persons in all households within the selected EAs; matching of characteristics and particulars collected during the PES listing and those obtained from the corresponding census questionnaires; and field reconciliation.
56. Complete independence between the census and PES operation is an ideal situation, which, in most cases, is not attainable under field conditions. Notwithstanding the above, the planners for the PES in Cambodia took measures to make the PES as operationally independent as possible from the census. Such measures included:
· Taking care in selecting better-qualified and trained enumerators. The quality of fieldwork was maintained by close supervision by well-trained and experienced supervisors. It was ensured that no enumerator involved in the census could enumerate for the PES the same EA as in the census;
· Enumerators for the PES had no access to information/data collected in the census in their respective areas of operation;
· The location of selected EAs for the PES was not disclosed to the field staff beforehand;
· The PES did not begin until all completed census records were transported and stored at the National Institute of Statistics headquarters.
57. The purpose of the matching operation was to classify each person listed in the PES and the census according to whether he or she was correctly enumerated in the census or tentatively missed in the census. For the matched person the operation entailed a person-to-person match between the PES and census records. For the tentatively missed person they verified whether the person was actually present or not in the household on census night. Owing to the confusion of reported names and sometimes relationship to the head of household between the census and PES, some people were initially classified as missed.
c. Coverage error
58. Cambodia used a single rather than a dual system of estimation, which is said to be a simpler method not subject to a strict independence requirement between the census and the PES (Turner, 1997). This method, however, tends to underestimate coverage error.
59. Net coverage error was estimated at 1.78 per cent. The percentage of underenumeration was highest among infants (children under the age of one), followed by people in the age group 20-29. Cases of omission of households were minimal.
d. Content error
60. Content or response error includes mistakes in reporting and/or recording of items. This was measured for selected variables such as age, mother tongue, literacy, main activity, employment period, children ever born and children surviving. The index of inconsistency, which is the relative number of cases for which responses vary between the census and the PES, ranged from 4.97 to 44.34. Mother tongue and employment period were the most and least reliable, respectively. The other remaining indices of inconsistency were below 20 and therefore considered low.
61. The PES was conducted two weeks after the population and housing census. The survey was designed to allow for the estimation of coverage error. Results provided for quantitative evaluation of coverage errors in more than one domain depending on the sample size. The dual-system method was used to measure coverage error. The method provided an estimate of the census population from the PES and the population counted in both the census and the PES. Turner (1998), however, cautions that the method of dual-system estimation requires that the PES and the census be completely independent in all respects. This ideal condition is not usually met in actual practice.
a. Sample design
62. A 4.7 per cent sample of EAs was selected with equal probability. The survey results were used to assess coverage error at the national level, Gaza Strip, rural, urban and refugee camps. A single-stage stratified systematic random sample design was adopted. It consisted of 140 census EAs, some of which were composite. All households and persons in the selected EAs were re-enumerated for the PES. About 21,000 total households were in the selected EAs. An area-based sample was necessary to ensure that missed households had a chance of being enumerated in the PES. The presumed census undercoverage rate for purposes of predicting reliability of the PES was taken to be 5 per cent. The sampling frame for the PES was the final list of 3,308 EAs in the West Bank and Gaza, used in the population census (Abu-Libden, 1999; Turner, 1997 and 1998).
b. Matching and coverage error
63. The dual-system method was used to obtain coverage estimates. The matching was done by comparing all census questionnaires in an EA with all questionnaires in a corresponding selected EA in the PES. The net undercoverage was estimated at 1.8 per cent. Due to cost and time considerations, the PES did not measure content error.
c. Quality and independence
64. Independence and quality issues were handled as follows: fieldwork for the PES started a week after census questionnaires were collected from the field. Special instruction manuals, training and field procedures were designed specifically for the PES. The sample of EAs was treated as confidential until enumerators were in the field, and those involved in the PES were better qualified. For example, the best crew leaders in the census served as enumerators, the best supervisors served as crew leaders and the best district directors managed the fieldwork. No single person was allowed to work in the same area where he or she worked during the main census. The same approach was adopted for data processing, which was done away from the main census data-processing area to avoid any possible contamination.
65. After the census of 1990, which was completed in September, a PES was conducted in December 1990, two months after the census. The objective of the PES was to measure both coverage and content errors. In this regard the PES combined the post-censal matching survey for measurement of coverage error and the re-interview survey for evaluation of content errors. The alternative methods to a PES were not used because of the unavailability of reliable records and a registration system, as well as limited demographic data in the country (Zambia, Central Statistical Office, 1995).
a. Sample design
66. A stratified cluster probability sample design was adopted, standard enumeration areas (SEAs) were selected with probability proportional to size within each stratum and all households were enumerated in the selected SEAs. The population covered by the survey excluded persons living in institutions and collective dwellings.
67. The sampling frame consisted of all SEAs demarcated for the population census. The frame was stratified by province, rural and urban. Using a sampling fraction of 1 per cent, a national sample of 160 SEAs was selected and distributed equally between the rural and urban strata. The 160 SEAs were allocated to each of the provinces proportionately to their measures of size based on the 1990 projected population.
68. The PES was not carried out until December for the following reasons: The census operation including mopping up was completed towards the end of September 1990; some key staff members of the Central Statistical Office (CSO) who were to be involved in the PES were engaged in the summary counts; and additional resources had to be mobilized for the PES.
b. PES enumeration
69. Although the PES methodology required independence between the census and the PES, in reality, it was impracticable to achieve complete independence. An attempt was made, therefore, to maintain some independence between the two operations, to the extent possible, in addition to maintaining quality by taking the following measures: census supervisors were used as PES supervisors so as to take advantage of their experience and qualifications; they were, however, assigned to work areas that were different from their census areas. The PES planners also took advantage of the experience and qualifications of regular CSO enumerators by using them as PES enumerators. Most of these enumerators did not participate in the census. The few who participated in the census had different work areas. PES field staff did not know preliminary census results of areas to which they were assigned.
70. There were 160 interviewers, one for each selected SEA and 51 supervisors, giving a ratio of about one supervisor for every three enumerators. The low ratio ensured effective supervision, thereby enhancing the quality of the data collected by enumerators. Instruction manuals and control forms were prepared for the enumerators and supervisors. The enumeration in the PES was on a de jure basis, while that of the census was both de jure and de facto. To facilitate matching, the questionnaire contained pre-coded answers and shaded spaces for recording census responses.
71. The PES questionnaires were matched to the census questionnaires by pairing each household and each person with corresponding census records. A two-way matching system, which was done manually, was adopted to identify omissions and erroneous census enumerations. Non-movers and out-mover categories were matched, while in-movers were identified but not matched. The matching was done at two levels, initially by identifying housing units and households and subsequently by matching individual characteristics. The matching status was classified into the following categories: perfect match, possible match and non-match.
72. At the initial stage, while using strict matching rules, there was a high incidence of possible matches and a low incidence of perfect matches. This was due mainly to differences in reported names of members of households between the two sources, particularly in rural areas. The use of alternate names is very common in Zambia (Banda, 1985). Large age variation, especially among older members of households, also posed problems to matchers. Subsequently, age tolerance limits for the old age groups were somewhat relaxed. Another reason for the high incidence of possible matches was partly caused by the differences in the use of the concept of household between the PES and census enumerators with respect to polygamous families. There were a number of areas where polygamous families were regarded as one household in one case and two or more households, depending on the number of wives, in other cases. This caused some persons to be counted more than once in the PES or census.
d. Field reconciliation
73. Field reconciliation was considered necessary because some unmatched cases identified in the office could be matched in the field. The non-matched questionnaires were, therefore, referred back into the field in order to identify erroneously enumerated cases and to resolve cases which had insufficient matching information. The exercise contributed to the increase in perfect matches.
e. Coverage error
74. The net coverage error was 1.92 per cent. The urban net coverage rate was 2.57 per cent, much higher than that of the rural areas (0.92 per cent). Some provinces had high net coverage errors mainly due to poor mapping and demarcation of areas into SEAs.
f. Content error
75. In general, the index of inconsistency for age was higher for rural areas for all ages compared to urban areas. The results also showed an unexpectedly high index of inconsistency for the relationship of son/daughter to head of household. The African extended family system, which, in most cases, does not distinguish between one’s children and those of one’s brother or sister, was identified as the main reason. This finding helped in framing better questions with respect to this item in the subsequent census and surveys.
76. Plans for census evaluation were less elaborate in Mongolia, due mainly to a shortage of funds. Following the completion of fieldwork of the 2000 census, a limited PES was undertaken in Mongolia, with the main objective of evaluating the coverage of the census. The PES was, therefore, not intended to be a major source of information on quality of responses. Only a few items were considered for evaluating content error. For a subsample of the PES an attempt was made to compare the census and PES responses for the same questions to provide some insights into consistency of interviews. The PES was conducted three days after the completion of the census fieldwork. Emphasis was placed on quality assurance rather than a good evaluation. The PES was therefore restricted to the most difficult areas, mainly in urban areas.
a. Sample design and data collection
77. A two-stage cluster sample design was adopted with first-stage units selected purposively. The PES form was adapted from the census questionnaire. The National Statistical Office used the best available staff that had not worked as enumerators in the census. The enumerators were not notified of their selection until after the census fieldwork had been completed. Similar instructions and definition of concepts were used in the PES and during the census.
78. Following the PES, names and characteristics of persons included in the census and PES were matched and the results compared. The matching, as expected, was confined to persons enumerated in either the census or the PES or in both.
c. Problems with the Mongolia PES
79. The importance of probability sample selection was not recognized. The selection for the PES was ad hoc and purposive. The operations of the census and the PES were not always independent, as it was possible for some managers to see to it that high agreement between the census and the PES was achieved. For example, in four areas there were no differences between census and PES results, which was unusual, while some other areas displayed large errors. A solution adopted was to exclude the extreme high and low ends of the distribution of sample areas to remove most, if not all, of the suspect areas from the analysis (Mongolia, National Statistical Office, 2000).
80. Dauphin and Canamucio (1993) have reported on successful PES’s conducted in Burundi and Rwanda during the 1990 round of censuses. The procedures and results are summarized below.
a. Sample design
81. In both Burundi and Rwanda the smallest areas for which boundaries could be verified on maps were EAs. The sampling frame of EAs in Burundi contained a total of 5,500 EAs while in Rwanda there were 6,200 EAs. In both countries a single-stage stratified cluster sample design was adopted. All households in the selected EAs were enumerated to facilitate matching against the census records. In Burundi the strata corresponded to the domains of estimation, namely, the urban and rural areas. Similarly, for Rwanda, the first level of stratification corresponded to domains of estimation, namely, urban, rural and the capital city. The second level of stratification was geopolitical subdivisions, even though no estimates were expected for these subdivisions. Seventy EAs were selected in Burundi and 120 in Rwanda.
b. Data collection
82. PES questionnaires were prepared based on final census questionnaires. The questionnaires allowed for the classification of each listed person in the households into non-mover, in-mover, out-mover, or erroneously enumerated.
83. In Both Burundi and Rwanda, the data collection aimed at identifying all usual residents at the time of the census in addition to those found as of the date of the PES. Data collection started two weeks after the completion of the census fieldwork. The PES response rates in Burundi and Rwanda were 98.0 and 99.9 per cent, respectively.
c. Field staff
84. To control non-sampling errors and ensure the quality assurance of the PES, 70 enumerators were deployed in Burundi and 167 in Rwanda, at least one in each EA. The enumerators were detailed to verify EA boundaries, to conduct preliminary listings and to enumerate all households within EAs independently from the census count.
85. In Burundi PES staff were selected from the census pool, taking advantage of their better qualifications, but they were assigned to different areas for the PES. In Rwanda some of the PES enumerators had not participated in the census.
86. The PES methodology that was adopted included a two-way matching of households and household members with census records of the same EAs. The matching operation resulted in the classification of all PES and census-enumerated persons within the selected EAs, in specified categories that permitted the calculation of coverage error and the determination of cases for which content error was calculated.
87. Namibia conducted its first PES in 1991. Sample selection was made from a frame of EAs compiled from the census. At the time of selecting EAs, they were assumed to be almost equal in size with respect to population. The selection of EAs was therefore based on equal probability. During the enumeration, however, EAs showed wide variation in population size.
88. The PES for Namibia faced a number of operational problems, among them:
· There was confusion over boundaries of EAs. In certain parts of the country many enumerators did not follow the EA boundaries during enumeration. The use of well-defined and identifiable EAs, on the ground, with clear written boundary descriptions, is essential for a successful PES;
· Many enumerators did not cover their EAs in a systematic way. For example, some households and houses were omitted;
· Some enumerators started fieldwork without pre-listing, especially in small towns and villages without street names and house numbers;
· Callbacks were, in some cases, not effected;
· There were no pre-tests of questionnaires and procedures owing to staffing, time and financial constraints; and
· Reconciliation was not done. Therefore, possible matches were not verified.
89. A listing was made of all persons, who were classified as non-movers, in-movers and out-movers. The first stage of matching was not satisfactory, as there was a low percentage of matched cases. Some respondents may have used different names during the census and PES, especially illegal immigrants. There were, at times, different interpretations of concepts of house and locality. Households, in some cases, were classified as one during the PES and two during the census. If a field reconciliation exercise had been done, some of the possible matches could have been converted into perfect matches. The low percentage of matches could also be attributed to unqualified staff, who were initially involved in the matching exercise. Corrective measures were taken later and the situation somewhat improved (de Graft-Johnson, 1992).
90. The assumption was that the unmatched persons were presumed to have been missed during the census. This assumption is not necessarily true. Those with high rates of non-matching included children below the age of four and female heads of households.
91. Two items included in the census but not in the PES could have been better variables for matching purposes. They are birthplace and place of usual residence. Tests in some African countries have shown that the former has a low index of inconsistency. The latter could have assisted in the search for addresses of persons who stated that they were enumerated during the census in a particular household but who could not be matched.
a. Introduction and sample design
92. The United States undertook an extensive coverage measurement survey during its last census. Census day was 1 April 2000. The survey included 314,000 households, a sampling rate of about two tenths of 1 per cent. The survey results had to be produced by April of 2001 for a decision about whether or not to adjust the census counts. Many operational decisions were made based on the need to do the survey quickly but with the utmost attention to quality. The survey did not include several things: any content assessment, any use of administrative records or any inclusion of demographic analysis results into the estimation process. Demographic analysis results were used instead as an independent benchmark for the survey results.
93. The first stage in drawing the sample was to divide the entire country into clusters of blocks such that each cluster had an expected 30 housing units. Enumerators then listed the addresses of all the housing units in the cluster. A stringent quality assurance (QA) programme was imposed upon the listing operation. Listing occurred well before the census itself and was scrupulously independent of it. The population residing in group quarters (typically, housing units with temporary or transient occupants) and institutions (such as nursing homes for the elderly) was not included in the survey. Subsampling was undertaken in the manner described above so that a sample was used that was efficient with respect to field operations.
94. Interviewing by telephone of about 30 per cent of the sample began three weeks after census day, and personal visits of the remaining cases began in mid-June. All interviewing was completed on personal computers by 6,500 field staff. These laptops permitted interviewing and data capture to be executed much more quickly than a paper instrument would have allowed. Since completed interviews were downloaded each evening, an added benefit was that reports on the quality of these interviews were available to supervisors the next day. QA on interviewing was extremely fast and effective. The final two weeks of an eight-week personal interviewing period were dedicated to converting non-response cases into interviews. The non-interview rate for housing units occupied on census day was 2.9 per cent (Hogan, 2001).
95. Matching was carried out in two stages: computer matching and clerical matching. About two thirds of the time, people in the census and the PES were matched by a conservative computer-matching algorithm. To prepare for clerical matching in the remainder of the cases, the computer linked census and PES households where the residents had a high probability of matching. Clerks examined the linkages using an automated software system. The system allowed them to access computerized address lists, census forms, PES interview results and maps for the cluster and use this information to make objective matching decisions. This software allowed the matching workforce to be one tenth the size it was for the 1990 PES. All the matchers were able to work at one site. This improved the ability to ensure that everyone was doing the same task with the same quality.
d. Reconciliation and estimation
96. Many cases that were not matched were sent to the field for reconciliation. All E‑sample cases were sent, as were P‑sample cases with proxy respondents. (The reasoning for this is given in the section above on reconciliation). After this follow-up, the last cases were given match codes and sent to estimation (described in the section on estimation).
97. The PES estimated the net national undercount to be 1.18 per cent (0.13 per cent standard error). This compared well with the previous (1990 census) measurement of 1.61 per cent (0.20 standard error). Additionally, the undercount of some minority groups in the United States was reduced compared to the 1990 census. The undercount for non-Hispanic blacks dropped from 4.57 per cent (0.55 per cent standard error) to 2.17 per cent (0.35 per cent standard error), and the undercount for Hispanics dropped from 4.99 per cent (0.82 per cent standard error) to 2.85 per cent (0.38 per cent standard error) (Hogan, 2001).
98. However, these results differed significantly from the demographic analysis estimates of the undercount. As of the 1 April 2001 deadline, based on these differences and other considerations (all of which needed time to be researched), the government of the United States of America decided not to adjust the official census counts using the PES results. Research continues and decisions about adjusting the counts for other uses will be made in the near future.
99. The use of PES results should be viewed within the context of the many possible specific objectives, which fall under the umbrella of measuring coverage and content errors. Countries can therefore emphasize all or some of these specific uses. Such uses include:
· Providing a statistical basis for adjusting census results;
· Evaluating the EAs as sampling units for intercensal surveys;
· Identifying procedural and conceptual improvements needed for future censuses;
· Providing to users of census data, such as researchers, information on sources and causes of errors.
100. As already stated, one of the major criticisms against PES’s is that, in some cases, the results are not used for adjusting census data. It should, however, be pointed out that while estimates of undercount may be made, adjusting census numbers is not universal. In some countries the question of adjustment is settled as a matter of policy rather than as a statistical decision. It is clear that the uses of PES results are multifaceted, so the question therefore is whether countries should fulfill all the objectives or some of them to justify the conduct of PES’s. In this connection, it has been suggested that it is better to use the PES not for adjustment, but as an evaluation tool to help assess and analyse the quality of the census and to uncover potential problems to correct in future censuses (Turner, 1998). We highlight below some of the practical uses of PES results in selected countries.
101. The PES is part of the methodological work which contributes to additional knowledge that may help to improve future censuses and intercensal surveys. The Zambian 1990 de jure official census figures were adopted after studying the de jure PES results. In addition, the content error analysis showed that age reporting was more accurate than anticipated in developing African countries. As a result of the high consistency index for age data, there was no need to smooth the data. In fact, the PES results helped analysts to notice, at an early stage, the unexpected age distribution, which, as mentioned earlier, was not due to age misreporting but was the result of a possible effect on the age structure of the HIV/AIDS epidemic, especially for women aged 15 to 34. AIDS is at its peak between these ages. Without the PES results it would have been assumed that there were age-reporting errors and analysts would have missed the opportunity to address the problem at an opportune moment..
102. PES results were used to adjust the census results of the Occupied Palestinian Territory. In the case of Cambodia, the PES results threw some light on the potential sources of undercount and overcount in the census. The findings will help in planning future censuses. In addition, while it was not possible to adjust population figures down to the village level based on the national undercount, for the purpose of projections, the base population at the national level was adjusted taking into account net coverage error. Swaziland plans to use the PES results to adjust population projections, and South Africa plans to use PES results to adjust its population census figures.
103. Adjusting of census data based on PES results can at times be controversial at both political and technical levels. At a political level adjustment may introduce possible realignment of polling districts, which may be undesirable in some quarters, and therefore would be resisted. At a technical level, in many countries the sample sizes for the PES are small; therefore, results can only be inferred for larger domains such as national, rural and urban domains. It is, however, tempting after getting the adjusting factors from the PES results to make adjustments at lower administrative domains. The latter may not be justified if the sample sizes for such domains are small and inadequate, thereby rendering the results unreliable.
It is evident from the observations above that some countries have used the PES to adjust census results and also as an evaluation tool considered crucial to the completion of the census task.
It is clear from country examples that PES’s had mixed results. For some countries relatively successful PES’s were conducted. It is, however, also true that some countries had unsuccessful PES’s, mainly because the best practices of conducting PES’s were not followed. The examples are revealing as they highlight problems that tend to crop up in PES’s which require a focused approach in order to resolve them. A summary of problems experienced by various countries whose PES’s have been described in this paper are:
· Poor cartographic work;
· Neglect of operational independence, as it relates to field staff, between the census and PES activities;
· Inadequate resources allocated to the PES;
· Lack of qualified personnel to manage and carry out various PES activities;
· Inclusion in the PES questionnaire items which were not useful for matching purposes;
· Not following basic principles of probability sample design and selection;
· Not carrying out field reconciliation for possible matches;
· Difficulties in matching alternate names because of the common practice, especially in Africa, for individuals to have more than one name;
· Difficulties of matching names because of the absence of unique addresses, especially in the rural areas;
· Neglect of pre-testing PES procedures;
· Flawed inferences based on relatively small samples; and
· Delay in undertaking the PES in the case of one country.
104. Post-enumeration surveys are worth conducting if they are carefully planned and function within operational and statistical constraints (United Nations, 1998). The PES methodology is adaptable to many circumstances, such as the use of single or dual methods of estimation. While independence between the census and the PES is a fundamental requirement, in practice operational independence seems to suffice because it is not possible to make all the various aspects of the census and PES operations mutually exclusive.
105. Since there is no error-free census, there is a need to continue to consider PES’s as part of census programmes. In many developing countries the PES seems to be the most feasible method of evaluating census results owing to the lack of comprehensive and, at times, accurate data from other sources, such as civil registration and population registers. For the PES to be useful in measuring coverage and content errors, it must be well planned and implemented. We therefore propose that efforts should be made to:
· Develop good area frames, with well-defined and mutually exclusive enumeration areas;
· Design plausible probability samples to facilitate objective generalization of PES results to relevant domains;
· Adopt efficient but realistic matching rules;
· Adhere, to the extent feasible, to the ideals of independence between the PES and census operations;
· Harmonize definitions and concepts used in both the census and the PES;
· Ensure that items included in the PES for matching purposes are relevant and useful;
· Involve well-trained and qualified field staff;
· Train key staff, involved in the design of PES samples, in survey sampling methods;
· Carry out pre-tests for the PES process and field reconciliation;
· Allocate adequate funds to the PES within the framework of the census; and
· Keep the PES as simple as possible and stick to objectives that are attainable.
106. Notwithstanding the conventional wisdom, a PES may not be as complex as perceived. It normally covers few variables compared to other household surveys and is usually based on a comparatively smaller sample. The PES calls for additional resources, but the fact that it is carried out immediately after the census means that its overhead costs may be greatly reduced. Such overhead costs as vehicles, office space, computers and census maps are usually covered by the census programme, thus making the PES more affordable. Onsembe (1999) observes, from his African experience, that the direct cost of a PES programme is about 1 per cent of the census costs. For example, the cost of the PES programme for Kenya is anticipated not to exceed 1 per cent of the total census costs
Abu-Libdeh (1999). Evaluation Census Data: Coverage Evaluation. Paper presented at the UNSD/ECE Workshop on Methodology for Census Evaluation and Dissemination for the CIS and Baltic Countries, Almaty, Kazakhstan.
Banda Jeremiah P. (1993). Census coverage evaluation: Recent experience in some African countries. Paper presented at 49th session of International Statistics Institute. Florence, Italy, 25 August-2 September.
_____ (1985). Integrated data collection for the rural sector: the Zambia experience. Paper presented at tenth conference of Commonwealth Statisticians. Nuffield College, Oxford, 31 July-9 August.
Biemer, Paul P. et al. (2001). Enumeration accuracy in a population census: an evaluation using latent class analysis. Journal of Official Statistics, vol.17, No.1, Statistics Sweden.
Cambodia, National Institute of Statistics (1999). Report of 1998 Census. Phnom Penh: Ministry of Planning.
Cantwell, Patrick, et al. (2001). Accuracy and coverage evaluation: missing data results. US Bureau of the Census, Decennial Statistics Studies Division, Census 2000 Procedures and Operations Memorandum Series, B-7* 28 February.
Dauphin, Marjorie, and Anne Canamucio (1993). Design and implementation of post-enumeration survey: developing country example. Washington: International Statistical Programs Center, US Bureau of the Census.
De Graft-Johnson, Kweku T. (1992). Mission Report on the Post Enumeration Survey, Windhoek, Namibia.
Hogan, Howard (2001). Data and analysis to inform the ESCAP Report. US Bureau of the Census, Decennial Statistics Studies Division, Census 2000 Procedures and Operations Memorandum Series, B-1*, 28 February.
Mongolia, National Statistical Office (2000). Population and Housing Census 2000: Mongolia Administrative Report, Vol. 1. Ulaanbaatar, Mongolia: NSO.
Onsembe, Jason (1999). Post enumeration survey for coverage and content error evaluation. Training Workshop for National Census Personnel within the Framework of 2000 Round of Population and Housing Censuses. Addis Ababa, Ethiopia, 15-19 November.
Turner, Anthony G. (1997). Report on Mission to Palestine National Authority, 9-21 November. United Nations Statistics Division, New York.
_____ (1998). Report on Mission to Palestine National Authority, 15-21 April. United Nations Statistics Division, New York.
United Nations (1998). Principles and Recommendations for Population and Housing Censuses, Revision 1. Sales No. E.98.XVII.8.
UNSD/SADC (2000). Workshop reports on sample survey design. Lusaka, Zambia, 28 February-4 March.
United Nations, Economic Commission for Africa (1975). Technical report on the post-enumeration survey for coverage evaluation of Africa population censuses. Paper presented at the ninth Session of the Conference of African Statisticians. Lome, 6-15 October.
United States Bureau of the Census (1985). Evaluating Censuses of Population and Housing, Statistical Training Document, ISP-TR-5. Washington: US Bureau of the Census.
Zambia, Central Statistical Office (1995). Post Enumeration Survey. Lusaka, Zambia: CSO.
* This document was reproduced without formal editing.
** US Bureau of Census, USA and UN Statistics Division, New York respectively. The views expressed in the paper are those of the authors and do not imply the expression of any opinion on the part of the United Nations Secretariat.
 This scenario for the PES is called the Procedure-A, Definition II approach. The Procedure-B approach differs in that it establishes census day residency at the time of the interview, establishes where persons in the subject housing unit lived on census day and searches for matches around their census day address. Definition I differs in that it asks for alternative census day addresses and searches them also.
 Based on Cambodia, National Institute of Statistics, 1999.
 Based on Abu-Libdeh, 1999, and Turner, 1997 and 1998.
 Based on Zambia, Central Statistical Office, 1995, and Banda, 1993.
 Based on Mongolia, National Statistical Office, 2000.
 Based on de Graft-Johnson, 1992, and Banda, 1993.