4.
Census
taking is improving constantly throughout the world. As censuses improve and people get used to using information from
them, censuses undergo more and more scrutiny. For instance, given one census,
the science of demographic analysis can produce independent estimates of
population size for comparison with the results of the following census.
Differences between the demographic analysis predictions and the actual results
of the new census are inevitable. These differences can take the form of
unreasonable sex ratios or wild disparities in age cohorts.
5.
We
reiterate that as censuses improve, they are used more and examined more. A
country’s last census may not have undergone much scrutiny, but its next one
could be a bombshell waiting to explode. As time passes, the need for
self-evaluation of census results inevitably increases.
6.
It should
also be noted that a population census is the most extensive and expensive
data-collection exercise any country can undertake. With vast amounts of
resources spent, there is usually tremendous pressure on census takers to
ensure that census results are accurate. As a result of the massive nature of
the census operation, it is inevitable that some inaccuracies arise from
deficiencies, including errors of coverage and response. The major difference
among countries is the extent of such errors. This, however, does not diminish
the importance of the census as long as users understand the limitations of the
data and the errors do not affect the major uses of the data (Cambodia,
National Institute of Statistics, 1999).
7.
It is
against this background that a number of methods for evaluating censuses have
been developed. Such methods include demographic analysis, comparison of census
totals with figures from other sources and matching census returns with those
from interviews selected on the basis of a probability sample in a
post-enumeration survey (PES). This paper focuses on the PES as a method of
evaluation. The paper discusses the purpose of PES’s; problems and constraints
associated with the PES; design and methodological issues; country experiences;
uses of PES’s and suggestions to improve PES programmes.
8.
While a
number of methods have been developed to evaluate census data, for many
developing countries the PES seems to be the most ideal owing to paucity of
appropriate data to facilitate the effective use of other methods. The lack or
incompleteness of registration systems and absence of regular population and
demographic surveys contributes to the lack of use or limited use of other
methods of census evaluation. In general, a number of countries have relied
primarily on PES methodology to evaluate the census undercount (Biemer et al.,
2001).
9.
Post-enumeration
surveys are an accepted census self-evaluation tool. Typically, a PES is an
independent survey that replicates a census. The survey and the census results
are then compared (matched). The results of the comparison are used to measure
the coverage and/or errors in content of the census. Estimates of net coverage,
the number of people omitted in the census, the number erroneously enumerated
and content error rates for specific questions are typical products of a PES.
10.
Additionally,
these estimates can be broken down further into their component parts. One can
design the survey so that reliable estimates of undercount or overcount can be
obtained for the entire census, for geographic areas of interest in the census,
and for any of a host of demographic characteristics, such as age, race and
sex, for which one might desire census coverage statistics.
11.
The survey
results also enable one to be able to uncover census methodologies or
operations that, when implemented, produced less than desirable results.
Suppose, for instance, that a high census omission rate was observed in rural areas.
One might then use specific PES results to examine whether the rural errors
were due to the omission of whole housing units. If so, this might well imply
an incomplete census frame and cause one to re-examine the methodology for
building an address list in rural areas.
12.
PES results
can be used to adjust census results. Using a carefully designed survey, under-
or overcounts can be converted into adjustment factors and the census
population increased or decreased accordingly by these factors. Later in this
paper we will discuss post-stratification and the need to ensure homogeneity
within each adjustment cell. It has been reported that in some African
countries, PES results have been used to support or defend census results when
the accuracy of the census is challenged (Onsembe, 1999).
13.
In
addition, censuses are used for many other purposes, such as updating
population estimates; developing and updating sampling frames; correcting and
updating population registers and the establishment and updating of key
components of the Geographic Information System (GIS). These many uses suggest
that there is a need to use an objective method for assessing coverage and
content errors as a crucial step for concluding a census operation (Abu-Libdeh,
1999). Quality assurance alone, introduced at various stages of census
operations, cannot ensure a complete evaluation of the qualitative and
quantitative accuracy of census data (UN Economic Commission for Africa, 1975).
14.
In
summary, post-enumeration surveys have many good purposes. They basically
inform users regarding the quality of the census data. As stated earlier,
providing limitations of published census data increases the confidence of
informed users in such data. On the other hand, there are distinct limitations and
constraints in managing and implementing the evaluation survey.
15.
Although a
PES can be an important component of a census programme and can contribute to
the process of building confidence in the census results, a poorly designed and
executed survey can inflict considerable damage to census legitimacy. We list
below some of the problems and constraints associated with PES’s:
·
Planning
and management of a PES, ideally, have to be undertaken by a staff that is
separate from the census staff. This is not usually the case in many countries;
·
The design
of the survey—especially the matching step—is relatively complex. For example,
in the United States planners continue to find design flaws in the matching
system. However, as corporate experience grows, these flaws become more and
more minor;
·
The survey
must be independent of the census. In the survey’s sample areas, census results
must not be biased by the
implementation of the PES;
·
The PES interview
itself is demanding. Usually it incorporates questions to determine if the
respondent should “really” be counted at the residence in question. Also, the
PES interview usually transpires after the census interview, at which point the
respondent may feel overburdened and not be as forthcoming with accurate
information;
·
Some of the
developing countries lack technical personnel with experience and skills in
survey methodology in general and PES in particular;
·
Past
failures in some countries in conducting PES’s discourage such countries and
others from conducting PES’s in the subsequent rounds;
·
Some of the
countries, such as the United States of America, which have conducted PES’s,
have not used the results to adjust population census figures. In such cases
questions have been raised about the rationale for conducting PES’s;
·
In some
countries, census planners feel it is enough to institute good-quality
assurance procedures at various stages of census activities; therefore they see
no need for a PES (UNSD/SADC, 2001).
16.
Lastly,
some of the countries are ambivalent about conducting a PES because a census is
usually a grueling and taxing operation, which saps the energy of those
involved. The general fatigue it generates may be sufficient to discourage the
conduct of the survey. Additionally, by the time the census enumeration is
completed there is usually a feeling of accomplishment among census planners.
They may, therefore, not see the need for conducting a PES, which, after all, may
just expose glaring discrepancies between census and PES results to the
detriment of the reputation of the census or statistical organization.
17.
Considering
whether a post-enumeration survey is worth it or not leads immediately to some
decisions that have to be made regarding the design of the survey. These
decisions revolve around what goals one has for the survey and what answers
best suit the individual situation in which the survey will be conducted.
18.
We will
assume in this paper that the goal of the PES interview is to establish
carefully who lived in the subject housing unit on the day the census was
officially taken. In the next step we match the results from the interview to
appropriate census forms in a well-defined area around that subject housing
unit.
19.
The central
facet of a PES is measurement of census omissions. PES methodology calls the
sample used to measure omissions the P sample or population sample.
Roughly, one interviews this sample and compares (matches) it to the census
results.
The resulting tallies can be represented
in a two-by-two table:
|
|
In census
|
Out of
census
|
|
|
In PES
|
|
|
|
|
Out of
PES
|
|
|
|
|
|
|
|
|
where
is the estimate of the number of people counted
in both the census and the survey,
is the estimate of the number of people
counted in only the survey,
is the estimate of the number of people counted in only the
census,
is the estimate of the number of people
missed by both the census and the survey,
is the total estimate of the number of
people counted in the survey,
is the estimate of the total number of
people.
20.
The
dual-system estimation model assumes independence between inclusion in the
census and in the PES. (We elaborate on how independence is implemented later in
this paper.) The dual-system estimate (DSE) of the total population is given by
=
21.
Simply
stated, the DSE raises the census total by the ratio of the total estimate of the
number of people in the PES divided by the estimate of the number 
that matched to the
census.
22.
In the
section above on purposes of a PES, we discussed breaking down the DSE
estimates by geographic areas and for any of a host of demographic
characteristics, such as age, race and sex, for which one might desire census
coverage statistics. If direct
estimates are desired for any of these breakdowns, one might post-stratify the
sample results into the categories desired. The objective of
post-stratification is to include in each dual-system estimate people who have
similar capture probabilities in the census.
23.
Omissions
are not the whole story in evaluating a census. Errors can be made in the
census itself that affect the overall under- and overcount measurement:
·
The census
can contain duplicate or multiple enumerations;
·
The census
could have people or housing units ascribed to the wrong geographic location
(and thus not matching the PES interview);
·
People
could be less than perfectly enumerated—that is, there could be insufficient
information for matching to the PES interview;
·
The census
could have erroneously enumerated someone who should have been enumerated
elsewhere or the enumerator could have made up a fictitious person.
24.
So, if one
is interested in quantifying these errors and their effect on census coverage,
a sample of the census enumerations has to be checked to tally the number of
times these types of errors, called erroneous enumerations, occurred. In PES parlance,
this sample is called the E sample. The section below on estimation
explains how the quantification of these errors is incorporated into the
dual-system estimation formula.
25.
A desirable
option for the E sample is to draw it directly from the census for the
sampled areas used in the P sample. This facilitates matching and helps
ensure that the survey is balanced—that is, that one is searching for omissions
in the exact same area where one is searching for erroneous enumerations. The
area one searches for omissions and erroneous enumerations is called the search
area.
26.
For
instance, the person might have lived elsewhere for the rest of the year. Some
E‑sample units and the people in them will not match any of the PES
interviews. They might have been missed in the P‑sample frame or truly
erroneously enumerated. Since these people have not been asked the battery of
questions to determine if a person should have actually been counted at the
particular housing unit on census day, a follow-up operation is needed to
determine if the unmatched people in the E‑sample unit were or were not
erroneously enumerated—that is, whether one of the census errors listed above
occurred or did not occur. More about
this follow-up interview is presented in the section below on reconciliation.
27.
Two other
design decisions have to be made: What is the primary sampling unit for the
survey? and What is the definition of cases to be included in the survey? This leads into our next topic, the sampling
frame of the survey.
28.
A popular
choice for a sampling frame is to use an area sample for the coverage
measurement survey. The primary
sampling unit can be the block. Blocks
are land areas surrounded by visible geographic features such as roads and
streams. The frame, therefore, consists of creating a universe of blocks in the
country and dividing those into sets of blocks (or clusters of blocks) that can
be interviewed by a single interviewer within the allotted time.
29.
Another option
is to use a survey that is already in place that is being taken around the time
of the census. This has the large
advantage of using an existing organization to manage the PES. It also has several disadvantages:
·
The
existing survey may not be large enough, and supplementing it may be as complex
as creating a specially designed survey;
·
Procedures
may have to be augmented with the result that the quality of the existing
survey and the PES suffers;
·
The
ultimate sampling units may not lend themselves to being an efficient erroneous
enumeration sample, where duplication, geocoding errors and so forth need to be
easily discernible.
30.
Above, we
mentioned the option of using block clusters as a frame for the survey. One might want to design the sample in
several stages by first choosing a group of these clusters and then optimizing
the sample by subsampling. For instance, the housing unit totals from a
previous census might be used to choose the initial sample of clusters; then,
after the addresses of all of the housing units in the sample clusters are
listed, the block clusters might be divided into small, medium and large
sampling strata, where
·
Small
clusters might consist of 0-2 housing units;
·
Medium
clusters might consist of 3-79 housing units; and
·
Large
clusters might consist of 80 or more housing units.
31.
The next
step could be to subsample some of these clusters:
·
Medium and
large clusters might be subsampled whenever their actual counts of housing units
differed significantly from what was expected (from the previous census);
·
Probably,
there are many small block clusters in rural areas. These can be subsampled to
keep the fieldwork manageable;
·
Again, to
keep the fieldwork manageable, some within-cluster
subsampling can be undertaken in very large clusters.
32.
For the
block clusters in which interviewers
are to enumerate, interviewers need maps to find each subject housing unit and
a listing of all the housing units in the block cluster. The listing operation is done independently
of any census activity. Not only are
the addresses of existing housing units listed, but inquiries are made at
commercial structures and other structures to ensure that no people live in
them. One option is to give listers
blank maps upon which they put a numbered spot representing each living quarter
or potential living quarters in the cluster.
33.
Listing
needs to be of high quality. It must ensure not only that the correct blocks
are listed but also that a complete list is obtained within the cluster. So a quality assurance plan needs to be
created to ensure correct listings—for instance, to ensure that commercial
structures have, indeed, been checked to see if they contain residences.
34.
The interview
approach is currently the common method used in PES. The questionnaire asks
about all people who resided at the sample address on census day and asks
questions to ensure that the respondents should have been counted at this
address. Subsequently it searches for them at that address and in the search
area surrounding it.
35.
Obviously,
since people move, it is most efficient if the PES interview can occur as soon
after census day as practical. Getting
information about out-movers (those who move out of the sample address between
census day and the PES interview) is usually difficult.
36.
The
training book, Evaluating Censuses of
Population and Housing (U.S. Bureau
of the Census, 1985), gives a set of basic questions for PES’s that are still
valid today. Essentially they are:
·
What are
the names of all people living here on census day?
·
What are
their relationships to each other?
·
What is
each person’s age and sex?
·
Is each
person still residing here? If not what
is his/her current address?
·
What are
the names and relationships of other people living here on census day?
37.
The last
two questions are examples of probing questions. To ensure that people didn’t live elsewhere on census day, the interview
could include more probing “other residence” questions that fit the country’s
existing situation.
38.
Since this
information is going to be matched to the census, it is obviously important
that it be complete. Some countries
have chosen to have the PES interview completed on a laptop computer which can
check within its program to ensure complete data. Quality control measures need to be employed in any case.
39.
When
measuring a small fraction of the population, those not counted in the census,
it is very difficult to deal with a high non-interview rate. To keep the non-interview rate low, a second
and final phase of interviewing might be planned during which the best
interviewers attempt to convert the remaining non-interviewed cases.
40.
After data
capture is completed for the PES interviews and the census data prepared for
each PES cluster, the next step is to match the two. One approach is to accomplish this in two steps: computer
matching, which “skims the cream” (that is, makes the easiest matches),
followed by clerical matching of the remaining non-matches and possible matches
as determined by the computer process.
41.
Of course,
matching can be completed manually. The
process involves clerks first gathering materials to facilitate matching in a
particular cluster. The materials
include:
·
Address
lists from both the census and the PES;
·
Census
forms for the cluster;
·
PES
interview results for the cluster; and
·
Maps for
the cluster from the census and the PES.
42.
Gathering
materials for a cluster is indeed a cumbersome part of the matcher’s job.
Regardless of how it is done, the basic process of matching remains the same:
comparing people’s names and demographic characteristics between forms—the
census form and the PES interview form. One design has clerical matching
comprised of four basic steps:
·
The clerk
must first determine if he or she has sufficient information for matching—that
is, whether the names and demographic information are complete enough to be
able to discern a definite match. The
clerk must determine this before
looking at households from the other survey.
It is a violation of independence to have the census or PES results from
a household biasing a clerk’s ability to read, for instance, a sloppily written
name;
·
The clerk matches
the P sample to the census throughout the cluster (or search area);
·
The clerk
then looks for duplicates within the E sample; and
·
Lastly, one
option is to have clerks do some surrounding block matching—that is, looking
for matches in the blocks surrounding the cluster if and only if one balances
this by looking for E‑sample correct enumerations with equal exactness.
43.
We
mentioned previously (in the section on the E sample) that many PES interviews
ask a battery of questions to determine if a person should have actually been
counted at the particular housing unit on census day. If whole households of E‑sample housing units have been
not matched to anyone in the P sample, they would not have been asked the
residence questions. A field follow-up
operation is needed to determine if the unmatched people in the E‑sample
unit were or were not erroneously enumerated.
44.
Additionally,
in a coverage measurement survey follow-up operation, P‑sample interviews
with unmatched people that had been completed by proxy respondents can be
followed up. Some research has indicated that PES interviews by proxy
respondents need this additional attention to ensure accuracy.
45.
After the
follow-up operation, forms are received in the processing office and their
final match status is coded—probably by the same people who did the earlier
matching. This completes the operations for the PES, and we move on to
estimating the under- or overcount.
a. Dual-system estimate
incorporating an E sample
46.
The
section above on the P sample delineated the basic dual-system estimation
formula for estimating the total population using the PES P‑sample
results and the census results. The section on the E sample specified errors
that could be made in census taking, called erroneous enumerations, which could
also be taken into account in the total population estimates. These census
errors included:
·
Duplicate
or multiple enumerations;
·
Housing
units ascribed to the wrong geographic location; and
·
People
with insufficient information for matching to the PES interview.
A
dual-system estimate of the total population (U.S. Bureau of the Census, 1985)
that included an E sample would subtract weighted totals of the errors above
from the census count. The refined estimate would be:
= 
where
= the total number of
whole person imputations in the census,
= the weighted number of people erroneously enumerated in the
census,
and
= the weighted estimate of duplicates,
= the weighted estimate of geographic errors,
= the weighted estimate
of census people having insufficient information for matching,
= the weighted
number of people estimated in the survey,
= the total number of people counted in the census, and
= the estimate of the total number of people.
b. Missing data
47.
Inevitably
in any survey, missing data are encountered.
Interviewers, however tenacious, cannot obtain every answer to every question
and, in fact, given time constraints in surveys, cannot interview every
household. During estimation for
coverage measurement surveys we must account for missing P-sample data and
missing E-sample data.
48.
Missing
data can be separated into categories with different approaches taken for each.
For instance, missing data can be divided into three types:
·
Entire
households that were unable to be interviewed in the P sample. With this
type of missing data, planners can take the approach of redistributing the
sampling weight assigned to each of these households to other households living
in similar-type dwellings interviewed in the same block;
·
Missing
demographic characteristics data. These
data are to be used in post-stratification (see next section). When they are missing, substitute data can be
imputed in their place using a “hot deck” procedure. This procedure chooses data from a completed case that are very
similar to the case with the missing data;
·
Unresolved
match status or residence status. In the P sample there are cases in
which, even though they have undergone a reconciliation interview, match status
and/or residence status cannot be resolved.
Match status is whether a person matches a census enumeration or not,
and residence status is whether or not a person actually should have been
counted in the census at the subject residence. In the E sample similar
missing data are encountered when there isn’t enough information to determine
if someone is correctly enumerated in the census. Cases without match or residence status can be assigned a
probability of matching and/or a probability of being a census day resident
based on all the information collected about them and cases with similar
characteristics (Cantwell et al., 2001).
c. Post-stratification
49.
The
object of post-stratification is to include in each dual-system estimate people
who have similar capture probabilities in the census. For instance, young people are usually more mobile than the
elderly and so more difficult to count.
To mix young and old in one dual-system estimate would lead to a bias in
the estimate. However, to have separate
post-strata for age groups and separate estimates for each and then to add the
estimates across the age groups avoids this bias problem.
50.
On
the other hand, having too many post-strata such that each one does not receive
a large enough sample will increase variance of the estimates. Post-stratification is a balancing act that
has to be carefully thought out.
51.
Some
examples of coverage measurement survey post-stratification variables are, for
instance: race, Hispanic origin, age, sex, tenure (whether one owned or rented
his/her home), degree of urbanicity and type of enumeration area of the country
and mail return rate of census forms.
52.
Country
examples, which are presented below, highlight recent national approaches to
planning and executing of PES’s in selected countries. They are also aimed at
illustrating that a PES can successfully be carried out, even for countries and
areas that conducted censuses for the first time. It is, however, important for
the PES to be included in the overall strategic plan of the census. If a PES is
conducted half-heartedly, as an afterthought, is poorly planned and executed
and has inadequate resources, the exercise is bound to fail. The country
examples are illustrative and are used, in this paper, in a positive sense of
bringing out issues and lessons learned.
53.
A
PES
was conducted in March 1998, about two weeks after the census. This was
essential because an accurate recall of persons in households on census night
is critical for the evaluation of de facto census coverage. The main objective
of the PES was to provide national-level estimates of coverage and content
errors in the census.
a. Sample design
54.
A
sample of 100 enumeration areas (EAs) was selected for the PES from a
population of about 24,918 EAs. The EAs
in this case were clusters of households. The sampling frame used in the PES
was the final list of EAs that were covered during the population census. All households in selected EAs were
enumerated. The PES, like the census,
excluded inmates of institutions such as hotels, hospitals and prisons. The homeless and transient population was
also excluded. The following
assumptions and requirements were set for the sample design: (1) a probability
as opposed to a purposive sample was selected; (2) an area-based sample was
selected to ensure that missed census households had a chance to be enumerated
during the PES; (3) the presumed census undercoverage rate, for purposes of
predicting reliability of the PES, was taken to be 3 per cent. Reliable
estimates were expected at the national level only; however, separate estimates
were calculated for the rural area and large regional groupings of provinces.
No estimates were considered for the urban area.
55.
The
PES for Cambodia had three distinct operations, namely, listing and enumeration
of persons in all households within the selected EAs; matching of
characteristics and particulars collected during the PES listing and those
obtained from the corresponding census questionnaires; and field
reconciliation.
56.
Complete
independence between the census and PES operation is an ideal situation, which,
in most cases, is not attainable under field conditions. Notwithstanding the above, the planners for
the PES in Cambodia took measures to make the PES as operationally independent
as possible from the census. Such
measures included:
·
Taking
care in selecting better-qualified and trained enumerators. The quality of fieldwork was maintained by
close supervision by well-trained and experienced supervisors. It was ensured that no enumerator involved
in the census could enumerate for the PES the same EA as in the census;
·
Enumerators
for the PES had no access to information/data collected in the census in their
respective areas of operation;
·
The
location of selected EAs for the PES was not disclosed to the field staff
beforehand;
·
The
PES did not begin until all completed census records were transported and
stored at the National Institute of Statistics headquarters.
b. Matching
57.
The
purpose of the matching operation was to classify each person listed in the PES
and the census according to whether he or she was correctly enumerated in the
census or tentatively missed in the census.
For the matched person the operation entailed a person-to-person match
between the PES and census records. For
the tentatively missed person they verified whether the person was actually
present or not in the household on census night. Owing to the confusion of
reported names and sometimes relationship to the head of household between the
census and PES, some people were initially classified as missed.
c. Coverage error
58.
Cambodia
used a single rather than a dual system of estimation, which is said to be a
simpler method not subject to a strict independence requirement between the
census and the PES (Turner, 1997). This method, however, tends to underestimate
coverage error.
59.
Net
coverage error was estimated at 1.78 per cent. The percentage of
underenumeration was highest among infants (children under the age of one),
followed by people in the age group 20-29.
Cases of omission of households were minimal.
d. Content error
60.
Content
or response error includes mistakes in reporting and/or recording of items.
This was measured for selected variables such as age, mother tongue, literacy,
main activity, employment period, children ever born and children
surviving. The index of inconsistency,
which is the relative number of cases for which responses vary between the
census and the PES, ranged from 4.97 to 44.34.
Mother tongue and employment period were the most and least reliable,
respectively. The other remaining
indices of inconsistency were below 20 and therefore considered low.
61.
The
PES was conducted two weeks after the population and housing census. The survey
was designed to allow for the estimation of coverage error. Results provided
for quantitative evaluation of coverage errors in more than one domain
depending on the sample size. The dual-system method was used to measure
coverage error. The method provided an estimate of the census population from
the PES and the population counted in both the census and the PES. Turner
(1998), however, cautions that the method of dual-system estimation requires
that the PES and the census be completely independent in all respects. This
ideal condition is not usually met in actual practice.
a. Sample design
62.
A
4.7 per cent sample of EAs was selected with equal probability. The survey
results were used to assess coverage error at the national level, Gaza Strip,
rural, urban and refugee camps. A single-stage stratified systematic random
sample design was adopted. It consisted of 140 census EAs, some of which were
composite. All households and persons in the selected EAs were re-enumerated
for the PES. About 21,000 total households
were in the selected EAs. An area-based sample was necessary to ensure that
missed households had a chance of being enumerated in the PES. The presumed
census undercoverage rate for purposes of predicting reliability of the PES was
taken to be 5 per cent. The sampling
frame for the PES was the final list of 3,308 EAs in the West Bank and Gaza,
used in the population census (Abu-Libden, 1999; Turner, 1997 and 1998).
b. Matching and coverage error
63.
The
dual-system method was used to obtain coverage estimates. The matching was done by comparing all
census questionnaires in an EA with all questionnaires in a corresponding
selected EA in the PES. The net
undercoverage was estimated at 1.8 per cent.
Due to cost and time considerations, the PES did not measure content
error.
c. Quality and independence
64.
Independence and quality issues were handled as follows: fieldwork
for the PES started a week after census questionnaires were collected from the
field. Special instruction manuals, training and field procedures were designed
specifically for the PES. The sample of EAs was treated as confidential until
enumerators were in the field, and those involved in the PES were better
qualified. For example, the best crew
leaders in the census served as enumerators, the best supervisors served as
crew leaders and the best district directors managed the fieldwork. No single
person was allowed to work in the same area where he or she worked during the
main census. The same approach was adopted for data processing, which was done
away from the main census data-processing area to avoid any possible
contamination.
65.
After
the census of 1990, which was completed in September, a PES was conducted in
December 1990, two months after the census. The objective of the PES was to
measure both coverage and content errors. In this regard the PES combined the
post-censal matching survey for measurement of coverage error and the
re-interview survey for evaluation of content errors. The alternative methods
to a PES were not used because of the unavailability of reliable records and a
registration system, as well as limited demographic data in the country
(Zambia, Central Statistical Office, 1995).
a. Sample design
66.
A
stratified cluster probability sample design was adopted, standard enumeration
areas (SEAs) were selected with probability proportional to size within each
stratum and all households were enumerated in the selected SEAs. The population
covered by the survey excluded persons living in institutions and collective
dwellings.
67.
The
sampling frame consisted of all SEAs demarcated for the population census. The frame was stratified by province, rural
and urban. Using a sampling fraction of 1 per cent, a national sample of 160
SEAs was selected and distributed equally between the rural and urban strata.
The 160 SEAs were allocated to each of the provinces proportionately to their
measures of size based on the 1990 projected population.
68.
The
PES was not carried out until December for the following reasons: The census
operation including mopping up was completed towards the end of September 1990;
some key staff members of the Central Statistical Office (CSO) who were to be
involved in the PES were engaged in the summary counts; and additional
resources had to be mobilized for the PES.
b. PES enumeration
69.
Although
the PES methodology required independence between the census and the PES, in
reality, it was impracticable to achieve complete independence. An attempt was made, therefore, to maintain
some independence between the two operations, to the extent possible, in
addition to maintaining quality by taking the following measures: census
supervisors were used as PES supervisors so as to take advantage of their
experience and qualifications; they were, however, assigned to work areas that
were different from their census areas. The PES planners also took advantage of
the experience and qualifications of regular CSO enumerators by using them as
PES enumerators. Most of these enumerators did not participate in the census.
The few who participated in the census had different work areas. PES field
staff did not know preliminary census results of areas to which they were
assigned.
70.
There were 160 interviewers, one for each selected SEA and 51
supervisors, giving a ratio of about one supervisor for every three
enumerators. The low ratio ensured effective supervision, thereby enhancing the
quality of the data collected by enumerators. Instruction manuals and control
forms were prepared for the enumerators and supervisors. The enumeration in the
PES was on a de jure basis, while that of the census was both de jure and de
facto. To facilitate matching, the questionnaire contained pre-coded answers
and shaded spaces for recording census responses.
c. Matching
71.
The
PES questionnaires were matched to the census questionnaires by pairing each
household and each person with corresponding census records. A two-way matching
system, which was done manually, was adopted to identify omissions and
erroneous census enumerations.
Non-movers and out-mover categories were matched, while in-movers were
identified but not matched. The
matching was done at two levels, initially by identifying housing units and
households and subsequently by matching individual characteristics. The matching status was classified into the
following categories: perfect match, possible match and non-match.
72.
At
the initial stage, while using strict matching rules, there was a high
incidence of possible matches and a low incidence of perfect matches. This was due mainly to differences in
reported names of members of households between the two sources, particularly
in rural areas. The use of alternate
names is very common in Zambia (Banda, 1985).
Large age variation, especially among older members of households, also
posed problems to matchers. Subsequently, age tolerance limits for the old age
groups were somewhat relaxed. Another
reason for the high incidence of possible matches was partly caused by the
differences in the use of the concept of household between the PES and census
enumerators with respect to polygamous families. There were a number of areas
where polygamous families were regarded as one household in one case and two or
more households, depending on the number of wives, in other cases. This caused
some persons to be counted more than once in the PES or census.
d. Field reconciliation
73.
Field
reconciliation was considered necessary because some unmatched cases identified
in the office could be matched in the field. The non-matched questionnaires
were, therefore, referred back into the field in order to identify erroneously
enumerated cases and to resolve cases which had insufficient matching
information. The exercise contributed to the increase in perfect matches.
e. Coverage error
74.
The
net coverage error was 1.92 per cent.
The urban net coverage rate was 2.57 per cent, much higher than that of
the rural areas (0.92 per cent). Some
provinces had high net coverage errors mainly due to poor mapping and
demarcation of areas into SEAs.
f. Content error
75.
In
general, the index of inconsistency for age was higher for rural areas for all
ages compared to urban areas. The results also showed an unexpectedly high
index of inconsistency for the relationship of son/daughter to head of
household. The African extended family
system, which, in most cases, does not distinguish between one’s children and
those of one’s brother or sister, was identified as the main reason. This
finding helped in framing better questions with respect to this item in the subsequent
census and surveys.
76.
Plans
for census evaluation were less elaborate in Mongolia, due mainly to a shortage
of funds. Following the completion of fieldwork of the 2000 census, a limited
PES was undertaken in Mongolia, with the main objective of evaluating the
coverage of the census. The PES was,
therefore, not intended to be a major source of information on quality of
responses. Only a few items were considered
for evaluating content error. For a
subsample of the PES an attempt was made to compare the census and PES
responses for the same questions to provide some insights into consistency of
interviews. The PES was conducted three
days after the completion of the census fieldwork. Emphasis was placed on quality assurance rather than a good
evaluation. The PES was therefore
restricted to the most difficult areas, mainly in urban areas.
a. Sample design and data collection
77.
A
two-stage cluster sample design was adopted with first-stage units selected
purposively. The PES form was adapted from the census questionnaire. The
National Statistical Office used the best available staff that had not worked
as enumerators in the census. The
enumerators were not notified of their selection until after the census
fieldwork had been completed. Similar
instructions and definition of concepts were used in the PES and during the
census.
b. Matching
78.
Following
the PES, names and characteristics of persons included in the census and PES were
matched and the results compared. The matching, as expected, was confined to
persons enumerated in either the census or the PES or in both.
c. Problems with the Mongolia PES
79.
The
importance of probability sample selection was not recognized. The selection
for the PES was ad hoc and purposive. The operations of the census and the PES
were not always independent, as it was possible for some managers to see to it
that high agreement between the census and the PES was achieved. For example,
in four areas there were no differences between census and PES results, which
was unusual, while some other areas displayed large errors. A solution adopted
was to exclude the extreme high and low ends of the distribution of sample
areas to remove most, if not all, of the suspect areas from the analysis
(Mongolia, National Statistical Office, 2000).
80.
Dauphin
and Canamucio (1993) have reported on successful PES’s conducted in Burundi and
Rwanda during the 1990 round of censuses. The procedures and results are
summarized below.
a. Sample design
81.
In
both Burundi and Rwanda the smallest areas for which boundaries could be
verified on maps were EAs. The sampling
frame of EAs in Burundi contained a total of 5,500 EAs while in Rwanda there
were 6,200 EAs. In both countries a single-stage stratified cluster sample
design was adopted. All households in
the selected EAs were enumerated to facilitate matching against the census
records. In Burundi the strata corresponded to the domains of estimation,
namely, the urban and rural areas. Similarly, for Rwanda, the first level of
stratification corresponded to domains of estimation, namely, urban, rural and
the capital city. The second level of
stratification was geopolitical subdivisions, even though no estimates were
expected for these subdivisions. Seventy EAs were selected in Burundi and 120
in Rwanda.
b. Data collection
82.
PES
questionnaires were prepared based on final census questionnaires. The questionnaires allowed for the
classification of each listed person in the households into non-mover,
in-mover, out-mover, or erroneously enumerated.
83.
In
Both Burundi and Rwanda, the data collection aimed at identifying all usual
residents at the time of the census in addition to those found as of the date
of the PES. Data collection started two
weeks after the completion of the census fieldwork. The PES response rates in
Burundi and Rwanda were 98.0 and 99.9 per cent, respectively.
c. Field staff
84.
To
control non-sampling errors and ensure the quality assurance of the PES, 70
enumerators were deployed in Burundi and 167 in Rwanda, at least one in each
EA. The enumerators were detailed to verify EA boundaries, to conduct
preliminary listings and to enumerate all households within EAs independently
from the census count.
85.
In
Burundi PES staff were selected from the census pool, taking advantage of their
better qualifications, but they were assigned to different areas for the PES.
In Rwanda some of the PES enumerators had not participated in the census.
d. Matching
86.
The
PES methodology that was adopted included a two-way matching of households and
household members with census records of the same EAs. The matching operation resulted in the
classification of all PES and census-enumerated persons within the selected EAs,
in specified categories that permitted the calculation of coverage error and
the determination of cases for which content error was calculated.
87.
Namibia
conducted its first PES in 1991. Sample selection was made from a frame of EAs
compiled from the census. At the time of selecting EAs, they were assumed to be
almost equal in size with respect to population. The selection of EAs was
therefore based on equal probability. During the enumeration, however, EAs
showed wide variation in population size.
88.
The PES for Namibia faced a number of
operational problems, among them:
·
There was
confusion over boundaries of EAs. In
certain parts of the country many enumerators did not follow the EA boundaries
during enumeration. The use of
well-defined and identifiable EAs, on the ground, with clear written boundary
descriptions, is essential for a successful PES;
·
Many
enumerators did not cover their EAs in a systematic way. For example, some
households and houses were omitted;
·
Some
enumerators started fieldwork without pre-listing, especially in small towns
and villages without street names and house numbers;
·
Callbacks
were, in some cases, not effected;
·
There were
no pre-tests of questionnaires and procedures owing to staffing, time and
financial constraints; and
·
Reconciliation
was not done. Therefore, possible matches were not verified.
a. Matching
89.
A
listing was made of all persons, who were classified as non-movers, in-movers
and out-movers. The first stage of matching was not satisfactory, as there was
a low percentage of matched cases. Some
respondents may have used different names during the census and PES, especially
illegal immigrants. There were, at
times, different interpretations of concepts of house and locality. Households, in some cases, were classified
as one during the PES and two during the census. If a field reconciliation
exercise had been done, some of the possible matches could have been converted
into perfect matches. The low percentage of matches could also be attributed to
unqualified staff, who were initially involved in the matching exercise. Corrective measures were taken later and the
situation somewhat improved (de Graft-Johnson, 1992).
90.
The
assumption was that the unmatched persons were presumed to have been missed during
the census. This assumption is not necessarily true. Those with high rates of
non-matching included children below the age of four and female heads of
households.
91.
Two
items included in the census but not in the PES could have been better
variables for matching purposes. They are birthplace and place of usual
residence. Tests in some African countries have shown that the former has a low
index of inconsistency. The latter could have assisted in the search for
addresses of persons who stated that they were enumerated during the census in
a particular household but who could not be matched.
a. Introduction and sample design
92.
The
United States undertook an extensive coverage measurement survey during its
last census. Census day was 1 April 2000. The survey included 314,000
households, a sampling rate of about two tenths of 1 per cent. The survey
results had to be produced by April of 2001 for a decision about whether or not
to adjust the census counts. Many
operational decisions were made based on the need to do the survey quickly but
with the utmost attention to quality. The survey did not include several
things: any content assessment, any use
of administrative records or any inclusion of demographic analysis results into
the estimation process. Demographic analysis results were used instead as an
independent benchmark for the survey results.
93.
The
first stage in drawing the sample was to divide the entire country into
clusters of blocks such that each cluster had an expected 30 housing units.
Enumerators then listed the addresses of all the housing units in the cluster.
A stringent quality assurance (QA) programme was imposed upon the listing
operation. Listing occurred well before the census itself and was scrupulously
independent of it. The population residing in group quarters (typically,
housing units with temporary or transient occupants) and institutions (such as
nursing homes for the elderly) was not included in the survey. Subsampling was
undertaken in the manner described above so that a sample was used that was
efficient with respect to field operations.
b. Interviewing
94.
Interviewing
by telephone of about 30 per cent of the sample began three weeks after census
day, and personal visits of the remaining cases began in mid-June. All
interviewing was completed on personal computers by 6,500 field staff. These
laptops permitted interviewing and data capture to be executed much more
quickly than a paper instrument would have allowed. Since completed interviews were downloaded each evening, an added
benefit was that reports on the quality of these interviews were available to
supervisors the next day. QA on interviewing was extremely fast and effective.
The final two weeks of an eight-week personal interviewing period were dedicated
to converting non-response cases into interviews. The non-interview rate for
housing units occupied on census day was 2.9 per cent (Hogan, 2001).
c. Matching
95.
Matching
was carried out in two stages: computer matching and clerical matching. About
two thirds of the time, people in the census and the PES were matched by a
conservative computer-matching algorithm. To prepare for clerical matching in
the remainder of the cases, the computer linked census and PES households where
the residents had a high probability of matching. Clerks examined the linkages
using an automated software system. The system allowed them to access
computerized address lists, census forms, PES interview results and maps for
the cluster and use this information to make objective matching decisions. This
software allowed the matching workforce to be one tenth the size it was for the
1990 PES. All the matchers were able to work at one site. This improved the ability to ensure that
everyone was doing the same task with the same quality.
d. Reconciliation and estimation
96.
Many
cases that were not matched were sent to the field for reconciliation. All E‑sample cases were sent, as were
P‑sample cases with proxy respondents.
(The reasoning for this is given in the section above on
reconciliation). After this follow-up, the last cases were given match codes
and sent to estimation (described in the section on estimation).
e. Results
97.
The
PES estimated the net national undercount to be 1.18 per cent (0.13 per cent standard
error). This compared well with the previous (1990 census) measurement of 1.61
per cent (0.20 standard error). Additionally, the undercount of some minority
groups in the United States was reduced compared to the 1990 census. The
undercount for non-Hispanic blacks dropped from 4.57 per cent (0.55 per cent
standard error) to 2.17 per cent (0.35 per cent standard error), and the
undercount for Hispanics dropped from 4.99 per cent (0.82 per cent standard
error) to 2.85 per cent (0.38 per cent standard error) (Hogan, 2001).
98.
However,
these results differed significantly from the demographic analysis estimates of
the undercount. As of the 1 April 2001 deadline, based on these differences and
other considerations (all of which needed time to be researched), the
government of the United States of America decided not to adjust the official
census counts using the PES results. Research continues and decisions about
adjusting the counts for other uses will be made in the near future.
99.
The
use of PES results should be viewed within the context of the many possible
specific objectives, which fall under the umbrella of measuring coverage and
content errors. Countries can therefore emphasize all or some of these specific
uses. Such uses include:
·
Providing a
statistical basis for adjusting census results;
·
Evaluating
the EAs as sampling units for intercensal surveys;
·
Identifying
procedural and conceptual improvements needed for future censuses;
·
Providing
to users of census data, such as researchers, information on sources and causes
of errors.
100.
As
already stated, one of the major criticisms against PES’s is that, in some
cases, the results are not used for adjusting census data. It should, however,
be pointed out that while estimates of undercount may be made, adjusting census
numbers is not universal. In some countries the question of adjustment is
settled as a matter of policy rather than as a statistical decision. It is
clear that the uses of PES results are multifaceted, so the question therefore
is whether countries should fulfill all the objectives or some of them to
justify the conduct of PES’s. In this connection, it has been suggested that it
is better to use the PES not for adjustment, but as an evaluation tool to help
assess and analyse the quality of the census and to uncover potential problems
to correct in future censuses (Turner, 1998). We highlight below some of the
practical uses of PES results in selected countries.
101.
The
PES is part of the methodological work which contributes to additional
knowledge that may help to improve future censuses and intercensal surveys. The
Zambian 1990 de jure official census figures were adopted after studying the de
jure PES results. In addition, the content error analysis showed that age
reporting was more accurate than anticipated in developing African countries.
As a result of the high consistency index for age data, there was no need to
smooth the data. In fact, the PES results helped analysts to notice, at an
early stage, the unexpected age distribution, which, as mentioned earlier, was
not due to age misreporting but was the result of a possible effect on the age
structure of the HIV/AIDS epidemic, especially for women aged 15 to 34. AIDS is
at its peak between these ages. Without the PES results it would have been
assumed that there were age-reporting errors and analysts would have missed the
opportunity to address the problem at an opportune moment..
102.
PES
results were used to adjust the census results of the Occupied Palestinian
Territory. In the case of Cambodia, the PES results threw some light on the
potential sources of undercount and overcount in the census. The findings will
help in planning future censuses. In addition, while it was not possible to
adjust population figures down to the village level based on the national
undercount, for the purpose of projections, the base population at the national
level was adjusted taking into account net coverage error. Swaziland plans to
use the PES results to adjust population projections, and South Africa plans to
use PES results to adjust its population census figures.
103.
Adjusting
of census data based on PES results can at times be controversial at both
political and technical levels. At a political level adjustment may introduce possible
realignment of polling districts, which may be undesirable in some quarters,
and therefore would be resisted. At a technical level, in many countries the
sample sizes for the PES are small; therefore, results can only be inferred for
larger domains such as national, rural and urban domains. It is, however,
tempting after getting the adjusting factors from the PES results to make
adjustments at lower administrative domains. The latter may not be justified if
the sample sizes for such domains are small and inadequate, thereby rendering
the results unreliable.
It is evident from the observations above that some
countries have used the PES to adjust census results and also as an evaluation
tool considered crucial to the completion of the census task.
It is clear from country examples that PES’s had mixed
results. For some countries relatively successful PES’s were conducted. It is,
however, also true that some countries had unsuccessful PES’s, mainly because the
best practices of conducting PES’s were not followed. The examples are
revealing as they highlight problems that tend to crop up in PES’s which
require a focused approach in order to resolve them. A summary of problems
experienced by various countries whose PES’s have been described in this paper
are:
·
Poor
cartographic work;
·
Neglect of
operational independence, as it relates to field staff, between the census and
PES activities;
·
Inadequate
resources allocated to the PES;
·
Lack of qualified personnel to manage and carry out various PES
activities;
·
Inclusion
in the PES questionnaire items which were not useful for matching purposes;
·
Not
following basic principles of probability sample design and selection;
·
Not
carrying out field reconciliation for possible matches;
·
Difficulties
in matching alternate names because of the common practice, especially in
Africa, for individuals to have more than one name;
·
Difficulties
of matching names because of the absence of unique addresses, especially in the
rural areas;
·
Neglect
of pre-testing PES procedures;
·
Flawed
inferences based on relatively small samples; and
·
Delay
in undertaking the PES in the case of one country.
104.
Post-enumeration
surveys are worth conducting if they are carefully planned and function within
operational and statistical constraints (United Nations, 1998). The PES
methodology is adaptable to many circumstances, such as the use of single or
dual methods of estimation. While independence between the census and the PES
is a fundamental requirement, in practice operational independence seems to
suffice because it is not possible to make all the various aspects of the
census and PES operations mutually exclusive.
105.
Since
there is no error-free census, there is a need to continue to consider PES’s as
part of census programmes. In many developing countries the PES seems to be the
most feasible method of evaluating census results owing to the lack of
comprehensive and, at times, accurate data from other sources, such as civil
registration and population registers.
For the PES to be useful in measuring coverage and content errors, it
must be well planned and implemented.
We therefore propose that efforts should be made to:
·
Develop
good area frames, with well-defined and mutually exclusive enumeration areas;
·
Design
plausible probability samples to facilitate objective generalization of PES
results to relevant domains;
·
Adopt
efficient but realistic matching rules;
·
Adhere,
to the extent feasible, to the ideals of independence between the PES and
census operations;
·
Harmonize
definitions and concepts used in both the census and the PES;
·
Ensure
that items included in the PES for matching purposes are relevant and useful;
·
Involve
well-trained and qualified field staff;
·
Train
key staff, involved in the design of PES samples, in survey sampling methods;
·
Carry
out pre-tests for the PES process and field reconciliation;
·
Allocate
adequate funds to the PES within the framework of the census; and
·
Keep
the PES as simple as possible and stick to objectives that are attainable.
106.
Notwithstanding
the conventional wisdom, a PES may not be as complex as perceived. It normally
covers few variables compared to other household surveys and is usually based
on a comparatively smaller sample. The PES calls for additional resources, but
the fact that it is carried out immediately after the census means that its
overhead costs may be greatly reduced. Such overhead costs as vehicles, office
space, computers and census maps are usually covered by the census programme,
thus making the PES more affordable. Onsembe (1999) observes, from his African
experience, that the direct cost of a PES programme is about 1 per cent of the
census costs. For example, the cost of the PES programme for Kenya is
anticipated not to exceed 1 per cent of the total census costs
