ABS Policy on Release of Data Subject to Error
Summary
From the design to the dissemination stage of a statistical collection there are many ways in which errors can enter the statistics. Some errors, resulting from the use of sampling, are random and their magnitude measurable. Other errors of a non-sample nature can introduce bias into the results. The ABS has a responsibility to place "quality labels" on all disseminated data to assist in interpretation.
ABS Policy
2 It is ABS policy that estimates for individual cells of a published table should not be suppressed solely because they are subject to large sampling errors, provided users are given adequate caution of the lack of reliability of the data.
3 It is ABS policy that data with serious bias, which may be misleading to users, should not be published.
4 It is ABS policy that tables that are not published at a given level of detail, because most of the estimates would be highly unreliable in terms of sampling errors, may be released to users with appropriate indications of the sampling errors. Greater caution should be exercised in releasing data that are subject to serious biases when the data have not been published.
5 It is ABS policy that all ABS publications should include a statements about data quality and methodology.
6 It is ABS policy that information indicating the size of sampling and non-sampling errors should be provided.
7 It is ABS policy that sampling errors may be presented in publications in at least one of the following ways:
a) incorporated in the table with the estimates to which they refer;
b) as a separate table of actual errors; or
c) in illustrative form as a table or graph from which the reader can estimate the sampling error associated with particular statistics.
(ABS policy was originally approved by a Division Heads Meeting, February 1988, following consideration of a paper "Policy on the Release of Statistics Subject to Error". Data Quality is also presented in the Publishing Manual, April 1990.)
8 It is ABS policy that, in addition to standard errors corresponding to the published estimates, methods should be provided for approximating the standard errors of derivative statistics such as estimates of change, ratios of different estimates for the same period, changes in ratios over different periods, etc. If necessary, supplementary data needed for this purpose, such as correlations, should be presented.
9 It is ABS policy that, for releases on electronic media, statements on sampling and non-sampling errors should be available for distribution to users of these products and services.
Clarification of Policy
10 If more than one-third of the cells in a table are subject to sampling errors of 25 per cent or higher, authors areas are advised to combine some of the classifications or to withhold the table from publication.
11 The seriousness of bias in data depends on the uses to which the data are likely to be put. If the data are likely to be misleading, they should not be published.
12 If users wished to invoke Freedom of Information legislation, data with high sampling errors would have to be made available. Data with high sampling errors may be released to users, but not published, on the understanding that it is the users' responsibility to use the data wisely. The ABS should not accept this responsibility although it does have a professional obligation to put "quality labels" on data.
13 Appropriate cautions should always be provided with the release of unreliable data, and users should be requested to present these cautions in any publications, reports etc. in which they may quote data. If the data are known to be misleading for the use to which they are to be put, the user should be discouraged from acquiring the data. However if they insist, the data should be provided if resources are available. (Such requests would have a lower priority.)
14 The errors that affect data are an important characteristic of the data and there should be some discussion of these errors in all ABS publications. Technical terms such as sampling error should be explained, possibly by example. For guidelines, see the Publishing Manual.
15 Discussions on error should cover both sampling variability (applicable to sample surveys only) and non-sampling variability (which includes response variability, response bias, non-response, imputation, and processing error).
16 Statements on methodology should refer to aspects of the design of the survey which affect the magnitude of error from these various sources. The wording of each statement for Central Office publications should be checked with Statistical Services Branch prior to publication. For State publications the statements should be checked with the SSUL unit.
17 An asterisk should be used to indicate those figures that are subject to very high sampling or non-sampling error.
18 For releases on electronic media, a statement on sampling and non-sampling errors should be included in the documentation prepared for the release. For micro data releases, the documentation should include methods for deriving sampling errors for the main type of estimates which might be derived from the micro data.
Guidelines on the Description of Data Quality and Methodology
19 Users generally recognise that data are subject to error and therefore need to know whether the data are sufficiently accurate to be useful to them. To make this assessment, they need to be informed of the likely principal sources of error and, where possible, the size of the error.
20 Indicators of data quality cannot be expected to be comprehensive, and users also require a knowledge of the data collection and processing methodology in order to assess whether the data adequately approximate what they wish to measure and whether the estimates they wish to use have tolerances acceptable for their intended purpose.
21 The guidelines are to be found in the Publishing Manual. They are primarily intended for internal use within the ABS when the documentation and dissemination related to a statistical program are being planned or reviewed.