Protocols for Official Statistics
Preface
Official statistics are one of the cornerstones of good government, and public confidence in good government. The Statistics Act 1975 provides a strong foundation for the integrity of official statistics. These protocols embody the key principles that underpin the Statistics Act. The protocols are designed for the use of agencies which operate under the Statistics Act, but without the explicit obligations placed on the Government Statistician. In meeting the protocols, any agency would fulfil these obligations.
Part of our British inheritance, as noted below in the instruction to Hobson, has been a need for statistics that we all can trust, to be fit for the uses they have. The Statistics Act 1975 is still one of the leading examples of statistical legislation in the world.
Instructions to our trusty and well beloved William Hobson esq. our Governor and Commander-in-Chief in and over our Colony of New Zealand or in his absence our Lieutenant Governor......Given at our Court at Buckingham Palace on 5th day of December 1840 - Victoria R.
Lord John Russell to Governor Hobson 9th December 1840.
"And whereas you will receive through one of our Secretaries of State a book of tables in blank commonly called the Blue Book, to be annually filled up with certain returns relative to the revenue and expenditure, militia, public works, legislation, civil establishments, population, schools, course of exchange, imports and exports, agricultural produce and other matters in the said Blue Book, more particularly specified to the state of our sociology, now we hereby signify our pleasure that all such returns be accurately prepared and punctually submitted to us from year to year through our Secretaries of State and that no officer in our said Colony within whose Department it may be to contribute any return or returns for the purpose aforesaid or to prepare the same when so contributed, shall be entitled to receive or shall receive from you any warrant for the payment of his official salary which may become due and payable to him so long as such duty as aforesaid shall be in arrear or remain unperformed."
Extract from "Copies or extracts of correspondence relative to New Zealand in continuation of papers presented to the House of Commons on 14th April 1840 in pursuance of address of 8th April."
These protocols represent a 20th century version of this very necessary commitment to excellence in official statistics.
Len Cook
Government Statistician
_____________________________________
Introduction
1 Official statistics(1) are produced by government to inform debate, decision making and research both within government and by the wider community. Objective, reliable and accessible official statistics give people confidence in the integrity of government and public decision making. Hence, in producing official statistics it is important that the relevance, reliability and integrity of official statistics is maintained, and is perceived as such. It is also important that co-operation from respondents and providers of information is maintained, and information given on a confidential basis must remain so.
2 The range of New Zealand official statistics is wide, covering all key areas of national activity and life. The greater share of official statistics is produced by Statistics New Zealand and over the years Statistics New Zealand has established policies and processes for achieving a high degree of public confidence in the objectivity and reliability of statistics it releases. However, a good deal of official statistics are produced by other departments from surveys conducted by them or outsourced, or from administrative records.
3 The Statistics Act gives the Government Statistician responsibility for ensuring that all official statistics are produced using methodologies and practices which are statistically correct and will provide outputs in which users can have confidence.
(1) See Appendix - Definition of Official Statistics
Principles
4 With the dual aim of maintaining public confidence in official statistics and promoting uniformly acceptable standards for official statistical surveys, Statistics New Zealand has developed these protocols as a code of practice for the production and release of official statistics. They are based on the following principles:-
1. the need for a survey must be justified and outweigh the costs and respondent load for collecting the data
2. a clear set of survey objectives and associated quality standards should be developed, along with a plan for conducting the many stages of a survey to a timetable, budget and quality standards
3. legislative obligations governing the collection of data, confidentiality, privacy and its release must be followed
4. sound statistical methodology should underpin the design of a survey
5. standard frameworks, questions and classifications should be used to allow integration of the data with data from other sources and to minimise development costs
6. forms should be designed so that they are easy for respondents to complete accurately and efficient to process
7. the reporting load on respondents should be kept to the minimum practicable
8. in analysing and reporting the results of a collection, objectivity and professionalism must be maintained and the data impartially presented in ways which are easy to understand
9. the main results of a collection should be easily accessible and equal opportunity of access is enjoyed by all users
10. be open about methods used and documentation of methods and quality measures should be easily available to users to allow them to determine fit for use.
5 Guidelines for implementing these principles are set out in this document.
Use of these protocols
6 The protocols do not set out how to conduct a survey. Such information is found in books such as "A Guide to Good Survey Design" published by Statistics New Zealand. More detailed procedures for various aspects are also available and these are referenced where relevant. The purpose of the protocols is to set out principles for establishing and maintaining confidence in official statistics.
7 Where Government agencies, including Statistics New Zealand, undertake or commission statistical surveys, these protocols should be explicitly adhered to. They should be followed also for official statistics derived from administrative data produced as a part of service delivery or regulation. Such statistics are usually used for informing policy development or evaluation, and require the same principles to be followed as surveys to achieve public confidence in their integrity and the processes they inform.
8 Where a Government agency other than Statistics New Zealand has agreed to abide by these protocols, the requirement that the survey should be submitted to Statistics New Zealand for the approval of the Minister of Statistics, under section 6 of the Statistics Act 1975, has been waived.
9 Where there is no agreement to abide by these protocols, and the Government agency proposes to conduct a statistical survey from which it intends to publish statistical output, then a survey proposal must be submitted to Statistics New Zealand for Ministerial approval under the Statistics Act, unless there is a waiver from the Minister of Statistics.
10 Where Statistics New Zealand is partly involved in a survey such as through providing survey design or other expertise, or in providing survey frames, it will only do so if the survey has been approved by the Minister of Statistics without a waiver, or an agency has agreed to abide by these protocols.
11 Where a survey is to produce statistics to be used in-house as an input into programme evaluation and monitoring, but not for publication (e.g. client satisfaction surveys and market research for improving operational activities), Ministerial approval is not necessary. Nor is approval necessary for surveys collecting 'physical' measurements (e.g. cadastral, topographic). However, Statistics New Zealand recommends that Government agencies follow the protocols for all surveys to ensure that the outputs are reliable, integrate with other surveys and that wherever possible some results be made publicly known and accessible. Such outcomes are consistent with the Policy Framework for Government Held Information.
12 The methodological aspects of these survey protocols are those which any reputable survey house should be using and as such should not put additional demands on either the agency or the survey taker.
Principle 2
A clear set of survey objectives and associated quality standards should be developed, along with a plan for conducting the many stages of a survey to a timetable, budget and quality standards
14 Once it is established that a survey is needed and justified in terms of costs and benefits, care at the planning stage is necessary to ensure a satisfactory result. A key to success is a clear statement of survey objectives, based on the main uses of the statistics. Imprecise objectives for a survey may lead to results which do not meet key requirements, and can lead to inefficiency in design. At its worst, the result can be data produced which is not fit for use and hence the money spent on producing the data and efforts by respondents is wasted and the credibility of official statistics impaired.
15 Objectives must be developed with the key users, with trade-offs made to suit the budget and what is feasible. Objectives should be expressed in terms of key variables, and the main classificatory variables, along with standards for accuracy, timeliness and presentation. A survey should be undertaken to provide data required by key users to a minimum standard to suit use, using best practice processes to achieve efficiency as well as effectiveness. All design and operation decisions should be made with reference to achieving the survey objectives to the standard required.
16 While standards for various survey processes will help achieve reliable and objective results, planning for the many stages of running a survey is necessary to ensure survey objectives, timeframes and budgets are met. Consistent inability to produce reliable results on time weakens credibility with users and respondents.
17 To a large extent, applying these protocols is a key ingredient to assuring quality of outputs fit for use. In particular, the use of sound statistical techniques and of standard frameworks, statistical units, questions and classifications are important elements for assuring quality (see Principles 4 and 5).
The "Guide for the Collection of Community Information" is available from Statistics New Zealand.
Principle:
A clear set of survey objectives and associated quality standards should be developed along with a plan for conducting the many stages of a survey to a timetable and budget
In more detail, the required elements are
- the main objectives and outputs of a collection should be specified along with quality standards for accuracy, timeliness and presentation
- assess against those standards all design and operation decisions for their impact on the resulting data quality
- develop a plan to cover the following stages:
1. Funding
2. consultation & content development
3. Timetables
4. Design
5. Testing
6. instructions & training
7. Collection
8. non-response follow up
9. Processing
10. Validation
11. Analysis
12. Release
- be clear from the outset on issues such as ownership and custodianship of collected data, as well as access and release polices, particularly with collections involving other parties
Principle:
Quality assurance processes should be followed to ensure data is fit for use
In more detail, the required elements are
- resources should be allocated to ensuring quality standards are achieved. Use of resources to achieve quality in excess of the standards is not good use of scarce resources.
- processes and roles and responsibilities should be clearly defined and documented
- documentation on methods used to produce data and any measures of quality should be available for users of the data to understand the data and judge quality of fit·
- all decisions made to depart from agreed design processes should be documented and users should be kept informed of any changes made to survey processes which impact on quality
- for regular collections, maintain a quality assurance and improvement program whereby processes are reviewed regularly to ensure adherence to design
Principle 3
Legislative obligations governing the collection of data, confidentiality, privacy and its release must be followed
18 Several Acts of Parliament and some government policies (such as that for information management) directly or indirectly govern how official statistics should be collected and released. These Acts and policies recognise the need to balance cost and intrusion necessary to collect information with obligations to protect confidentiality and privacy, and to make results generally available. In addition, in some areas ethics committees exist and where appropriate should be consulted. Following these protocols will ensure that these various obligations are met.
Principle:
Legislative obligations governing the collection of data, confidentiality, privacy and release must be followed
In more detail, the required elements are
- all official statistics are subject to Part 1 of the Statistics Act which covers prescribed matters, Ministerial approval, joint collections and periodic reviews
- the Statistics Act also covers confidentiality and release requirements for information collected under that Act
- in some health and medical areas, research ethics committees exist and should be consulted at an early stage
- the Privacy Act 1993 outlines 12 Information Privacy Principles which should be followed by all official collections
- access to official statistics is subject to the requirements of the Official Information Act 1982. The general principle of the Act is that information should be made available on request, and compelling reasons why not are unlikely to exist for official statistics once they have been compiled and the main results made available.
- information will be preserved or destroyed in accordance with the Archives Act 1957
- management and access to the data should be in accordance with the Policy Framework for Government Held Information
- the United Nations Fundamental Principles for Official Statistics summarises the international commitment to key principles.
19 Reliable official statistics depend on public co-operation and goodwill to provide accurate and timely information requested in surveys. Such co-operation and goodwill is maintained by protecting the confidentiality of information provided by respondents. Without such co-operation, response rates can be too low and threaten the accuracy of the end statistics, or legal enforcement may be required. Public concerns about inadequate protection of confidentiality by any government department can lessen public confidence generally for official statistics.
20 As well as the need to maintain public confidence that confidential information will be protected from release to others, there is the extra issue of privacy. All surveys represent a degree of intrusion, which must be justified on the basis of the need for public information on issues of importance.
21 The Statistics Act 1975 provides the legal authority for the collection of official statistics and sets out obligations to protect the confidentiality of information provided by persons and businesses. The use of information collected under the Act is restricted to statistical purposes, and the Act prohibits disclosure of identifiable information. To achieve the purposes of the Act, the Government Statistician makes office rules for avoiding the release of information identifiable to individuals in published aggregated statistics.
22 Applying the code of practice developed by Statistics New Zealand in all official statistics, regardless of whether the collections are governed by the Statistics Act or not, will underpin trust in the integrity of official statistical processes.
Principle 4
Sound statistical methodology should underpin the design of a survey
23 A sound statistical survey involves the collection of information from some or all units of a population using a frame and well-designed concepts, methods and procedures based on statistical theory for selection and estimation. This allows measurement of the levels of bias, variability and cost so that a survey design can be developed to achieve the stated objectives.
24 In practice, frame imperfections and compromises on methodology are inevitable and careful consideration needs to be given to their possible impact on the reliability of the final results. For example, higher proportions of Maori and Pacific Islander households are without a telephone compared with the general population. This means that sole use of a telephone-based frame for surveys of the New Zealand population may not be acceptable.
25 High levels of non-response may produce serious levels of bias in results if the non-responding population is markedly different to the responding population. A survey plan should include time and effort for follow-up of tardy respondents and sound procedures for estimating for the non-responding population.
26 Sampling methods used for the production of official statistics are usually probability based. In a probability-based sample, properly carried out, there is no risk of bias from subjective judgements in sample selection. The validity of the survey results from a probability-based sample does not, in principle, depend on any model. A model may suit some uses of the data but not others, and official statistics need to be able to be used confidently in a range of uses. Furthermore, valid sampling errors can be calculated from the design. These are strong points in favour of probability-based sampling.
27 Occasions arise, for reasons such as cost or because of difficulty in getting respondent co-operation, when purposive or quota methods of sample selection may be considered. Such methods always have a speculative element - a sample taken by such methods will only yield conclusions about the target population if certain model assumptions are valid. (This holds even if there is good data available about the target population to set the quotas.) In selecting a sample by quotas, there is the possibility of bias arising from subjective judgements. For non-probability-based samples, sampling errors have no validity, although they can be calculated on the assumption of randomness. In summary, very good reasons are needed to justify the use of non-probability-based sampling in official statistics. The case for such sampling is strengthened if it is shown that the major users accept the model assumptions which underlie the purposive or quota design.
28 The survey design should be implemented through procedural rules (for sample selection, interviewing, non-response follow-up etc). These rules should not be so complicated that they are not able to be carried out in practice with resulting introduction of bias.
29 The methodology needs to be adequately documented so that users of the data can understand its limitations. It is sound practice to have an independent review by a technical expert.
Principle:
Sound statistical methodology should underpin the design of a survey
In more detail, the required elements are
- the population of interest should be defined, along with any subgroups of interest
- a survey frame used which covers all units in the population of interest once and only once, with each unit clearly distinguishable from other units. Ideally, each unit has information on significant characteristics such as size and type which are related to the key data items of each and can be used to improve sample efficiency.
- scope and coverage of the collection defined, along with any process for extrapolating to population of interest
- the method of collection, and sample size & design determined with regard to the accuracy required for key data items, the variability of the attributes being measured, the ability of respondents to provide the information and costs.
- a method of selection which gives each unit a measurable chance of selection so that generalisations can be made about the whole population and estimates made of sample error. This requires randomisation of selections which reduces the chance of a non-representative sample. Any decision to use a non-random method of selection should be made with full regard to the potential for bias in the results.
- an efficient method of estimation based on statistical theory which takes account of information already known about the population, the probability of selection of each sample unit, non-response and missing values
- balance is achieved between sample error and non-sample error by directing resources available within the budget where they have the greatest impact on the quality of the results
Principle 5
Standard frameworks, questions and classifications should be used to allow integration of the data with data from other sources and to minimise development costs
30 The use of statistical frameworks for presenting statistics and of standards for statistical units, data definitions, questions and classifications permits the integration of data over time and across different collections, thereby increasing the value of information from any single collection. Many standards are based on international recommendations, which allow for international comparisons. Using standards can also reduce the resources required for developing and conducting a collection.
31 Statistics New Zealand has developed standards for many aspects of statistical collections, usually based on international standards where they exist modified to suit special needs. Standards usually exist for the following aspects of statistical collections:-
- classifications
- codefiles
- concordances
- statistical units
- definitions
- questions/question modules
- frameworks
32 Classifications and associated coding rules and codefiles are one of the cornerstones of reliable and comparable official statistics. They allow accurate and consistent arrangement of data according to common properties for further manipulation, integration and presentation of results. Statistics New Zealand has developed a comprehensive range of classifications, and conducts regular reviews to achieve a balance between reflecting contemporary circumstances and consistency over time. Concordances are available which allow comparisons between different versions of a classification and between common non-standard classifications. New Zealand classifications are either integrated or closely aligned with international standards, and in some cases harmonised with the equivalent Australian classification.
33 Statistical units are the units of observation ie the basic entities about which data are recorded and classified, and then aggregated and integrated to provide official statistics. Examples of statistical units include businesses, farms, motor vehicles, building sites, persons, households and families.
34 Questions and data definitions determine what data are collected and then classified for aggregation and analysis. They should be determined by the objectives for a survey. Again, consistency in questions and definitions across collections allow for integration. Statistics New Zealand is developing standards for questions, with variations to suit the method of collection.
35 Statistical frameworks exist for integrating and presenting statistics in many fields. The most important framework for economic statistics is the System of National Accounts (SNA) which is used to summarise economic activity and its various elements. The use of standard questions, definitions, statistical units, classification etc for producing statistics presented in frameworks greatly reduces the effort in integration and reconciliation required when bringing data together.
36 Documentation on standards for use in collections can be obtained from Statistics New Zealand by contacting Classifications and Standards Section on 03 374 8884, and will be available from the Statistics New Zealand Internet WWW site at http://www.stats.govt.nz
Principle 7
The reporting load on respondents should be kept to the minimum practicable
41 The Statistics Act 1975 outlines a number of responsibilities of the Government Statistician in relation to the demands placed on respondents. These include a requirement to consult with users to ensure that official statistics are both relevant and meet their broad needs. Any significant changes to a survey or any new survey must also be approved by the Minister with appropriate justification being provided, unless a waiver applies such as for agencies adhering to these protocols. The Act also requires the Statistician to co-ordinate the wider statistical needs of the government sector to ensure that any such collections do not duplicate other collections and that sound statistical practices are used.
42 Statistics New Zealand has a number of policies and practices in place for managing relationships with respondents with the aim of keeping reporting load to the minimum practicable and maintaining co-operation for ensuring the quality of collections. These are summarised below and should be applied across all official statistics to manage the load placed on respondents and to continue to obtain co-operation with collections.
Principle:
The reporting load on respondents should be kept to the minimum practicable
In more detail, the required elements are
- the users' needs for the data justify the collection, its frequency and the range of data items involved
- data suitable for users' needs are not already available from some other source, particularly administrative records
- the required data can be produced with less respondent load by modifying an existing collection rather than instituting a new collection
- the new data requirement of users demonstrates a shift in their needs or priorities such that an existing data collection can be eliminated or reduced in size or content
- sampling techniques can be used rather than a complete enumeration
- statistical techniques or "synthetic" (indirect or pro rating) estimation techniques can be used to produce estimates at a finer level of detail than standard estimation procedures allow or to produce estimates of data items not directly collected
- size cut-offs, or other coverage restrictions can be used, in order to exclude from collections those units which in total do not have a significant effect on the important statistical aggregates produced by the collection, or which can be adequately accounted for in some other way
- the full range of data can be collected from a sample of units and a more limited range of data from a larger sample or all the remaining units (i.e. short form/long form approach)
- selection techniques can be used which minimise overlap between collections, and in periodic collections respondents rotated out after a specified period of inclusion.
Principle:
All collections should be designed so that reporting units can provide information easily
In more detail, the required elements are
- the statistical units about which requested information refers, e.g. person, family, business establishment, building site should be clearly defined
- questions should be framed in such a way that the data requirements can be readily understood and answered - see standards for form design
- structured interviews or tailored forms should be used, wherever appropriate, in order to reduce the number of questions that each respondent has to answer
- a collection method used which suit the respondent. In particular, data should be collected in electronic form if it is already available in that form from respondent's records.
- query action arising from editing should minimise the number of follow-up contacts with respondents. Computer-based estimation of missing data should be considered.
- respondents should be informed of the purpose of a survey and the main uses of the data collected.
- documentation on a survey should be freely available to respondents and they should have ready access to staff for enquiries.
43 In addition, for surveys of businesses some additional principles and processes should apply:-
Principle:
Additional considerations apply for reducing the load placed on businesses
In more detail, the required elements are
- respondents should be informed in advance of the details and frequency of the data to be requested so that they can implement any changes necessary to their record keeping systems, and be provided with assistance to do so
- where possible, the requests for information made of businesses by different collections of the same respondent should be co-ordinated
- every effort should be made to frame data requests in terms of the kinds of records normally kept by businesses in that industry or type of activity
- the timing of the conduct of, and reference period for, a collection should take into account the likely timing of the availability of the required data
- careful estimates and/or rounded figures and reporting of data presented in ranges may be accepted
- standard data item definitions and classifications should be used, but because these may impact adversely on respondent load, judgement will need to be exercised about how the standards are applied
- registers or lists of statistical units used as a frame should be up to date so that sample sizes can be kept to a minimum and only in scope businesses included in the sample. Accurate information should also be maintained on the structure of businesses and contact information.
Principle 8
In analysing and reporting the results of a collection, objectivity and professionalism must be maintained and the data impartially presented in ways which are easy to understand
44 Consistent with the Statistics Act, the results of a survey should be made publicly known and available. This is best done by having the main results analysed and reported as statistical tables along with graphs and commentary to facilitate understanding and to get the main findings widely known. In analysing and reporting the findings of a collection, it is important that objectivity and professionalism is maintained. Information on how the data was compiled and analysed, qualifications and known limitations should accompany the findings. Also, in presenting statistics in tables and graphs care must be taken to present the data in ways which make it easy for the results to be understood and to avoid misleading or incorrect conclusions.
45 Statistics are enhanced when comparisons are made over time to determine trends, or across subgroups in the population to show differences. In doing such comparisons it is important that effects which are the result of seasonal or special effects, or structural differences in the groups being compared are allowed for (e.g. different age and sex structure which might be a significant determinant of say death rates).
46 Of particular interest are time series which are a statistical record of a particular social or economic activity. The data is measured at regular intervals over a period of time. The data is usually recorded on a monthly or quarterly basis, though some data may be recorded weekly. Time series are collected on this basis to assist understanding of the current situation, enabling the most recent data observations to be placed in a meaningful historical perspective. Users of the data may then determine if levels or the direction of movements suggest improvement or deterioration of the current situation. Guidelines for presenting time series data are being prepared by Statistics New Zealand.
Principle:
In analysing and reporting the results of a collection, objectivity and professionalism must be maintained and the data impartially presented in ways which are easy to understand
In more detail, the required elements are
- the main results should be analysed and reported as statistical tables along with graphs and commentary to facilitate understanding
- analysis and reporting must be objective and professional
- information should be available on how the data was compiled and analysed
- results should be validated against other information where practicable
- tables and graphs should follow standards so that they are easy to read and do not mislead
- comparisons should be made over time or across subgroups of the population to show differences
- effects which are the result of seasonal or population differences should be allowed for in any comparisons
47 Research has been undertaken over the years on how best to present data in tables and graphs so that the information is easy to understand and misinterpretations minimised. Guidelines for presenting data in graphs are available from Statistics New Zealand. Guidelines for presenting data in tables are being prepared.
Principle 9
The main results of a collection should be easily accessible and equal opportunity of access is enjoyed by all users
48 Exclusive or privileged access to official statistics can provide an advantage to an individual or group. In particular, for key economic statistics this advantage can be used for financial gain. Similarly, denial of access to individuals or groups can mean disadvantage to them in many ways, including uninformed input to the political process of representation and of public administration. It is for these reasons that the Statistics Act requires the results of surveys to be published.
49 To achieve impartiality in the release of official statistics, the main findings of statistical collections and analyses should be made widely known, be easily accessible and equal opportunity in access enjoyed by all users of the statistics. Releasing official statistics should be separate from the advocacy of policies. Furthermore, to be useful and to maintain public confidence in them, official statistics need to be timely. Guidelines for Writing About Statistics for the Community are available from Statistics New Zealand.
Principle:
The main results of a collection should be easily accessible and equal opportunity of access is enjoyed by all users
In more detail, the required elements are
- release the data in a manner that promotes equality of access by users
- disseminate results as soon as practicable after they have been compiled
- announce release dates for key statistics in advance, so that it is known that the timing of release is not determined by the results
- maintain strict security in the preparation and release of key results because of the potential for financial, political or other gain.
- present statistics clearly , without advocacy, alongside commentary and analysis to enable wide understanding
- promote widespread access to key findings by distribution through major public information service providers
- make catalogues and directories available so that potential users know what statistics are available
- protect confidentiality of individual respondents in all releases
- provide information on methods, practices and quality to users
- notify users of errors in release as soon as they are known
- archive unit records for possible future use
Principle:
Care should be taken when releasing data subject to error
In more detail, the required elements are
- data with serious bias, which may be misleading to users, should not be published.
- estimates for individual cells of a published table should not be suppressed solely because they are subject to large sampling errors, provided users are given adequate caution of the lack of reliability of the data.
- tables that are not published at a given level of detail, because most of the estimates would be highly unreliable in terms of sampling errors, need to be released to users with appropriate indications of the sampling errors. Greater caution should be exercised in releasing data that are subject to serious biases when the data have not been published.
all publications and releases should include a statement about data quality and methodology. In most cases this will be guidance to where comprehensive information can be obtained.
51 There are often trade-offs that need to be made when making decisions about quality, what to publish and when. Usually the trade-offs are between accuracy, timeliness and cost, but can also be between using standards to make outputs more widely usable and tailoring outputs to meet specific needs of key users.
52 The classic trade-off is between timeliness and accuracy. Users understandably want statistics as soon as possible, particularly if the statistics are needed to monitor time-sensitive policy issues. However, for the statistics to be useful for decision making, they need to be accurate, and accurate statistics take time to compile. The traditional approach is to release broad detail statistics first, followed by a series of more detailed releases based on more complete responses and analyses.
53 Doing this, however, can result in revisions if later releases are based on more complete responses. Revisions which do not significantly improve the quality of decision making based on the data can be more of a hindrance than a quality improvement. Furthermore, regular unexpected revisions which do not significantly improve the quality of data can result in loss of confidence in the statistics and the producer. However, errors in published data which can mislead should be corrected as soon as possible.
Principle:
The release of revisions to data should balance the need for users to have the best estimates with the uncertainty created by frequent revisions
In more detail, the required elements are
- The publication of revisions needs to follow a predictable pattern, otherwise users will be uncertain of the status of the statistics they are using.
- The unexpected release of a revision will cause major dissatisfaction. Users should have a statement of our revision experience, as a guide to likely future revisions.
- Regardless of the causes of the revision, the over-riding consideration should be to ensure that consistent time series are maintained through adequate backdating. Breaks in time series significantly reduce their usefulness.
- Consistent revisions policy should be applied across macro-economic statistics, particularly those that contribute to consolidated series such as national accounts and balance of payments.
- Revision should only be generally introduced when a substantial changes has occurred.
- All revisions should be accompanied by documentation which adequately explains their nature and provides good analysis of the differences between the series. This may include a parallel series of tables which clearly show the differences.
- Errors should be notified and corrected as soon as discovered.
Principle 10
Be open about methods used and documentation of methods and quality measures should be easily available to users to allow them to determine fit for use.
54 From the design to the dissemination stage of a statistical collection there are many ways in which errors can be introduced into the results. Some errors, resulting from the use of sampling, are random and their magnitude measurable. Other errors of a non-sample nature can introduce bias into the results.
55 Care needs to be taken when releasing data with high errors. For all releases, information on methods used and on known sources of errors should be made available so that users can determine if data is fit for use. This information can be included with the results or available as accompanying documentation. Guidelines for Publishing Data Subject to Error are available from Statistics New Zealand.
56 Information on the following aspects of a collection should be readily available to users:-
Documentation: data quality
Aspects to be covered
- coverage of frame
- response rates
- sampling error
- imputation effect
- other major sources of error
- comparability over time
- comparability with other data sources
- benchmarking and revisions
Documentation: methodology
Aspects to be covered
- objectives of the survey
- target population and any differences between this and the survey frame used
- questionnaire used
- data definitions and classifications
- sample design, method of selection and estimation procedures
- collection method
- method and extent of follow-up of non-respondents
- processing method (coding, data capture)
- editing and imputation
- impact of any confidentiality protection procedures
- evaluation studies
57 Statistics New Zealand provides information on data sources and methods in several ways. For key outputs, regular publications and manuals are available, with references to them accompanying any data release. Of increasing value is the Survey and Output Information database available on the Statistics New Zealand WWW site which contains information on the population covered by a survey, statistical units, methodology used for selection and estimation, analysis, sources of error, output variables and classifications, and key outputs. This system is available for agencies to use to provide documentation on their surveys.
Statistics New Zealand
August 1998
APPENDIX - DEFINITION OF OFFICIAL STATISTICS
Official statistics are statistics produced by government agencies to
- shed light on economic and social conditions
- develop, implement and monitor policies
- inform decision making, debate and discussion both within government and the wider community.
Government and its adminstrative arms need official statistics for policy development, implementation and evaluation. The public at large have similar information needs in order to evaluate government policy, to ensure public accountability, and to be adequately informed about social and economic conditions.
The Statistics Act 1975 states
"Official statistics shall be collected to provide information required by the Executive Government of New Zealand, Government Departments, local authorities, and businesses for the purpose of making policy decisions, and to facilitate the appreciation of economic, social, demographic, and other matters of interest to the said Government, Government Departments, local authorities, businesses, and to the general public."
It defines "Official statistics" as statistics derived by Government Departments from:
(a) Statistical surveys as defined below; and
(b) Administrative and registration records and other forms and papers the statistical analyses of which are published regularly, or are planned to be published regularly, or could reasonably be published regularly.
"Statistical survey" means a survey of undertakings, or of the public of New Zealand, whereby information is collected from all persons in a field of inquiry or from a sample thereof, by a Government Department pursuant to the authority of this Act or any other Act, or without specific provision in any Act, wholly or primarily for the purpose of processing and summarising by appropriate statistical procedures and publishing the results of the survey in some statistical form.
Surveys conducted to produce statistics to be used in-house by a Department as an input into improving procedures, but not for publication (eg client satisfaction surveys, market research for improving operational activities) or for collection physical measurements (eg cadastral, topographic) are not considered to be official statistics. However, such information is covered by other legislation such as the Privacy Act and the Policy Framework for Government Held Information.
APPENDIX - LIST OF ASSOCIATED DOCUMENTS
Government Legislation and Policies
- Statistics Act
- Privacy Act
- Official Information Act
- Archives Act
- Policy Framework for Government Held Information
- United Nations Fundamental Principles for Official Statistics
SNZ Policies
- Confidentiality Protection
- Microdata Access Protocols
- Population Census Record Matching Policy
- Respondent Management
- Release of Statistics
- Writing about Statistics for the Community
Standards and Guidelines
- Classifications
- Frameworks
- Objective setting
- Forms
- Tables
- Graphs
- Presenting time series
- Revisions
- Publishing data subject to error
Reference material
- A Guide to Good Survey Design
- Guide for the Collection of Community Information
- Characteristics of an Effective Statistical System by Ivan Fellegi, Statistical Journal of the United Nations ECE 13 (1996) pp 89-117
- United Kingdom Government Statistical Service: Code of Practice
- Public Confidence in the Integrity and Validity of Official Statistics by J Hibbert, Journal of Royal Statistical Society, A (1990) 153 Part 2 pp 123 - 150
- Handbook of Official Statistics in ECE Member Countries, United Nations
- Australian Bureau of Statistics - Policy Manual
SUPPLIER ACCREDITATION ASSESSMENT
- access to expertise and knowledge
-- subject matter experts involved in collection design and analysis
-- statistical methodologists used for sample design and estimation
-- links to relevant professional and international organisations
- respondent and public trust
-- sound record with handling confidential information
-- privacy principles followed
-- openness about processes
-- regular consultation with stakeholder groups and custodians of privacy
-- confidence from special target populations
- absence/management of conflicts of interest
-- separation of regulatory and statistical functions
-- independence of statistical processes from political intervention
- performance management
-- techniques, processes and tools used are best practice
-- risks managed
-- peer reviews of survey designs and analyses regularly arranged
-- open to innovation and adaptation
-- protocols known to staff and followed
-- partnerships exist where necessary to complement skills and knowledge
- objectivity and impartiality
-- impartial selection of survey objectives, content and questions
-- main findings of surveys published
-- objective and professional analysis of data
-- impartial presentation of survey results and analysis
-- equality of access to data
-- content and timing of releases not influenced by results