ABS Access to Microdata Policy
Objectives
The objective of the disclosure of unidentifiable individual statistical records (microdata), which is allowed under clause 7 of the Statistics Determination, is to assist and encourage informed decision making through enabling wider access to ABS data for social and economic research and analysis.
Legislation
2 The release of information in the form of individual statistical records is authorised by clause 7 of the Statistics Determination made under subsection 13(1) of the Census and Statistics Act 1905 (CSA).
3 The following legislative requirements apply to the release of microdata files:
a) There is no obligation on the Statistician to release microdata files. The Determination is an enabling provision only.
b) All directly identifying information must be removed prior to release. This includes name, address and any other information (such as register numbers issued by other bodies) that identifies particular persons or organisations.
c) The information may be released only in a manner that is not likely to enable the identification of the particular person or organisation to which it relates.
d) The information may be released only after a relevant written undertaking acknowledging the conditions of release has been signed by the client, and after the Statistician has provided written approval. Persons who are authorised to sign an undertaking are specified in clause 7(2) (a), (b) and (c) of the Determination. Terms and conditions which must be included in the undertaking are specified in clause 7(2) (d), (e) and (f) of the Determination. Additional conditions which might be included in the undertaking if the Statistician considers it necessary are shown in clause 7(2)(g) and (h).
An undertaking given on behalf of an official body or organisation covers all members of that organisation. In the case of universities, this includes both staff and students.
e) It is a condition of release that the recipient of microdata files must not release the information, or any part of the information, to another person or organisation.
f) The Statistician can only authorise the release of unidentifiable information for statistical purposes, and then only after an undertaking has been received to this effect. Some judgement is involved in determining whether the intended use of a microdata file is for statistical purposes or not. In practice, as long as the data are not likely to enable identification, it is highly unlikely that the information could be used successfully for anything but statistical purposes.
The selling of analyses and/or aggregate data by the recipient of a microdata file is not, in itself, contrary to the requirement that the file be used for statistical purposes only, nor is it contrary to the condition that the recipient must not release any part of the information contained in the microdata file to another person or organisation.
g) It is a condition of release that the recipient of microdata files must not attempt to identify particular persons or organisations.
Delegation
4 Every release of microdata files, including microdata files for which the Statistician has previously approved release to another client, must be approved by the Statistician; the Statistician has not delegated this authority.
ABS Rules
5 In addition to the legislative requirements, the following ABS rules must be complied with regarding the release of microdata files:
a) There will be no release of microdata files without close examination and assessment of the risk of disclosure associated with such a release. The Microdata Review Panel (MDRP) comprising the First Assistant Statistician, Methodology Division (Chair) and two Assistant Statisticians, at least one of whom has responsibility for Household Surveys, has been established for this purpose.
b) There will be no release of microdata files where the intended use includes the matching of ABS data against data holdings from other sources. To prevent such matching, microdata files should not contain data items which could be matched with other data holdings that are generally available. The Statistician may approve restricted access under supervised conditions to the file in a "data laboratory" on ABS premises where identification of a person or organisation is a possibility through matching with another non-ABS file or data store.
Where external data sources are not generally available and are not available to a recipient, their existence need not be taken into account when determining what data items should be retained on the ABS records. However, the recipient should be required to undertake not to attempt to seek access to these data sources.
c) Microdata files shall not be released before the first publication from the relevant collection has been released.
d) The prospective client demand for microdata files must justify the ABS resources involved in preparing these files. In the case of social surveys where a microdata file will generally be a standard output, the resource allocation for the survey should include provision for the preparatory work.
e) Clients must be charged for microdata files in accordance with ABS Pricing Policy.
f) Secondary distribution arrangements where disclosure to a third party end-user is arranged at the discretion of an intermediary are not permitted.
g) The ABS will include in its Annual Report a table of releases of unidentified information under clause 7 of the Statistics Determination approved during the previous year.
ABS Policy
6 The following policies apply to the release of unidentifiable microdata files: Any suggested departure from policy must be strongly supported by a proposal to the Statistician.
a) The ABS will release microdata files from most social surveys unless there is an expectation of no demand from major clients or it is not possible to produce a useful file because of confidentiality restrictions.
Microdata files will be released from other collections only following a rigorous assessment of client demand and intended use of data.
b) Because the confidentiality provisions of the CSA are not enforceable outside Australia, releases under clause 7 of the Statistics Determination will not normally be made to clients overseas.
c) Geographic codes on microdata files released under the Determination generally should not be below the level of the Capital City Statistical Division/Rest of State. Where there is justified user demand for additional geographic detail, a departure from this policy may be permitted if the MDRP is satisfied that the confidentiality of the file is preserved.
d) In order to reduce costs, whenever possible a standard file or files (rather than tailor-made files) should be made with the requirements of several clients in mind. This may require some liaison with prospective clients to assess their likely needs before proceeding with the design of a file.
e) The ABS can release a microdata file to an administrative body that provides limited access to approved users (e.g. a group of departments and authorities within a sector of government). Access to any user can only be provided after the user has given a signed undertaking to the Statistician and the Statistician has approved the access in writing. The administering body must sign an undertaking that includes a condition that it will permit access only on these grounds.
7 The CSA also enables unit record data to be returned to the person/organisation from whom/which the ABS obtained the information, even if the information can be identified. The recipient of the information can legally disseminate the information to other persons or organisations. On some occasions, the recipient may further disseminate these microdata files. This "return to source" provision is used for some administrative by-product collections (eg cause of death) to provide detailed microdata that could not otherwise be disclosed.
Procedures
The following procedures describe the steps to be followed to obtain the Statistician's approval for the release of unidentified information (microdata).
Summary of procedures
2 The following is a summary of the procedures to be followed for the use of clause 7:
a) Assessment of demand
Pro active approach to identification of client demands for a microdata file. (see paras 3 - 5)
b) De-identifying the file
Aspects to be considered in determining whether information is likely to be identifiable. (see paras 6 - 13)
c) Preparation of proposals
Responsibilities of SMAs, MDRP, Secretariat Section, DAS/FAS/Regional Directors towards seeking approval for clause 7 disclosures (see paras 14 - 18).
d) Seeking the Statistician's approval
Seeking the approval of the Statistician for the disclosure of unidentified information using clause 7 of the Determination is a three step process undertaken in the following order:
-- approval in principle (see paras 20 - 24)
-- obtaining signed undertakings from the client (see paras 25 - 26)
-- formal approval (see paras 27 - 29)
e) Breach of conditions (see paras 30 - 31)
f) Pro forma documents (see para 32)
3 The likely client demand for access to microdata files from a particular collection should be assessed at the same time as demands for other types of statistical output from that collection are being assessed. This will assist both the ABS and clients to plan ahead, although in any preliminary discussions with clients it should be made clear that the availability of microdata files is at the Statistician's discretion. However, cases will arise where demand is not expressed until after output plans have been developed. In those cases preparation of microdata files will have to be assessed as supplementary action superimposed on other plans.
4 Any plan to produce a microdata file should take account of the time, resources and expertise required to negotiate with clients, design the file, develop any appropriate access software and prepare appropriate documentation to accompany the file. It is important that these activities do not disrupt the approved ABS forward work program. The relevant Program Head is responsible for assigning an appropriate priority to the preparation of microdata files, having regard to the overall release strategy of the program.
5 It is not appropriate for the ABS to investigate in detail the applications proposed by each potential client, although an understanding of likely uses of the microdata file is necessary to help make choices in tradeoffs for de-identifying records. The ABS has a responsibility to caution, at the beginning of discussions with potential clients, against use of the information in a statistically inappropriate manner, e.g. analysis of small area detail or infrequently occurring characteristics using sample files.
6 Clause 7(1)(b) of the Statistics Determination provides that information in the form of individual statistical records may be disclosed in a manner that is not likely to enable the identification of the particular person or organisation to which it relates. The following aspects should be considered in determining whether information is likely to be identifiable or not.
Level of detail in each record
7 The greater the number of characteristics, the higher will be the probability of identification. This will be further increased if the amount of detail provided for particular characteristics is high. Thus, the number of characteristics shown and the detail for each characteristic should be assessed carefully to see whether they enable identification.
Level of geographic detail
8 The probability of identification depends very much on the level of geographic detail identified. Where there are justified user demands for additional geographic detail, a departure from the policy that geographic codes on unit records released under the Determination should not be below the level of the Capital City Statistical Division/Rest of State may be permissible if identification is not likely. In such cases, regions down to 250,000 people should be generally acceptable, although if a large number of variables was being released, problems could arise. If data was restricted to only a few variables, regions down to 150,000 could be acceptable, however this depends heavily on the degree of detail available on other data items eg occupation and industry coding.
Order of records
9 If the records are held by ABS in an order which might assist identification, particularly with respect to geographic characteristics, the records should be randomly sorted.
Sample of records
10 A further means of reducing the probability of identification for some data bases is to provide a sample of records rather than all records. This will introduce a further element of uncertainty as a user cannot prove that a unique record in a sample is also unique in the whole population without having some further information from a non-ABS source.
Other methods
11 If identification is still a concern, other methods that could be considered without unduly reducing the amount of statistical information provided include:
a) not releasing values as collected, but showing them as classes
b) randomly perturbing the value by some small number
c) swapping variable information between records
d) dropping individual records from the file
e) substituting imputed for collected information
12 Further information regarding de-identification of data is available on the MDRP database .
Advice to users
13 Some of the methods for reducing the likelihood of identification may affect the usefulness of the information for some applications, particularly those that involve statistical analysis. It is therefore important that clients are alerted that procedures to ensure confidentiality have been followed and where possible, clients are advised of the likely impact this might have on their analysis. It should be noted, however, that the details of the methods used to de-identify records should not be provided to clients where this will assist them in undoing the effect of the de-identification.
14 Proposals for release of unidentified information from national collections should be prepared by the relevant subject matter area (SMA) in Central Office (CO). Regional Directors may prepare proposals for State-specific collections.
A range of information is required to enable the Statistician to consider whether approval should be given to the release of a microdata file.
15 As well as preparing the microdata file, the responsible SMA is required to:
a) consult with Statistical Services Branch for advice on any special measures that may be needed to make the records unidentifiable and for an assessment of the likelihood of identification
b) seek the assessment of the Microdata Review Panel (MDRP) of the methods proposed to de-identify the unit records
c) in consultation with Information Services Division, determine the price of the proposed microdata file.
d) for non-standard microdata files, obtain written assurances from clients that they will agree to ABS terms and conditions, and to meet the appropriate charges should the Statistician approve release
e) prepare documentation outlining relevant sample details (sampling fractions, stratification plan, etc), data definitions and other statistical information relevant to using the information, as well as file specifications, conditions of sale, and any relevant information on data quality to accompany the microdata file.
16 The Microdata Review Panel is required to:
a) assess all proposals for the release of microdata prior to any submission being made to the Statistician
b) prepare a written report signed by the Chair of the panel, covering its assessment of the methods proposed to de-identify the data (individual unit records). The report should point out any special conditions that should apply to the release of the microdata file to ensure that it is released in a manner that is not likely to enable the identification of the particular person or organisation to which it relates.
17 Secretariat Section is responsible for:
a) advising the SMA on rules, policies and procedures
b) clearing the draft written undertaking to be signed by the clients.
18 The DAS of the Group/Office preparing the proposal is responsible for:
a) verifying that resources are available for the activities involved in preparing the microdata file
b) ensuring that the MDRP has endorsed the proposal for release, and that the file is consistent with the requirements of the Statistics Determination and any special conditions which may have been imposed
c) signing off the documentation seeking the Statistician's approval for the release. (All requests for release of microdata files, including requests for subsequent release of a file, must be signed off by the relevant DAS).
Seeking the Statistician's approval
19 Seeking the approval of the Statistician for the release of unidentified information using clause 7 of the Determination is a three step process undertaken in the following order:
a) approval in principle
b) obtaining signed undertakings from the client
c) formal approval
20 The approval in principle stage requires a minute to the Statistician from the Branch Head/Regional Director of the SMA preparing the proposal. This minute, which should be signed off by the relevant DAS (and if appropriate, the relevant FAS), should be sent through the Assistant Statistician, Policy Secretariat Branch. The minute to the Statistician for approval in principle should include the following:
a) the nature of the data to be released
b) details of known and anticipated users and usage
c) method/s employed to reduce the probability of identification
d) ABS resource costs
e) the assessment of the methods proposed to de-identify the unit records by the MDRP, signed by the chair of that panel;
f) any special circumstances affecting user demands or the likelihood of identification
g) a statement that the proposal meets all legislative requirements
h) a statement that the proposal conforms with ABS rules and policies, or if not, a statement as to why departure from ABS rules or policy is recommended
i) draft undertaking/s for subsequent signature by the client/s.
21 Secretariat will inform the SMA of the Statistician's decision. Approval in principle by the Statistician does not imply automatic formal approval.
22 If the Statistician has not agreed in principle to release of unidentified information, the SMA should advise the client concerned. Alternative services which can be offered, when appropriate, include:
a) development of PC based metadata as a standard and early product to allow users to design and specify tables
b) provision of a test file to allow users to develop models and analyses to be run against a full file by ABS
23 If the Statistician has requested further information before making a decision regarding in principle approval, the request for in principle approval, together with the additional information requested should be resubmitted to the Statistician through the Assistant Statistician, Policy Secretariat Branch.
24 Following approval in principle by the Statistician, the likely availability of the file can be advised to known interested users by publication of an information paper, or by other suitable means. However, the Statistician must specifically authorise the disclosure of the file to each user and each user must provide an appropriate written undertaking before any file can be disclosed. Approval in principle needs to be obtained only once for each microdata file.
25 If the Statistician has given in principle approval to a request, arrangements should be made by the SMA for undertaking/s to be signed. Details of who can sign such undertaking/s are in clause 7(2) (a) - (c) of the Statistics Determination. In the case of information to be disclosed to an official body or an organisation other than an official body, the undertaking should be signed either by the head of the client department or organisation, or by a person who has sufficient status to sign on behalf of the department or organisation, generally a person at a senior executive level. The SMA should also obtain a written agreement from the client to meet any ABS charges and any other conditions imposed by the ABS.
26 The legal significance of the undertaking and the penalties for breaching it should be explained clearly in a covering letter to the undertakings. (Failure to comply with an undertaking is an indictable offence under subsection 19(2) of the CSA, punishable on conviction by a fine not exceeding $5,000 or imprisonment for a period not exceeding 2 years, or both.)
27 When signed undertaking/s have been obtained, the SMA should prepare, in consultation with Secretariat, a formal document for approval in writing by the Statistician. That document, together with the original proposal showing the Statistician's approval in principle, the endorsement of the MDRP and the signed undertakings, should be submitted for the formal approval of the Statistician through the Assistant Statistician, Policy Secretariat Branch.
28 Following the Statistician's formal approval, Secretariat will file the documents in the register of clause 7 approvals (which will form the basis of the table presented in the ABS Annual Report), and forward a copy of the signed approval in writing to the relevant SMA.
29 The SMA should ensure that the letter to the client advising of the Statistician's approval to the release of the unidentified information covers all of the rule and policy requirements relating to that release.
Breach of conditions
30 Any suspicion of a breach of an undertaking by a client is to be brought to the immediate attention of the DAS of the Group/Office sponsoring the request. The DAS should take any action necessary to investigate the possible breach and, if necessary, make a recommendation to the Statistician through the Assistant Statistician, Policy Secretariat Branch on action to be taken.
31 If it is thought necessary, the Statistician can make it a condition of release that the ABS may require a compliance audit concerning observance of the terms and conditions under which the information is disclosed.