Page History

18.1.         Chapter 18 describes the concept and structure of metadata (or data that define and describe other data and processes)

...

^[1] for use within the statistical framework for describing the international supply of services, as well as good metadata compilation practices. It underscores that metadata are relevant for the correct understanding of the content, coverage and limitations of the data, and should guide users on their correct interpretation. The chapter contains the following sections: a summary of good practices (section A); an overview of the basic concepts, definitions and role of the Statistical Data and Metadata Exchange (SDMX) (section B); an indicative list of metadata items (section C); metadata standards of international and regional organizations (section D); and country practices (section E).anchor

...

1. The present Guide recognizes that metadata (a) play a crucial role in the statistical production process, as they enable and facilitate sharing, querying, understanding and using data over the different stages of their collection, compilation and dissemination, and at their various levels of aggregation and (b) are indispensable for assessment of the quality of data, as their availability and wide dissemination constitute a basis for the correct interpretation of publicly available statistics and their effective use.

1. It is advised that compilers take into account that statistical metadata cover the following items: Guidelines on Integrated Economic Statistics, para. 5.91. statistical description, unit of reference, reference period, institutional mandate, confidentiality, release policy, frequency of dissemination, dissemination format, accessibility of documentation, quality management, relevance, data accuracy and reliability, timeliness, comparability, coherence, cost and response burden, data revision and statistical processing.

1. The use of standard terminology for metadata across the various statistical domains and use of the SDMX information model is also advised, as they will facilitate further integration of statistics, the standardized sharing of data and the international comparability of data.

1. It is further advised that compilers design their metadata systems in the most efficient way so that metadata items can be conveniently retrieved from the relevant databases, be used in the generation of the intermediate and final data sets or in the production of other metadata, and be updated and synchronized. In that context, compilers are encouraged to design and use a warehousing system for data and metadata to integrate the dissemination of data and metadata with the collection and processing components of the statistical production process.

1. With regard to compiling metadata, compilers are encouraged to follow standardized metadata concepts, make use of the metadata developed in related statistical domains and already being applied in their national statistical system, define layers of metadata, establish metadata registries, incorporate structural metadata items into the data processing as early as possible, establish clear links between data and metadata and compile reference metadata.

1. In the case of countries with less developed statistical systems, it is good practice to begin by setting up an exhaustive, consistent and detailed repository (possibly in the form of a metadata registry) with both structural and reference metadata, adopting, as much as possible, metadata concepts that are standardized across all statistical domains, both nationally and internationally. The present Guide strongly advises that the next immediate priority be to grant equal, easy, extensive and timely access to metadata to all user groups, including the general public.

1. The structural metadata items promoted by the present Guide are those defined within the framework of the Balance of Payments Data Structure Definition (BOP-DSD) defined by the Manual on Statistics of International Trade in Services (MSITS 2010) and the present Guide for FATS and for additional indicators on the international supply of services. Metadata items should cover both monetary and non-monetary (quantitative) data items whose compilation is encouraged by the present Guide.

1. The metadata standards of international organizations should be carefully considered by compilers, both to improve their metadata collection and compilation and ensure better compliance with their international and regional data and metadata reporting obligations.

1. It is good practice for any deviations from international standards, as well as the use of estimations and modelling to compile certain data series, to be clearly documented in metadata.

...

...

1. The Statistical Commission recommends the use of standard terminology for metadata across the various statistical domains to facilitate the international comparison of data. Guidelines on Integrated Economic Statistics, p. 40. The Commission is also increasingly encouraging countries to treat metadata compilation and dissemination as integral parts of the statistical process in any statistical domain, and promotes the standardization of the compilation and dissemination of metadata. The important role in this respect was played by the ECE publication Terminology on Statistical Metadata, Conference of European Statisticians Statistical Standards and Studies, No. 53, (United Nations publication, Sales No. E.00.II.E.21).
2. The way forward: metadata warehousing Statistical agencies have traditionally developed separate databases for each statistical output. While that practice may simplify development processes, it can hinder the successful integration of statistics, especially if there is no effort to standardize variable definitions, labels and formats. Use of a centralized data warehousing system for data and metadata can make creating, maintaining and accessing metadata more efficient and can contribute to the integration of economic statistics. The process is being facilitated as better information and communications technology tools become available. See chapter 21 for more information on the use of information and communications technology (ICT) in the statistical process.
3. With well-designed data warehouses, the dissemination of data and metadata becomes integrated with the collection and processing components of the statistical production process. A data warehouse should establish a simple and efficient process for accessing data to provide the following:

(a) Comprehensive metadata to facilitate understanding and analysis;
(b) Consistent and coherent long-term time series;
(c) Reliable information about the availability of data;
(d) Information about the availability of updated versions of published series;
(e) Contact details for the people who can provide more information about a statistical output.

1. The implementation of a more comprehensive metadata system is an important prerequisite for developing an integrated questionnaire in the statistical system. The metadata will eventually provide the necessary coherence among the various estimates and data collection tools involved in the production of statistical information. For sophisticated users, metadata are not only relevant for concepts related to units, variables and classifications, they are also relevant for the quality of data.

B.1 Role of the Statistical Data and Metadata Exchange

1. The SDMX project was developed by an international consortium Bank for International Settlements, Monetary and Economic Development,Guidelines for Reporting the BIS International Banking Statistics (2013), para. 5.123. for use in data and metadata management. The SDMX information model is applicable for much of the information stored and processed within statistical organizations and its use by such organizations is promoted by the Guidelines on Integrated Economic Statistics of the United Nations. Ibid.
2. The use of the standardized information management model is very important for compilers of statistics on the international supply of services, as various agencies participate in data collection and compilation at different stages of the statistical production process, and the establishment of a standardized data sharing among them results in additional efficiency.
3. The development of global DSD, which define the structure for the exchange of data (see section C), by the SDMX consortium and international organizations, enables the broader adoption of the SDMX standard for data collection, exchange and dissemination.

Box 18.1 Statistical Data and Metadata Exchange
The Statistical Data and Metadata Exchange (SDMX) is an international cooperative initiative aimed at developing standards and employing more efficient processes for the exchange and sharing of statistical data and metadata among international organizations and their member countries.

1. The rationale of SDMX is the standardization of statistical data and metadata access and exchange. With the ever increasing ease of use of the Internet, the electronic exchange and sharing of data are becoming easier, more frequent and important. This heightens the need for the development of a set of common standards for the exchange and sharing of statistical data and metadata, and for making processes more efficient. As statistical data exchange takes place continuously, the gains to be realized from adopting common standards are considerable, for both data providers and users.

...

1. Structural metadata contain the list of concepts and attributes of variables necessary to codify the reporting requirements of four international agencies, the European Central Bank (ECB), Eurostat, the International Monetary Fund (IMF) and the Organization for Economic Cooperation and Development (OECD), The four agencies are the European Central Bank, Eurostat, IMF and OECD. for data collection exercises of external sector statistics, including international trade in services. At the time of writing, there is not yet a data structure definition (DSD) available neither for FATS nor for non-monetary indicators of modes of supply. A FATS DSD would include specific dimensions such as FATS characteristics, e.g., number of enterprises, number of persons employed, etc. and resident economic activity vs. non-resident economic activity. Items specific to non-monetary indicators on modes of services supply would include direction of trips: inbound, outbound; and country of origin or destination. The Balance of Payments-Data Structure Definition (BOP-DSD) structure is based on the methodology defined in the IMF Balance of Payments and International Investment Position Manual, 6th edition (BPM6), MSITS 2010, and the OECD Benchmark Definition of Foreign Direct Investment, 4th edition (BD4). In order to code trade in services data, the Extended Balance of Payments Services Classification (EBOPS), including the complementary groupings, is included in the "international account item" dimension of the DSD.
2. The BOP-DSD, presented on the SDMX website, See http://sdmx.org/?page_id=1747. includes 16 concepts and 13 attributes (see box 18.2). Concepts are used to uniquely identify a time series and, when joined together, provide the series code or "time series keys," which are the unique identifiers for a time series. When defining a time series key using SDMX, a valid code must be assigned to each concept of the DSD. Attributes are used to further describe the data.
3. When coding detailed trade in services by partner country statistics, a number of BOP-DSD concepts are fixed, e.g., the reference and counterpart sectors are defined as the total economy (S1) when the data refer to total trade between related and unrelated parties. In order to eliminate the possibility of having multiple ways of coding the EBOPS 2010 complementary grouping "total services transactions between related entities", that item is not coded in the international accounts dimension of the DSD, but is coded in the annual international trade in services dataflow as follows (example on the credit side): A.N.#.%.S1.S1A.T.C.S._Z._Z._Z.$._T._X.N. However, trade in services between related parties can also be coded by using code S1A "affiliates", in the counterpart sector dimension, whereas S1B should be used for unaffiliated parties.

1. Other concepts of the DSD are not fixed, such as the "counterpart area", which is used to identify the territory of the non-resident entity of individual time series. The country code list in the counterpart area follows the International Organization for Standardization (ISO) classification and is a "cross-domain" code list. The codes used for various regional groupings were harmonized across international agencies that use the BOP-DSD, wherever possible.

...

1. The times series key (A.N.US.FR.S1.S1.T.C.SC._Z._Z._Z.USD._T._X.N) provides an example of codification for a trade in services series. That times series key stands for a series in annual frequency, "A", with no adjustment indicator, "N". The reference country is the United States, "US", the counterpart area is France, "FR", the reference sector is the total economy, "S1" and the counterpart sector is the total economy, "S1". The flows and stocks indicator dimension indicates a transaction, "T", the accounting entry dimension indicates that this is a credit entry, "C", the international accounts item specifies that the series relate to transport services, "SC", the functional category is not applicable, "Z", the instrument and assets classification is not applicable, "Z", and the maturity is not applicable, "Z". The unit of measure is the United States dollar, "USD", and the currency of denomination is set to "all currencies of denomination" ,_T. The valuation is unspecified, _X, and the compilation methodology is National, "N".

1. As noted above, attributes are used to qualify observations further, that is, they provide information on the "confidentiality" status or the "compiling organization". Attributes do not contribute to the identification of a time series, as that is already done by using the dimensions.

C.3Reference metadata

1. The following items are typically part of the reference metadata associated with statistics on the international supply of services:

(a) Legal framework and institutional arrangements: references to relevant laws and regulations, the role of all institutions involved in compilation and the description of the coordination of the dissemination of statistics and data-sharing agreements among those institutions, either distinctly or as part of broader statistics (e.g., BOP and other external sectors statistics);
(b) Underlying concepts and definitions: definition of residence, non-residence, residence of units, as applicable, definition of statistical value, scope of statistics on the international supply of services and their relationship to national accounts and international merchandise trade statistics, distinction from other international transactions and classification under relevant services item according to BPM6 and EBOPS 2010 and any deviations from international standards, if any, the ultimate controlling institutional unit (UCI) concept, the definition of a foreign affiliate, the definition of direct or indirect control and definitions of statistical and reporting units, etc.;
(c) Description of core data sources: ITRS, enterprise/establishment surveys, surveys of households or persons, administrative records, statistical models, partner country data or a combination of sources, including specific notes on services categories or activities for which particular data collection arrangements or a combination of sources are employed and comments on limitations of source data in terms of coverage, frequency, level of detail, reliability and availability, etc.;
(d) Description of data collection, data compilation methods and data-processing procedures, including the frequency of data collection, the description of specific procedures used for data collection, validation, editing and aggregation, etc., adjustments made to source data, such as imputations, misclassification, adjustments for non-response or under-coverage, adjustments to standard data processing procedures, such as coding, tabulation errors, etc. and indications of departures from international standards, if any;
(e) Estimation methods, such as descriptions of methods for estimating non-reported transactions or transactions falling below customs and/or ITRS thresholds (e.g., cost, insurance and freight (CIF)-free on board (FOB) adjustments for the transportation item);
(f) Dissemination policy, including release and revision schedules, an indication of the presentation format of data, the level of disaggregation and eventual commentaries accompanying the data, etc.;
(g) Additional explanations and footnotes concerning the data as required: explanatory notes on revisions, breaks in series, definitions of confidentiality flags, etc.;
(h) Quality reporting, including the publication of regular quality reports that use the quality dimensions in the Template for a Generic National Quality Assurance Framework (NQAF) See https://unstats.un.org/unsd/dnss/QualityNQAF/nqaf.aspx; and see chapter 19 for more information on the national quality assurance framework (NQAF) and quality reporting and management in general. and include definitions of such quality dimensions as timeliness, accessibility and comparability;
Confidentiality, including descriptions of confidentiality rules and indications of how much data is affected by such rules. See chapter 20 for more information on statistical confidentiality.

1. Compilation of metadata Metadata are compiled at all stages of the statistics production process. The present Guide encourages countries to use the following good practices, as applicable, in metadata compilation:

(a) Use standardized metadata concepts. In the same way as any data item, metadata items must also be clearly defined. Even though each statistical domain, including statistics of international trade in services, has its specific metadata items, it is good practice to use applicable standardized concepts that are relevant across statistical domains (e.g., by adopting cross-domain concepts from the SDMX framework or the OECD Glossary of Statistical Terms). The aim should be to promote the harmonization of statistical information and their related high-level metadata across various institutions and statistical domains, even if some specific metadata concepts are not applicable or are organized differently in different domains or institutions;
(b) When developing metadata for statistics compiled within the framework for describing the international supply of services, use the metadata developed in the related statistical domains and used in your country. Statistics of international trade in services is a relatively new statistical domain in many countries. However, it is very likely that the metadata policy is already in place in related statistical domains. Compilers are advised to carefully review and use such metadata;
(c) Define layers of metadata. It is good practice to compile metadata in layers of incremental detail and provide clear links between high-level and specific metadata concepts. Such a layered structure of metadata will allow data users and analysts to access necessary metadata items and to minimize the risk of misinterpretation of data content when, for example, compiling data from various data sources. It will also ensure the clear presentation of metadata to diverse groups of users;
(d) Establish metadata registries. A metadata registry is a central repository (usually a database itself) with information that allows for the linking of the detailed definitions (semantics) with the codes (representations) of the metadata items used to describe a particular statistical data set. It is good practice for compilers to put special emphasis on the development, maintenance and dissemination of metadata registries to improve the harmonization, standardization, use, reuse and exchange of their metadata; The Euro-SDMX registry includes harmonized structural metadata, the DSDs designed for the statistical domains and metadata structure definitions, e.g., Euro-SDMX Metadata Structure (ESMS) and other related information.
(e) Confidentiality and access to metadata during the compilation process. As metadata for statistics on the international supply of services might be compiled by various units of the same agency or by units located in different organizations, there might be cases in which metadata describes data that is subject to confidentiality rules. It is good practice, in that context, for confidentiality rules to be set up in a such way that they will allow compilers to obtain non-confidential data aggregates with the same metadata content;
(f) Incorporate structural metadata items into the data processing as early as possible (e.g., as parts of the records structure). That step will facilitate data processing, including the identification of viable options for data aggregation and subsequent presentation. It is advisable for structural metadata to be made an integral part of the database containing statistics compiled within the framework for describing the international supply of services in such a way that it can be extracted together with any data item and used in data processing to obtain meaningful combined data sets;
(g) Establish clear links between data and metadata. As metadata are generated and processed during every step of the data compilation process, there is a strong requirement to ensure that the appropriate metadata retain their links with data. In that connection, it is good practice to implement metadata-driven management in the various stages of the statistical production process; There are several information model specifications that can contribute to achieving that goal (most notably SDMX and the Data Documentation Initiative (DDI), which are designed to perform different functions, but can be used together in the same system, or complement each other in the compilation and exchange of data and metadata.
(h) Compilation of reference metadata. Reference metadata can be presented as a detailed explanatory note describing the scope, coverage, and quality of a data set and can be made available electronically alongside the database or in special publications.

1. Priorities in metadata management Although, ideally, the management of metadata would take into account all the recommendations highlighted thus far, countries with less developed systems for statistics on the international supply of services should begin by setting up an exhaustive, consistent and detailed repository (possibly in the form of a metadata registry) with both structural and reference metadata, adopting as much as possible standardized cross-domain metadata concepts, both nationally and internationally. The next immediate priority should be granting to the general public easy, extensive and timely access to metadata. In subsequent phases, the system could be improved by the gradual incorporation of more advanced features, such as a layered presentation of metadata and active links between data and metadata.

...

1. The international availability of appropriate metadata is of great interest to all organizations with global or regional responsibilities. Those organizations have made efforts to standardize their requirements for the scope and structure of the data and metadata that they would like to obtain from countries. Those requirements should be carefully studied by countries, both to improve their metadata collection and compilation, as well as to ensure better compliance with their international and regional data and metadata reporting obligations.

...

1. The Special Data Dissemination Standard (SDDS) and the General Data Dissemination System (GDDS) are part of the IMF data standards initiative aimed at enhancing member countries' data transparency and promoting the development of sound statistical systems. A dedicated electronic bulletin board on the IMF website http://dsbb.imf.org. posts information that SDDS countries provide to IMF on their dissemination practices, and offers direct links to the economic and financial data that countries disseminate under the SDDS, as well as information that GDDS countries make available to IMF on their statistical practices
2. For both standards, metadata are organized by country and topic. The SDDS metadata are available in two presentations, the current SDDS format and the Data Quality Assessment Framework (DQAF) format, while the GDDS uses the DQAF format alone. Revisions to metadata made regularly and are available on the IMF website. Metadata aspects related to statistics compiled within the framework for describing the international supply of services are embedded in the various quality dimensions of the BOP framework.

D.2The Statistical Data and Metadata Exchange content-oriented guidelines on metadata of Eurostat

1. On the basis of SDMX information, model DSDs can be created for data on the international supply of services (including FATS). Available from http://ec.europa.eu/eurostat/data/metadata/metadata-structure. The SDMX content-oriented guidelines have been used to define reference metadata for the European Statistical System (ESS). See also the Commission of the European Communities recommendation 2009/498/EC of 23 June 2009 on reference metadata for the European Statistical System (Official Journal of the European Union, L 168,30.6.2009). Table 18.1 lists the main components of the ESS reference metadata. The present Guide advises compilers of statistics on the international supply of services of other regions to take the European Union recommendations into account, as applicable, when setting up the conceptual structure of their own reference metadata for such statistics.

D.3Ensuring the consistency of metadata and data reported to international organizations

...

Table 18.1 Main components of the European Statistical System reference metadata

1. If there is no acceptable way to allocate all services, compilers should use a category to be labelled "not allocated", which will be shown at the same level as the main services items (i.e., not included within any of the services items) or main partners. The corresponding values should be included at the total services level only if such a category corresponds to the classification shown, or at the total world level if it relates to the partner breakdown shown.

1. A "services not allocated" item is included in the BOP-DSD because it is part of the reporting requirements of ECB and Eurostat for the quarterly Balance of Payments (QBOP) and of the requirements by Eurostat, OECD and the Statistical Division for annual international trade in services data. By detailed partner countries for the former and by detailed EBOPS 2010 for the latter. However, such a category is not included in the QBOP data reporting requirement of IMF. Consequently, for countries that use the category "services not allocated", the individual services items will not add up to the total services in their report to IMF. In that case, the compiler should indicate in the metadata provided to the agencies that that total services do not correspond to the sum of the main services items owing to the presence of some transactions that are impossible to allocate across services. Table 18.2 presents an example of data reporting in such a case.

...

...

1. Bank of Italy adopts an integrated approach to the broad range of its statistical programmes. The approach of the Bank to the management of metadata is based on two pillars: (a) an information model, capable of fully describing the data, the processing steps and the elaboration algorithms and (b) a software platform, which supports the entire statistical production chain.

1. Bank of Italy has designed a proprietary model called "Matrix," Vincenzo Del Vecchio, Fabio Di Giovanni and Stefano Pambianco, "The 'Matrix' Model:unified model for statistical data representation and processing" (Banca d'Italia, 2007). Available from https://www.bancaditalia.it/statistiche/raccolta-dati/sistema-informativo-statistico/modellazione/matrixmod.pdf. based on mathematical and statistical theory, to support all phases of the statistics production process (data definition, collection, compilation and dissemination) and all the data of interest (micro/aggregated, registers, questionnaires, etc.). A fundamental infrastructural component of the Bank's system, representing a core part of the actual implementation of the Matrix model, is the central statistical dictionary, a repository describing the entire content of the statistical data warehouse, in terms of structural metadata (e.g., concepts, classifications, data structures and processing rules) and reference metadata (e.g., methodological notes).
2. The Matrix model was also designed to take into account major international standards, so that, for example, the Matrix data and metadata can be easily transformed into SDMX and other metadata formats. Another essential feature of the Matrix model is that it enables a metadata-driven system by employing a recently introduced software platform for statistical processing called Infostat. Fabio Di Giovanni, Daniele Piazza, "Processing and managing statistical data: a National Central Bank experience", paper presented at the "International Statistical Conference", Prague, 14-15 September 2009. Available from http://www.academia.edu/6455962/Processing_and_managing_statistical_data_a_National_Central_Bank_experience. Consistent with the underlying holistic approach, Infostat supports the statistical production chain end-to-end, by providing the following services:

...