C. Data processing and database management at the responsible agency
1. Characteristics of data processing, data flow and data transformations
8.18. Characteristics of data processing at the responsible agency in contrast with other statistical activities: The statistical processing of merchandise trade data involves dealing with large numbers of data sets of relatively simple structure. These data sets are in general obtained from customs declarations and received from customs. Further characteristics are the use of extensive and usually automated validation and quality checking procedures, the storing of processed data and metadata in well-maintained databases capable of performing customized data queries, and timely provision to users of large data sets in various formats. All these activities imply the intensive use of information technology which frequently requires that significant IT resources be specifically dedicated to trade statistics. Particular challenges for statistical data processing can arise when revisions or corrections need to be coordinated and agreed between customs and the responsible agency. A further potential difficulty is the integration of data from other sources, as those data, for example, might not follow the required standard format.
8.19. Data transformations. The following data transformations are often executed at the responsible agency: suppression or removal of certain information (due to issues of confidentiality or quality), correction of existing data and supplementation of existing data through estimation or other means (i.e., if certain characteristics are not provided).
8.20. The role of customs. Custom declarations are the main and usually preferred data source for merchandise trade statistics. Not only are customs authorities providing this information to the responsible agency, but, they have a very strong influence on the quality of the information provided (see chapter IX for details, in particular para. 9.5 on data processing and validation). In this context, it is critical that customs work with the traders or brokers who enter the information to ensure that the data required for statistical purposes are adequately captured in the customs declarations. At the same time, the responsible agency needs to make customs aware of these requirements (see chap.V for details).
2. Examples of data processing systems at the responsible agency
8.21. Eurotrace software: data- processing software for external trade statistics. The Eurotrace software, distributed free of charge by Eurostat and implemented in many developing countries,[15] allows (a) the importation and management of the data necessary to the development of the external trade statistics (in particular the customs data), (b) the treatment of these data, in particular through carrying out quality controls and the application of standards, (c) the working out and calculatation of a certain number of aggregates, in particular indices of foreign trade and (d) their export for dissemination and publication. Eurotrace consists of the following separate applications that work together: Eurotrace DBMS, the Eurotrace Data Editor and the Comext Standalone Data Browser.
8.22. Eurotrace applied in Trinidad and Tobago.[16] The Central Statistical Office (CSO) of Trinidad and Tobago has developed a Eurotrace application which has transformed its trade statistics data dissemination. As a result of the implementation of the Eurotrace Trade Statistics application, the time taken to respond to a wide array of ad hoc data requests from international, regional and local data users has been significantly reduced.[17] Further improvements depend largely on the implementation at customs of ASYCUDA which would replace the current system of manual data capture based on copies of declaration forms. The proposed future data flow will be greatly simplified and will consist of data reception from ASYCUDA, importation to Eurotrace, validation in Eurotrace, upload of validated data and data extraction/direct data download through the Comext Browser.