Data Exchange Suite (DES)
DES consists of web service application and synchronization program. Due to size, REST web service is selected because its ability of steaming and using on-the-fly compression. The data exchange format used is SDMX Cross Sectional Format (see http://www.sdmx.org/ for further information about SDMX). The synchronization program (called ComtradeTools) is, currently, command line program that can be run under .NET Framework machine. In the future, ComtradeTools might be upgraded to have better user interface.
Data Exchange Architecture
1. UN Site contains a web server and database server. Comtrade, Tariff Line, and Total Trade Data are available through the web services.
2. Web Services can stream compressed trade data in SDMX-ML format or Element based XML. The web services also have some metadata such as the data availabilities and references.
3. Organization #1 connects to the internet and obtains trade data using the internet browser. He/she can save-as the data to XML-file.
4. Organization #2 uses ComtradeTools (developed by UNSD) to obtain the trade data and convert it into text files. Other tools can also be used.
5. Organization #3 set up the automatic one-way synchronization so that the local trade database will be always up to date with UN site.
6. All three organizations are authenticated by IP Addresses and Web Service Access Control List.
DES database has similar design with UN Comtrade: one table holds data for a year. Multiple tables are needed for multiple years and partition view is created to cover them. There will be two tables: Tariff Line Tables and Comtrade Tables. Both of them have the same table design. In addition to that, several tables are created for footnotes, explanatory notes and memorandum items.
Application Design: Approach and Strategies
1. The strategies of the Trade Data Dissemination are
· Using a standard data format. SDMX (http://www.sdmx.org) version 1.0 has been released and it is being used as one of the format for trade data dissemination;
· Leveraging Internet as a low cost transfer medium and for its standards, such as http protocol, web services, etc;
· Platform independent means that client can use any operating systems, any database, and program languages to obtain trade data;
· Be able to transfer a very large data. The REST Web Service and internet stream compression play important role here;
· The possibility for unattended one-way synchronization.
2. The implementation of the strategies:
· Architecture: “Pull” architecture due to the flexibility and tight security (such as UN Server can’t initiate http request to any web sites). The client has to initiate a request.
· Synchronization Approach: By comparing the data availability between two sites for creating the list of new/updated datasets. By using time stamps.
· Data Format: SDMX-ML version 1.0 is currently being used.
· Medium and Protocol: Internet (with http protocol) via Web Services.
· Compression: The SDMX-ML is compressed during the data transfer. It reduces 70-80% of the bandwidth usage and downloads time.
· Encryption: None at the moment. If it is necessary, the https will be implemented instead of http.
· Security: Based on IP Address and Access Control List (to use Web Service).
· Tools: A small command line program (beta) has been created that downloads Comtrade SDMX-ML via Web Services and (can) import them into the SQL Server. If it is necessary, a full client program can be developed. Please bear in mind that this tool is not the only one that is able to obtain trade data. The client can obtain the trade data using an Internet Browser such as Netscape or Internet Explorer.
Synchronization program: ComtradeTools
For a start, ComtradeTools is created as command line program. The main disadvantage is that it is not user friendly and use needs to understand how to use it. However, the ComtradeTools is not necessary to get data. User can create his/her own program.
The advantages of ComtradeTools are providing one-way synchronization, compression and direct import to SQL database.
Add Your Comments
Last Modified:Thursday, January 14, 2010
Last Modified By: comtrade_admin
Article not rated yet.
Article has been viewed 11,585 times.