Trade Statistics Knowledgebase » Knowledgebase » Popular Articles

Data Extraction Using Comtrade Web Service

comtrade_admin — Thu, 04 Apr 2013 09:44:17 GMT

Data Extraction Using Comtrade Web Service
General Information

Comtrade Web Service is a REST Web Service (see http://en.wikipedia.org/wiki/Representational_State_Transfer) so that data extraction requests need the right URL to be submitted.

Data, metadata (such as explanatory notes) and data availability can be accessible within web service framework.

Data Format

The web service offers two kinds: SDMX (see http://www.sdmx.org/) and element-based-XML. Depending on what you need, both formats can be used independently.

In order to promote SDMX format, command line data extraction program was created (ComtradeTools). You can use it to extract data and metadata within SDMX and save it as CSV text format.

SDMX vs. Element-Based-XML

SDMX merges data and metadata information in one XML file. For an example, exchange rate and trade system (both of them are metadata) are included on data header. Due to its complexity, SDMX might require extensive learning effort.

On the other hand, element-based-XML is a very simple format, on which it can also be imported easily with other spreadsheets or databases, such as Microsoft Excel. However, metadata may not integrated with data.

Note: Complex query which includes commodity/country grouping is only available on element-based-XML.

Security & Authentication

Access to UN Comtrade database is restricted with IP authentication. A user must have site license account in UN Comtrade and Web Service access right is granted. However, access to metadata and data availability is not restricted.

(Feb 2010) For those who like to access UN Comtrade Web Service outside of IP address ranges, you can use the authorization code (can be obtained from http://comtrade.un.org/db/u/MyAccountInfo.aspx).

Click here to check your access rights: http://comtrade.un.org/ws/CheckRights.aspx

Web Service Methods and Parameters

To access REST web service, you need page name (method) and query string (parameter). In fact, UN Comtrade database was built based on query string, such result page of Ireland Total Export in 2006
(http://comtrade.un.org/db/dqBasicQueryResults.aspx?cc=TOTAL&px=H2&r=372&y=2006).

dqBasicQueryResults.aspx is the page name (method to show result) and cc=TOTAL&px=H2&r=372&y=2006 are the query strings, of which cc is commodity code, px is commodity classification, r is reporter and y is year.

In order to get the result via web service, you need to change dqBasicQueryResults.aspx to get.aspx (for Element-Based-XML format) or getSdmxV1.aspx (for SDMX format) and change /db to /ws (web service) and add comp=false (set compression flag to disable, so that it can be viewed on a browser):
(http://comtrade.un.org/ws/get.aspx?cc=TOTAL&px=H2&r=372&y=2006&comp=false or http://comtrade.un.org/ws/getSdmxV1.aspx?cc=TOTAL&px=H2&r=372&y=2006&comp=false
or for those who like to use authorization code
http://comtrade.un.org/ws/getSdmxV1.aspx?cc=TOTAL&px=H2&r=372&y=2006&comp=false&code=yourcode
)

Here is the list of common query string parameters:

Px = Commodity Classification
R = Reporter
Y = Year
CC = Commodity Code
P = Partner Country
Rg = Trade Flow

Data (restricted access)

The following data is available (depends on your access rights):

-UN Comtrade (get.aspx, getSdmxV1.aspx)
-TotalTrade (getTTotal.aspx, getConvFactor.aspx)
-Tariff Line (getSdmxTariffLineV1.aspx)

Metadata and data availability (free access)

Reference Tables:
-Country List (refs/getCountryList.aspx)
-Commodity List (refs/getCommodityList.aspx)
URL examples:
-Get 2 digit HS1996: http://comtrade.un.org/ws/refs/getCommodityList.aspx?px=H1&cc=??
-Get all country codes: http://comtrade.un.org/ws/refs/getCountryList.aspx

Metadata:
-Explanatory Notes (refs/getExplanatoryNotes.aspx)

Data Availability
-Data Availability (refs/getDataAvailablity.aspx)

Asynchronous Web Service (DEPRECATED)

comtrade_admin — Thu, 04 Apr 2013 09:42:26 GMT

To previous users of the asynchronous web service:

The asynchronous web service feature is no longer available. If you would like to submit a query for future execution please use the batch processing feature.

How to use the "Basic Selection" feature in the Data Query

comtrade_admin — Tue, 02 Apr 2013 05:34:16 GMT

How to use the "Basic Selection" feature in the Data Query?

Basic Selection is a sub-menu of "Data Query" in the UN Comtrade database.

This feature allows the user to build a query by selecting items from provided lists. This is considered as a traditional data selection tool.

Note: Basic Selection only in Internet Explorer v 6.0 and above.

Two methods for Basic Selection:

Method 1: Load Save Query (available only for subscribers)
You can load your saved query in 'step 0'

Method 2: Input/Select basic items query

Step 1 (Required): Select the commodity classification you are searching for. Description of the classification and its commodities listing can be found at:
http://comtrade.un.org/db/mr/rfCommoditiesList.aspx
http://unstats.un.org/unsd/cr/registry/regct.asp?Lg=1

Should you need more information on how to choose the commodity classification, please refer to the following KB article: Which classification system is appropriate to use?

Step 2: Enter selection items

Commodities: Allows one to search for commodity by ‘commodity code’ or by ‘item/description’. For example you can search for ‘steel’ in HS2002. Searching by name yields several results and the user has the option to choose one or more which appear in the Select Items column. See figure below:

If you want to load your favorites, choose the right source in step.1. Special Items source enables selection of TOTAL, 1 digit code, 2 digit codes … etc.

Reporters, Partners, Years: Select one or more of each and click the ‘add’ button.

Others: This feature allows one to select the trade flow, filter by value, select a preset sort order, and choose the aggregation option desired. NOTE: Currently only one trade flow can be saved if you choose to save your query. Also, AutoBatch can be set to ‘full’ or ‘incremental’.

Step 3: Save Query (optional)

You can save your query for future use, and enable notification or AutoBatch.

Please review the following article regarding the above features for save query: http://unstats.un.org/unsd/tradekb/Knowledgebase/Saved-Query-Alert-and-Notification

Related questions:

Q: I often get "NO RECORDS", when running the data extraction. What could I do to avoid it?

A: Since not every country reports its trade data in the same classification, you have to make sure which classification is available for which countries or commodities or years. Please use "Data Availability" menu.

UN Comtrade Data Availability (via Comtrade Web Services)

comtrade_admin — Mon, 25 Mar 2013 08:50:24 GMT

UN Comtrade Data Availability (via Comtrade Web Services)

Query string parameters: px (classification), r (reporter codes), y (years), lowT (start date), highT (end date), so (sort order), isOri (original flag).
Available classifications (px): HS2002 (H2), HS1996 (H1), HS1988 (H0), SITC Rev.3 (S3), SITC Rev.2 (S2), SITC Rev.1 (S1), BEC (BE)
lowT and high T format is YYYY-MM-DD (ex. 2006-06-15). Time can be added using ‘T’ (2006-06-15T17:44:01)
Available sort orders (so): 1 (by Upload Time), 2 (by Upload time descending), 10 (by year), 11 (by year descending)
“Original Flag true” returns original reported classification only.

Example:

Get loaded data sets after 15 June 2006 17:44:01
http://comtrade.un.org/ws/refs/getDataAvailability.aspx?lowT=2006-06-15T17:44:01

Get original loaded data sets after 15 June 2006 17:44:01
http://comtrade.un.org/ws/refs/getDataAvailability.aspx?lowT=2006-06-15T17:44:01&isOri=true

Get all available H2 data sets of year 2005 order by upload time descending
http://comtrade.un.org/ws/refs/getDataAvailability.aspx?y=2005&px=H2&so=2

Tariff Line Data Availability

Please note that tariff line data is only available to specific partner organizations. Email comtrade@un.org if you feel that you require access to tariff line data.

Query string parameters: px (classification), r (reporter codes), y (years), lowT (start date), highT (end date), so (sort order)
Available classifications (px): HS2002 (H2), HS1996 (H1), HS1988 (H0), SITC Rev.3 (S3), SITC Rev.2 (S2), SITC Rev.1 (S1), BEC (BE)
lowT and high T format is YYYY-MM-DD (ex. 2006-06-15). Time can be added using ‘T’ (2006-06-15T17:44:01)
Available sort orders (so): 1 (by Upload Time), 2 (by Upload time descending), 10 (by year), 11 (by year descending)
“Original Flag true” returns original reported classification only.

Example:

Get loaded data sets after 15 June 2006 17:44:01
http://comtrade.un.org/ws/refs/getDATariffLine.aspx?lowT=2006-06-15T17:44:01

Get original loaded data sets for Germany
http://comtrade.un.org/ws/refs/getDATariffLine.aspx?r=276

Get all available H2 data sets of year 2005 order by upload time descending
http://comtrade.un.org/ws/refs/getDATariffLine.aspx?y=2005&px=H2&so=2

Timetable for data availability of country

Editor SL — Fri, 01 Mar 2013 10:43:40 GMT

Timetable for data availability of country

Q: How soon will country data for imports and exports be available?

A: There isn't an expected date or time when a specific country's data will become available. We request the annual data in the month of February. However, we receive data from various countries at different time periods. As soon as our office receives the data, we validate it and upload into the UN Comtrade database.

You can set up a notification procedure where the system would inform you when a particular country period is uploaded. (Note: This option is only available to subscribers.)

For monthly data, we follow-up on a monthly basis with countries that have indicated they will regularly provide monthly data but haven't done so in several months. Some countries provide their latest monthly data within a month of the close of the reference period, while others provide their much data later. As of February 2013, for countries having provided data on at least three separate occasions, and measuring the delay as the number of days between the closure of the reference period and the first receipt of the latest data in our office, 7 countries have a median delay of less than one month, 43 countries have a median delay of less than two months, and 76 countries have a median delay of less than three months, out of a total of 81 countries having submitted data on at least three separate occasions. 28 countries have provided data only once or twice, for a total of 109 countries providing monthly data. This includes countries having provided data but whose data has not been released due to resource constraints. Many countries have not yet established regular provision of monthly data, and we are working with data providers to improve the regularity of data provision.

Related question:

Q: The Comtrade Database has annual data available online. Do you also have monthly data?

A: We have monthly trade flows in the UN Monthly Comtrade (beta version) database. The data can be accessed at http://comtrade.un.org/monthly.

UN Comtrade SDMX

Editor MP — Thu, 10 Nov 2011 10:40:58 GMT

What is SDMX?

The name stands for “Statistical Data and Metadata Exchange” refers to an international cooperation initiative aimed at developing and employing more efficient processes for exchange and sharing of statistical data and metadata among international organizations and their member countries.

Please refer to Brief Introduction to SDMX for the full introduction document.

For further information, please visit SDMX web site.

Comtrade SDMX
1. Comtrade uses SDMX cross-sectional format version 1.0.
2. Each commodity classification has a specific Key Family. Therefore, there are 6 key families: for HS2002, HS1996, HS1992, SITC Rev.3, SITC Rev.2, and SITC Rev.1 Download the all key families here.
3. Explanatory notes are integrated with Comtrade SDMX.
4. This is a beta version. The SDMX structure can be modified.
5. Comtrade Web Services are under the development and testing. With web services, trade data transfer and synchronization will be simplified.

How to read SDMX
SDMX is a text file which is readable by any text editors. A converter is required, in order to convert SDMX into CSV format.

Users can download data in SDMX format using the direct download feature. Comtrade Batch is not able to generate data in SDMX format.

Difference BoP - Foreign Trade statistics - Eurostat, 2004

Editor NK — Mon, 28 Mar 2011 09:26:26 GMT

Differences between Balance of Payments and Foreign Trade Statistics

Balance of Payments (BOP) and Foreign Trade statistics (FTS), both provide data for the external trade in goods of a given country; the two data sets generally do not show the same figures. The two data sources have very different detail and serve different users’ needs. Different methodologies are the main reason for the differences between the two data sets. This paper provides a summary of the main methodological differences between the two data sets and describes all the adjustments that have to be made by BOP compilers to go from FTS to BOP data. The results of a recent survey of the main adjustment practices in EU member states and candidate countries are also provided.

Download

Sectoral Classification List (W/120)

Editor NK — Mon, 21 Mar 2011 06:04:33 GMT

The services sectoral classification list (W/120) is a comprehensive list of services sectors and sub-sectors covered under the GATS. It was compiled by the WTO in July 1991 and its purpose was to facilitate the Uruguay Round negotiations, ensuring cross-country comparability and consistency of the commitments undertaken. The 160 sub-sectors are defined as aggregate of the more detailed categories contained in the United Nations provisional Central Product Classification (CPC).

The list is available at the WTO website at: http://tsdb.wto.org/Includes/docs/W120_E.doc

Report on business registers (UNSC, 2011)

Editor NK — Mon, 14 Mar 2011 07:11:33 GMT

The report provides a brief summary of the themes discussed during the 22nd meeting of the Wiesbaden Group on Business Registers. The meeting was held from 27 to 30 September 2010 in Tallinn, hosted by Statistics Estonia. It was prepared in accordance with a request of the Statistical Commission at its forty-firstsession (see E/2010/24, chap. I.A) and transmitted to the Commission for information.

Report of the Wiesbaden Group on Business Registers

38th UNSC - Report of the SG and the WTO on tourism statistics

comtrade_admin — Mon, 14 Mar 2011 07:03:35 GMT

Joint report of the Secretary-General and the World Tourism Organization on Tourism Statistics

The report was submitted to the Statistical Commission in accordance with a request of the Commission at its thirty-fifth session. It contains a description of activities undertaken in response to the actions of the Commission at that session with regard to the revision of international recommendations for tourism statistics. The draft version of the International Recommendations on Tourism Statistics 2008 (IRTS 2008) was submitted to the Commission as a background document for consideration and adoption.

Joint report of the Secretary-General and the World Tourism Organization on Tourism Statistics (40th UNSC, 2008)

Improvement in business registers (ECLAC, Rio de Janeiro 2009)

Editor NK — Mon, 14 Mar 2011 06:16:06 GMT

This document was presented during the workshop on Statistics of international trade in services in the Latin American and the Caribbean countries organized by UNSD, in collaboration with the Instituto Brasileiro de Geografia e Estatística (IBGE) and Banco Central do Brasil and Economic Commission of Latin America and the Caribbean (ECLAC). The workshop was hosted by the Instituto Brasileiro de Geografia e Estatística (IBGE) in Rio de Janeiro between 1 and 4 December 2009.

Improvement in business registers (ECLAC, Rio de Janeiro 2009)

Foreign Affiliates Statistics (WTO, Rio de Janeiro 2009)

Editor NK — Mon, 14 Mar 2011 06:10:29 GMT

This document was presented during the workshop on Statistics of international trade in services in the Latin American and the Caribbean countries organized by UNSD, in collaboration with the Instituto Brasileiro de Geografia e Estatística (IBGE) and Banco Central do Brasil and Economic Commission of Latin America and the Caribbean (ECLAC). The workshop was hosted by the Instituto Brasileiro de Geografia e Estatística (IBGE) in Rio de Janeiro between 1 and 4 December 2009.

Foreign Affiliates Statistics (WTO, Rio de Janeiro 2009)

Trade in services in ECLAC countries (ECLAC, Rio de Janeiro 2009)

Editor NK — Mon, 14 Mar 2011 06:02:00 GMT

This document was presented during the workshop on Statistics of international trade in services in the Latin American and the Caribbean countries organized by UNSD, in collaboration with the Instituto Brasileiro de Geografia e Estatística (IBGE) and Banco Central do Brasil and Economic Commission of Latin America and the Caribbean (ECLAC). The workshop was hosted by the Instituto Brasileiro de Geografia e Estatística (IBGE) in Rio de Janeiro between 1 and 4 December 2009.

Trade in services in ECLAC countries (ECLAC, Rio de Janeiro 2009)

Tourism statistics in the CIS countries (CISSTAT, Chisinau 2010)

Editor NK — Fri, 11 Mar 2011 06:39:20 GMT

This document was presented during the workshop "Developing National Systems of Tourism Statistics: Challenges and Good Practices " organized by the United Nations Statistics Division (UNSD) in collaboration with the UN World Tourism Organization (UNWTO), the Interstate Statistical Committee of the CIS (CISSTAT) and the National Bureau of Statistics of the Republic of Moldova (NBS), with the support of the United National Development Programme – Moldova. The Workshop was hosted by the NBS in Chisinau, Moldova between 29 June and 2 July 2010.

1- Background document (CISSTAT, Chisinau 2010)
2- Tourism statistics in the CIS countries (CISSTAT, Chisinau 2010)

Overview of EU tourism statistics (Eurostat, Chisinau 2010)

Editor NK — Fri, 11 Mar 2011 06:34:22 GMT

This document was presented during the workshop "Developing National Systems of Tourism Statistics: Challenges and Good Practices " organized by the United Nations Statistics Division (UNSD) in collaboration with the UN World Tourism Organization (UNWTO), the Interstate Statistical Committee of the CIS (CISSTAT) and the National Bureau of Statistics of the Republic of Moldova (NBS), with the support of the United National Development Programme – Moldova. The Workshop was hosted by the NBS in Chisinau, Moldova between 29 June and 2 July 2010.

Overview of EU tourism statistics (Eurostat, Chisinau 2010)

EBOPS classification (UNSD, Hanoi 2010)

Editor NK — Fri, 11 Mar 2011 06:14:02 GMT

This document was presented at the Workshop on Statistics of International Trade in Services for the South-East Asian countries which took place from 30 November to 3 December 2010 in Hanoi, Vietnam.

EBOPS classification (UNSD, Hanoi 2010)

Enterprises survey (UNSD, Hanoi 2010)

Editor NK — Fri, 11 Mar 2011 06:13:36 GMT

This document was presented at the Workshop on Statistics of International Trade in Services for the South-East Asian countries which took place from 30 November to 3 December 2010 in Hanoi, Vietnam.

Enterprises survey (UNSD, Hanoi 2010)

Overview of the BPM6 (UNSD, Hanoi 2010)

Editor NK — Fri, 11 Mar 2011 06:13:13 GMT

Overview of the MSITS 2010 (UNSD, Hanoi 2010)

Editor NK — Fri, 11 Mar 2011 06:12:41 GMT

Foreign Affiliates Statistics (Eurostat, Hanoi 2010)

Editor NK — Fri, 11 Mar 2011 06:12:13 GMT

FDI statistics (Eurostat, Hanoi 2010)

Editor NK — Fri, 11 Mar 2011 06:11:49 GMT

Modes of supply (WTO, Hanoi 2010)

Editor NK — Fri, 11 Mar 2011 06:11:08 GMT

Data dissemination (UNSD, Hanoi 2010)

Editor NK — Fri, 11 Mar 2011 06:09:30 GMT

Overview of the IRTS 2008 (UNSD, Hanoi 2010)

Editor NK — Fri, 11 Mar 2011 06:06:38 GMT

Data quality and metadata (UNSD, Hanoi 2010)

Editor NK — Fri, 11 Mar 2011 06:04:04 GMT