1. Unit values from customs records
28.A.1. Frequency and volume of data from customs records processed. StatisticsNorway receives administrative data from the Norwegian customs administration authority (TAD), every day. The number of customs data records used by Statistics Norway in the creation of exports statistics has increased 23 per cent since 2000, and for imports this increase has been more than 80 per cent. In 2010, Statistics Norway used about 1.4 million data records of exports and almost 11 million records of imports (representing 44.3 per cent and 99 per cent ofNorway's total value of exports and imports, respectively).
28.A.2. Two-step process for unit value calculation. The main body of information onNorway’s external trade statistics comprises administrative data from customs declarations (the single administrative document (SAD)). As the single administrative document does not contain a price variable, unit values are derived from the variables value and quantity. This is carried out based on the total commodity value and quantity, after a two-step validation process which involves stratification to identify commodity codes whose the data may be utilized for price statistics.
28.A.3. Stratification below the commodity code level. In the first step, data in each commodity code are stratified below the HS-code level. The aim of the stratification is partly to arrive at more homogeneous strains within the code and, simultaneously, to reduce the overall variation observed at the code level. There are three stratification variables: enterprise (VAT number), partner country (ISO code) and quantity group (based on weight or supplementary unit). The choice of the best stratification variable for a commodity code in the reference year is made by means of automated analyses run on the data of the previous (base) year.
28.A.4. Outlier detection and data editing. Before the estimation, the data are run through an editing procedure controlling for extreme prices. The data are subjected to a Hidiroglu-Berthelot (HB)-based procedure for identifying extremes, both on the stratum level and within the strata. Extremes are excluded from further calculations. For each stratum within an eight-digit commodity, a set of control variables is calculated. The purpose of this step is to evaluate statistical properties of unit prices resulting from each method of stratification (enterprise, country and quantity group). The indicators are:
(a) Regularity of transactions (number of months in year T-1 with no transactions < six)
(b) Price variation (coefficient of variation < 0.5);
(c) Value (> 1 per cent of the total value on commodity level);
(d) Quantity (> 1 per cent of the total quantity on commodity level).
28.A.5. Selection of customs data for the computation of unit value indices. The ratio of the arithmetic average to the quantity-weighted average of the monthly unit values, at the transaction level, is used as a background variable for evaluating the stratification of the data and choosing which method to use for each HS code. Taken together, these indicators give information on stability, and magnitude and concentration of the strata. A stratum is accepted if the values of all the indicators are within the required limits. If one or more of the limit values are exceeded for a stratum, the stratum is rejected and therefore does not become a part of the calculation of price indices.
2. Producer price indices (PPIs) for external trade
28.A.6. Integration of survey data from producer price indices. For some important commodities, data from customs records are deemed too heterogeneous to yield acceptable price information. To compensate for such shortcomings, survey-based price indices are used as indicators in external trade statistics. In Statistics Norway, the survey that yields producer price indices (PPIs) covers the domestic, export and import markets. An important characteristic of theNorway’s system of price statistics is the fact that external trade considerations guide and influence the PPI production, particularly in respect of determining which commodity codes are included in the sample.
28.A.7. Survey implementation. Data collection is mainly conducted through questionnaire, whereby respondents also receive guidance in the form of an information brochure as well as semi-annual messages from Statistics Norway. The statistics register employed by Statistics Norway includes all resident firms that produce or deal with the commodities in question and have 10 or more employees. The sample is based on a scheme of probability proportional to size. Prices are collected over time for selected well-defined products, all of which are classified according to the HS nomenclature. In practical terms, this means that a survey questionnaire makes reference to a specific HS commodity classification, and the respondent must provide price data for a product model that best suits this commodity description; the price of this product is reported monthly.
28.A.8. Index formulas. Elementary indices are calculated at the HS level, using a geometric mean. Indices at the HS level are then aggregated, using a weighted average, to form a Classification of Products by Activity (CPA) index, and from the CPA level, indices are aggregated to CPA level of four-, three-digits, etc. This is carried out for each of the three markets (domestic, export and import). Indices above the elementary level are calculated using the Laspeyres formula.
28.A.9. Imputation. During the process of compilation of PPIs, missing HS data are imputed using higher levels of aggregation. Sequences of 13 consecutive months are used to calculate a short-term index, where the base is always December of the previous year.
3. Other data sources
28.A.10. Alternative sources of price data. In addition to survey data and customs records, there are special data-collection mechanisms in place, including the use of price information from international commodity exchanges and foreign statistical agencies. For internationally traded commodities (refined oil products, nickel, etc.) price data are collected from the London Stock Exchange and London Metal Exchange.
28.A.11. Use of foreign indicators of price trends. For other products (especially exports and imports of capital goods), international price indicators are in some cases considered to be representative of the price development of the same product group in Norway’s trade. For instance, data from the United States Bureau of Labor Statistics are used for about 80 export products and 40 import products.
4. Data validation and editing
28.A.12. Validation of customs data. Data validation procedures are routinely put in place in order to detect errors in the statistical values reported in the customs declarations. In this regard:
(a) Tests were introduced in 2011 that are applied directly on data as they are entered by declarants of exports or imports. These tests aim at identifying obvious errors or data inconsistencies at the first step of data flow, and include: validity checks for commodity and country codes, price verifications based on upper and lower thresholds, quantity checks, and checks for implausible data by commodity or partner. These and other controls are also applied within the customs service’s own information systems;
(b) Prior to loading customs data into the Statistics Norway database, some data editing is conducted. Only the transactions involving commodities above 1,000 Norwegian kroner (NOK) and less than one year old are selected, and incomplete declarations are rejected. After loading, automatic corrections are carried out, and the validity of codes is checked again. Also, with the aid of statistical tools, probable errors are identified, which may involve unusual prices, partners or commodities, as well as code combinations that seem suspect. All large declarations are subject to data quality control, in which the experience of staff members specializing in the checking of data of specific groups of commodities plays a key role;
(c) In cases where Statistics Norway does not have sufficient information to correct obvious errors, a report is sent to customs specifying the nature of the problem with each suspect transaction. This report is reviewed by customs and sent back to Statistics Norway with a comment indicating whether any corrective action was taken.
28.A.13. Validation of price survey data. Validation mechanisms are also applied to price survey data submitted to Statistics Norway. These mechanisms include detection of high and low outliers, control of CPA classification, and checks on aggregated data at different NACE levels. If errors are suspected and the data in the questionnaire are insufficient, Statistics Norway will establish direct contact with the respondent in order to obtain further clarification.
28.A.14. Most frequent kinds of errors. Some of the most frequent kinds of errors detected are related to the wrong currency and/or exchange rates, as well as errors in the quantities reported. Two specific examples illustrate some of the kinds of errors that have been dealt with in the past. The first case involved salmon exports to the European Union that were subject to a punitive duty. As firms filling out the declarations were not able to report separately the duty, Statistics Norway had to expend significant efforts to correct the statistical value. Another situation was created by some companies using computer software to speed up the filling out of customs declarations, which automatically distributed total quantity (weight) of all declared goods according to their individual value shares. As a result, all commodities declared in a single document were implicitly given exactly the same unit value, rendering the information useless for unit-value calculations.
5. Institutional framework
28.A.15. Cooperation between Statistics Norway and Norway’s customs administration. A good working relationship exists between Statistics Norway and theNorway’s customs administration authority in terms of providing data for statistical purposes, as required by the Statistics Act of 1989. Cooperation between the customs administration authority and Statistics Norway is regulated by a formal agreement, which establishes responsibility for contacts between both parties. It stipulates that changes made to the existing administrative data systems should be communicated to Statistics Norway, regulates data transmission between the customs administration authority and Statistics Norway, confers on Statistics Norway the responsibility for compiling a list of all statistical surveys being conducted, and requires a yearly report on cooperation. As cooperation with customs personnel is essential during the data validation process, Statistics Norway provides regular training for Customs employees, allowing for improvements at the data source level.