Scanner Data

Task Team of the UN Committee of Experts on Big Data and Data Science for Official Statistics

Introduction

The statistical community has the obligation of exploring the use of new data sources, such as scanner and web scraped data, to meet the expectation of society for enhanced products and improved and more efficient ways of working. The task team on scanner data is created as a separate team, since scanner data is one of the Big Data sources which is used more and more in national statistical systems for the calculation of price indices. Many of the price measurement issues and methods for scanner data from supermarket chains and other retailers apply also to other big data sources. For example, online prices obtained from web scraping.

Objectives

The Task Team objectives are:

  1. Guidance on using ADS for consumer price indices - Produce an e-handbook (wiki) on using alternative data sources (ADS) to produce consumer price statistics from the initial stages on data acquisition to implementing these new data sources in live production. Make code available for NSIs to test out different methods that can be applied to ADS to produce consumer price indices.
  2. Classification - Guidance on the process for classifying scanner data to produce data ready for price index compilation. This guidance will include advice on various international approaches to preparing these data including the use of machine learning techniques and, where appropriate, will make methods/code available to share with NSIs.
  3. Training - Ongoing capacity building efforts: expand existing training material; work with the UN Training, Competencies and Capacity Development task team to produce a new certified training course on using these data in consumer prices

Deliverables

Workstream 1 - Guidance on using ADS for consumer price indices

  • Produce an e-handbook (wiki) on using alternative data sources (ADS) to produce consumer price statistics
  • Make code available for NSIs to test out different methods that can be applied to ADS to produce consumer price indices

Workstream 2 - Classification

  • Draft new guidance on potential methods available for classifying scanner data to produce data ready for price index compilation – via e-handbook
  • Initial methods/code available to share with NSIs

Workstream 3 - Training

  • Production of new training content for trusted learning (targeted at different entry levels)
  • Delivery of new training course on using alternative data sources for consumer prices

Events

Workshop on Scanner Data and Web Scraping
Virtual   12-14 Oct 2021
The objectives of the workshop are to exchange on practical experiences related to the use of new data sources in consumer price statistics, and to present and discuss some of the work conducted by the Task Team and by the European Statistical System.

Task Team members

Countries

  • Australia
  • Austria
  • Belgium
  • Brazil
  • Canada
  • Denmark
  • Finland
  • Germany
  • Indonesia
  • Italy
  • Malaysia
  • Mexico
  • Netherlands
  • New Zealand
  • Norway
  • Poland
  • Switzerland
  • Thailand
  • Turkey
  • United Kingdom
  • United States

Agencies

  • Eurostat
  • IMF
  • UNSD

Universities

  • University of Graz, Austria