Scanner Data

Home   UN Big Data GWG Task Teams


The statistical community has the obligation of exploring the use of new data sources, such as Big Data, to meet the expectation of the society for enhanced products and improved and more efficient ways of working. Big Data could also support the monitoring of the Post-2015 development goals by improving timeliness and relevance of indicators without compromising their impartiality and methodological soundness. The report of the Global Working Group (GWG) to the Statistical Commission (E/CN.3/2015/4) provides additional background to the work of the task team, where the Terms of Reference of the GWG serves as general reference, but each task team also has its own specific terms.

Scanner data is a Big Data source being increasing used in national statistical systems for the calculation of price indices as statistical offices explore ways to meet the expectation of society for enhanced products and improved, more efficient ways of working. Many of the price measurement issues and methods for scanner data from supermarket chains and other retailers apply also to other big data sources (for example, online prices obtained from webscraping).


The task team on Scanner Data is created as a separate team, since Scanner Data has surfaced in recent years as one of the Big Data sources with a lot of promise. Indeed, a number of countries are already using the data in the compilation of Consumer Price Indexes and seeking to expand the use of the data to new products and purposes. The task team will share developments and practices from across countries with the aim of providing a set of tools and explanatory material which can be used by national statistical offices from around the world.

Workshop and Training Materials

GWG Meeting on Big Data
for Official Statistics   Nov 2017


  1. The delivery of an open source application with an associated Application Program Interface (API), which can be shared among all partners in the statistical community. This application will take cleaned and classified 1 scanner data 2 (i.e. Analysis Ready Data (ARD)) and will apply a range of analysis and monitoring processes before enabling a range of methods to be used for estimation of price indexes. The exact method can be specified by the user.
  2. The development of training and instructional material on the use of the application
  3. The development of accompanying methodological guidance material which will a) summarise the relevant literature on methods, b) point to internationally-agreed recommendations on which methods are appropriate in which situations and c) catalogue existing and intended practice across NSIs in the use of Prices big data



  • Australia
  • Belgium
  • Canada
  • Denmark
  • Netherlands
  • New Zealand
  • United Kingdom

International Institutions

  • UNSD