UNSD Document

Ranking: right or wrong?
Some problems in comparing national statistical offices and systems

Willem de Vries

1.Introduction

At the 47th plenary session of the Conference of European Statisticians at Neuchâtel in Switzerland in June 1999 the topic for substantive discussion was ‘Performance indicators’. A couple of papers that were discussed in this regard generated some excitement. One in particular, an invited paper by Robert Chote, Economics Editor of the Financial Times (Chote, 1999), was rather furiously attacked by several participants, some of whom even requested it be excluded from the official proceedings of the conference. My own paper (De Vries, 1999) for the conference was less controversial, except for a small table I showed, which was not part of the paper itself, comparing statistical systems on costs. In particular, some eminent statisticians whom I greatly admire, said that ‘benchmarking on costs’ was nonsense. Admittedly, it is a very difficult exercise, and in fact my table was taken from a report which is mostly about precisely these difficulties. Therefore, the purpose of this article is to explain how the table came about and what it may or may not be worth.

2. Risks of ranking

When the newspaper The Economist published rankings (or league tables) of national statistical offices some years ago, quite a few senior statisticians were upset and although there was little public discussion about the method that The Economist had used, there was broad agreement that the assessment had been superficial and that the results were debatable, to say the least. I will not go into the technicalities of The Economist’s ranking approach, except to say that a combination of ‘objective’ criteria and judgement by peers was used.

Robert Chote’s ranking was based on assessing the performance of national statistical systems against the IMF Special Data Dissemination Standard (SDDS), a voluntary standard, agreed by about 50 countries now. Its purpose is to make the quality of official statistics more transparent for the users. The main dimensions of the SDDS are:
- data characteristics (coverage, periodicity, timeliness);
- access by the public (advance dissemination of release calendars, simultaneous release to all interested parties);
- integrity and transparency (government access to data before public release, government comments on data, revisions etc.);
- quality (methodology, sources, possibilities to crosscheck).

Chote’s ranking results were based on a kind of weighted aggregate score on five main dimensions (coverage, periodicity, timeliness, release calendar and the presence of a hyperlink, allowing users to move directly from the IMF bulletin board to national statistical websites). He is the first to admit that his approach has its flaws; he himself says: ‘The table at the end of this paper illustrates just one way in which this (ranking) might be done ... This framework is open to all the same criticisms of incompleteness and inappropriateness that was levelled at The Economist ... But I am the first to concede that it would have to be a rather more rigorous and carefully thought out exercise than the back-of-an-envelope attempt I have made here’.

In spite of all these reservations from the author, the table came as a shock. Latvia headed the table, followed by Canada, Slovenia, Peru, the United Kingdom and Japan. Well-respected statistical offices such as those of Switzerland (27th), Australia (30th) and Iceland (40th) were shown as lagging far behind. The Netherlands found itself in 20th place, although if we took more care of our hyperlinks, we could get into the top ten. In particular the ‘composite’ nature of the total score was one of the points raised in criticism of the list.

One may question the significance of statistical ranking exercises as a matter of principle, but the fact of the matter is that statistics are used for ranking all the time. And rarely does this seem to shock the statisticians very much, even when the ranking is based on highly composite and/or relatively ‘soft’ measures, such as per capita GDP, research and development expenditure as a percentage of GDP, illiteracy rates, or poverty and development indicators etc. (1) So it seems curiously paradoxical that while statisticians can live with many kinds of rankings, they get jumpy when they themselves are the subjects of such an exercise. And a second paradox, which I shall discuss further below, is that official statisticians are supposed to be good at measuring nearly anything of relevance to society, but are definitely not very good at measuring their own activities and performance, at least not in my experience.

3. Why performance measurement?

Government budgets are under continuous pressure in most countries. There is a general tendency towards reducing government spending and therefore the spending on official statistics is also regularly scrutinised. Statistical offices are often relatively large and costly operations, and, as everyone knows, continuous technological and other developments make it easier to compile statistics more efficiently. So governments are likely to ask us, from time to time: can’t you do it cheaper? And in asking this question, it is practically inevitable that they also ask: and how do you compare with similar institutions in other countries? (2)

In exactly such a scenario, the Dutch government and parliament recently asked us to do a summary comparison exercise. The remainder of this article represents our answer to this request. It should be noted here, that the results have little to do with ‘ranking’ in the proper sense, although the order of countries in the small comparison table of the document, may be construed as such (from ‘expensive’ to ‘less expensive’ statistical systems).

4. Comparing statistical offices: the difficulties

The question was whether Statistics Netherlands is big and costly, compared with similar organisations in other countries. It is difficult to answer this question with any degree of precision as only a very global comparison is possible. The following reasons make comparisons of this kind rather problematic:

- The general organisational structure of statistics (centralised or decentralised);
- Coverage of the statistical work programme, i.e. which subject matter areas are covered; the work programme of Statistics Netherlands, for example, covers a very wide range of subject matter areas);
- The size of the country (there are certain economies of scale; on the other hand it should be recognised that large countries often have some sort of regionalised administrative and statistical structure, which may imply inefficiencies);
- Administrative and legal infrastructure (which may be relevant for the possibilities to use registers for statistical purposes);
- Special responsibilities that some national statistical offices may have (e.g. economic analysis).

Some explanations of concepts

- What is meant by ‘central’ and ‘decentral’? In some countries, for example the Netherlands, Sweden, Australia and Canada, the production of national statistics is the responsibility of one single organisation, regardless of how many ‘offices’ it may have. In other countries, however, there is some kind of decentralisation, either regionally - often, but not always, in combination with a ‘federal’ administrative structure - or departmentally, i.e. ministries producing statistics for their own policy areas, or a combination of various forms of decentralisation. Pure forms of ‘central’ and ‘decentral’ are relatively rare, but it is widely believed that any form of decentralisation implies certain negative efficiency effects.

- A factor that is difficult to measure is the role and significance of statistics in individual countries? To what extent are political and administrative decisions (e.g. the financing of regional and local authorities, budget policies, wage bargaining etc.) based on statistics? Many countries apparently have forms of ‘formula use’ of statistics, which give statistics some extra weight. In countries with some kind of federal structure this is certainly true, but it also applies to the European Union. It is hard to say what effects this phenomenon has on statistical expenditure. For the countries of the European Union included in the comparison in this article, it may be said that they all have to comply with the so-called acquis communautaire (for Statistics Netherlands, 70% of the work programme is covered by European regulations and directives), and are therefore in a more or less comparable situation.

- As to ‘administrative and legal infrastructure’, an important question is to what extent registrations exist: in the Anglo-Saxon countries, for instance, there are no population registers; and secondly, whether there are legal and other arrangements that enable statistical offices to use these registrations for statistical purposes. In Scandinavia these arrangements are very well developed. If there are no registrations, or if they may not be used for statistics, other - relatively more expensive - forms of data collection are necessary: population censuses, for example.

The comparison presented here deals with some larger and medium-sized, economically developed countries, which have a statistical system that is generally considered to be good or adequate (e.g. according to the league table developed by The Economist in 1993, which ranked Canada, Australia and the Netherlands as the top three). Besides the Netherlands, eight countries were compared, six in Europe: Sweden, Finland, Denmark, France, Germany, and the United Kingdom, and two outside: Canada and Australia.

Brief outline of the statistical systems involved

Germany, France and the United Kingdom (although recently measures towards more centralisation have been taken) have fairly decentralised statistical systems.

The Netherlands, Sweden, Finland, Denmark, Canada and Australia have a centralised system, though some of these countries have more than one statistical ‘office’, either offices per state or province (Australia, Canada), or two locations (Sweden, the Netherlands).

In Germany there is a Statistisches Bundesamt in Wiesbaden, roughly 500 km from the new government centre Berlin. The former fairly large branch office in Berlin recently moved to Bonn, the former government centre, as a compensation for the ministries that are being relocated from Bonn to Berlin. However, most of the data collection and dissemination is done by the statistical offices of the Länder.

France has both regional and departmental decentralisation. Apart from the central office INSEE (Institut Nationale pour la Statistique et les Analyses Économiques) in Paris, the ministries have their own statistical departments which are loosely connected with INSEE, which appoints the senior statistical managers, however. In addition INSEE has some dozens of regional offices for data collection and dissemination.

The United Kingdom traditionally had a small central statistical office, set up by Winston Churchill during World War II, but most statistics used to be produced by ministries. Co-ordination and confidence problems in the nineties led to centralisation of the most important economic and social statistics in one office: the Office for National Statistics (ONS). Part of ONS is located in London, but there are some other offices as well, including one in Newport, Wales, roughly 250 km from London.

Statistics Canada has its central office in Ottawa, but there are branch offices in each of the Canadian provinces as well.

In Australia the central office of the Australian Bureau of Statistics (ABS) is located in Canberra, but in addition there is an ABS office in each of the states. These regional offices do some specific statistical work for the government of their state, as well as some data collection and dissemination for the ABS, but they are also responsible for a certain part of the national statistical programme, for example business registers and service statistics (Melbourne), mining (Adelaide), or the financial sector (Sydney).

Sweden (Stockholm/Orebro) and the Netherlands (Voorburg/Heerlen) have one statistical office with two locations (in both cases about 200 km apart). Norway and Ireland have a similar structure. Finland and Denmark, lastly, have one organisation and one location.

5. Comparison of some indicators

The table below compares some indicators for the nine countries mentioned, in particular with respect to inhabitants and number of ‘official statisticians’, as well as the increase or decrease in this number over a certain time period, and government spending on official statistics, related to GDP (1998).