December 12, 2022

We are hopefully coming out of a pandemic. A pandemic where in the first 2 years, excess mortality associated with Covid-19 was almost 15 million people.

Not all of these 15 million died of Covid-19, but many died indirectly of Covid-19 owing to overwhelmed health systems. Were it not for the extraordinary scientific progress made in recent decades, and the rapid development and rollout of vaccines, that death toll could have been so much worse.

The Covid-19 pandemic is not over. In October 2022 alone, more than 50,000 Covid-19 deaths were reported to WHO. We await with trepidation the coming winter months.

Public health officials however are already thinking about the future. Covid-19 has naturally provoked a desire to be better prepared for future health emergencies. In this age of quantification, the solution in the minds of many is to develop a dashboard or more indicators. But there is a saying among soldiers that generals always fight the last battle. I worry that we may be guilty of doing the same thing. By building dashboards are we preparing to fight the last pandemic, the last health emergency?

We don't know what the next health emergency will be, but we can be reasonably certain it will not be what we expect. We can be almost certain it will surprise our dashboards. Dashboards and indicators can play a hugely important role to transmitting key messages. My point is that they have their limitations. We should not put all our faith in dashboards. A dashboard alone is not adequate surveillance. We should always remember that statistics play a dual role. They not only measure reality - statistics also shape or define reality. A dashboard will monitor the statistics presented in that dashboard, nothing more. A dashboard measures the reality defined by that dashboard.

As an aside, after the SDG indicators were selected, they were criticized in some quarters as being reductionist. This struck me as an odd criticism, as statistics are by definition reductionist. That is their very purpose - to distill the complexities of life into a set of metrics that help us to understand patterns and associations.

Thus, a dashboard may prepare us against emergencies, but it may also blinker us against the unexpected. It may give us a false sense of security. A dashboard is reductionist - that's not a criticism, simply a reality. If dashboards told us everything we need to know, there would be no road accidents.

In this age of measurement and quantification, where druckeresque phrases like ‘what cannot be measured cannot be managed’ are pronounced as self-evident truths, where anything unmetrified is not to be trusted, but everything that is quantified is regarded as empirical and objective, we have a responsibility. I would argue, this is not only to improve the quality of our data, but also to do more to convey uncertainty, to communicate the limitations of our data and statistics, so that decision makers understand that they cannot delegate their responsibilities to a dashboard.

We need to look beyond scientific management and the simplistic rules of Taylorism¹. If we have learned anything from Covid-19, it is that not everything important can be measured. None of the pre-pandemic dashboards or composite indices, such as the Global Health Security (GHS)² Index incorporated the importance of leadership. It's not clear how this can be done, yet it was one of defining success factors in many countries. This is not a condemnation of the GHS, it is easy to be wise with hindsight, but we should learn lessons.

My reflections lead me to conclude that foresight will not be improved by a better dashboard or index. Rather I would argue that there is no short cut for analytical capacity. We need to foster and encourage, creative, analytical capacity.

I would like to offer one final reflection. When I began work as a statistician in the early 1990s, it was understood our mission was to provide evidence to inform decision making. I stress the word inform. It was implicit that decision makers would weigh the evidence, but take into consideration other factors, political, cultural … and so forth. Evidence was just one element in the complex equation of decision making.

Since then, that phrase has morphed into data-driven decisions. To my mind, this is a completely different mission. Data-driven implies data are making the decisions, not just informing them. Taking this change to its logical conclusion, data-driven decision-making removes accountability from politics - the politician can now claim they had no choice, the data made the decision. To an extent, this is already happening; this is where we are going with Artificial Intelligence (AI) and Machine Learning (ML), but it is a noteworthy shift in language. It is a noteworthy change in the role of data; one worth reflecting on.

I would like to conclude with a quote from one of my favourite books - The Tyranny of Metrics by Jerry Muller. ‘Measurement is not an alternative to judgement: measurement demands judgement.’

---

1. Frederick Taylor founded what is known as Scientific Management Theory. He is associated with making routine processes more efficient.

2. The Global Health Security (GHS) Index is the first comprehensive assessment and benchmarking of health security and related capabilities across the 195 countries that make up the States Parties to the International Health Regulations (IHR [2005]). The GHS Index, which is developed in partnership by the Nuclear Threat Initiative (NTI) and the Johns Hopkins Center for Health Security at the Bloomberg School of Public Health, working with Economist Impact.