I love BBC’s “More or Less – behind the statistics”. It’s compulsory listening, I think, for anyone in the data business. Constantly pointing out how statistics and be misused, misunderstood or just fabricated, often leading to game-changing reactions driven by poorly informed media and hence the public (and politicians!)
Recently they put forward their 5 key tips to avoid being misled by catchy statistics. I thought I’d summarise here because they are useful to all of us. Useful to agencies presenting data to clients, and to recipients assessing data they are receiving.
- Observe your feelings. It influences your belief in the data. We tend to reject data that doesn’t sit comfortably with what we already know. It’s called confirmation bias. We are also affected by what we would like to be true. We are more likely to reject data that is awkward or problematic for us.
- Understand the claim properly. Eg “The world’s 15 largest ships emit as much pollution as all the cars in the world”. One could be quite shocked by this. In reality, this is about only sulphur emissions and only applies if the ships used the worst possible polluting fuel and the cars all used the best. So, the headline whilst not incorrect is quite misleading.
- Get the backstory. Who has put forward the data, and what is their agenda? Eg “chocolate cures Dementia”. Most data is presented by people with some kind of goal or purpose. Particularly in the media. Market researchers should, of course, be totally objective. But where money is changing hands or at stake it’s pretty hard to avoid at least some kind of selectivity, right?
- Put things into perspective. Is 7m a big number? Is it likely that 7m disposable cups are thrown away each year in the UK? Given this is proportionate to one cup per UK adult per week. It’s sort of reasonable. Better though to ask how it looks compared to other things or perhaps is it up or down on last year? If it doesn’t pass some simple relative evaluations it could well be plain wrong.
- Embrace imprecision. Does it really matter if the answer is 3.1 or 3.2? Focus on the main message, don’t get distracted by too much detail. Better to be roughly right than precisely wrong (particularly if this precision is wrong!) And if detail comes with a high price tag.
- Finally, an overarching point, be curious. First, seek to understand. Be willing to explore, with a sceptical mind any data you are presented with. Facts matter, but they are often puzzles rather than what you see at first sight.
Any you’d like to add to this? Share your thoughts on Twitter or LinkedIn.