Big data, little questions, cognitive biases

Nearly 17 years ago I was a member of a team that implemented the BBC’s first data warehousing project. It gathered information from a number of operational systems, and then using a combination of products provided reports about the profitability of various products that the commercial wing of the organisation produced and sold.

Technically it was a reasonable success – we learned a lot (I was responsible for the user experience and user interface), and the legacy of those services I believe still exist in pockets these many years later. From a business outcomes point of view, however, the project’s success was somewhat questionable at the time. The Finance Director had wanted to affect a change in the sales organisation to get them thinking less about volume and more about profitability. The data warehousing service was seen as a way to make that change happen, but it became something of a lesson of how expecting technology to change organisational beliefs and behaviours is a fools game. The fact that many of the sales teams continued to be bonused on revenue made the change even less likely. More great learning…

I’ve been thinking more about those days of data warehousing as I see the relentless coverage of the “big data” wave of technology hype. The data processing we were doing back in the late 1990s would probably be able to be processed on my smartphone these days. But whilst the processing power available to us has grown exponentially, I’m not sure that the data sources available in many sectors necessarily has. The sales data generated back then (all wholesale) would I imagine be little different today. There might be many sources of data available in addition, but I’m not convinced that they would make much difference.

I’ve got no doubt that the abilities to store and process vast volumes of data that we have today in comparison to the late 1990s are advancing certain fields (genome research, for example) exponentially. But the widely distributed view that “big data” is somehow going to revolutionise decision making in every field is, I believe, in need of some tempering. Here are four reasons why:

1) data-driven decision making hasn’t had the impact that data warehousing promised

Data warehousing and decision support systems have been in common use for fifteen years or more. And yet the core data in most organisations is still either spreadsheet- or (even worse) PowerPoint-based. The drive for “dashboards” in the name of simplicity has resulted in many people in many organisations being further away from data than ever before. We see “traffic lights” – mostly selected on the basis of human not machine decision – and that doesn’t lead me to think that we’re all going to suddenly immerse ourselves in big data.

2) it’s not about big data, it’s about big questions

I wish I had been paying more attention when I first heard this quote, but I wasn’t so I can’t tell you who I heard it from. It’s a key point, though – it’s not about the data, or the volume of the stuff. It’s knowing what questions to ask of it. And that takes us back to point 1 – if most organisations couldn’t do much with a data warehouse, what is there to make us think that even more data is going to improve matters?

3) evidence-based decision making is incremental – and incremental decision making isn’t what you need in times of disruption

We are in disrupted times. To lesser or greater extents depending on the industry you are in and the hype you decide to believe. But the impact of the Internet, consumerised and commoditised technology, globalisation and changing economic patterns after the banking collapse all combine to make it a different world. In disrupted times, you need to innovate – and innovation is really, really hard if you shackle yourself to make decisions surrounded by the information of past performance. Nothing in the data warehouses of the, say, music industry could have helped them avoid the impact of digitisation of their business. Less than helped, the data of past performance could quite possibly give unrealistic comfort in one’s position until it became too late.

4) very few things are objective – and human beings suffer terribly from confirmation bias

And this is the killer. We are subjective beasts, and we work in subjective, human-constructed organisations. No measure is objective, and through the psychology of confirmation bias, we seek out the numbers and the data that supports our position – not the other way around. More and more data just gives us greater and greater ability to find the data we need to support our position.

Is this all doom and gloom? Far from it. There are things we can do with the vast reserves of storage and processing we now have available to us to find cures for diseases, new planets, superconductors and who knows what else. But my hunch is that past experience shows that decision making – something that we humans are both excellent and terrible at in equal measure – will probably remain with us for quite some time to come. The big data hype is probably mostly the usual wave of silver bulletry that tech companies love to spin.