The legends of Big Data

On Friday I was at a presentation given by one of the big IT vendors on the subject of Big Data and analytics. During the session the story was told of how US retailer Target had got so good with predictive analytics that they had infuriated the father of a teenage girl by sending her marketing materials for products related to pregnancy.

What had happened, so the story goes, was that through the analysis of purchasing history, key shopping traits of a pregnant woman had been identified, and the marketing focus to that household was accordingly changed. Furious father a few weeks later visits the local Target store, tail between legs, to reveal that the teenage girl was pregnant. This story brought to you by Big Data.

In the session, I noted how I was beginning to smell a rat. This story has been doing the rounds for a few years, so surely now if it was the case then we’d all by now seeing the results in much more (or even over-) focused marketing materials arriving on our real and virtual doorsteps. Maybe Target had just got lucky? Maybe it hadn’t happened at all? A few words tapped into Google and I found that the story’s substantiation is at best a bit flakey.

Ironically sitting alongside me was a chap from the Met Office. If there is one organisation in the UK that I’d say is almost certainly using Big Data at the core of its activities, it’s the nation’s weather forecasters. Where are the Big Data stories from the world of meteorology? Well, I’m sure that they might exist, but I reckon that “7% more accurate weather on a 72-hour window of prediction” or some such isn’t a compelling enough narrative in comparison to the furious father of a pregnant teenager.

And in that, you have the key to understanding why so much of the Big Data hype is hyperbole. Big Data and analytics demand a model of human behaviour based on evidence-based logical decision making that just isn’t the way we operate. We are emotional creatures. We make decisions on our gut instincts a lot of the time, and seek out data to justify our decisions, not to inform. When we sell the concept of Big Data we turn to an emotional narrative which with a little bit of Internet investigation seems to be unsubstantiated.

Twenty years ago the legend of Beer and Diapers was the narrative used to sell the concepts of Data Warehousing. Today it’s the teenage pregnancy. Both myths. Both poorly substantiated and discoverable as such with meagre research. Both demonstrating how little evidence actually fits into the ways in which we operate as human beings, but how important stories are to how we operate.