Skewed questioning

“Back in the day” (a phrase I feel it’s inevitable I will get to use more and more as I get older) if you wanted to find something out about a group of people, you’d ask them through a survey. The process was riddled with issues: people generally needed some sort of incentive to complete the survey and that would skew the results; questions would be interpreted by different people in different ways, and that would skew the results; a sample representative of the while group being investigated would be hard to select, and the results would skew the results. But most of all, the whole darn process of administering a survey face-to-face, on the phone our by post was remarkably expensive, and collating and interpreting the results even more so. And that expense kept things under control…

In August I conducted a survey of my own… surveying the number of times I was asked to complete surveys, either at work our in my personal life. I gave up counting at 30 requests (and that was the point at which I vowed, unless the objectives of a survey were completely transparent to me up front, I would no longer complete them).

Online survey tools, email and databases have effectively reduced the administrative cost of social surveys to nothing, and the increasing use of inter- and intra-organisational metrics about customer satisfaction that can only be calculated by surveying have likely drowned the practice.

I can’t be the only survey refusnik, because a whole new world of social research seems to new opening up… the analysis of social media data. Whilst I have no doubt that Tweet steams and Facebook updates must be interesting to analyze, I’m not sure that they will offer a replacement for the old world of surveying now seemingly broken by volume.

The people who express opinions are skewed as a sample in two fundamental ways. Firstly, because they are the sort of people who by definition talk about stuff on social media; whilst the world’s fourth biggest country might now be Facebook (or whatever stat its membership has now reached), only a relatively small proportion seem to be particularly active. Secondly, people expressing an opinion on social media are likely to be expressing a more extreme view than the norm because “Got on a train and it ran a couple of minutes behind time for most of the journey, but ended up a minute early” is not the stuff or Facebook or Twitter.

Social surveys seem to be going the same way as penicillin… A valuable tool for society that had been wasted through overuse and irresponsible prescription. I’m not sure that analysing Twitter streams and the like will be much more than the metaphorical equivalent of a switch to homeopathy.