It’s easy to glibly assume that technology is relentlessly getting better. How often do we stop to assess how true that assumption actually is?
I’m currently re-reading John Seely-Brown and Paul Duguid’s 2000 book The Social Life of Information. The second chapter looks at the rise of intelligent agents, or bots, which seems somewhat timely given the recent Siri/Cortana/OK Google developments on smart devices in recent months.
In the chapter the authors check out the relatively simple Sherlock search engine aggregation tool that was part of MacOS back in the day. They perform a very simple search, for the word Knobot (I’m presuming pronounced “know-bot” not “nob-bot” which is a totally different type of intelligent agent…).
They then analyse the results, concluding
We cannot read too much into this outcome of a single search. Sherlock is a relatively crude engine and our single-word search, with no refinements, was itself yet more crude. Our results [of 51 returned web pages] primarily serve to remind us that the Web is a vast, disorderly, and very fast-changing information repository with enormous quantities of overlapping and duplicate information and that all its catalogues are incomplete and out of date. That in itself, however, is a useful lesson when we consider whether the Web, self-organized but patrolled by bots, might be an alternative to libraries. Automatic organization, like automatic search, has its perils.
So what of today? In Brown and Duguid’s original analysis showed that the 51 responses contained masses of duplication, missing links and just stuff that wasn’t relevant. a quick Google search (try it yourself) results in 100 times the results these days – over 5100 documents on the web indexed by Google are returned on that search term. Yet the first few pages show that the results are possibly even less relevant these days with the term appearing to have been picked up by a musician in Brazil amongst other things.
What’s really fascinating, though, is that “Sherlock’s objective search, conducted on relatively friendly territory, hints at difficulties that might lie ahead in sending out a Sherlock-like surrogate with requests to “get me a knobot,” let alone to the the “best”.
So what happens if I search Google to “get me the best Knobot”? The results – all 36,500,000 of them – are gibberish. References to whisky, to knobs, to nobs, to rap music. Nothing about intelligent agents. As with the first experiment, we can’t read too much into this. But I was very surprised how, if anything, the power of our modern systems seems to have taken us backwards in this one very limited case.