Eight or so years ago, I was doing some work for a UK law firm. At the time, across the legal industry, there was considerable interest in AI, primarily machine learning, to automate high-volume work such as commercial conveyancing for mortgage lenders. Ultimately, the aim was to reduce costs.
I remember a conversation with one of the senior partners who couldn’t understand why we weren’t able to do more, more quickly. The analogy I used with him seemed to help – “Would you issue a contract that had been translated by Google Translate without human review?”
Eight years on, and I think this analogous benchmark still holds up. The reality of machine translation is that, despite a further decade of iteration in the technology, I doubt any lawyer would still today issue out a machine-translated contract without review.
But with the remarkable tools we have in LLMs, I think the rule holds too; whilst we are seeing plenty of LLM-generated slop being published, I’m not sure that there’s much of the importance of a legal contract that’s getting out into the wild without it being reviewed by actual flesh-and-blood humans.
There’s incredible utility, but that last 10%, or 5% or even 0.1% to get to output that passes this “contract” test could end up simply not being worth the effort. That in turn of course brings up fascinating questions of who will be the next generation of professionals who will be able to review the LLM output if they haven’t had the apprenticeships of writing the drafts.
mmitII 