What this law firm learnt from experimenting with AI

“While [the blind study] was quite consistent in terms of what produced the better outputs, which was still the humans in all but one case, the thing that was really interesting is that [the assessors] could not consistently identify whether the output was from a human or from the technology.”

The blind study required 16 lawyers to produce a summary of a well-known case without using AI, before asking them to produce three more versions of that summary with three different generative AI tools. (Ashurst declined to name the AI tools but described them as a general productivity-based tool, a general enterprise-level large language model and a major legal industry-specific large language model.)

A common assessment framework was used across each experiment to ensure comparability of data and included questions to evaluate the output’s accuracy, usability, efficiency, trust, value and experience.

Ashurst partner and chief digital officer Tara Waters said the lawyers were allowed to adjust and amend the AI-generated output to more closely reflect the way in which the technology might be used in future. But this human oversight failed to bring the quality of work produced by AI to the same level as that produced entirely by humans in all but one case.

Ms Waters said this suggested the AI tools were less useful at writing the first draft than at helping lawyers produce it by assisting them in collating and better understanding the relevant information.

The trial, parts of which are still ongoing, also involved small group experiments and feedback sessions and surveys.

In the group experiments, lawyers were observed via video call using AI to perform certain tasks and simultaneously asked questions about what they were doing and the accuracy and quality of the AI tool’s responses.

One experiment involved locating clauses across a range of similar agreements to compare and contrast differing precedent positions. Another used AI to interrogate public authority policies to create a table of key points.

Ms Goodier said the trial’s findings suggested the uptake of AI could bring an end to the billable hour by forcing law firms to charge for the value of work completed rather than the time taken to do it. She said it also had implications for training, as AI would most likely automate the tasks that today’s senior lawyers were trained on.

“Some of those tasks are actually incredibly valuable to help you understand how to navigate and interpret a contract, how to synthesise a case or a piece of legislation for a client,” Ms Goodier said.

“If we are now getting generative AI to do that first draft for us, how do we think now about training our lawyers, and how are they going to develop the knowledge that we all developed through doing that work manually?”

Asked whether the potential time savings offered by AI paved the way for fewer lawyers or lower fees for clients, Ms Waters said the technology was not yet cheap enough or sufficiently advanced to affect the firm’s fees or recruitment policies, and was unlikely to reach that stage for years.