Login
AI v human lawyers, in mock trial form
BigLaw firm Lander & Rodgers recently undertook an experiment in which a senior associate was pitted against artificial intelligence in a courtroom setting, which offered insights into the future of advocacy.
Grace Robbie
Speaking on a recent episode of The Lawyers Weekly Show, Jeanette Merjane, senior associate and innovation manager at Lander & Rogers, and Kenneth Leung, a digital economy lawyer at Lander & Rogers, reflected on their participation in a recent mock trial at SXSW Sydney, which provided a revealing look into how AI might reshape legal practice in the coming years.
Merjane emphasised that the live experiment provided an intriguing opportunity to test and compare and assess the capabilities of AI within the legal field.
“The experiment was a unique opportunity to look at the evolving role of AI in the legal profession. So we ran a live mock trial between a human lawyer, being myself, and an AI-assisted layperson,” Merjane said.
During the mock trial, Merjane explained that two audience members were randomly chosen to participate in the courtroom simulation, but with distinct reparations: one was assisted by a human lawyer – herself – and the other represented themselves using the AI platform.
“We had two defendants who were randomly chosen from the audience, and both were arguing the same case, but with a twist. So one had me as their lawyer, and then the other was self-represented using an AI platform, NexLaw, with some help from Kent,” Merjane said.
Merjane noted that both she and the AI technology were presented with identical evidence and took turns presenting their cases, thereby ensuring a fair and equitable process.
“Both myself and the AI platform were given the same evidence. So there were witness statements, police reports, mobile phone records, driving records and photographic evidence. Then we took turns to present our opening statements, submissions and closing statements,” Merjane said.
Audience members were invited to engage with the proceedings and cast their votes on the outcome, though Professor David Lindsay, a UTS academic, made the final decision.
In the face of opposition from the AI, Merjane disclosed that the judge ultimately ruled in favour of the human lawyer, confirming the efficacy of traditional legal expertise in the courtroom and underscoring the significance of human lawyers.
“In the end, both the audience and the judge agreed that the human lawyer came out on top. So I continue to have a job, which is excellent,” Merjane said.
Leung underscored that the primary aim of the mock trial was to illustrate the strengths and weaknesses of human lawyers and AI technology.
“The aim was to demonstrate what both humans and AI can do. What are the strengths and weaknesses of each case? Where is it that the human shines? What are the capabilities of AI?” Leung said.
“The platform that we used is not intended to be used by an individual; it’s for law firms. But we wanted to see, well, where can that really take us in the future?”
While the AI system demonstrated commendable performance in its overall presentation, Leung expressed concerns regarding its capability to understand and effectively utilise essential details for the case.
“AI definitely had some hiccups, like referencing the incorrect legislation, despite us asking it to be New South Wales law, and also not addressing each of the key elements of the offence.
“It really does show why it’s crucial to double-check the AI outputs and our ethical obligations as lawyers,” Leung said.
Leung also pointed out that another area in which the AI system fell short – an issue raised by both the mock trial’s presiding judge and the audience – was its failure to reference crucial evidentiary elements and details.
“Where it really did fall short, and what Professor Lindsay acknowledged and some members of the audience also noticed, it did fall short on the details, and it didn’t focus on all elements of the evidence presented as well, which an audience member came up to me afterwards and queried about,” Leung said.
“Whilst it had great language and was great as an overall high-level approach, it didn’t go into that detail, and it didn’t step through all those elements, which was what the court was looking for as well.”
