Blueprint Prep was one of the first to analyse the LSAT results, which showed that the ChatGPT-3.5 series was unable to meet the typical scores required for admission to a top 14 law school in the US.

Gene Suhir, LSAT academic manager at Blueprint Prep, said that the bot’s main failures were in relation to applying logical and critical reasoning and that it had an inability to distinguish essential information from superfluous material.

“We know that the LSAT is designed to measure a student’s analytical reasoning, critical thinking, and reading comprehension skills, which are essential for success in law school,” he said.  

“These skills can be significantly strengthened, but to do so requires leveraging proven strategies and top-scoring instructors who can help students get into the mindsets of the test makers. This form of LSAT test prep not only enables the student to process information like a lawyer would, but it’s been proven that strengthening reasoning skills via LSAT test prep can help wire students’ brains to think more like a lawyer. This is not the specific kind of reasoning that ChatGPT is innately useful for, although it can learn these skills.”

However, ChatGPT also had the ability to improve its LSAT test scores with a new update; the 4.0 model indicates that the AI tech is able to “steadily improve its analysis of LSAT questions,” according to Mr Suhir.

“When analysing ChatGPT’s initial LSAT results, we found that it frequently failed in its application of logical and critical reasoning, and was often unable to distinguish essential information from superfluous details. However, it demonstrated — with an updated system — that it made great strides in applying principles to a set of facts, rendering a supported judgment, and processing dense blocks of text, which are the building blocks of skills for legal work,” he told Lawyers Weekly.  

“This being said, it’s difficult to foresee a team of ChatGPT lawyers in a courtroom because the chatbot still lacks vital legal reasoning skills and the expertise needed to practise law.”

VIEW ALL

The LSAT is scored on a scale of 120 to 180, with 151 being roughly average. Students usually need a typical score in the 170s to be admitted to one of the top law schools. On the two tests it took, ChatGPT scored a 148 (37th percentile) and a 157 (70th percentile).

However, Terri Mottershead, executive director of the Centre for Legal Innovation at the College of Law, said it’s worth noting that these types of tests may not be around forever for prospective lawyers.   

“First, it’s worth noting that the use of LSAT (and other standardised tests) in admissions for new students is not without controversy. We may actually see this abolished in August 2023, depending on how the ABA House of Delegates and its Council of the Section of Legal Education and Admissions to the Bar meetings go in/around August 2023,” she explained.

“Second, like every large language model, ChatGPT works from the data available to it and, specifically for ChatGPT, the prompts (questions) it’s asked. It’s not a human and it’s not a test taker. It connects the dots, i.e. helps assemble information and do tasks. It’s never going to demonstrate uniquely human traits like critical thinking, empathy, and legal judgment (expertise) — these are the capabilities lawyers will increasingly rely on to add value.”

The technology is also still in its infancy, meaning that it’s still too early to say just how much or how little it will disrupt the legal industry.

“ChatGPT isn’t the first time AI is being applied to the legal field; artificial intelligence has aided legal research into relevant case history and deposition questions. But how much actual legal reasoning will the AI chatbot be able to employ as it progresses? That’s largely unknown right now, and it will be interesting to see to what extent (and at what speed) the reasoning capabilities develop. Interestingly, there was an LSAT passage back in December of 2006 that spoke to AI being helpful to aid legal research, but falling short in providing expert legal advice,” Mr Suhir added.

“This was for two reasons: Inputting rules hasn’t been effective enough to demonstrate critical or creative thinking since laws are often purposefully written to be vague, and training the AI chatbot based on precedents hasn’t been effective because ChatGPT has had difficulties determining which precedent to use.

“These two limitations of AI were already on the minds of lawyers at least 17 years ago, and even with the great strides in technology since then, ChatGPT has not yet proven to us that it has overcome these two obstacles. Over the years, technology’s role in the legal field has grown and evolved, and if ChatGPT starts to close the gap between its reasoning skills and those of a human, it’s quite possible that someday, jurists will look back on the launch of ChatGPT as a watershed moment.”

But while it may have limitations, Ms Mottershead emphasised that AI is going to completely redefine a number of sectors, including legal.

“ChatGPT or more broadly, generative AI (GAI) is the future. It is going to redefine everything about legal work — what it is, who does it, when, where and how — and we will continue to see legal roles contract, expand, and new ones evolve, too. ChatGPT is already prompting us to ask and answer what it means to be a lawyer. It promises to assist, if not close, the very wide access to justice gap,” she opined.

“As GAI is customised and has access to internal client data, its accuracy will improve, as will our comfort level in using it inside and outside our practices. There are huge privacy, IP and potential data bias issues to navigate here along with ethical ones too, but I’m confident we will work these out. It follows that different capabilities are needed now, during the transformation, and afterwards too.”

Legal educators need to be prepared for this, as well as legal employers, Ms Mottershead added.

“Legal practice will become even more multidisciplinary, multigenerational, and multicultural and with an emphasis on working collaboratively versus in silos and in competition. Legal educators need to be all over this and prepare students for this reality by working together on the solutions that run the breadth and depth of the whole legal education continuum,” she said.    

“We are starting to see deep change at a scale and pace that we have not seen before in the legal ecosystem. It will take a toll. A laser focus on mental health and wellbeing [and] support to help us adapt and manage ongoing change will be critically important here. We need to proactively place the human bits at the centre of this tech/AI driven change.”