Judges around the world have made headlines for illicitly using generative AI in their work. But in Pakistan, a large-scale trial of a specially designed AI tool for judges found the technology–together with appropriate training–boosted the number of cases resolved by 6.3% with no obvious drop in the quality of judgements.

With a backlog of 2.26 million cases and fewer than two judges per 100,000 people – compared to 22 in the EU and 8 in Brazil – Pakistan’s judiciary was in sore need of help. So, in consultation with the judiciary, economist Sultan Mehmood, of the New Economic School in Moscow, Russia, and collaborators tested whether AI could ease the burden.

They built a custom tool combining OpenAI’s GPT-4 large language model (LLM) with a knowledge base of nearly 130,000 Pakistani judicial opinions and statutes, to help judges with legal research and drafting judgements. They began offering the tool in 2024 to 1,559 trial judges – roughly half the country’s justices.

“We do find an increase in cases resolved, and we don’t find any corresponding decrease in decision quality,” Mehmood says.

First of its kind

“It’s pretty amazing that he’s able to pull this off,” says David Autor, an economics professor at the Massachusetts Institute of Technology. “It’s not easy to do large-scale field experiments in civil service, but especially where the stakes are so high.” The 6.3% productivity boost is not overwhelming, he says, but it’s credible and likely to improve as the tool is more widely used.

AI tools for judges are already being rolled out in Brazil and India, and prominent U.S. law professor Eric Posner has compared LLM judgments to human judgements in a single case study, but until now there have been no major independent assessment of ongoing judicial use of AI. The new study focused on Pakistan’s trial courts; Mehmood says judges there were enthusiastic from the start.

“They were more techno-optimist than we were,” he says. “The delays are so huge this is something which they thought was worth trying anyway to reduce people’s suffering.”

Some judges were also already using AI chatbots, Mehmood says, but commercial offerings performed poorly on Pakistani legal queries, frequently hallucinating case law. So the team built a tool tailored to the Pakistani context, called JudgeGPT.

They used retrieval-augmented generation (RAG), which allowed the model to query a database of 128,292 Pakistani judicial opinions and 943 statutes. Responses included footnotes linking to cases and laws.

“It turns out that actually the way to fix [hallucinations] isn’t just more intelligent models,” says study co-author Elliott Ash, an associate professor of law, economics, and data science at ETH Zurich in Switzerland. “It’s to attach the models to a tool that can do a search and verify the sources.” However, the researchers do not report hallucination rates.

The team also put 1,197 judges through six 90-minute Zoom training sessions where Ash covered how LLMs work, their limitations, the risk of bias and hallucinations and the importance of verifying outputs. Another 180 judges only underwent general training on technology in legal research, while a final group got no training.

By the time 487 judges had been through the program, the median district saw a jump of 6.3% resolved cases, and the more trained judges in a district, the bigger the effect. Appeal rates also fell slightly, suggesting faster resolution wasn’t leading to sloppier decisions.

The JudgeGPT can be used to surface relevant case law with a simple text query and results provide links to the full text of the related judgements.Sultan Mehmood, Christoph Goessmann, and Elliott Ash

Addressing limitations

The team also assessed the quality of judgements. Having legal experts evaluate large numbers of judgements was infeasible, Mehmood says, so the team asked OpenAI’s GPT-5-mini to choose between pairs of judgements from the same judge before and after training. The LLM chose post-training judgements 59% of the time. Two experienced Pakistani lawyers also evaluated the model’s analysis of 90 judgement pairs. They agreed with GPT-5-mini 70.6% of the time, compared to 73% agreement with each other.

Training turned out to be vital. On average, JudgeGPT-trained judges logged in 56 times and sent 212 prompts over the study period, compared to 10 logins and 25 prompts after generic training. Those who had no training tended to use the tool for around a month and then drop off entirely, Mehmood says. “Just giving people the technology does not necessarily make them use it persistently,” he says.

A 6.3% increase sounds modest, but the researchers calculated that a trained judge was resolving 38.5 more cases a month than the baseline, translating to roughly US $38.50 saved in judicial costs for every dollar spent running the tool. Ash also notes that these figures come from a nine-month period at the start of the trial, and that they’ve since updated both the underlying AI model and the database.

For users the tool has been a lifeline. One participating trial judge, who spoke on condition of anonymity, says the number of cases assigned to them hasn’t dropped below 1,000 in more than a decade. The tool saves significant time, in particular searching for case law and summarising lengthy documents. “For research, it’s just one prompt away, whereas before I had to search for the precedents and laws for hours,” the judge says. “If I have to read 10 pages of a precedent, now I ask JudgeGPT to just summarise it for me and give me the crux, and it does that work in seconds.”

But efficiency isn’t the only thing you want out of a justice system, says John Zeleznikow, professor of law and technology at La Trobe University in Australia. “What they’ve tried to do is be effective, [to] deal with more cases more quickly, and they’re able to do that,” he says. “What’s not that clear is whether what you call the quality of justice is better.”

Zeleznikow saysAI can be useful but only if judges are diligent about evaluating and verifying the output. However, the working paper’s authors found that roughly a fifth of participant’sthe prompts given to JudgeGPT involved what they call “substantial AI delegation” – asking the tool what the best decision is, to produce legal reasoning or write opinions with little input from the judge. On the bright side, training lowered the proportion of inappropriate delegation.

But given that judges are already using AI, Ash says better tools and training are crucial. “There are risks for using these AIs for sure, even with all these safeguards, but at some point you have to just put the judges in as strong a position as you can,” he says. “Have technological safeguards, but then try to encourage the judges not to rely on it too much.”