Use Of AI Hallucinations In Court Cases

As readers well know, I’ve been extensively covering and analyzing a myriad of facets regarding the intersection of AI and the law for many years. You can find my writings not only in my Forbes column but also as posted in Bloomberg Law, ABA Law Journal, The National Jurist, The Global Legal Post, Lawyer Monthly, The Legal Technologist, MIT Computational Law Journal, and so on.

I have extensively examined the repercussions for attorneys who have used AI-hallucinated elements in their legal briefs; see my coverage at the link here and the link here. The situation usually is that an attorney opts to include fake legal citations or fictitious quotations about legal cases, having generated those via the sloppy use of generative AI and large language models (LLMs).

I’ve frequently noted that until the courts impose stiffer penalties, or until lawyers wake up and use suitable double-checking procedures and tools, this phenomenon is going to continue. There is a constant drumbeat of lawyers contending they had no idea that AI could produce bad references or bogus quotes. Though this might have seemed plausible when generative AI first hit the scene, the claims of innocence now seem quite hollow. Most savvy attorneys raise their eyebrows when a caught red-handed lawyer tries to plead ignorance of these matters. It does not compute anymore.

Background On The Happenings

Daily bulletins in the legal community keep highlighting lawyers who have used AI to prepare their court filings and ended up with fake legal citations and false quotations in their documents. These errors in legal briefings can potentially occur due to AI hallucinations. An AI hallucination is when generative AI or large language models (LLMs) such as ChatGPT, Claude, Grok, Gemini, CoPilot, Llama, and other AIs veer into generating fictitious confabulations. For my in-depth coverage of AI hallucinations, see the link here and the link here.

I had long ago predicted that attorneys using AI might get careless in their legal efforts and allow the AI to produce bogus content and not double-check by hand what the AI has generated for them (see my prediction in 2023, at the link here).

The problem for attorneys is that when they submit formal court filings, they are supposed to be responsible for the contents of the filings; thus, if AI has slipped in faked citations or false quotations, the lawyer is likely to be held accountable since they didn’t catch the erroneous content. I say “likely” accountable because judges and courts have been quite lenient so far, overall, and allowed excuses such as “the computer did it”. Attorneys often incur nothing more than a minor hand slap or mild rebuke, asserting that AI is new to the legal beagles and they were caught unawares. Sometimes, the reprimands are accompanied by a modest financial sanction or penalty, which is gradually ratcheting up as these instances continue to climb.

Judges and courts are beginning to lessen their patience and sense of charity in giving lawyers the benefit of the doubt. For the time being, the matter is still generally being treated with kid gloves. The expectation is that lawyers are facing a learning curve, and a bit more time is required before they will be able to properly handle the use of AI. In my view, more than enough time has already passed.

Stop coddling legal pros that are supposed to remain on top of the latest technological aspects when performing their legal services (see my coverage of ABA Formal Opinion 512, at the link here, which was promulgated on July 29, 2024, nearly two years ago!).

Twists And Turns Aplenty

I recently examined the somewhat rare situation of attorneys that fail to detect that the opposing side has made use of AI hallucinations in their legal filings in a court case; see the link here. The question there is whether the attorney that doesn’t catch the mistakes of their opposition ought to receive any sanctions or penalties. There are tradeoffs involved, which I analyzed at length in my posting.

There is another new twist that is garnering attention, namely the exasperating circumstance of both sides of a case relying on AI hallucinations at the same time. You might be shocked to think that this could happen. It’s one thing for an attorney on one side to make such an error, but having both simultaneously do this is astonishing and dismaying.

On the other hand, given that attorneys seem to be ignoring the blaring headlines or otherwise opting to fall asleep at the wheel when using AI, the chances of both sides committing the same type of error are perhaps higher than we might want to admit. Sliding into quicksand is heightened when both parties opt to stand ill-prepared at the same edge.

For my analysis on the situations of both sides relying upon AI hallucinations, see the link here.

Attorneys Explain Themselves

Once a court realizes that an attorney has relied upon AI hallucinations, the natural line of inquiry is for the judge to ask the attorney why this has occurred. An attorney must come up with some explanation. In theory, the explanation would simply be the unvarnished truth. That is the expectation underlying the duties of an attorney to their professional obligations and to the court.

Not all attorneys will necessarily proffer the unvarnished truth when sharing their explanation. Attorneys are quite good at finding nuances. It’s their job and their training. Whether you consider this to be shading of the truth, or merely providing truth that has suitable context, that’s up to you to decide.

One of the most common ways to explain the AI hallucinations is by claiming that the attorney is utterly shocked, dismayed, upset, and chagrined that they made use of AI hallucinations. Golly, they would have never used AI-hallucinated content if they had known that such a possibility exists. As a side tangent, I’ll remind you of my position, which is that it doesn’t matter that AI hallucinations exist, since the attorney, in my view, is entirely responsible for their filings, regardless of how the content was devised.

Anyway, we know that so far, the claim of not knowing of the possibility of AI hallucinations has been a relatively safe and smart way to defend your circumstance. Judges seem to buy into it. The attorney gets to paint themselves as victims. They are victims of AI. They are victims of those who built the AI.

It’s all a ready-made escape hatch.

The Inquiry Can Go Deeper

Courts typically take an attorney at their word. If an attorney says that they didn’t know of the possibility of AI hallucinations, voila, might as well believe them. An attorney is expected to tell the truth. The repercussions for lying to the court can be quite severe, possibly denting a career, maybe even carrying the possibility of criminal conduct depending upon the matter at hand.

But the tide is slowly turning on rubber-stamping the popular claim. A court might not just take at face value the word of the attorney that they don’t know of the possibility of AI hallucinations. This hesitation and doubt means that attorneys going this route will have to up their game. Whether telling the truth or possibly stretching the truth, or outright lying, the courts are starting to press for details.

In that sense, an attorney needs to think of themselves as a witness that is going to testify in a legal matter. The judge might pepper the attorney with questions. If the answers seem inconsistent, the odds are the judge is going to press further. This is something that few attorneys have had to undergo. They are accustomed to being taken at their word.

Let’s discuss what kind of inquiry might occur.

Getting Tripped Up By The Facts

I’ll give you a heads-up example of how an attorney can get themselves into hot water when answering questions on this topic.

One judge asked the attorney whether they had taken any courses as part of the need to comply with attorney CLE (continuing legal education) requirements. CLE is nearly mandatory in each of the states. A specified number of credit hours must be undertaken each year. Furthermore, the credit must be well-documented so that proof of attaining the CLEs is readily available. Without some valid proof, monkey business can arise.

The question seems innocuous. The attorney said they had indeed met all their CLE requirements for the last several years. They were proud of this. The judge then asked for a listing of the CLE credits, along with a detailed description of what was encompassed. Lo and behold, the attorney, now trapped, provided a description of the classes – one of which included the topic of AI hallucinations.

Yikes, nabbed!

You can imagine the immediate backpedaling. The course was a long time ago. The judge noted it was last year. Does that constitute a long time ago? Really? The course might have skipped the topic. The course might have covered the topic, but it was done confusingly. The course did cover the topic, but no practical application was shown, and the attorney didn’t see how it would ever apply to them. Blah, blah, blah.

Do you see how the attorney has boxed themselves in? You would assume that an attorney worth their salt that knows how to examine and cross-examine witnesses would be better prepared and better able to cope with such guffaws. I suppose it is akin to the tale of the cobbler’s children that had no shoes.

The Closing In Of The Walls

The gist is that an attorney ought to think carefully about how they are going to defend their claim of not knowing about the possibility of AI hallucinations. By and large, I would suggest that it is no longer a viable excuse. If the court digs in, you are likely going to be found out.

I will walk you through a series of ways that an attorney’s story can fall apart. Let’s do this on a question-by-question basis.

Here we go:

• Do you read any legal journals, bar association articles, or blogs about the law, and/or any other national or state materials that have ever discussed anything about AI?

If you say yes, the follow-up question is that there is almost certainly some mention of AI hallucinations in that type of reading material, and so how did you miss seeing it? The court might ask for specific content that you’ve read, requiring you to provide links to the content so that the court can inspect the materials to ascertain whether AI hallucinations have been mentioned.

If you say no, namely that you have read any legal or law-related materials, that would certainly raise the eyebrows of the judge. How could that be? An attorney is generally obligated to keep up with the current times. In what manner are they remaining informed about the law?

• Have you completed any CLEs in the last few years, and if so, have any of them covered the AI topic, of which AI hallucinations might have also been covered?

An attorney would certainly have to say yes that they have been completing CLEs. They might profess to not remember whether AI was ever covered. The gig will be up once the judge asks for the CLE descriptions, though an attorney could hope that perhaps providing bland summaries of the CLEs would prevent them from disclosing that any AI topics had been discussed. This form of manipulation is venturing onto thin ice.

• Have you attended any judicial conferences, bar association meetings, or any kind of get-togethers that cover aspects of the law, and if so, was AI ever discussed, including possibly AI hallucinations?

The same rules as above apply to this question – if you’ve not gone to any meetings of any kind, that might seem questionable. If you have gone to meetings, the next angle would be to assert that AI never came up. That seems questionable in this day and age.

Wiggling Like A Caught Fish

An attorney might try to be clever by saying that AI did come up, but that the specific topic of AI hallucinations didn’t arise. This seems to be more reasonable. You are giving just enough of a whiff to seem credible but remaining outside the firm grip of being nabbed.

It is also a likely slippery slope. How could AI have come up in a legal context, and yet the topic of AI hallucinations never arose? By saying that you have read, seen, or discussed AI, especially in a legal milieu, it opens Pandora’s box of suspiciously never having discussed AI hallucinations.

Another attempt would be to use the subterfuge of time -- I don’t remember when I last read or discussed anything about AI. It’s the tried-and-true classic of a loss of memory (conveniently). Hopefully, the judge lets you off the hook at that point. The reality is that modern judges aren’t going to let you get away with the same trickery that they know legal beagles use all the time.

If you can’t seem to remember, please go ahead and look at your papers, logs, calendars, and the like, and get back to me with a detailed list of where and when the topic of AI might have arisen. The aim is to force the attorney into being specific and not allowing them to dance around the matter.

The cat-and-mouse game can keep going, and the attorney is likely digging a bigger and bigger hole for themselves. Their main hope is that the judge tires of the game. If the judge has this firmly in their gaze, it is going to play out until the bitter end.

Claiming Incompetence

There is an entirely different way to interpret the claim by an attorney that they never knew of the possibility of AI hallucinations. I’ve been concentrating on the truthfulness of that assertion. Suppose we take the claim as being absolutely true.

The rub is this. The attorney is essentially admitting that they used a tool to aid their legal work and did not understand sufficiently how to use it. An attorney might try to argue that AI hallucinations are not a material element to learning about the use of AI, but that’s not going to fly. For lawyers, learning about AI almost inherently involves learning about AI hallucinations.

An admission by the attorney that they didn’t know about AI hallucinations has become their own worst nightmare. If they hadn’t used AI, they would be somewhat safe, since they would not need to explain how they encountered AI hallucinations. By using AI, and by admitting that the confabulations were AI hallucinations, they have put a nail in their own coffin.

By the professional obligations of an attorney, you can clearly see in the ABA Code of Conduct that knowing how to properly use a tool is an essential credo. The attorney has possibly violated their professional obligation, as admitted in their own words. The corresponding question is whether they have also undercut their obligation to their client. The client might have a basis for malpractice against the lawyer. It’s a mess.

Expectations Are Rising

A judge might make this proclamation: "Perhaps you genuinely did not know. But the profession expected you to know." Ignorance is no longer the free pass that it once was. Is it sensible that an attorney exercising their ordinary diligence in 2026 is truly unaware of AI hallucinations, and yet opted to use AI as a tool for their legal work?

Something there is out of whack.

The Victorian novelist Robert Smith Surtees said this about lawyers: “There are three sorts of lawyers – able, unable, and lamentable.” Don’t be the last two. Be the first. Get to know about AI, be wary of AI hallucinations, double-check your work, and you’ll never be in the awkward and career-dampening role of having to defend your being naïve about AI.