The new courtroom gotcha? Check whether your opponent left in brainless AI hallucinations, and tattle on them to the judge.

This is precisely how the prestigious Wall Street law firm Sullivan & Cromwell, which boasts clients including Donald Trump, was recently humiliated. On Saturday, Andrew Dietderich, the cohead of the firm’s restructuring group, apologized to federal judge Martin Glenn after it emerged that its filing in a major case was riddled with bogus citations that an AI made up.

“We deeply regret that this has occurred,” Dietderich wrote in the groveling letter.

The errors were caught by an opposing firm, Boies ​Schiller Flexner, in a motion S&C filed in a federal bankruptcy court in Manhattan. In some passages, the filing straight-up misquoted US bankruptcy code, and cited the decisions from other cases that were either inaccurately summarized or misidentified. One of the cases cited by S&C in the motion wasn’t even a case, Boies Schiller Flexner lawyers found.

No doubt terrified of potentially being sanctioned, Dietderich extolled S&C’s rigorous practices and safeguards around AI in the apology letter, which include “training requirements governing the US of AI tools in legal work,” and lamented that its AI policies “were not followed” while preparing the motion.  

“I want to assure the Court that the Firm’s policies governing AI use are both clear and rigorous,” he stressed. 

Dietderich also claimed that the firm undertook “immediate remedial measures,” including launching a review into how the AI hallucinations slipped through the cracks. A corrected version of the filing was later submitted to the court.

Which AI model, or models, were involved in the blunder wasn’t specified. Sources close to the firm told the Financial Times, however, that it has an enterprise license for OpenAI’s ChatGPT.

The gaffe is the latest high profile example of lawyers tripping over themselves by using hallucinating AI tools. The AIs can incorrectly cite or misquote case law and may even invent nonexistent cases, egregious errors that are often overlooked by the lawyers using them. Other major firms have been embarrassed by AI blunders, including Morgan & Morgan.

Judges have been happy to admonish lawyers that leave in AI garbled citations, including by imposing sanctions, and threatening to refer them to the state bar. Some get creative with their punishments: when two lawyers from the firm Cozen O’Connor were caught leaving in erroneous AI material, the judge said they could either choose to be sanctioned, or to write a letter to their law school deans explaining how they screwed up.

More on AI: Things You Told ChatGPT or Claude May Have Already Doomed You in Court