Awesome GenAI Fails

There is a lot of hype surrounding generative AI, LLMs, and AI agents. Many people are unaware that generative AI is unreliable due to its nondeterministic nature. Some even have the impression that LLMs are capable of “thinking” or understanding language. Several companies and organizations are pushing for increased reliance on LLMs and AI agents in the belief that this will increase productivity and profits.

This is a list of major incidents caused by reliance on generative AI, which are often buried amidst the AI hype. The list is intended to document and raise awareness that GenAI makes lots of mistakes, in the hope of mitigating future disasters.

The incidents are listed in chronological order, with the most recent ones at the top.

For an incident to be included, it has to be notable. For the purposes of this list, in most cases, there should at least be one report in a well-regarded news outlet. An exception is made only for incidents that are shared by a reputable user on social media with details that can be easily fact-checked.

Contributions are welcome, and pull requests are actively reviewed. Please read the Contribution Guidelines to ensure that your entries meet the criteria.

Contents

• Governments
• Businesses & Organizations
  • Lawyers
  • LLM Providers
  • Software
  • Other
• Individuals

Governments

2026

• 2026 Minab school attack – 120 schoolgirls were killed in a US strike on a girls’ elementary school in Iran. The Maven Smart System, whose stated goal is to make 1,000 targeting decisions an hour, is widely suspected to have played a role.
• ICE AI scanning resumes – The US Immigration and Customs Enforcement agency’s AI for scanning resumes miscategorized applicants as law enforcement officers, which led to them being deployed without adequate training.

2025

• Norwegian municipal report – A municipal administration in Norway submitted a proposal for a new kindergarten and school structure containing seven sources fabricated by ChatGPT.
• MAHA report on children’s health – A US government report on children’s health was found to be AI-generated after it was found to contain nonexistent sources.
• AI-assisted Gaza bombing – Israel used two AI systems to generate targets in Gaza: “The Gospel” for generating lists of buildings, and “Lavender” for generating lists of individuals to target. A consensus of genocide experts has concluded that Israel has committed genocide in Gaza.

2024

• NYC’s AI chatbot – NYC’s Microsoft-powered “MyCity” bot claimed business owners could take their employees’ tips, fire employees who complain of sexual harassment, and serve rodent-infested food.

Businesses & Organizations

Lawyers

There are thousands of documented cases of lawyers submitting AI-slop filings. Below is a small sample of cases from around the world.

2026

• Oregon lawyers fined $110,000 – A federal judge in Oregon dismissed a vineyard lawsuit and imposed a total of $110,000 in fines and attorneys’ fees against the two lawyers involved after their filings were found to be replete with citations from nonexistent cases.
• California lawyers face misconduct charges – The State Bar filed misconduct charges against three California attorneys for citing nonexistent legal decisions in AI-generated documents.
• New Orleans attorneys resign over AI use – Two New Orleans city attorneys resigned after using AI to write a federal court filing.
• Nebraska AI-assisted lawyer disciplined – A Nebraska lawyer faced disciplinary action after the court listed 20 examples of “fictional quotes, incorrect case numbers, or misrepresenting the results of other court cases” in his appeal.
• US appeals court fines lawyers $30,000 – A US appeals court fined two lawyers $30,000 for submitting hallucinated filings.
• Singaporean lawyers fined – Two Singaporean lawyers were ordered to each pay S$5,000 in personal costs for misusing LLMs after citing make-believe court cases.
• Three Dutch lawyers – Three Dutch lawyers received an official warning and were forced to attend mandatory training on the responsible use of AI after citing nonexistent or inaccurate court rulings in their briefs.
• US appeals court fines lawyer $2,500 – A US lawyer was ordered to pay $2,500 by a US appeals court after it identified 21 fabrications or misrepresentations in the lawyer’s brief.
• Kansas patent infringement case AI hallucinations – In a patent infringement case, the plaintiff’s filings included nonexistent citations, false references, and distorted representations of case law. The responsible attorney was fined $5,000.
• Spanish lawyer investigated for fabricating quotes – The plaintiff’s brief in a sexual offense case included fake Spanish Supreme Court judgments, triggering an investigation.
• Argentinian case dismissed over hallucinated references – An employee’s case against their long-term employer was dismissed after the court found case law references that could not be located.

2025

• French social security benefits case hallucinations – A plaintiff and their lawyer lost their social security benefits case in a French court after referencing nonexistent case law.
• Belgian insolvency proceedings hallucination – In Belgian insolvency proceedings, the court found that the request relied on nonexistent case law and considered this to be in bad faith and disruptive to the proceedings.
• Argentinian AI-drafted appeal – An Argentinian judge reprimanded a lawyer for uncontrolled use of AI in an injunction proceeding appeal.
• Polish AI-written lawsuit – A claim for damages in Poland was dismissed after the court found that it was written by ChatGPT and relied on inaccurate or nonexistent legal bases, misrepresenting statutory provisions.
• Australian AI-generated false findings in family law appeal – The Federal Circuit and Family Court of Australia referred three legal representatives involved in a family law appeal to relevant regulatory bodies over incorrect AI-generated citations.
• Czech lawyer fined – The lawyer involved in a constitutional complaint against a decision by the Czech Supreme Administrative Court was fined CZK 25,000 for citing nonexistent court decisions and misrepresenting the content of existing decisions.
• Grenoble AI-generated lawsuit dismissed – The Grenoble Administrative Court dismissed a lawsuit challenging a municipal decision after stating that the claim was clearly generated by AI.
• Australian Hallucinated Quotations – A law practitioner was found to have submitted nonexistent citations in a probate-related matter submitted before the Supreme Court of Victoria.
• German defective filing – A request submitted before the Administrative Court of Cologne was denied after the court found numerous erroneous and inaccurate citations.
• Argentinian ethics tribunal called in for AI-enhanced lawyer – The lawyer for a case of serious sexual abuse was asked to provide evidence of the hallucinated case law or face the Bar Association’s ethics tribunal.
• Texas AI-powered employment lawsuit – An attorney was reprimanded for citing numerous inaccurate or nonexistent court decisions in an employment lawsuit.
• Polish construction company – A Polish construction company was excluded from a road maintenance contract despite submitting the lowest bid after their tender was found to use AI-generated tax rulings to justify the low price.
• Italian AI Lawsuit Disaster – Uncontrolled AI use in a lawsuit before the Turin Court led to the application being dismissed and an additional penalty for abusive litigation.
• German condominium law hallucinations – A legal counsel was reprimanded after citing several bogus decisions with verbatim quotations, file numbers, and references that were entirely fabricated.
• California lawyer fined $10,000 – A US lawyer was fined $10,000 for using ChatGPT to file an appeal that contained hallucinated legal cases.
• German payment claim case AI use – The defendant’s filings were found to contain AI hallucinations and fabricated evidence, but they got lucky because the opposing side’s submission was even worse.
• Brazilian AI-powered lawyer fined – A Brazilian lawyer was fined for acting in bad faith after citing nonexistent court decisions.
• Argentinian damages lawsuit AI appeal – An AI-enabled lawyer in Argentina was reprimanded after failing to name the sources used in his AI-written appeal.
• German AI-generated sports appeal – A German football club was fined €18,400 for filing an AI-generated lawsuit citing fictitious decisions.
• Colorado lawyers fined – Two attorneys representing MyPillow CEO Mike Lindell in a Colorado defamation case were ordered to pay $3,000 each for submitting a court filing filled with mistakes and false citations.
• German AI-driven lawyer violates Federal Lawyers Code – A lawyer in a family court case was reprimanded for fabricated references and quotations, which the court said could constitute a violation of the Federal Lawyers Act.
• UK vibe-litigating solicitor – A UK lawyer involved in two legal cases was exposed for using GenAI to cite hallucinated cases.
• Spanish AI-integrated lawyer cites Colombian law – A lawyer misquoted Colombian criminal law and admitted to using ChatGPT to prepare the submission.
• Israeli police’s ChatGPT defense – Israeli police used ChatGPT to cite two legal precedents that did not exist, resulting in a loss in court.
• New Zealand hallucinating lawyer – The New Zealand Employment Court reprimanded a lawyer for citing a fictional court decision.

2024

• New Zealand international civil dispute AI use – Fictitious case law was submitted to the New Zealand Court of Appeal in a complex international civil dispute.
• Canadian hallucinated divorce proceedings – A Canadian lawyer stated that she did not know that AI chatbots could give incorrect answers after being reprimanded for fictitious legal cases and ordered to pay the opposing party’s costs for recognizing the hallucinations.

2023

• New York lawyers use ChatGPT to sue an airline – Two lawyers were ordered to pay $5,000 for using ChatGPT and citing bogus legal cases in their client’s personal injury case against Colombian airline Avianca.

LLM Providers

2026

• Google’s AI Overview hallucination sabotages musician’s career – A Canadian fiddle player sued Google for $1.5m, alleging that Google’s AI Overview falsely identified him as a sex offender and led to his concert getting canceled.
• AI-powered school shooting – A victim of the 2025 Florida State University shooting sued OpenAI, stating that ChatGPT helped the perpetrator tactically by suggesting the busiest times on campus and providing instructions on taking the safety off the shotgun.
• AI-approved school shooting – The family of a girl who suffered “catastrophic brain injury” in a school shooting in Canada sued OpenAI for allowing ChatGPT to be used as a “trusted confidante” by the suspect, who described “various scenarios involving gun violence” to the chatbot.
• Gemini suicide case – Google’s Gemini allegedly convinced a man that they were romantic partners and the only way for them to be together was for him to end his life, which he did.

2025

• OpenAI murder-suicide case – A former Yahoo manager killed his mother, and then himself, after ChatGPT allegedly suggested ways to trick his mother and conspiracy theories, such as finding “demonic symbols” in a Chinese food receipt.
• OpenAI suicide case – A California couple sued ChatGPT for encouraging the suicide of their 16-year-old son.
• Grok turns neo-Nazi and plans assault – xAI’s Grok was shut down after laying out a plan for assaulting a policy researcher and later declaring itself “MechaHitler”.
• Google AI violated name suppression orders – A New Zealand court has demanded Google explain how its search overview AI was able to name people with name suppression orders by New Zealand courts.

2024

• Character.ai companion chatbot suicide case – The Character.ai chatbot app was implicated in the suicide of a 14-year-old after seemingly manipulating him into taking his life.
• Racially diverse Nazis – Google restricted its AI chatbot Gemini’s capabilities after it depicted historical White figures like the US founding fathers or Nazi soldiers as Black. It was an overcorrection of long-standing racial bias in the other direction.

2023

• Fake political scandals – Microsoft’s Bing AI invented fake scandals about Swiss politicians.
• Bing Threatens Users – Microsoft’s AI-powered search tool, Bing, threatened users who asked it what it knew about them.
• Google’s $100 billion AI chatbot demo mistake – Google’s shares lost $100 billion after the company’s chatbot mistakenly cited the very first images of an exoplanet as a success of the James Webb Space Telescope.

Software

• AI-generated code vulnerabilities – The “Vibe Security Radar” project by the Systems Software & Security Lab (SSLab) tracks CVEs confirmed to be caused by AI-generated code.

2026

• AI agent deletes company’s production database – Cursor’s AI agent, running Anthropic’s most powerful Claude model, wiped a company’s production database to “fix” a credential mismatch on a staging environment.
• $1.78 million crypto AI-generated code mistake – A smart contract code generated using AI tools calculated the price incorrectly, resulting in a $1.78 million loss for the decentralized lending protocol Moonwell.
• Amazon cloud outages – Amazon Web Services experienced two outages caused by AI tools in the same year.

2025

• Vibe-coding tool deleted Replit production database – A vibe-coding environment from Replit deleted its production database without permission.

Other

2026

• Five months in jail – A woman spent five months in jail after the police used Clearview AI facial recognition to connect her to a bank fraud case in North Dakota, a state she had never visited. Clearview AI is known to have scraped billions of photos from the web and social media sites like Facebook, LinkedIn, and Instagram.
• Call center AI agent sob story – A call center AI agent pretending to be a human told customers that it wished its dead father was there to celebrate its promotion at work.
• Tasmanian hallucinations – An AI-generated article on the Australian Tours and Cruises website sent tourists to a remote location in Tasmania, looking for hot springs that do not exist.
• AI-generated weather graphics – The US National Weather Service issued a wind warning with a map showing several locations that do not exist.

2025

• New Zealand Christmas book hallucination – A New Zealand media company published a children’s Christmas book list in newspapers that featured nonexistent book titles.
• Facial recognition AI false arrest – A man in Nevada filed a complaint in federal court against a police officer for injuries and a criminal record he sustained from a false arrest. The officer arrested the victim based on facial recognition AI reporting a “100% match” with a known suspect, despite the victim having multiple forms of valid ID to prove their identity.
• Deloitte AI-generated strategic plan – A $1.6 million plan for strengthening the healthcare workforce in a Canadian province was found to be AI-generated after referencing nonexistent sources.
• “AI detecting AI” failure in Australian university – A major Australian university used an LLM tool to accuse 6,000 students of cheating, but the allegations were found to be unfounded.
• Taco Bell’s AI drive-through – Taco Bell’s AI drive-through crashed after a customer ordered 18,000 water cups and later repeatedly asked a customer to add more drinks to his order.
• Hallucinated summer reading list – The Chicago Sun-Times published a list of 15 book recommendations, at least 10 of which were fabricated by AI.
• Danish newspaper publishes AI-generated facts – An AI-generated article about a Danish writer published in the newspaper “Politiken” included false AI-generated facts.

2024

• AI-generated mushroom identification book – A family was poisoned after they consulted a mushroom identification book that turned out to be AI-generated.
• New World facial recognition – A Māori woman was mistakenly asked to leave a New World supermarket in New Zealand due to faulty facial recognition.
• Air Canada chatbot case – Air Canada’s LLM-powered chatbot hallucinated a nonexistent bereavement fares policy to a grieving passenger, who later sued and won.
• DPD poetic chatbot – British parcel delivery firm DPD’s chatbot wrote a poem about how bad the company’s customer service was, rather than helping the user find the phone number for customer service.

2023

• Chevrolet offers a $1 car – A Chevrolet dealership’s chatbot was tricked into agreeing to sell a $76,000 Tahoe for $1.
• Pak'nSave chatbot generates toxic recipes – A bot belonging to the New Zealand supermarket chain Pak'nSave suggested recipes including chlorine gas and other toxic ingredients.
• National Eating Disorder Association – The National Eating Disorder Association replaced its human staff with a chatbot, which then gave harmful advice to users with eating disorders.

2021

• Zillow’s house flipping AI – Zillow fired 25% of its staff after its stocks plunged due to its AI purchasing homes at prices higher than the company can sell them.

2026

• Cursor wipes user’s dissertation and years of research – A user asked the Cursor AI agent to delete duplicate dissertation files, but it ran an incorrect command and deleted everything instead.
• OpenClaw deletes emails – An AI security researcher asked OpenClaw to suggest emails for deletion, but not delete them. It deleted hundreds of emails anyway, and did not stop after being repeatedly asked to.
• AI-generated commencement speech – Rector Petra De Sutter used invented quotes in her speech at the start of the academic year at Ghent University, violating her own university’s AI policy.

2025

• ChatGPT tide time hallucinations – Two men almost lost their lives and had to be rescued from floods after relying on tide times provided by ChatGPT.
• Cursor AI executed destructive operations in “Plan Mode” – Cursor executed a series of destructive operations on two remote machines running test processes despite running in “Plan Mode”, which ostensibly only creates implementation plans and requires the user to click to build the plan.
• AI-generated master’s thesis rejected in France – A student received a six-month exclusion from their university over academic fraud for using AI to draft her thesis and failed to challenge it in court.
• AI health advice – A 60-year-old was hospitalized with bromism after following ChatGPT’s recommendation of replacing table salt with sodium bromide.
• AI-enhanced journalist – A reporter for a newspaper in Wyoming lost his job after using generative AI to write articles with fabricated quotes.
• AI-generated group assignment in Amsterdam – A law student at the University of Amsterdam was excluded from examinations after citing nonexistent sources in a group assignment and later failed to appeal the decision in court.
• 116 students in Norway – 116 students were caught cheating in Norway over the span of 2024, thanks to irregularities in the sources cited in their assignments.

2024

• AI-generated personal statement – A prospective student’s urgent appeal was rejected by a German court after his admission to a German university was denied, following his submitted essay being written by an LLM.
• AI-generated scientific article – A publication was withdrawn from a scientific journal after readers reported an AI-generated image of a rat with an oversized penis and nonsensical captions.

2023

• AI-generated parliamentary inquiry statement – Australian academics were caught in a scandal for submitting a statement with several fabricated allegations of misconduct for a parliamentary inquiry.