If you thought AI customer service bots were bad, just wait until you need to convince a large language model that yes, you have been involved in a car accident, actually.

According to new reporting by the Shreveport Times, officials at the Orleans Parish Communication District — the emergency dispatch agency for the city of New Orleans — are now integrating AI answering agents into its workflow.

The new tool is called AI Emergency Call Triage, offered by a company called Carbyne. Its function is essentially to triage incoming calls to dispatchers, bumping high-priority emergency calls to humans, and automatically handling repeat calls related to ongoing incidents, the Times explains.

Starting in April, New Orleans experimented with implementing the AI for 311 calls, a phone line meant to connect callers to local government information and non-emergency services.

That the AI will now be sorting incoming emergency calls is a significant escalation for the city’s dispatch agency. New Orleans will now join other major cities including Seattle and Atlanta, which have also implemented AI into their emergency dispatch services in recent weeks.

Though officials claim the new tech will help cut through the hundreds of calls inundating human dispatchers, it nonetheless represents an added layer of friction between callers in distress and trained dispatchers who can connect them with help.

It’s also worth noting that AI voice chat remains highly flawed even when powered by frontier AI models, with plenty of opportunities for errors both in how they process information, and how they dish it out. A February study by the media software company NewsGuard found that in voice mode, AI bots like ChatGPT and Gemini produced false claims with realistic audio up to 50 percent of the time. Emergency calls to 911, needless to say, are often extremely chaotic.

Meanwhile, AI voice systems are also persistently vulnerable to outside attacks, with cybersecurity researchers successfully manipulating voice model’s behavior with an average success rate between 79 and 96 percent.

What all of that means for situations where connecting with a human dispatcher could mean the difference between life and death remains to be seen — but if other AI voice experiments are any indication, it won’t be good.

More on first responders: Emergency Responders Say They’re Now Unpaid “Roadside Assistance” for Confused Waymos