Popular AI models fail to effectively transcribe Indic languages, mishearing one in three words or dropping English words altogether in mixed speech, as per a study by physical and voice AI data infrastructure company Humyn Labs.

Founded by gaming veteran Manish Agarwal, the startup looks to create a Benchmark of Regional & International Data for Global Evaluation (BRIDGE) to evaluate commercial AI speech-recognition tools. The study looked at tools like ElevenLabs Scribe v2, Deepgram Nova-3, Gemini 2.5 Flash, OpenAI GPT-4o, and Indian providers Sarvam saaras v3 and Gnani vachana v3 on real Indian language data.

The study showed that even the most widely deployed tools have a fundamental problem of mishearing words in Indian language audio. Worse still, in cases of a natural mixing of Hindi or any Indic language with English mid-sentence, most AI tools either drop the English words or convert them into transliterated script, breaking the meaning for anyone reading the transcript.

“The models are grading their own work. ASR providers published their own accuracy scores using benchmarks built on English-first, internet-trained datasets, with little independent validation. Meanwhile, enterprises are making million-dollar deployment decisions on numbers that rarely reflect how their users in Global South actually speak,” said Manish Agarwal, Co-founder, Humyn Labs, adding that theirs is the first independent benchmark for real-world conversational audio across non-English markets.

The scores reveal that Deepgram Nova-3 leads in terms of the semantic gap at 0.906. Amazon Transcribe scores 0.199. OpenAI’s models fall below 0.4. Most enterprises using these tools were unaware of the errors because the standard industry measure, Word Error Rate (WER), was never designed to catch the failures that define real Indian speech.

Comparing global models against Indian providers, the study showed that Sarvam AI’s saaras v3 ranks third overall on WER at 20.2 per cent, ahead of Google Gemini, Microsoft Azure, and AWS Transcribe, a strong result for a model built specifically for Indian languages. However, in terms of mixed speech, Sarvam scores 0.588, placing it in the partial-reliability category where performance varies by language and English density. This means the gap between headline accuracy and code-switch reliability applies to domestic and international providers alike.

Humyn applies a seven-metric stack to test whether the AI models preserve the meaning of what was said, ensure the LLM accurately tracks English words embedded in Indian language speech, how Indic phonology is transcribed as well as Word Information Lost in case of under- or over-transcription.

“The models aren’t the only problem the metrics are. You cannot evaluate non-English speech with a scoring system designed for English phonology and call it rigorous. The performance leaderboard for Hindi is not the leaderboard for Tamil, Bengali and Marathi. A single aggregate benchmark score cannot support cross-regional deployment decisions,” said Ishank Gupta, Co-founder, Humyn Labs.

The study highlights how a model that leads on Spanish may not lead on Vietnamese. Similarly, the model that leads on code-switching does not lead on word accuracy, stressing the need for enterprises to evaluate the language, dialect, and speech pattern that matches their actual users.

Published on May 11, 2026