Two users type the same question in Kannada. One uses the Kannada script, the other Roman letters. A large language model (LLM) reads both, interprets both correctly, and even paraphrases them with comparable accuracy. Yet, it makes the correct triage decision for the native script version and hesitates or mis-classifies the romanised one.
This unexpected divergence sits at the heart of ‘Script Gap: Evaluating LLM triage on Indian languages in native vs Roman scripts in a real world setting’ (Khullar et al, 2025), which analyses maternal and newborn care chats across six Indian languages and English.
AI stability
In every model evaluated — including GPT 4o, Claude 4.5, LLaMA 4, DeepSeek, Qwen and Indic-specialised systems — romanised Indian language messages produced a consistent and sizeable performance drop. The researchers use the F1 score, which measures precision and recall, to show how reliably a model identifies the right category without overreacting or overlooking important cases. Romanised messages saw declines of 5-12 points compared with native scripts.
Kannada fell from 83.7 per cent in native script to 57.3 per cent in romanised form. Marathi declined from 78.6 per cent to 61.4 per cent, and Nepali from 83.1 per cent to 59.8 per cent. Hindi showed a smaller but still meaningful gap. English, however, remained strong. The problem is not the Roman alphabet itself but the unpredictable, informal ways in which users write Indian languages using Roman characters.
The most surprising finding is that the models generally understand these romanised messages perfectly well. The systems routinely paraphrase romanised queries accurately. They extract symptoms, identify intentions and interpret the context sensibly. In other words, the meaning is not lost. Yet, the decision is wrong.
Romanised messages are far more likely to be labelled “insufficient information”. Orthographic noise — the highly inconsistent and improvised spellings used when writing Indian languages in Roman letters — narrows the boundary between the classification categories. It introduces instability in how the model processes the text even when it understands the meaning.
Several indicators reveal this instability. Romanised inputs reduce cross-model agreement, with only about half achieving full consensus compared with roughly two-thirds for native or English messages. Queries that are plainly non-urgent in native script are frequently downgraded to “insufficient information” when written in romanised form.
Normalisation
One of the most counter-intuitive findings in the study is that code mixing improves performance. Users who blend English words with romanised Indian language text receive more reliable classifications. For instance, a caregiver writing, “baby ko fever hai please tell what to do”, which mixes Hindi and English words. The English tokens act as anchors, stabilising tokenisation and reducing ambiguity. A style often regarded as messy helps AI systems behave more predictably.
Even more striking is the impact of script normalisation. When romanised messages are automatically transliterated back into their native scripts, model performance rebounds sharply.
GPT 4o, for example, rises from 75.3 per cent to 80.1 per cent after normalisation, almost matching its native script baseline. Translating romanised inputs into English also improves performance, but not as reliably. These results confirm that the underlying semantic content is not the issue. The difficulty arises from orthography and tokenisation rather than meaning.
This matters because romanised writing is extremely common in India due to habit, device limitations or familiarity with English keyboards. When an AI system penalises this writing style, it inadvertently penalises those most in need of accessible digital tools. In the healthcare setting studied, 56 per cent of user messages arrived in romanised form.
Beyond healthcare
Although the study focuses on clinical triage, it applies to any system that relies on language models to route, escalate or classify user messages. A financial chatbot may mishandle a fraud alert, a public grievance portal may downgrade a complaint or an educational platform may offer weaker guidance to students. In emergencies, a romanised plea for help can be misread as too vague to act on.
The implication for system designers is clear. AI must adjust to the full range of how people actually write.
More Like This
Published on December 15, 2025