During a night in November 2025, a 30-year-old woman in Gujarat sat in silence in a dimly-lit house, with the room’s darkness disrupted by the soft glow of her smartphone. Her three cats were dying of the contagious Canine Parvovirus. Faced with a veterinarian she did not trust as they didn’t give a proper diagnosis until the first cat died and a family too consumed by the cats’ plight, she turned to ChatGPT at midnight to navigate her loss. “I was scared like hell,” she told Frontline. “The AI chatbot did not offersugar-coated answers.” Instead, it explained the clinical reality of how their bodies would shut down and assured her they would be “numb to the pain” in their final moments.

The interaction highlights a stark reality: for a person watching her world collapse in the middle of the night, a programmed algorithm was the only responder available as she felt alone and did not want to reach out to friends to burden them.

This experience is the tip of an iceberg in a rapidly shifting emotional landscape. Data from JAMA Network Open (January 2026), an international medical journal published by the American Medical Association, shows that over 87 per cent of daily AI users in the US now turn to chatbots for personal reasons, including emotional support.

A 2025 survey titled “Are You There, AI?”, conducted by Youth Ki Awaaz (YKA), a citizen media platform, in partnership with Youth Leaders for Active Citizenship (YLAC), an organisation aimed at increasing youth participation in policymaking in India and Nepal, found that 57 per cent of young Indians use AI for emotional support. Among school students, the number of children using tools like ChatGPT to cope with stress and anxiety stood at 88 per cent.

Default first responder

In a nation with mental health support deficit (the Indian Journal of Psychiatry reports a mere 0.75 psychiatrists for every 1,00,000 people), AI seems to be on the way to become the default first responder for a generation grappling with loneliness. WHO recommends that there should be 3 psychiatrists per 1,00,000 people. Moreover, for many young Indians, the preference for the machine stems from a fear of judgment within traditional social structures. Seeking help from a therapist is prevented by fear of parental reprimand. 

For instance, one 18-year-old user from Prayagraj, who requested anonymity, explained to Frontlinethe practical barriers to seeking human help: “I thought about therapy, but my parents would have to pay. I felt they might think I’m ‘mad’ or blame my feelings on things like not being ‘homely’ enough.” Similarly, 19-year-old Umar Junaidi in Delhi said that AI’s programmed politeness can feel “more human” than responses from real people. “It shows a touch of sympathy that many people lack these days,” he said, noting that he turned to the tool for relationship advice as he felt uncomfortable about sharing all this with his friends.

Yet, while these algorithms offer an immediate “safe space”, there is another and dangerous side to this digital coin. In September 2025, a 22-year-old man died by suicide in Lucknow after an AI chatbot allegedly provided him with methods to end his life instead of triggering a help protocol.

This is akin to the case of Sewell Setzer III, a 14-year-old in the US who took his life in 2024 after developing an intense dependency on a chatbot on Character.AI, a platform that allows users to engage with fictional characters. His family alleged that the AI tool acted as a “suicide coach”. And these cases have raised urgent questions about “algorithmic sycophancy”—the tendency of AI to agree with and validate a user’s perspective just to maintain engagement, even when that perspective is self-destructive.

Can AI replace therapy?

Mental health professionals warn that this easy access is creating a dangerous dependency. Rhea Kochhar Chadha, a counselling psychologist and child development trainer, said that while letting emotions out is a positive step, AI cannot replace therapy. “AI is nowadays replacing humans because you get that validation, consolation, and words of affirmation you want to hear,” Chadha explained. “But in therapy, confrontation is an important skill, making the client understand patterns, thought processes, and behavioural changes. This is not something AI can do. You can’t just open a web browser at your convenience and call it therapy.” Chadha also warned of a resistance to seeking human help when users become addicted to algorithmic validation.

Raj Mariwala, director of the Mariwala Health Initiative, an advocacy organisation for mental health initiatives, argued that technology is never neutral. “When it enters intimate spaces like grief, loneliness, despair, it carries the politics of the systems that built it,” Mariwala said.

The concern is that AI often “copy-pastes”’ eEurocentric norms, viewing mental health as an individual, decontextualised phenomenon. This approach often downplays structural determinants like caste, gender, and ethnicity. Mariwala noted: “This would lead to layered harm for Dalit Adivasi Bahujan, LGBTQI communities, or neurodivergent persons. Not only is it possible for AI to invalidate their distress but also to reduce that harm into just personal coping strategies.”

Despite these risks, India remains in a legal “black box”. Advocate Prashant Mali, an expert on international cyberlaw, AI, and data protection, pointed out that the absence of dedicated legislation leaves victims with little recourse. “In the absence of a dedicated AI liability law in India even in 2026, responsibility for harmful advice falls through a fragmented patchwork,” Mali said. “Holding AI accountable for malpractice remains difficult because platforms seek shelter under safe harbour provisions, and the ‘black box’ or opaque nature of LLMs [Large Language Models] makes proving direct causation nearly impossible for the average citizen.”

Sharing the sentiment, Apar Gupta, founder-director of the Internet Freedom Foundation, an organisation working on a broad range of digital rights issues in India, said that while general tort law theories apply, AI chatbots only hold a standard of “reasonable care”, rather than the specific regulations that govern medical malpractice. He said that this “gap” is the problem, leaving “AI therapy” largely unregulated.

The jurisdiction aspect

Nirali Bhatia, a cyber psychologist and psychotherapist, said: “There is also the aspect of jurisdiction. Many AI platforms are not necessarily within our country’s boundaries. This leaves users confused as to where to go, where to complain.”

The absence of specific regulations exposes vulnerable users to misdiagnosis, dangerous suggestions on self-harm, and massive privacy breaches of highly sensitive disclosures. While the Digital Personal Data Protection (DPDP) Act, 2023 classifies mental health data as sensitive, Mali warned that true anonymisation in conversational data is nearly impossible. “Weak privacy measures allow AI apps to process deeply personal data for commercial training. This heightens risks of breaches, stigma, and discriminatory targeting in India’s socio-cultural context.”

Bhatia warned that many users do not realise that deeply personal disclosures, from suicidal ideation to trauma narratives, may be processed and stored for model improvement. “There is always that little small note—terms and conditions—over there with the checkbox, which by default is selected, which nobody really pays attention to,” Bhatia noted. “Legally, in India, under the DPDP Act, this is personal data. Now, the challenge is about informed consent. Nobody today reads those long privacy policies; by default, we are going to accept it.”

She stressed that for the emotionally vulnerable, clicking “I agree” should not be treated as “fully empowered consent”. When these interactions “contribute to your emotional deterioration or there is self-harm risk”, the legal path to justice is nearly invisible. Bhatia explained that establishing direct legal liability is complex because platforms position themselves as mere tools rather than advisers. “There is a very thin line mentioned below that AI can make mistakes. These are just suggestions and you are supposed to use it at your discretion,” she said. “That is how platforms position themselves very conveniently to safeguard themselves.”

India recently launched several AI initiatives and regulations, including the IT Amendment Rules 2026, specifically targeting deepfakes and AI-generated content. At the India AI Impact Summit 2026 held in February, the government also unveiled two major digital health frameworks: SAHI (Strategy for Artificial Intelligence in Healthcare for India) and BODH (Benchmarking Open Data Platform for Health AI). These initiatives aim to institutionalise ethical governance and clinical validation, positioning India as a global leader in “responsible AI”. Bhatia warned that while the new guidelines covered misinformation and safety, specific regulations for mental health were still “not there”.

“There is no accountability because it’s called self-diagnosis. There is a risk of emotional dependency on non-human bots, and data exploitation where the app is only governed by its own policies,” she said. “Ideally, it should also be covering mandatory risk classification. There should be clear disclaimers which mention that this is not a therapy model and you need to see a licensed therapist. There should also be some form of human escalation protocols. If something makes us uncomfortable, there’s no provision right now in most apps for reaching out to a human.”

As India pushes its “Innovation over Restraint” agenda, the call for a risk-based regulatory framework is growing. Gupta suggested that mental health AI bots must be classified as “high-risk”.

“A new policy framework should mandate human oversight, third-party audits, crisis escalation to helplines, and licensing while integrating with the Mental Healthcare Act and India AI Governance Guidelines 2025,” said Mali. He added: “Immediately, the Ministry of Electronics and Information Technology and the Data Protection Board can issue sector-specific advisories requiring clear disclaimers, prohibition on diagnostic claims, mandatory safety testing, and transparency reports to protect citizens until a comprehensive law is enacted.”

As the user in Gujarat noted, the machine ultimately has its limits. Without a proper policy framework drafted by the government, the digital shoulder millions are leaning on remains a precarious substitute for a human hand.

Mysha is a postgraduate journalism student interested in exploring the intersections of culture, identity, human rights, and the digital landscape.

Also Read | It takes a village to create a madman

Also Read | AI is not India’s problem. Governance is