Authors:Shreya Shah, Akshay Swaminathan, Meghana Simhadri, Ivan Lopez, Sharang Phadke, Divyanjali Verma, Abhay John, Luke Zhao, Fiona Cai, Sharon Zhang, Gloria Ye, Ivy Pham, William Wang, Sebastian Garcia, Sarah Wornow, Angelina Wang, Nigam H. Shah
Abstract:As artificial intelligence (AI) tools get increasingly deployed for mental healthcare, public trust in these systems remains uncertain. It is unclear how clients perceive AI involvement in counseling interactions, particularly in moments of crisis that require empathy and connection. To address this gap, we analyzed 75,777 crisis counseling conversations from a human-staffed WhatsApp helpline in India to characterize how often clients suspected they were speaking to AI, what triggered those doubts, and how counselors responded. Though no conversations actually involved AI assistance, the proportion of conversations where clients suspected AI use increased from 0.8% in June 2024 to 2.6% in March 2025. Within suspicious conversations, 21.5% of clients stated an explicit preference for humans. Client suspicion primarily arose in the first half of messages (68.3%), and when counselors offered reassurance (e.g. 'I assure you; this is not ai!'), clients continued to press or ended the conversation 17.6% of the time. As AI tools get increasingly integrated into counselor workflows, understanding these dynamics is essential for designing AI systems that preserve the therapeutic relationship between counselors and clients.
From: Sarah Wornow [view email]
[v1]
Sun, 10 May 2026 22:36:06 UTC (1,023 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。