Therapy is predicated on trust. You can’t be honest and vulnerable, and share how you’re really feeling, if you don’t believe in the embodied-concerned-frown sitting in the armchair across from you.

So you can understand why one woman, 31-year-old Molly Quinn, was taken aback when her trusted therapist suddenly whipped out an AI model to start recording their private conversations, NPR reports. 

“She wasn’t taking notes like she usually did,” Quinn recalled realizing halfway through one session. “The iPad was just propped up.”

Where were her words being processed and stored? Will they one day become training data? It’s not something you have to ask yourself when your therapist jots stuff down on a clipboard. But those questions were now racing through Quinn’s head, leaving her uneasy.

“The more I thought about it, the more I just started getting more and more sick to my stomach,” she told NPR. “This person who I’m supposed to be able to trust with some very private and very intense emotions had just completely disregarded something I said I was not comfortable with. I felt completely violated.”

Though her therapist offered to stop using the AI tool, Quinn cut her off and found another one.

“The trust was gone,” she told NPR.

Like doctors, therapists across the country are adopting AI tools for notetaking and generating transcripts. AI companies offering these services frame it as a way of cutting down on the drudgery of paperwork and other administrative tasks, freeing up more time to focus on patients — a permutation of a common AI industry refrain: let us do the tedious stuff for you. 

The reliability of AI tools remains fairly dodgy, though, and even setting aside questions of hallucinations creeping into clinical notes — which is something we’re already seeing happen — it’s not clear whether patients are even comfortable with the tech yet. In a YouGov survey cited by NPR, only 11 percent of Americans said they would be open to using AI in mental health care. An even slimmer eight percent said they would trust AI being used this way, while 40 percent said they don’t trust the technology at all.

“Even the presence of AI changes the therapeutic experience,” Marisa Cohen, a couples and sex therapist in New York, told NPR. “Clients know or feel like something else is listening to them. That awareness can subtly alter their disclosure.”

“When you introduce something that’s being stored electronically, it raises additional questions about trust and safety,” Cohen added. “It’s essentially a third party.”

Tal Salman, the CEO a popular AI scribe tool for therapists called Berries, insists that conversation recordings are deleted immediately and that transcripts are stored on HIPAA compliant servers in the US. Even if this is true, if AI companies’ tools are to ever have a place in private mental health settings, they need the trust of patients — and that’s something the AI industry clearly hasn’t earned yet. Quinn fears that AI-recorded conversations could one day be exposed by hackers.

“We’re going to see breaches,” she told NPR. “Maybe not tomorrow, maybe not next week. But in a few years? I think we’re going to see them. And I don’t want my therapy session to be part of that.”

More on AI: The Pope Just Low Key Declared Holy War on Artificial Intelligence