In late March around 15 religious thinkers met with the artificial intelligence company Anthropic to discuss one of the strangest and most consequential questions now facing the AI industry: How do you teach a chatbot to be good?

The invitations to these meetings had arrived in different ways. Greg Cootsona’s came via e-mail. Brian Patrick Green’s came via a friend of a friend after Anthropic asked for suggested names. Both ended up in a series of conversations with the company about Claude, Anthropic’s chatbot, and the moral framework meant to guide how it behaves.

The aim wasn’t to make the chatbot Bible-thumping or pious. But it was an acknowledgment that centuries-old traditions of moral reasoning might offer insights to a five-year-old frontier AI lab whose systems are becoming more capable, more persuasive and harder to govern by simple rules.

On supporting science journalism

If you're enjoying this article, consider supporting our award-winning journalism by subscribing. By purchasing a subscription you are helping to ensure the future of impactful stories about the discoveries and ideas shaping our world today.

“I think they have reached a point where they realize that the power is kind of outstripping their in-house wisdom,” says Green, director of technology ethics at the Markkula Center for Applied Ethics at Santa Clara University and one of the leading scholars working at the intersection of technology and theology. “They realized that they needed help.”

Cootsona, executive director of AI and Faith, an organization that advises tech companies on the ethics of AI, remembers the conversations similarly. “These questions have become too big for us,” he recalls Anthropic staff saying. “We can’t answer them on our own.” (Anthropic did not respond to an interview request for this story.)

The conversations took place amid a broader religious reckoning with AI. On May 25 Pope Leo XIV presented his first encyclical, Magnifica Humanitas: On Safeguarding the Human Person in the Time of Artificial Intelligence, an about 40,000-word treatise calling for AI to be “disarmed”—not rejected but freed from the assumption that “technical power automatically confers the right to govern.” Anthropic co-founder Christopher Olah was among those who attended the Vatican presentation that announced the treatise’s release.

The stakes extend far beyond Claude. Hundreds of millions of people now talk to AI chatbots every week, and the values their developers bake in via guardrails and corrective tuning shape what those models say about everything from end-of-life care to abortion to managing grief. There are few regulations, no agreed-upon method for doing this work and, until recently, little outside input. The fact that a leading company is now consulting theologians is either a rare sign of humility or of an industry improvising its ethics in real time—possibly both.

But what can religion offer AI—and what happens when religious values start shaping how a chatbot answers?

Religious traditions, for all their contradictions, have spent millennia considering the same underlying problem: how to form moral agents and instill those lessons in society. “Moral formation has been a topic that religions have been talking about for thousands of years,” Green says. “What insights can they give us that we can use to hopefully produce a model which will be better at doing what we want it to do, which is to be good and not do bad things?”

The goal of the meetings in late March, according to those who attended, was to help refine what Anthropic calls Claude’s constitution, a written set of principles the company uses to shape how the model responds, including by training Claude to critique and revise its own answers against those principles.

Anthropic is “looking for what works” and may try religiously informed ideas or techniques to see whether they improve model behavior, Green says. His understanding is that the company has recognized it “can’t make a regulation about every single case that the AI is going to come into contact with.” So instead of writing rules for every scenario, the aim is to shape something more like a model “persona” with a disposition toward good behavior rather than a checklist of prohibitions.

Not everyone is convinced that religious consultation solves the accountability problem. “I wonder, with these companies and types of executives, whether it makes sense to try to figure out whether they mean what they say,” says Carissa Véliz, an AI ethicist at the University of Oxford, “or whether it makes more sense to think about whether what they do is ethical or unethical, whatever their true intentions, while noting the incentives that their business model pushes.”

The easy criticism is that what Anthropic did was “ethics washing”—borrowing the moral seriousness of religion to burnish its reputation. But those who were in the room saw something different. “It’s not ethics washing,” Green says. “It’s sincere, from what I can tell.” He points out that inauthenticity with religious thinkers would be quickly spotted and that the resulting backlash would be hard to recover from.

Sincerity is no guarantee the company will act on what it heard. By multiple accounts, the late March meetings were not always polished. Green says the tone varied between sessions—some had stronger camaraderie, while others were “a little bit more awkward”—and that even the participants weren’t always clear on what was supposed to happen next. In the meeting he attended, he says, “everybody there was very interested in listening,” but there was also “a question of what do we do with this information now that we have it.”

Over time, Anthropic appeared to sharpen the format, learning how better to facilitate the discussions and produce more cohesive results. It has also widened the circle beyond Christian thinkers: a late April meeting brought together participants from several religious traditions, including Judaism, Hinduism, Mormonism, Sikhism and the Greek Orthodox Church.

Even if the earnestness is genuine, Véliz worries that the use of religious terminology and imagery around AI—deliberately or not—can make honest conversation harder to have.

“The increasingly religious notes of Silicon Valley do worry me, because they can inspire a kind of tribal mentality that can be harder to pierce through reason,” she says. “Religious feelings tend to be emotionally charged in ways that decisions purely based on business reasons are not,” Véliz says. They also “give leaders more leverage to inspire obedience in followers.”

In his encyclical, Pope Leo XIV argues that algorithmic power should not be imposed from above in an opaque and unilateral way. Anthropic’s experiment suggests how hard that principle may be to put into practice.