In the wee hours of Memorial Day, my friends and I stayed up past 4:30 AM California time to listen to the announcement of Pope Leo’s first encyclical, Magnifica Humanitas, on safeguarding the human person in the time of artificial intelligence. We were excited albeit sleepy, eagerly anticipating the event and upcoming essay by the world’s foremost religious authority on a question so central to our world. Still we were an odd audience for this presentation: none of us are practicing Catholics, and most of us didn’t really know what to expect.
I thought Pope Leo’s own speech was good, and addressed the current moment in AI with some of the seriousness it deserves. I thought the other speeches, including by Chris Olah, were less impressive. But that’s okay, I’m not the target audience!
A specific cardinal’s point struck me, however:
Cardinal Parolin made much of a specific prepositional choice in the subtitle: “sulla custodia della persona umana nel tempo dell’intelligenza artificiale,“ which the live translator translated to something like “on the safeguarding of the human person in the time of AI,” and not “sull’intelligenza artificiale“ – “on AI.”
This was supposed to be a big deal. “In the time of AI” supposedly centers the human person in the theological narrative, while a mere first papal encyclical on AI focuses too much on the technology itself and not on human and societal reactions. A fascinating position!
Though as my subsequent analysis will demonstrate, perhaps a more apt preposition here is “by.” As in, the world’s first papal encyclical written in large part by AI.
My article has the following claims, each of which I hope to convince you of:
Significant fractions of the recent papal encyclical are written by AI. I provide multiple lines of evidence for this.
We can corroborate the vibes and tonal indications with statistical evidence. Phrases and punctuation much more commonly used by AI are much more present in this papal encyclical than past encyclicals.
The best commercially available AI detector, Pangram, notes that some paragraphs are between 40% and 100% AI, while the majority paragraphs appear to be 0% AI.
In particular, 11% of the first 20 paragraphs are flagged as AI.
This is unlikely to be a false positive:
0% of paragraphs in past encyclicals I backtested are registered as AI.
Pangram in general has a very low false positive rate
This is overall very unlikely to be a translation artifact (including AI translation). We again have multiple lines of evidence:
All the most prominent signs of AI I observed in English are preserved verbatim in the Italian version, as well as in other translations.
The Italian version of the current encyclical also gets flagged as AI by Pangram (actually more so than the English version), though I’m not aware of academic research or rigorous testing of Pangram’s service when applied to Italian)
Backtesting AI translation of past encyclicals get 0% on Pangram
The specific AI used is most likely Claude, judging by both textual and circumstantial evidence.
Different sections of the encyclical have very different rates of apparent AI usage. This indicates to me that some Vatican officials used AI assistance for this encyclical and many (probably including Pope Leo himself) didn’t.
Each individual piece of evidence might be explained away, but the consilience of evidence across multiple angles and sources is in my opinion very hard to dismiss collectively.
I was initially very excited to read Pope Leo’s first encyclical, a long treatise on maintaining humanity in the age of AI. The intersection between AI and societal response is one of my greatest intellectual and personal interests, and it’s both exciting and a relief for the world’s foremost religious authority to share a substantial interest in my personal and career obsessions.
Nonetheless – as I kept reading – certain lines jumped out at me as too smooth, too triadic, too… inhuman:
“Technology has the power to heal, connect, educate and protect our common home; but it can also divide, exclude and generate new forms of injustice. In the abstract, technology in and of itself is not a solution to humanity’s problems, just as it is not inherently evil. In practice, however, technology is never neutral, because it takes on the characteristics of those who devise, finance, regulate and use it.”
“We must, then, avoid the “Babel syndrome,” namely the idolatry of profit that sacrifices the weak, a uniformity that neutralizes differences, and the pretense that a single language — even a digital one — can translate everything, including the mystery of the person, into data and performance. The risk of dehumanization — of building a future that excludes God and reduces the other to a means — is an ancient and ever-new temptation that today takes on a technical guise.”
“A dialogue with such kinds of knowledge does not diminish the power of the Gospel. On the contrary, it makes it possible to identify with greater clarity what genuinely fosters the lives of individuals and communities.”
“give stable form to this insight at the ecclesial and international levels, while bearing in mind the growing gap between rich and poor countries and the need for policies that genuinely promote more humane living conditions for all.”
“We cannot be satisfied with merely calling for the moralization of machines — the so-called “alignment” of AI with human values — without also having the courage to insist on a further condition: the possibility of openly discussing the ethical frameworks involved and subjecting them to shared standards of social justice”
I read AI-generated text as part of my job regularly, and believe I have acquired a very good intuition for discerning AI-generated text from those by humans, including in formal writing (both academic and otherwise). Still, any individual phrase that seems AI-generated can be a false positive on my end, the result of an oversensitive nose for AI1. However, the sheer density of these phrases and overall tone in specific paragraphs seem implausibly all a random artifact.
Still, I can definitely be wrong here, and you should not believe my gut intuitions or judgments of vibes on authority (“Trust me bro”).
Intuitions, self-proclaimed expert judgments, and loose verbal reasoning can be a good starting point for an investigation, but if we want any confidence in our conclusions, we need to investigate further and more systematically.
Three common and well-known tells in AI writing — sometimes genuinely deployed by humans but nowhere their profligate use by AI — are the regularity of em-dashes, the high frequency of specific words like “genuinely”, and the tendency to repeatedly invoke tricolons.
Let’s examine each of them in turn:
The em-dash (“—”) is punctuation that’s by far most strongly associated with AI. It is also used 127 times in Magnifica Humanitas, much more than previous encyclicals.
Magnifica Humanitas: 127 times em-dash, 6 times en-dash (–), the latter all in citations.2
Dilexit Nos (2024): 0 times em-dash, 26 en-dashes, including 2 in citations. Comparatively long document.3
Laudate Deum (2023): 0 times em-dash, 12 times en-dashes. Much shorter. Also not officially an encyclical.
Fratelli Tutti (2020): 0 times em-dash, 46 en-dashes, of which maybe 5-10 are in quotes or citations. Note that this is 50% longer than Magnifica Humanitas.
Laudato Si’ (2016): 0 times em-dash, 25 times en-dash, of which maybe 10 are in citations or quotes (the piece overall appears to have many quotes). Similar length to Magnifica Humanitas
Lumen Fidei (2013): 26 times em-dash, 0 times en-dash. Some em-dashes in citations.
Note that this comparison actually understates the weirdness of the em-dashes in Magnifica Humanitas. For example, in Lumen Fidei, many of the em-dashes function similarly to speech colons in standard English. A typical use looks like
What was handed down by the apostles — as the Second Vatican Council states — “comprises everything that serves to make the people of God live their lives in holiness and increase their faith. In this way the Church, in her doctrine, life and worship, perpetuates and transmits to every generation all that she herself is, all that she believes.”
Using em-dashes as speech colon replacements is moderately common in formal (human) English writing, but essentially absent in LLM-English. I also did not notice em-dashes used this way in Magnifica Humanitas (though with 127 instances, it was annoying to check all of them!)
“Genuinely” is a phrase repeatedly used by Anthropic’s model Claude. It is extremely obvious to anybody who regularly uses it. It’s gotten so bad that in leaked system prompts, Anthropic attempted to explicitly forbid Claude to use that word!
1.4 Tone & Formatting
[...]
Claude avoids saying “genuinely”, “honestly”, or “straightforward”. 4
As far as I could tell, this injunction does not and did not work.
Indeed, Anthropic’s own “Claude Constitution”, which many people believe to be substantially AI-assisted, used the phrase “genuinely” 33 times and genuine overall 50 times (inclusive).
Less so than in Anthropic documents, but substantially more than past papal writings.
Specifically “genuinely” was used 9 times and “genuine” overall (inclusive) 22 times in yesterday’s encyclical, compared to 0 and 5 times, respectively, in Dilexit Nos, which is of similar length. Across a number of other encyclicals I scanned, the highest occurrences were 3 and 10, respectively.
These tells are all statistical. Any individual instance of “genuine(ly)” is plausibly a result of normal human communicative intent that is, well, genuine. But the sheer frequency of these occurrences, vastly out of accord with prior norms and normal human speech, is strongly suggestive of synthetic origin.
An obvious rejoinder you might have is that word choices in essays are naturally not independent of subject matter. And it sure seems like an encyclical on AI might meditate more about genuineness more than other encyclicals! For example, an essay on AI deepfakes might be much more concerned about what makes a video “genuinely human” than an essay on climate change.
To investigate this hypothesis, I dived specifically into each use of genuinely in this encyclical:
[Par 23] “A dialogue with such kinds of knowledge does not diminish the power of the Gospel. On the contrary, it makes it possible to identify with greater clarity what genuinely fosters the lives of individuals and communities. Following this perspective, Pope Francis [...] recognizes the importance of listening to scientific research and of encouraging a serious and honest debate among experts while welcoming a diversity of opinions.”
“Genuinely” does not seem critical here, nor specific to questions of AI and authenticity.
[Par 35] The establishment of the Pontifical Commission Iustitia et Pax should also be seen in this light as an attempt to give stable form to this insight at the ecclesial and international levels, while bearing in mind the growing gap between rich and poor countries and the need for policies that genuinely promote more humane living conditions for all.
Also not critical here.
[Par 40] In his social Encyclical Caritas in Veritate, Pope Benedict XVI sought to reassess and expand the concept of development presented in Populorum Progressio, interpreting it in light of globalization. He noted that such development should translate into “real growth, of benefit to everyone and genuinely sustainable.” [42]
Appears to be in a quote, so will give it a pass.5
[Par 57] Along with a greater awareness of the value of every human person and their rights, recognition of minority rights has also grown. Yet, there is still a long way to go to ensure that the rights of a great many, namely women, are equally and genuinely guaranteed throughout the world.
Again, does not seem like “genuinely” was endogenously related to the subject matter
[Par 100] The artificial imitation of positive human communication — words of advice, empathy, friendship and even love — can be engaging and at times genuinely helpful. However, for less discerning users, it can also be misleading, creating the illusion of a relationship with a real personal subject. When words are simulated, they do not build genuine relationships, but only their appearance. The artificial imitation of care or support can become particularly risky when it enters contexts where real relationships and emotional bonds are lacking. Here, the danger is not so much that a person may believe they are communicating with another person, but rather that they may gradually lose the very desire to form genuine human connections.
…you get the idea.6
Indeed, of all 9 instances of “genuinely” in the encyclical, only the last use (“When people come to believe that nothing is genuinely true and that principles are hollow words, then the fuse in their hearts is lit for new eruptions of intolerance and aggression.”) seem semantically critical. If we drop that and the Pope Benedict quote we’re left with 7/9 suspicious uses.
Again, to be clear any individual instance is plausibly normal, authentic, genuine. However the statistical pattern of the repeated invocations is quite suspicious!
Another possibility you might have is that maybe this is just a personality/stylistic quirk of Pope Leo XIV specifically? Maybe he just genuinely likes the word?
Lord knows I too have odd personality quirks in writing, some of which have an unfortunate resemblance to AI.
Ultimately, I think this is plausible but unlikely. First of all, popes don’t typically draft the text of their own encyclicals that much. So it’s unlikely that stylistic quirks as specific as adverbial usage will bleed out to the final drafts as much. In contrast, I’m much more open to higher level constructs like the imagery, themes, or favorite Bible passages being much more prominent in some pope’s encyclicals than others.
Further, the specific phrases used are often next to other suspicious “AI tells” (more on that later).
Unfortunately, I don’t have easy access to many (pre-papacy) writings by Pope Leo to test against this alternative hypothesis. However, I did find Chapter 2 (“The Authority of the Local Prior”) of his 1987 PhD thesis here. In 14 pages (roughly the size of the post you’re reading), the future Pope Leo’s chapter has no uses of “genuine” or “genuinely,” and 0 em-dashes in his own prose.7
(I welcome extensions of my analysis by people with access to the full thesis in print).
A common mark of LLM writing is the repeated invocation of tricolons: a series of three parallel words, phrases, or clauses used for rhetorical effect.
I noticed quite a few invocations of the tricolons in Magnifica Humanitas. It was especially notable in sections that otherwise had other tells of AI.
Unfortunately, unlike “genuinely” or em-dashes, this is harder to directly observe or baseline, as I can’t use the automatic “find” feature on chrome, it’s annoying to count by hand, and there are numerous edge-cases.
Nonetheless, I attempted to use my AI Agent Claude Code ( Claude Opus 4.7 1M XHigh) to give it a good college try, testing Magnifica Humanitas against 3 encyclicals by Francis, 2 by Benedict, 1 jointly by Benedict and Francis and 1 by Leo XIII (who wrote Rerum Novarum, which the current encyclical on AI is supposedly strongly based on).
I think this is partial confirmation of my hypothesis. Strict tricolons seem noticeably more prominent in pope Leo XIV’s writings than that of past popes we tested against.
Unfortunately (for my hypothesis) there is also substantial variation in the encyclicals authored/commissioned by previous popes. In particular, tricolons are much more common in writings by Benedict than by Francis. So this simple test is suggestive but does not rule out normal human variation.
Further, the LLM scan is a rough estimate. There’s inherent subjectivity in a question of triadic markers (unlike more direct vocabulary or punctuation tells). I welcome replication of my attempts here, either via a different AI agent methodology, or (preferably) someone more patient than me willing to manually count and verify this.
Do we have other sources of evidence to look at? Yes, we can use automated AI detectors, specifically Pangram.
Pangram is by far the best commercially available AI detector. It is much better than other AI detectors, so much so that other ones are almost useless in comparison. In particular, Pangram optimizes very hard for getting a false positive rate of nearly zero, while being more okay with false negatives.
This means that if you see text online that Pangram flags as AI, you should have very high confidence that it’s AI. In contrast, if you have some text that Pangram flags as 100% human, you should still be appropriately skeptical, especially if you’re otherwise suspicious. So how did yesterday’s encyclical do?
When I pasted the first twenty paragraphs of the encyclical in8, Pangram flags 11% of it as AI.
In particular, Pangram is very suspicious of Paragraphs 7-8.
For what it’s worth, I too was rather suspicious of this section. “It was an impressive feat: a single language, a single technology, a single direction.” sounded just a bit too neat to me.
Spot-checking different sections of the encyclical, we see a repeated pattern.
Some sections register as essentially 0% AI, while others seem much more AI-y.
This indicates to me that some senior Vatican officials who contributed to the encyclical used AI assistance heavily and most (probably including Pope Leo himself) did not. More on this later.
You might naturally be skeptical of Pangram’s analysis here. After all, maybe you haven’t heard of Pangram (or myself) until today. And besides, Pangram’s likely trained on internet data and its main use case is detecting AI in blog posts and social media posts and movie reviews and academic papers and so forth. What evidence do we have that it’s suited for a task as off-distribution as papal encyclicals? Maybe the detector’s just befuddled by unusually pious tokens?
To test this hypothesis, I ran Pangram on the previous 4 encyclicals. The first 20 paragraphs on all of them register as 100% human, all with high confidence:
I also tested writings by Pope Benedict and John Paul in case Pope Francis had an unusually human touch. And against encyclicals by Pope Leos XIII and XII (in both English and Latin) in case Pangram is prejudiced against Leos. All 100% human, as expected.
I also tested it against a transcript of Pope Leo’s speech announcing yesterday’s encyclical. 100% Human on Pangram. This is evidence that Pope Leo himself and/or his primary speechwriter does not use AI to draft his speeches.
(Incidentally, “genuine(ly)” appears 0 times in the transcript. Though em-dashes appear three times).
I think the encyclical evidence specifically should be quite convincing. But separately, I also strongly believe Pangram has a very low false positive rate in general. I observed this both in academic research and my own tests:
In the past, I’ve tested writings that I’m very confident is not AI (e.g. writings by myself, or from before 2021) dozens if not hundreds of times against Pangram, and repeatedly gotten 100% Human. This indicates to me that their advertised very low false positive rate is real.
In contrast, the false negative rate is much higher: Sometimes I’d ask an AI to read an outline by me and generate a draft, as a test. Pangram only catches that sometimes, though my impression is that they’ve gotten better in the last few months.
So you should generally trust that text that Pangram flags as AI is probably AI, while continuing to maintain some healthy suspicion of text that Pangram flags as not AI.
One potential mitigating factor is that this is a translation artifact: (human) senior church officials wrote the original encyclical in their native language (maybe Italian) and the translators, not the writers, were lazy and used substantially AI assistance in translating to English. I cannot rule this hypothesis out but currently think it’s very unlikely.
If the strong AI signs I’ve observed (“tells”) are a result of lazy translators to English, we should expect them to look different in other languages, especially the source language.
To test this, I gave all the tells I noticed in the English encyclical as well as the Italian version of the encyclical to two software agents – Claude Opus 4.7 and ChatGPT 5.5 Pro, arguably the strongest commercially available language models out there – to see whether the tells were preserved in Italian. Both agents believe the tells were preserved verbatim (See Claude screenshot below):
That said, I don’t speak Italian myself, and cannot personally verify the results. I welcome replications. Suspicious readers who speak Italian, and especially Italian native speakers who are familiar enough with how AI sounds in Italian, should try to verify the work for themselves!
If the English Pangram results are due to overly technophiliac or lazy translators of Italian to English, we should expect that the original Italian results will get 0% while the English translations gets flagged. We do not observe this (first twenty paragraphs again).
Instead, the Italian flagged sections appear to be a superset of the sections flagged in English. This would naively indicate that more, not less, of the Italian sections were drafted by AI.
(See further analysis by Daniel Filan)
That said, I don’t know how accurate Italian Pangram is. All the academic research I am aware of on Pangram’s accuracy (and my own experiments) are in English.
EDIT May 28: This paper by Italian academics from Istituto di Scienza e Tecnologie dell’Informazione finds a very low false positive rate (below detectable margin of error) for Pangram in Italian texts. H/T Daniel for suggesting this paper.
Another angle on the translation artifact hypothesis: Suppose you 100% organically human-write an article and then use AI to translate it. Does this even get flagged by Pangram?
As far as I can tell, the answer is no! (H/T Daniel Filan for this methodology)
I took a random excerpt of Fratelli Tutti in Italian and asked 3 leading frontier models (Gemini 3.1, ChatGPT 5.5, Claude 4.7) to translate it to English, with web search off. Each translation is different from the official Vatican translation (and from each other).
Pangram reads all three texts as 100% human. (Gemini results, Claude results, ChatGPT results)
This indicates to me further evidence that any Pangram-detected signs of AI in Magnifica Humanitas come from stylistic features that are present across languages, rather than artifacts plausibly introduced by AIs in translation.
The AI I use the most often for work is Anthropic’s Claude. I have a decent sense of its underlying rhythm, how it argues, and its favorite vocabulary and syntactic choices.9
I believe I identified the same voice of Claude in the recent papal encyclical about safeguarding the human person in the age of artificial intelligence. Somewhat ironic, considering.
Unfortunately, my primary reasons for believing this are somewhat idiosyncratic and inscrutable, and thus more likely to be wrong. And the “Claude authorship” hypothesis is of course overall less important to nail than the “AI authorship” hypothesis, all things considered.
Still, I want to offer some textual evidence favoring Claude over other models10.
The aforementioned “genuinely” strongly bears the fingerprints of Claude. It is very much the house style of Claude/Anthropic and regularly used in both internal and external Anthropic communications about Claude’s nature and in Claude’s repeated outputs. To be honest, I see “genuinely” so much these days that the word no longer feels like a real word to me and I’ve completely lost the ability to spell the word correctly by myself, never mind appreciating it in context.
Maybe there’s a metaphor in there about authenticity in the age of AI? Nah, I’m probably overthinking it.
Regardless, the repeated uses of genuinely in Magnifica Humanitas is strongly suggestive of Claude-specific fingerprints.
Applying the same analysis against ChatGPT fingerprints gives us a negative result. The words that I most associate with ChatGPT (“delve”, “meticulous,” “tapestry”, “goblins”) over other LLMs all show up zero times in the English encyclical.
This is moderate evidence against significant ChatGPT usage.
I don’t know enough about the tells that are specific to other models (Gemini, Grok, Mistral, DeepSeek, Kimi, etc). I welcome replications!
Significant fractions of the text in both English and Italian are flagged as AI. This varies significantly from section to section and paragraph to paragraph.
Some paragraphs in Pangram, and also by visual inspection, are clearly AI, while others seem clearly not AI.
My understanding is that popes don’t usually draft the majority of the text of their encyclicals themselves.
Between these two facts, this indicates to me that some senior Vatican officials heavily used AI assistance for this encyclical and many (probably including Pope Leo himself) didn’t.
My tentative hypothesis is that Pope Leo does not approve of the AI usage in encyclicals, and plausibly was not even aware of significant AI usage in his own encyclical! Quite unfortunate if true.
I attempted to provide a number of different angles and evidence for the conclusion that the papal encyclical on safeguarding the human person in the age of AI is actually an article in large part neither on nor in AI, but by AI. Any individual method might be flawed, but I believe the consilience of evidence is very strongly suggestive, perhaps even overwhelming.
Nonetheless, I welcome good-faith debate and corrections. Please feel free to comment with contrary evidence and replications.
We’re soon entering a time of unprecedented danger in the world. Like children with flamethrowers, or Bronze Age peasants attempting to build the Tower of Babel, humanity is messing with forces we cannot hope to understand or control.
In this new age of AI, getting provenance genuinuely right isn’t just a question of human authenticity — it’s a matter of life and death.