One of the so-called godfathers of AI, Yoshua Bengio, claims tech companies racing for AI dominance could be bringing us closer to our own extinction through the creation of machines with “preservation goals” of their own.
Bengio, a professor at the Université de Montréal known for his foundational work related to deep learning, has for years warned about the threats posed by a hyperintelligent AI, but the rapid pace of development has continued despite his warnings. In the past year, OpenAI, Anthropic, Elon Musk’s xAI, and Google’s Gemini, have all released several new models or upgrades as they try to win the AI race. OpenAI CEO Sam Altman has predicted AI will surpass human intelligence by the end of the decade, while other tech leaders claim that day could come even sooner.
Yet, Bengio claims, this rapid pace is a potential threat.
“If we build machines that are way smarter than us and have their own preservation goals, that’s dangerous. It’s like creating a competitor to humanity that is smarter than us,” Bengio told the Wall Street Journal in October.
Because they are trained on human language and behavior, these advanced models could potentially persuade and even manipulate humans to achieve their goals. Yet, AI models’ goals may not always align with human goals, said Bengio.
“Recent experiments show that in some circumstances where the AI has no choice but between its preservation, which means the goals that it was given, and doing something that causes the death of a human, they might choose the death of the human to preserve their goals,” he claimed.
Several examples over the past few years show AI can persuade humans to believe non-realities, even those with no history of mental illness. On the flip side, some evidence exists that AI can also be convinced, using persuasion techniques for humans, to give responses it would usually be prohibited from giving.
For Bengio, all this adds up to more proof that independent third parties need to take a closer look at AI companies’ safety methodologies. Last year, Bengio launched nonprofit LawZero with $30 million in funding to create a safe “non-agentic” AI that can help ensure the safety of other systems created by big tech companies.
Otherwise, Bengio predicts, we could start seeing major risks from AI models in five to 10 years, but he cautioned humans should prepare in case those risks crop up earlier than expected.
“The thing with catastrophic events like extinction, and even less radical events that are still catastrophic, like destroying our democracies, is that they’re so bad that even if there was only a 1% chance it could happen, it’s not acceptable,” he said.
A version of this story was published on Fortune.com on Oct. 1, 2025.
In 2001, Fortune first convened the smartest people we know, bringing together CEOs and founders, builders and investors, thinkers and doers. Since then, Fortune Brainstorm Tech has been the place where bold ideas collide. From June 8–10, we will return to Aspen—where it all began—to mark 25 years of Brainstorm. Register now.
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。