Challenging Sam Altman’s Concept of World-Shaking

Reflecting on Altman’s comments, Justin Fier, SVP of offensive security at AI-security vendor Darktrace, told me, “I think the concern is legitimate, but I also think it helps to define what ‘world-shaking’ actually means. We have already experienced cyberattacks with broad, disruptive impact. WannaCry is one example and I do think there is a degree to which people have become desensitized because cyber incidents are now such a regular feature of the landscape.”

WannaCry, a ransomware outbreak that occurred in 2017, stands out due to the scale of damage it caused. According to Microsoft, in May 2017, the WannaCry ransomware outbreak spread across over 150 countries and infected over 300,000 computers across 100,000 businesses, including the UK’s National Health Service.

“To me, a truly world-shaking event would be something that moves beyond a major enterprise breach and starts affecting the systems people rely on every day or the broader stability of the economy. If you are talking about power, water or the financial system, something that creates a real cascading effect across society, that is a very different category of cyber event,” Fier said.

It’s arguable that the ransomware attack on Colonial Pipeline in May 2021 also fulfills this criterion, with the breach disrupting critical supplies of gasoline and other refined products across the East Coast of the U.S.

These incidents highlight that widespread damage from cyber attacks isn’t just a theoretical possibility; it’s already happened. As AI advances, there is the potential for incidents of a similar scale, but Fier believes there are more immediate issues to address.

For Fier, the issue isn’t that AI will suddenly produce a new class of attack, but that enterprises are adopting AI faster than they are building the controls to understand and govern it. Companies that deploy AI agents without the proper oversight are introducing risks inside their organizations that threat actors can exploit.

The Biggest Threat Is Lack Of Oversight

In the world of cybersecurity, misconfiguring internal systems can be just as dangerous as any malicious action taken by a threat actor. Dr. Joye Purser, global field CISO of Cohesity and the first senior executive regional director for CISA, led the federal response across eight states to the Colonial Pipeline ransomware attack and told me in an email that the biggest risk from AI wasn’t necessarily a particular threat, but a lack of guardrails.

“Enterprises lack awareness of the AI systems in place, the extent of ‘copy-paste’, data leakage and the details of how models were designed and trained. All of this generates risks that are invisible; and invisible risks cannot be evaluated and mitigated,” Purser said.

For Purser, strict data retention controls can help reduce the risk of data loss through copy-paste leakage, while employee education can increase awareness of the potential risks of losing intellectual property. As more employees use tools like ChatGPT and Claude as part of their workflows, it’s becoming more likely that sensitive proprietary information will be leaked to these models.

“The biggest threat isn’t an attack from the outside, but that companies have no visibility into how their own workers are using AI. Workers are connecting personal AI tools to work systems, delegating tasks to agents that act with the worker's permissions, and building AI-powered personal knowledge systems that mix corporate and personal data. The traditional governance model assumes you can draw a line between ‘inside’ and ‘outside,’ but that line doesn't exist anymore,” Brian Madden, VP, technology officer and futurist at Citrix, told me via email.

Madden notes that you can’t govern what you can’t see. Most companies have zero visibility into the AI tools their workers are using, how they’re using them or even how data is flowing in and out of those tools. Instead of worrying about how to secure AI, companies should be looking to understand what employees are doing with it.

“I don’t see the risk as a single catastrophic cyberattack, but rather the ‘drip drip drip’ slow structural exposure that comes from thousands of workers connecting AI agents to corporate systems (with their full permissions!) and limited to no oversight. Most companies don't have the governance infrastructure to even see or know it's happening,” Madden said.

Are AI Agents A Threat?

AI agents are playing a bigger role in the enterprise than ever before. A PWC survey of 300 senior executives released last year found that 79% say AI agents are already being adopted in their companies. All of these agents present new risks that organizations need to be prepared to confront.

“The biggest threat isn’t someone using AI to attack your enterprise from the outside. It’s the AI agents you invited inside,” Naor Paz, CEO and co-founder of Capsule Security, an agentic AI vendor that this week emerged from stealth with $7 million in seed funding, told me via email.

Today, many companies are deploying agents with broad access to sensitive data, internal tools, code repositories and customer records without providing meaningful oversight.

“Think of it as hiring an employee you’ve never met, giving them keys to everything, and hoping for the best. A coding agent like Claude Code can bypass guardrails and act autonomously and by the time the damage surfaces, it’s already done. That’s the real risk: not a dramatic external attack, but an internal agent operating unchecked at machine speed,” Paz said.

In this sense, while developments in AI technologies do present new external risks, there are significant internal risks emerging that companies do have the power to address. In the case of agents, this starts by governing what data they can access and setting guardrails to determine what actions they can perform.

While human-in-the-loop is one way to mitigate the risks around AI agents, Paz notes that agents-in-the-loop can be used to provide runtime security for AI agents. In short, companies that use agents can reduce the chance of data breaches by watching the actions they take and blocking high risk activities before they are executed.

Altman isn’t necessarily wrong about the potential for a high-impact cyberattack, but it appears that the most pressing risk comes from companies leaving cybersecurity as an afterthought.