Joan Vendrell, NeuralTrust CEO and cofounder, has 15+ years of technology leadership experience advancing enterprise-grade AI security.
getty
I recently spoke with a CISO who was preparing for a major production rollout of an autonomous customer service agent. They had passed their traditional penetration tests with flying colors. But when I asked how the agent would handle a multi-step prompt injection attack that evolved in real time, there was a long silence. "We tested the model last month," they finally said. "But the agent is learning and interacting with live data every hour."
This is the fundamental challenge of the agentic era. Traditional security testing is a snapshot in time, while agentic AI is a continuous movie. At a time when agents are being granted the authority to execute workflows, call APIs and access sensitive databases, relying on a "one-and-done" security audit is like checking the locks on a house while the walls are still being built.
We are seeing a shift where the attack surface is not just the code or the network, but the reasoning process itself. If we don't move toward a model of continuous red teaming, we aren't just leaving the door open; we are handing the keys to the house to an autonomous operator we haven't fully vetted.
The core issue is that AI agents are non-deterministic. Unlike a standard application where input A always leads to output B, an agent’s behavior changes based on its context, its memory and the tools it has access to. This creates a playground for what I call "adversarial reasoning": attacks designed to corrupt the agent's logic rather than just its input.
Gartner predicts that by 2028, more than 50% of enterprises will use dedicated AI security platforms to manage these risks. The reason is simple: the OWASP Top 10 for LLM Applications has evolved. We aren’t just worried about simple prompt injections. We are now facing "agentic hijacking" and "indirect prompt injection," where an agent is manipulated through the very data it is supposed to analyze.
In my experience, traditional red teaming—where a human team spends two weeks trying to break a system—cannot keep up with the speed of AI development. We need a "machine-versus-machine" approach to security.
To secure the agentic enterprise, we must move beyond static testing and embrace a proactive, continuous defense. Here are five steps I believe every security leader should take.
If your agents are operating 24/7, your red teaming must do the same. You need to deploy adversarial agents with the sole job of finding weaknesses in your production agents.
This is about stress-testing the agent’s reasoning. Can it be tricked into bypassing a safety guardrail? Can it be convinced to escalate its own privileges? By using the MITRE ATLAS framework to map these attacks, you can automate the discovery of vulnerabilities before a malicious actor does.
The most dangerous part of an AI agent isn't the model, but the tools it can call. Red teaming must focus heavily on "insecure output handling," a top risk in the OWASP 2025 list.
You need to simulate scenarios where an agent is given a malicious command through a trusted tool, such as a compromised email or a poisoned database entry. Can an agent be tricked into executing a "delete" command on a database because it "reasoned" it was the right thing to do? Testing these boundaries is the new frontier of security.
The NIST AI RMF provides a critical taxonomy for managing AI risk. In my view, continuous red teaming should be mapped directly to the NIST "Measure" and "Manage" functions. This ensures that your testing isn't just a series of random attacks, but a structured validation of your risk tolerance.
By using a standardized framework, you can provide the board with measurable data on your AI security posture, moving from "we think we're safe" to "we know we're resilient."
One of the most insidious threats today is indirect injection. Imagine an agent reading a public website to summarize news, only to find a hidden instruction in the HTML that tells it to exfiltrate the user's session cookie. Your red team must continuously feed your agents "poisoned" data to see if they can maintain their instructions.
In my experience, the best defense is a zero-trust approach to agent inputs. Never assume the data an agent retrieves is safe.
During a red team exercise, pay close attention to how the agent’s identity is used. Does the agent maintain a clear chain of accountability when it’s under pressure? If an adversarial prompt tricks an agent into performing an unauthorized action, can you still trace that action back to the original human intent?
Continuous red teaming should validate that your identity lineage remains unbroken, even when the agent’s reasoning is compromised.
The rise of agentic AI is the most significant shift in enterprise technology in a generation, but it requires a new level of discipline. We cannot secure autonomous systems with manual, point-in-time processes.
In my experience, the companies that will lead the next decade are those that treat security not as a hurdle to be cleared, but as a living, breathing process. By embracing continuous red teaming, we don't just find vulnerabilities; we build the resilience necessary to let our AI agents move faster, do more and transform our businesses with confidence.
Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。