“You can’t secure AI agents as a bolt-on” was one of the key technology recommendations at Friday’s Coffee With Digital Trailblazers. We were discussing From POC to Production: The Scaling Decisions That Make or Break Delivering Business Value.

I asked the speakers, “What implementation choices do teams most regret later — the corner they cut in the POC that comes back to bite them in production?” This was a softball question, pitched to Derrick Butts, founder and vCISO at Continuums Strategies, who answered by declaring that security by design, especially in AI implementations, must start before the POC.

I’ve heard some experts suggest that organizations treat AI agents like employees. Others are more pragmatic and equate AI as tools powered by non-deterministic language models. But a better analogy is suggested by Victor Coimbra, CTO at Artefact.

“AI agents are employees you can’t fire, and once deployed, they act autonomously, make decisions, and carry permissions you gave them,” says Coimbra. “Guardrails aren’t optional: in any factory, you don’t fire the machine when it malfunctions, you build the safety controls before it ever runs. CIOs need to treat every agent like critical machinery, define its scope tightly, monitor it continuously, and never assume it will self-correct.”

Why security first? CSA’s report on AI agent incidents, now common in enterprises, finds that 65% of respondents reported at least one in the past year. Impacts include 61% reporting data exposures and 43$ experiencing operational disruption (43%).

And in Genpact’s report on autonomy requires trust in AI, over 30% site compliance/regulatory and risk/reputational damage as the top two concerns, making it hard to trust agentic systems with autonomous action.

I had to follow up on this topic on the key requirements for securing AI agents. Experts weighed in.

1. Create separate identities for AI agents from users

AI agents are being given more open-ended capabilities than most machines, which are locked into their functions. They orchestrate workflows across MCP integrations, and a key requirement is knowing who can do what. I don’t believe AI agents should be managed like people, but they need identities.

“One of the biggest mistakes I’m seeing is having AI agents assume the identity of the delegating user rather than the agents having their own distinct identity,” says Gilad Shriki, co-founder at Descope. “Humans directly sharing credentials with AI agents leads to a greatly increased risk surface in case the agent goes rogue, the employee that provisioned the agent is offboarded, and so on. If you can’t audit who the agent is, on whose behalf it’s acting, and what specific actions it can perform, you’re trading current “go lives” for future security issues.”

2. Separate entitlements for data sources, people, and AI agents

I expect enterprise CIOs to manage thousands of production AI agents that connect to even more data sources and are deployed across several top SaaS platforms and emerging startups. Many will look to build and deploy proprietary AI agents. Top companies will employ a bottom-up approach to entitlements, defining them at the data source level.

“Every AI agent you deploy inherits the permissions and data access of the systems it connects to, which means your enterprise risk surface expands with each integration,” says Pranava Adduri, co-founder and CTO at Bedrock Data. “My advice to CIOs: before you worry about prompt injection or model guardrails, map every data source and identity credential your agents can access, enforce least-privilege access at the agent level, and treat agent-to-system connections with the same rigor you’d apply to a third-party vendor. If you don’t govern what an agent can see and whom it can act on behalf of, you’ve handed out skeleton keys to your enterprise.”

Others weighed in on the intersection of what data a user can access and how to simplify assigning entitlements to role-based AI agents.

“Define clear roles and permissions for each agent aligned to job functions; agents should not access sensitive data their human counterparts cannot,” adds Ross Meyercord, CEO at Propel Software. “Equip agents with access controls and process guardrails; without governance, they can’t reliably protect sensitive data or ensure compliant actions.”

3. Outpace AI with strong data governance fundamentals

Beyond establishing data owners and entitlements by data source, my data governance non-negotiables include benchmarking data quality, auditing for data biases, and assigning data privacy management responsibilities.

“As AI agents move from passive assistants to autonomous actors, CIOs must realize that the agent is the new identity and the data it accesses is the new perimeter,” says Rick Holland, CISO at Cyera. “It is mission-critical to have strong AI and data governance frameworks in place to ensure teams can innovate with AI agents and scale safely. Ultimately, true AI agent security is rooted in deep visibility — you must know your data’s posture before you can safely empower an agent to interact with it.”

Regulated industries such as financial services have a long history of implementing data governance. But even they have to extend their policies, requirements, and implementations to support AI innovation aligned with governance, and not governance trailing AI.

“One of the biggest challenges with securing the quality and consistency of the output of AI agents is making sure they understand the data they’re acting on,” says Saurabh Gupta, president and CEO at The Modern Data Company. “AI agents don’t have institutional knowledge or human judgment to fill in the gaps, and it’s like hiring a PhD with no real-world experience. They need data with a clear business context, governance, and built-in trust from the start. The organizations that will be successful with agentic AI are the ones that treat trusted data as the foundation for AI, not just fuel for the models.”

4. Keep the keys away from the AI agents

Several security-by-design principles stand out for organizations developing AI agents and applications using code generators, vibe coding tools, and spec-driven development practices. DevOps non-negotiables around key management have even more importance when implementing AI agents.

“For a CIO or AI team, the highest-leverage security play is moving credentials out of the image and into the runtime environment,” says Will Barker, cybersecurity advisor at Huntress. “Mandate that every agent image be built without keys, then use a secrets manager to mount tokens directly into an unprivileged service user at startup. This ensures your images are portable and, more importantly, that a leaked image doesn’t mean a leaked production environment.”

Nicole Beckwith, senior director of security engineering and operations at Cribl, adds that insider risk is equally important to address. “Developers are unintentionally pasting secrets and credentials into publicly trained AI tools, which can lead to accidental access to your environment,” says Nicole.

5. Standardize a secure software supply chain

In another episode of Coffee With Digital Trailblazers on AI coding competencies, one speaker relayed his experiences vibe coding applications. He’d written full functional specifications, but hadn’t included any non-functional requirements. AI generated a 10,000-line application, forcing him to look for tools to determine what’s inside. A more proactive approach is to define policies on which software components AI code generators are allowed to use.

“The biggest mistake CIOs can make is treating AI agents like trusted developers before they have access to trusted information, says Brian Fox, CTO and co-founder at Sonatype. “An agent that recommends an outdated, vulnerable, or malicious dependency can move that mistake through the SDLC faster than any human team. Secure agents by grounding them in real-time software supply chain intelligence and policy, so they can choose safe components instead of simply generating plausible answers.”

6. Plan for runtime anomalies with robust observability

“It worked fine in dev and test” was what developers used to say in the days before CI/CD, continuous testing, and infrastructure-as-code to standardize cloud environments.

But what works in an AI POC can yield drastically different results in production as AI is non-deterministic, models drift, and the data evolves. Without observability, site reliability engineers (SREs) will not know if an AI agent is cutting down the forest.

“CIOs should secure AI agents the same way they secure privileged human users with tightly scoped access, continuous verification, and full visibility into what those agents are actually doing,” says Quentin Rhoads-Herrera, VP of security services at Stratascale. “You need runtime observability like decision logs, action auditing, and drift detection so when an agent makes a call, you can see exactly what it decided, why, and what it touched.”

Safeguards for observable AI agents include developing evals for dangerous recommendations, integrating observability with threat detection, and evaluating AI agent performance.

This short list doesn’t include the alignment between AI strategy and governance. As models evolve, world-class IT organizations are updating their security requirements to reflect the risks of deploying AI agents.