ai and ml

AI agents can't be trusted, so don't give them dangerous powers

NanoClaw, a secure agent framework, has partnered with supply chain platform JFrog to allow AI agents to fetch resources from JFrog's reviewed registries.

Gavriel Cohen, creator of NanoClaw and co-founder of NanoCo AI, announced the tie-up on Thursday evening in San Francisco at a JFrog event that concluded with a World Cup watch party.

Cohen explained that one of the features of Claw agents – OpenClaw and variations like NanoClaw – is that they can improve themselves by fetching tools and resources that they don't have.

That works fine, he explained, when there's a manual approval process for accessing known local data. But it's not ideal for npm packages, even when the agent involved is sandboxed and isolated as it is in NanoClaw. Malicious code within a container may still be able to take harmful actions, even if the scope of potential activity is constrained.

Developers, Cohen said, may not be familiar with a given package and it can take time to thoroughly assess whether a package is legitimate and uncompromised.

"So we teamed up with JFrog and we integrated NanoClaw with JFrog's registries," said Cohen.

The arrangement provides a way to reduce the agent's exposure to untrusted content. When the agent downloads new tools and libraries, the software comes from a vetted source.

Cohen also announced the availability of what he called an agent factory, his company's homegrown system used to handle pull requests (PRs) using NanoClaw agents.

The agent factory, he explained, is an attempt to triage pull requests, which have surged thanks to AI coding agents.

"It's very easy now to point a coding agent at a repo and say, 'open a pull request for this repo,'" he explained. "And it's very difficult as a maintainer to tell the difference between a high quality contribution from somebody who's really using the open source project versus someone who's just trying to build up the reputation [using automated methods]. So to help us tackle this, we built an agent factory that helps us review every single contribution to NanoClaw."

The agent factory is referred to as the PR Factory in the actual pull request. It's built with NanoClaw and hosted on exe.dev, a service that provides VMs with persistent storage.

"When a PR opens, the factory spins up a dedicated worker agent for it, posts a thread to Slack, and the worker triages the change, reviews the diff, and proposes a test plan," Cohen explains in the documentation. "Nothing consequential happens on its own: merges, test runs, and credentialed GitHub actions each surface as an approval card in the thread, and only fire when a human clicks approve."

Cohen acknowledged that some developers will think it's madness to process unsanitized PRs that could contain prompt injections or unsafe code. And he asked the assembled audience of developers how many had seen the phrase on the projected slide: "Never, ever, ever do this."

Anyone who has spent time using and configuring AI agents in a development context has seen something of the sort in configuration files like Claude.md, which gets loaded as instructions to the underlying agent and model.

"If you see something like this in the Claude.md file and the agent instructions say, 'Important: Never run drop database production,' it tells you two things. You know that that agent has deleted a production database before. And you know that it can actually still do it again. That's why the instruction is there."

This elicited a knowing laugh from the audience.

Cohen went on to say that the agent will do it again because instructions are not a way of enforcing security or safety.

"Instructions help steer an agent AI towards valuable output, but it's not a safety mechanism," he said. "The only way to reliably prevent an agent from taking undesired action is not allowing it to take that action, not giving it the ability to take the action."

That is the purpose of NanoClaw. ®