SYSTEMS

AI agents are a general-purpose workload no different from any other

OPINION Do AI agents need a new kind of CPU? That's what Arm, Nvidia, and a growing number of chip designers would have you believe. 

Arm named its first datacenter silicon the "AGI CPU." Nvidia CEO Jensen Huang described Vera as a "CPU for agents," and AWS's Graviton 5 marketing is chock full of references to agentic AI.

None of these Arm-based processors are going to bring about the singularity. They're not even AI accelerators. Don't let the spin doctors fool you – these chips are nothing more than general-purpose processors that have received an AI glow-up.

Sure, AI agents and their harnesses need CPUs. No argument there. But agents aren't one workload. They're simply a bridge between the AI model and the same applications we've been running for decades.

And the tools those agents end up running often look wildly different. Some will benefit from a higher ratio of memory bandwidth to compute, some will perform better on chips with large unified caches or dedicated compression engines, while others will prefer high frequency over core count, or vice versa.

There's a reason AMD and Intel don't just build one Epyc or Xeon SKU, and why all of the "purpose-built" agentic CPUs look so different. 

If you look at what Nvidia has built with its 88-core Vera CPU, the chip promises high single-threaded performance with gobs of memory and interconnect bandwidth.

As Huang explained it during his GTC Taiwan keynote, this combination of compute and bandwidth is key to keeping latency as low as possible.

"There will be billions of agents and these agents are going to be using the CPUs with very little patience because the cost of the GPUs they sit next to is too high," he said.

But of course Huang would say that – he's in the GPU-slinging biz. Vera, just like Grace, was designed to keep data flowing between the CPU and GPU as smoothly as possible. Data movement is literally Vera's thing.

Arm's AGI CPU, meanwhile, looks to be a bog-standard Neoverse V3 processor with 136 cores that's been stripped of anything an agent is unlikely to need in order to keep power consumption as low as possible. No simultaneous multithreading or dedicated accelerators, minimal vector extensions, but loads of memory bandwidth.

Amazon's 192-core Graviton 5 processors, announced at Re:Invent last winter, are essentially a scaled-up version of Arm's AGI CPU, right down to the Neoverse V3 cores, but arguably even more generic.

To echo Corey Quinn, "please, for the love of all that's holy, stop calling them 'AI chips.'"

Not to be left out of the fun, Intel and AMD have also been keen to recast their flagship Xeons and Epycs as the ideal platforms for running AI agents.

At Computex earlier this month, Intel showed off a couple of reference rack designs packing as many as 36,864 x86 cores into a 100 kW rack.

Meanwhile, AMD, following an initial round of Vera CPU benchmarks, went on the defensive last week, arguing that concurrency, not latency, is the metric that matters most when running agents at scale.

The House of Zen projects that for a 100 kW power envelope, its 256-core Venice Epycs, due out later this year, would deliver 3.3x higher throughput per rack than Vera.

If it feels like everyone has a different opinion on what the ideal agentic CPU should look like, that's because, as with any other datacenter workload, there's rarely one right answer.

We see this in early benchmarks of Nvidia's Vera CPU. Late last month, FOSS-friendly publication Phoronix got early access to the chip and ran a subset of its test suite that Nvidia apparently felt was representative of its target market.

The chip achieved a geo-mean score 10 percent higher than AMD's 128-core Epyc 9575F, and 55 percent higher than Intel's 128-core Xeon 6980P. That's a strong showing. But looking closer at the results, it becomes clear that Vera performs better in some apps than others.

And this gets to the crux of it all. There has never been one CPU to rule them all, and as the AI hype cycle enters its agentic era, there certainly isn't one now. ®