paas and iaas

AWS better at running chip fabs than their mouths

Amazon, along with the rest of the industry, has gotten so used to framing everything that happens through the context of AI that it has lost the plot on their Graviton chip lineup, and along with it their own credibility. Which is a shame, because it's actually a triumph of a chip.

First, the Wall Street Journal breathlessly reported that Snowflake's $6 billion AWS commitment was "for agentic computing chips." Then AWS's own press release heralded the release of their latest chips "for the Agentic AI era." In both cases, they were referring to their Graviton line.

You could be forgiven for thinking this was some kind of GPU. No, that's Trainium. (Technically, Trainium isn't a GPU, nor is it a CPU, but rather a systolic array. Don't worry; most AI engineering software doesn't know what the hell that is, either.) Graviton is AWS's general purpose Arm CPU, which can be used for AI in much the same way as Excel can be used as a database. But that's far from its only, or even primary, purpose.

Let's dive into what Graviton actually is.

Price / Performance / Reality

For the longest time, Amazon refused to issue benchmarks, competitively positioning its then-nascent Arm line against Intel. Many of us thought this meant that the results would underwhelm — so you can imagine my surprise when real-world workload tests showed 35 percent to 40 percent better performance in a wide variety of situations. It was as if Amazon had built something amazing, but was somehow embarrassed to admit it.

Those days are long behind us; they trumpet in the subhead of their announcement that Graviton 5 means "apps run 35% faster, ML inference is 35% faster, and databases are 30% faster." To their credit, I was expecting those numbers to be against something ancient, but in a refreshing bout of honesty, they're comparing them to Graviton 4, itself no slouch.

They are also 9 percent more expensive.

Once upon a time, new generations of AWS instances were notably less expensive than their predecessors. Going from a c4.large to a c5.large meant you'd get better performance, and the instance itself was a whopping 15 percent cheaper. Upgrading was a no-brainer!

That started changing, and now upgrading means the instance becomes more expensive. AWS's position is that this is an incomplete analysis, since the improved performance means you'd pay less for a given workload. In some cases, this is correct, but in others, it's akin to saying that a Ferrari offers better price performance than my Honda CR-V because I can drive it to work three times faster. Logic, as well as traffic lights, disagree.

Amazon's contention is correct for customers who have large fleets of nodes that they run at high degrees of CPU utilization. Switching those fleets to the new hotness will absolutely result in a price performance improvement, provided the workload and the stars both align.

However, for customers who need a fixed number of nodes (think database companies, who offer each customer of theirs a set number of replicas, or workloads of the form "each environment gets three nodes, one in each AZ"), this represents a pure 9 percent price hike going from old generations to new ones. That puts many customers in a pickle: upgrade to new instance families, or stay on the old ones and watch availability become constrained in the coming years as AWS stops racking old chips. (Hi, Amazon PR! If you're about to pop into my inbox to tell me that won't happen, I have a customer I'd love for you to have a chat with!)

But this price hike isn't happening in a vacuum. It's happening against a backdrop of "an 8GB Raspberry Pi is now $175, over twice its launch price of $85." Components have become fiendishly expensive across the board as giant companies compete for capacity, and AWS has to be feeling that pressure. Two companies each asked to buy all of AWS's Graviton capacity for the year; AWS clearly has room to kick their prices into the stratosphere! Somehow, they're not only resisting the siren song of "please gouge me, business daddy," but also managing to keep availability strong for customers of all stripes; I upgraded my developer node in my tiny unremarkable AWS account yesterday, and it Just Worked.

And so...

Despite the nonsense marketing, I don't want to detract from just how amazing Annapurna Labs (Amazon's chip division) has been at churning out wildly performant silicon year over year. Their chips are legitimately great, and the Graviton 5 numbers are a triumph. Lost against the backdrop of "Agentic AI," the stuff underpinning all of it continues to work, improve, and largely pass by unremarked.

Keep going. ®