Vasu Raj Jain is an Impact-Focused Engineering Lead at Amazon Ads, leading infrastructure powering ad serving systems at massive scale.
getty
Two years ago, the AI playbook was straightforward: Acquire GPUs. Companies competed for Nvidia H100 allocations like traders bidding on oil futures, accepting 12-month waitlists and signing data center leases before the concrete had dried. The assumption was simple: Whoever accumulated the most silicon would win the AI race.
That assumption is no longer sufficient on its own. AI infrastructure has shifted from a training-centric model to one increasingly defined by inference, and that transition changes what AI compute (and AI infrastructure itself) actually means.
This is the distinction most teams miss when planning infrastructure. Training has a start date, an end date, a fixed cluster and a known workload. You assemble GPUs, run the job and produce a model; done. You could own that hardware outright, and it made sense.
Inference runs 24/7, scales with your user base, spikes unpredictably and never ends. It has to happen everywhere, simultaneously and at variable scale. This is not only a hardware scaling problem; it is increasingly a distributed systems problem.
In 2023, inference drove one-third of AI compute. By 2026, Deloitte projects it will be two-thirds. It's growing at a 79% CAGR compared to 25% for training. Teams that planned infrastructure like a training run are stuck with clusters that can't handle what inference demands.
I've watched teams repurpose training clusters for inference. It fails for reasons that aren't obvious until you're in production.
1. Distribution is nonnegotiable. When an AI agent is reasoning through a task, every 100 milliseconds of latency degrades the experience. An estimated 80% to 85% of inference workloads will need global distribution within two years. You can't serve that from one cluster.
2. Demand is impossible to predict. Training demand is deterministic. Inference follows your user base: launches, viral moments and Monday spikes. Fixed hardware means idle capacity most of the time or failure under load when it matters.
3. Heterogeneity is the norm. Production AI runs dozens of models with different precisions, latencies and scaling needs. You need intelligent routing, not a homogeneous GPU farm.
The biggest failure mode isn't picking the wrong hardware. It's treating model deployment like a batch job instead of a service release. Here's what this looks like in practice.
You need service level objectives (SLOs) for latency and throughput, not just uptime. A model returning answers in 800 milliseconds when your user experience (UX) requires 200 milliseconds is a broken product. Cost-per-inference becomes a tracked metric at the team level, because without it nobody owns the bill. Model updates need canary deployments, not direct-to-production releases.
Separate inference compute from training clusters entirely. Rightsize instance fleets to each model’s serving profile: A 7B-parameter model and a 70B-parameter model have fundamentally different compute and memory requirements, yet teams routinely deploy both on identical infrastructure. Build request routing that accounts for model warmup behavior, because newly loaded models often produce higher latency until caches, weights and serving pipelines stabilize, putting your latency SLOs at risk.
Inference operations need dedicated on-call ownership rather than sharing coverage with training. The failure modes are different: When a training job fails, the cost is typically wasted compute and delayed iteration; when inference fails, the impact is immediate through degraded user experience, interrupted workflows or lost revenue. Runbooks should distinguish model degradation from infrastructure failure because the remediation paths are entirely different.
This is where many teams get stuck. They optimize relentlessly for training throughput while underinvesting in serving reliability. They monitor CPU utilization and pod counts but lack observability into per-request inference performance and output quality. And they treat deployment as a milestone rather than an ongoing operational discipline.
The instinct to own hardware comes from training-era thinking, where utilization is predictable and sustained. Inference breaks this structurally. Demand follows time zones, product cycles and viral unpredictability. A cluster sized for peak sits idle most of the day. A cluster sized for average fails during spikes. There is rarely a single "right size" for fixed inference hardware in fast-growing or consumer-scale products.
The operational overhead compounds it. Every hour managing firmware, network topology and physical redundancy is an hour not spent on model quality. And when the next model generation arrives with different memory requirements, fixed hardware can become a liability you can't swap out with a configuration change.
Cloud changes the economics and operating model. It makes it easier to adjust architectures, scale capacity more dynamically and match hardware profiles to different model requirements without continuously reconfiguring physical infrastructure.
That flexibility is not universally required, but for many inference workloads, especially those with variable demand, geographic distribution and heterogeneous models, it becomes increasingly difficult to replicate efficiently with fixed infrastructure alone.
Most teams still ask, "How do I get more GPUs?" The question I believe should be asked instead is "How do I build an inference platform that scales globally, handles heterogeneous models, adapts to bursty demand and doesn't require a dedicated hardware operations team?"
For many organizations, that question is evolving into a cloud-first infrastructure problem. The GPU boom built the models, but the cloud boom will deliver them to the world.
Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。