The first wave of generative AI infrastructure was defined by model training. The next phase will be defined by inference.
As large language models move into production, organizations are discovering that the real challenge is not building models, it’s running them. Inference workloads must deliver responses in real time, operate at large scale, and do so within a cost structure that makes enterprise deployment sustainable. Those requirements are forcing a rethink of how AI infrastructure is designed.
The partnership announced between AWS and Cerebras reflects that shift. Rather than treating inference as a single computational problem, the two companies are introducing what they describe as a disaggregated inference architecture. The system separates the two fundamental stages of inference — prompt processing and token generation — and assigns each stage to the processor architecture best suited for the task.
In the proposed deployment, AWS Trainium3 handles prompt processing (“prefill”), while Cerebras’s CS-3 system performs token generation (“decode”). The two stages are connected through AWS’s high-performance networking infrastructure (Nitro EFA) and exposed to developers through Amazon Bedrock.
The objective is straightforward: increase token throughput and reduce latency by allowing each processor to perform the work it is best designed to do.
If successful, the approach could signal a broader shift in AI infrastructure design. Rather than homogeneous clusters built around a single processor architecture, future AI platforms may increasingly resemble distributed pipelines of specialized accelerators — each optimized for a specific stage of the AI workflow. But what is the exact nature of this partnership, and what does it means for customers — and the market in general?
In short, AWS and Cerebras announced a collaboration to deploy Cerebras CS-3 systems within AWS datacenters, delivering what the companies describe as the fastest AI inference capability available on a hyperscale cloud platform.
The architecture combines three primary components:
The system will operate natively within AWS infrastructure and be accessed through Amazon Bedrock, AWS’s managed platform for building generative AI applications. In this setup, developers interact with the system through APIs while AWS manages the underlying orchestration.
AWS and Cerebras have positioned the architecture as an inference system designed to maximize token throughput and response speed, two metrics that are becoming increasingly important as generative AI workloads scale.
To understand the significance of this architecture, it’s useful to dig into how inference works. Inference is often described to the user as a single operation: Submit a question (i.e., a prompt) and receive a response. The reality is that inference comprises two workloads: prefill and decode.
Prefill processes the user’s request and all of the contextual information supplied with it. This includes:
During prefill, the model ingests the full input sequence and computes the internal attention states required to generate output tokens. Because the system must load and process the entire context window, it benefits from large memory capacity and highly parallel computing. The bigger the memory stack is, and the more lanes there are, the faster context can be loaded.
If you think about this in really simple terms, prefill answers this question: What information does the model need to understand before it begins generating a response?
The second phase, decode, generates the model’s response one token at a time. In other words, it is inherently serialized. This means each token must be generated before the next token can be produced. Because of this, decode is a latency-sensitive process where token generation speed becomes the primary performance constraint.
Decode workloads typically require:
When looking at real-world inference, decode accounts for the majority of total inference time. While the HBM found in GPUs is fast (100- to 300-nanosecond latency), SRAM (static RAM) is much faster (1- to 5-nanosecond latency). This 10x improvement in decode speed means that responses to generative AI prompts come back much faster with SRAM.
The importance of decode performance is increasing as real-world enterprise AI adoption accelerates. Early generative AI workloads often produced relatively short responses, so prefill was where the most resources were required. As models evolve and newer reasoning-oriented models and agentic systems are deployed, we see significantly longer outputs being generated.
In these cases, a model may generate hundreds or even thousands of tokens while solving a problem, writing code, or completing a multi-step reasoning task. And because tokens must be generated sequentially, the time required to complete inference depends on how quickly each token can be produced.
This goes back to architecture. As decode performance becomes a primary factor in the responsiveness of real-world AI applications, architectures that accelerate token generation offer a direct advantage for agentic AI, interactive assistants, and code generation, among other applications. In other words, the workloads where we see AI showing up today.
And this is what makes the AWS–Cerebras partnership and architecture so compelling. It specifically targets this emerging bottleneck by assigning the decode stage to hardware that is optimized for rapid token generation.
Traditional inference deployments typically run both prefill and decode on the same processor architecture, most commonly GPUs. While this simplifies infrastructure design, it forces the hardware to perform two workloads with very different characteristics.
The AWS–Cerebras architecture separates those workloads.Trainium3 performs prefill, processing large prompt contexts using highly parallel compute married with HBM. And Cerebras CS-3 performs decode, generating tokens rapidly using its high-bandwidth wafer-scale architecture. These two stages are connected through AWS’s high-performance infrastructure stack.
We can think about this architecture as a pipeline:
Application
↓
Bedrock API
↓
Prefill — AWS Trainium3
↓
Nitro EFA high-speed interconnect
↓
Decode — Cerebras CS-3
↓
Generated tokens returned to application
From the developer’s perspective, the system behaves like a single inference service. The disaggregation occurs beneath the service layer.
For disaggregated inference to work efficiently, the handoff between prefill and decode must be seamless and fast, as any excess interconnect latency between chips would undermine the architecture’s performance gains. This is where AWS’s infrastructure becomes important as Nitro plays a critical, if unheralded, role. Nitro provides the hardware and software architecture that underpins AWS compute services. It offloads virtualization, networking, and security functions from the host CPU to dedicated hardware components, allowing compute resources to operate with minimal overhead while maintaining strong workload isolation.
On top of Nitro sits the Elastic Fabric Adapter networking. EFA is AWS’s high-performance networking technology designed for tightly coupled compute workloads such as high-performance computing and large-scale AI training. It enables extremely low-latency communication between compute nodes using a remote direct memory access (RDMA) model.
Within the Trainium–Cerebras architecture, EFA enables the prefill stage output to be transferred to the decode stage with minimal delay. And of course, this fast handoff is critical to the latency-sensitive decode function. In effect, Nitro EFA provides the data pipeline that allows the two processors to function as a coordinated inference system.
Cerebras occupies a distinctive position within the AI accelerator ecosystem. Most AI infrastructure platforms scale by distributing workloads across clusters of processors such as GPUs. In these systems, many processors work together, communicating continuously through high-speed interconnects.
Cerebras takes a different approach. The company’s Wafer-Scale Engine (WSE) integrates an entire AI compute fabric onto a single silicon wafer. Instead of cutting the wafer into many individual chips, the entire wafer is used as one processor. This creates the largest processor ever built for AI workloads.
To put it in perspective, think about it this way:
This comparison doesn’t indicate that WSE is better than the H100. Rather, it is an entirely different architecture addressing an entirely different challenge. The wafer-scale architecture enables massive on-chip memory bandwidth, extremely fast communication between compute cores, and the need for fewer inter-chip interconnects. And because the processor operates as a single large compute fabric, many of the communication bottlenecks that arise in large GPU clusters can be avoided.
These characteristics make the architecture particularly well-suited for decode workloads, where rapid access to model weights and attention states directly affects token-generation speed.
Another practical implication of the architecture is that AWS can expose multiple inference configurations within the same Bedrock environment. Enterprises using Bedrock will likely be able to choose between different infrastructure profiles depending on performance requirements. A standard Bedrock environment would run inference entirely on Trainium infrastructure, which is well-suited to a broad range of generative AI workloads.
For applications that require faster token generation or lower latency, organizations could select a Trainium–Cerebras inference path. In this configuration, processing of AI prompts occurs on Trainium while token generation is offloaded to Cerebras hardware.
This approach allows AWS to introduce differentiated performance tiers while maintaining a consistent interface through Bedrock.
The AWS–Cerebras collaboration highlights several broader trends shaping the next phase of AI infrastructure.
The partnership between Cerebras and AWS demonstrates how AI infrastructure is evolving as AI transitions from experimentation to enterprise production. Early deployments treated inference as a simplified extension of training, typically running on the same GPU clusters used to build models. As workloads scale, however, inference is emerging as a distinct infrastructure challenge.
By separating prompt processing (prefill) from token generation (decode) and assigning each stage to specialized hardware, AWS and Cerebras are introducing a new architectural model for large-scale inference. For AWS, the partnership also introduces a form of differentiation. By integrating Cerebras hardware into its infrastructure and exposing the capability through Bedrock, AWS can offer inference performance characteristics that are not currently available from any other CSP.
Is disaggregated inference the future? That depends on how solutions are packaged, sold, deployed, and managed. Regardless, one thing is clear: The next generation of AI infrastructure will increasingly favor specialization. Future AI platforms may look less like homogeneous clusters of identical processors and more like coordinated systems of specialized accelerators — each optimized for a specific stage of the AI pipeline.
And I think Cerebras is going to play a prominent role.
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。