Google recently unveiled the eighth generation of its Tensor Processing Units (TPUs) at Google Cloud Next. The most significant change from prior generations: Google bifurcated the TPU line into two distinct chips. The TPU 8t is purpose-built for training. The TPU 8i is purpose-built for inference. (Both chips are expected to reach general availability later in 2026.) This decision is the most meaningful aspect of the announcement, and it reflects a recognition that for too long the industry has been optimizing for two fundamentally different problems under a single architectural model.
Google Cloud’s broader announcements wrapped these chips in a tightly integrated system: the Virgo interconnect fabric, Axion Arm-based CPUs replacing x86 host processors, a Managed Lustre storage system delivering 10 TB/s aggregate bandwidth, and updates to Google Kubernetes Engine (GKE) for agent-native workload orchestration. This framing is important, as Google is presenting a controlled environment where silicon, software, and infrastructure are designed, optimized, and consumed together. This is not a set of components. It is a vertically integrated AI platform, aptly named the AI Hypercomputer.
It is worth digging into the details of the new Google TPU and how these announcements tie to the coming enterprise AI wave.
Training and inference place genuinely different demands on infrastructure. Engineering for one creates friction for the other. Google’s decision to stop the compromises of a one-size-fits-all approach is grounded in that reality.
Training is a throughput problem, with a dash of gradient computation required. The objective is to keep as many accelerators as possible doing useful work as much of the time as possible. Coordination overhead, network stalls, and failure recovery all directly impact time to completion. Google measures training effectiveness as “goodput” (the fraction of time the cluster is actually training). For this, the company claims a 97% goodput rate for TPU 8t clusters. In terms of frontier model scale, every percentage point of “goodput” translates into days of additional training time.
The TPU 8t seems to be designed specifically to address this: 12.6 petaflops of 4-bit floating point (FP4) compute, 216 GB of high-bandwidth memory (HBM) at 6.5 TB per second, 128 MB of on-chip SRAM, and 19.2 TB per second of chip-to-chip interconnect bandwidth. It also retains the SparseCore blocks from the prior TPU generation (Ironwood), which are specialized for recommendation models and mixture-of-experts architectures.
Inference behaves differently. In auto-regressive generation (the process behind most conversational AI), tokens are generated one at a time and must reference a growing key-value cache with every token. In this model, memory bandwidth becomes the constraint, and latency is the key metric.
TPU 8i, as one would imagine, was designed specifically to address this requirement. On-chip SRAM triples to 384 MB, enabling more of the key-value cache to reside on-chip. This prevents a slower round trip to HBM. And those SparseCore blocks in Ironwood are replaced by a Collective Acceleration Engine (CAE), which Google says reduces communication latencies between chips by up to 5x.
These design choices make perfect sense. The question, though, is who these chips are designed for.
The TPU 8t is an impressive piece of engineering. Optical circuit switches connect up to 9,600 accelerators in a single training pod. This is more like a telephone switchboard than a packet-switched network, as it physically routes light paths between chips on demand. The Virgo Network extends this to 134,000 TPUs within a single datacenter and, by Google’s account, more than one million TPUs across multiple sites. For comparison, NVIDIA’s NVLink domain supports up to 576 GPUs before requiring scale-out over Ethernet or InfiniBand, though NVIDIA would have a legitimate argument that this is not a fair comparison. Regardless of the NVIDIA comparison, this level of optimization implies that the customer is running at hyperscale.
Of course, there are very few enterprises running AI at this scale. They are not training frontier models, and they are not running clusters of thousands of accelerators. Even organizations that fine-tune models rarely approach the scale that requires this kind of system-level coordination. The TPU 8t is best understood as infrastructure for Google itself and for a narrow set of external partners.
For enterprise IT, the TPU 8t sits one layer removed. It’s influencing the capabilities of the models they consume rather than the infrastructure they deploy. Put more plainly, it’s a design point, not a procurement consideration.
Inference is where enterprise relevance should emerge, and Google has clearly invested in improving performance at the chip level. The TPU 8i targets the right constraints with more on-chip memory, higher HBM capacity, and reduced inter-chip communication latency. For MoE models and memory-intensive workloads, these are meaningful improvements.
What stands out to me has less to do with the improvements to the architecture. Rather, it’s about what I didn’t see.
To set this up, we need some context: Inference workloads are evolving beyond steady-state request-response patterns. Agentic AI is all the rage, and agentic systems introduce pauses, tool calls, state persistence, and asynchronous execution. In agentic AI, workloads become less predictable and more fragmented. In this environment, a tightly synchronized accelerator cluster can struggle to maintain high utilization.
In response to this dynamic, AWS recently announced a partnership with Cerebras to disaggregate inference into prefill (context) and decode (token generation). (I wrote about this in detail a couple of months ago.) Likewise, one of the major themes at NVIDIA’s GTC was this very same topic, as the company demonstrated at-scale inference, where Rubin and the Groq LPU combine to improve performance and token economics. (More on that here.)
Notably absent from Google’s TPU 8 announcement is a clear narrative (and perhaps strategy) for disaggregating inference. I saw no explicit separation of prefill and decode stages, no indication of independent scaling between compute and memory resources, and no visible orchestration layer built for fragmented, long-lived workloads. Parts of the market are moving in that direction. This announcement does not.
With this said, the TPU 8i chip has an abundance of both SRAM (critical for the serialized token generation/decode function) and HBM (critical for the parallelized nature of prefill). And its CAE helps with the interconnect, if not orchestration. But it feels like Google could have been a bit more explicit about its disaggregation story.
For enterprises running AI applications on Google Cloud (particularly Gemini-based services through Vertex AI or the new Gemini Enterprise Agent Platform), TPU 8i improvements will eventually flow through to those services. The claimed 80% improvement in performance per dollar for inference could be meaningful for organizations paying for token consumption at scale, though its real utility remains unclear. What would be better is a comparison to competing alternatives — not just generation-over-generation improvements by Google. For enterprise customers looking to place strategic bets, the 80% claim carries no weight. If I’m the CIO, give me real-world numbers around performance and cost that enable me to compare Google to everybody else.
So far, there are no published results from independent benchmarks such as MLPerf. No standardized comparisons across common workloads. No visibility into how these systems perform under real-world enterprise conditions — concurrency, variability, cost constraints.
For what it’s worth, this isn’t a Google-specific issue. The broader AI infrastructure market has normalized to announcing performance claims in proprietary benchmarks against prior-generation baselines on vendor-selected workloads. This is maddening to an enterprise ITDM because it limits the ability to make informed decisions. The industry should be expected to do better.
Google is pairing both types of TPU with its Axion processors — Arm-based custom CPUs it introduced in 2024. This approach is replacing the x86 host processors the company previously deployed. For reference, AWS made the same move by pairing Graviton with Trainium 3 earlier this year. The technical and strategic motivations are aligned, and enterprises should assume that both of those motivations are in play.
I think the performance argument tied to Axion is real. CPU cores handle the orchestration tasks surrounding AI workloads: agent logic, tool calls, feedback loops, reinforcement learning reward calculation, and inference scheduling. And it only makes sense that Google, like AWS (Graviton) and Azure (Maia), can tweak its Axion architecture for these specific performance characteristics — in addition to general-purpose compute.
With this said, the economic argument is also real. By replacing merchant x86 silicon with in-house designs, Google gains control over cost structure, supply chain, and product differentiation. As mentioned, this is the same path taken by other hyperscalers. Enterprises should factor both motivations into their evaluation.
Google frames all of this within its AI Hypercomputer concept — a full-stack platform for building and running AI at scale. AI Hypercomputer bundles TPUs or GPUs, the Virgo Network, managed storage, Google Kubernetes Engine, and a software layer supporting JAX, PyTorch, and vLLM.
NVIDIA uses comparable language for its AI Factory concept: integrated clusters of accelerators, networking, systems, and software delivered as a complete platform. The meaningful distinction arises when we look at delivery model and portability. Google’s Hypercomputer is a cloud service consumed on demand and billed by usage. NVIDIA’s AI Factory spans cloud deployments through hyperscaler partnerships and on-premises hardware available for direct purchase.
An enterprise that needs to own and operate its AI infrastructure has a clear path with NVIDIA. The equivalent path with Google’s TPU infrastructure doesn’t seem to exist outside Google’s own datacenters. This gap may not matter yet, but it will matter very quickly to regulated industries, government customers, and any organization with data sovereignty requirements, which will list on-prem deployment as a non-negotiable requirement. And in these on-prem deployments, even Google Distributed Cloud only supports NVIDIA.
As touched on a moment ago, Google Cloud TPUs support JAX, PyTorch, and the vLLM inference engine. The vLLM integration in particular has come a long way. It enables third-party open-weight models to run on TPU infrastructure; confirmed support includes Meta’s Llama family, Alibaba’s Qwen family, and Google’s own Gemma series.
But there are some real constraints. vLLM on TPU currently supports only single-host configurations on GKE. For models with more than 400 billion parameters, a multi-host deployment is required, which necessitates Google’s own JetStream inference engine. This can lead to portability issues (including, notably, no NVIDIA interoperability). Native PyTorch support (TorchTPU) in preview mode will address the model code portability challenge. But it doesn’t solve for the reality that JetStream is a separate inference engine. So there is no common inference server layer between TPU and GPUs.
The result is a platform that is already more open than it was and becoming still more open. But it’s not yet completely frictionless for enterprise adoption.
The customer picture is also still thin. Anthropic announced a significant partnership with Google for 3.5 gigawatts of TPU compute starting in 2027. Additionally, Google’s own Gemini models, Search, Photos, and Maps are the primary internal consumers.
But enterprise representation is still lacking, in my estimate. While Google listed “enterprise” customer adoption with names like Salesforce and Major League Baseball, I argue that these customers aren’t “real” enterprise customers, because they both act like hyperscalers in terms of their adoption and use of technology. In fairness, Google is not alone in marching out such “enterprise” customers. (Every time I see Uber referenced as an enterprise, my head spins.)
It’s fair to say we have moved beyond chip-level competition. We now see a competition between platforms, each with its own assumptions about how AI workloads should be built, deployed, and consumed.
On raw per-chip specifications, NVIDIA’s Blackwell Ultra claims up to 35 petaflops of FP4 training performance and 288 GB of HBM4 at 22 TB/s. These are numbers that exceed the TPU 8t individually. But per-chip performance is not how frontier training is evaluated. At scale, interconnect efficiency, cluster reliability, and goodput matter more than peak single-chip numbers. Google’s interconnect scale advantage claims are real (though until we see them play out in production, the emphasis is on “claims”). Whether that translates to better system-level economics depends on many factors, one of which is workload characteristics.
I think NVIDIA’s durable advantage — against any competitor — lies in its ubiquity and ecosystem reach. Its hardware and software stack operate across multiple clouds and on-premises environments. By contrast, hyperscaler silicon (including TPU) is more tightly coupled to a single environment, offering deeper optimization at the cost of portability. I think this tradeoff will become more pronounced as enterprise AI deployments span hybrid and multi-cloud architectures.
For enterprise IT, the right frame is this: Choosing between Google Cloud inference and NVIDIA-based infrastructure is a platform and ecosystem decision, not a chip decision. Portability, software compatibility, data residency, and operational model are at least as important as token throughput per dollar. In fact, I would argue that having to account for these other factors will ultimately increase token throughput per dollar, albeit at the expense of operational costs.
For most enterprise IT organizations, the TPU 8 announcement has no bearing on near-term infrastructure decisions. Training is firmly out of scope. And inference is typically consumed through managed services rather than deployed directly on custom silicon. Where TPU 8 may have an impact is in the economics and performance of those managed services. But that impact is indirect, downstream, and not yet quantified in pricing terms.
It’s worth highlighting that TPUs are surprisingly not part of the Google Distributed Cloud portfolio. Google’s on-premises and air-gapped distributed cloud deployment runs on NVIDIA Blackwell GPUs. Organizations with data sovereignty, latency, or regulatory requirements pushing them toward on-premises AI cannot access TPU infrastructure in that model. This is a prime example of the portability challenges associated with larger models.
If I were to offer advice to Google, it would be to qualify TPU 8 for Google Distributed Cloud. Immediately. If you want to capture the enterprise market, give yourself a chance. By only offering a competing solution (NVIDIA), you are effectively cutting yourself off at the knees.
While I’m at it, Google should address the benchmarking problem I cited earlier. Submitting TPU 8 to independent benchmarking through organizations like Signal65 would give IT buyers and analysts some comparative data to make informed decisions. And until such data exists, I would recommend that enterprise buyers view these numbers as directional only — not definitive.
A decade after the initial TPU launch, the TPU 8 program fits into an accelerating pattern for the industry. Every major hyperscaler is building custom silicon optimized for its own workloads, supply chain, and cost structure. AWS has Trainium and Inferentia (primarily Trainium). Microsoft has Maia. Meta has MTIA. The trajectory of these accelerators maps closely to AWS Graviton: a custom Arm CPU that went from curiosity to meaningful market share over a decade without displacing x86 entirely. (Interestingly, TPU launched a couple of years before Graviton.)
The era of NVIDIA as the only viable AI hardware option is ending at the hyperscaler tier. Not abruptly, and not completely, but directionally. And that’s okay — NVIDIA will surely continue to be the dominant voice in AI silicon for years to come.
For enterprise IT, the more relevant question is not which custom silicon will win — as though there will be a single winner — but rather which cloud platform delivers the right combination of performance, software portability, operational simplicity, and pricing transparency for the specific workloads being run. I believe this question remains not fully answered for TPU 8 for several reasons. Performance claims are unverified. Training is a hyperscaler concern, not an enterprise one. Inference, while improved at the chip level, has not been reimagined at the system level in a way that fully addresses where agentic workloads are heading. Model support is expanding, but still carries friction. And TPUs don’t appear in Google Distributed Cloud, which limits their relevance for on-premises and sovereign deployments.
Whether TPU 8 becomes broadly relevant to enterprise IT will depend on independent validation, the maturity of the software ecosystem, and clearer alignment between its design assumptions and the realities of how AI is deployed outside hyperscale environments. For now, that alignment is still a work in progress.
With this said, Google has built some pretty cool chips.
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。