Liran Zvibel, Cofounder & CEO, WEKA.
getty
Training AI demands raw GPU compute. Inference demands something else entirely: memory. The GPUs powering today's models carry limited high-bandwidth memory (HBM) before external memory is required—that's the memory wall, and at inference scale, every model hits it. As the industry shifts from training to inference, memory has become the defining constraint in AI infrastructure.
DRAM supply remains tight amid strong demand, and I’m seeing enterprises paying 300-1100% more for memory chips since last year, a number expected to climb fast. Procurement timelines that once took days now stretch to months. Yet the most common response I’m seeing across the industry is the wrong one: hoard more hardware.
I’ve spent the last year talking with customers across every segment of the AI market, neoclouds, large enterprises and AI model builders—the pattern is consistent. Organizations compensating for architectural inefficiency by buying more capacity are now exposed. With memory stockpiles constrained through late 2027, that strategy no longer works.
The shortage didn’t create the problem. It’s the forcing function that revealed what was always broken, and made it impossible to ignore.
When chip manufacturers in 2024 shifted wafer capacity away from Dynamic Random Access Memory (DRAM) and toward AI-friendly HBM, global DRAM supply contracted sharply. The inventory situation has deteriorated to two to four weeks of product on hand, and the shortfall is expected to persist through late 2027.
This crisis exposes a fundamental flaw that has been percolating for years: the AI industry has been papering over architectural inefficiency with raw capacity. With memory now genuinely scarce, that approach has run out of road. The organizations that win the AI race won’t be the ones who secured the most chips. They’ll be the ones who deployed software-defined architectures that transform underutilized resources, fresh NVMe and spare CPU capacity already sitting in GPU servers, into high-performance memory extensions.
The memory shortage will only intensify over the next 18-24 months, and organizations can’t afford to wait for manufacturing to pick up as AI projects accelerate. Instead, savvy teams will deploy software that can transform underutilized hardware into high-performance AI systems.
Here are five strategies to help you not only survive the memory shortage, but thrive in spite of it:
Audit your current GPU utilization before placing any new orders. You’ll likely find rates well below capacity—not because you lack GPUs, but because your storage can’t feed them fast enough. I regularly see enterprise customers discover they’re leaving 50%–70% of available capacity on the table. Benchmark your data delivery rates against GPU consumption first. The answers are usually already in your infrastructure.
The critical bottleneck isn’t storage capacity measured in terabytes—it’s GPU memory measured in gigabytes. When inference exhausts the limited HBM integrated into GPU packages, systems waste expensive compute cycles recomputing tokens they’ve already processed. You cannot buy more HBM; it’s fixed in the hardware and more constrained than any other memory type.
Implement GPU memory extension using direct communication technologies that transform abundant NVMe into a functional extension of scarce HBM. Since the aim is to replace or extend memory that is incredibly fast, focus on picking an NVMe flash-based solution that optimizes for lowest latency. With near-perfect KV cache hit rates, organizations can achieve dramatically faster time-to-first-token and serve significantly more concurrent users per GPU—without waiting months for new GPUs that face the same memory constraints.
The instinct during shortages is to secure more silicon at any price. But when new fabs won’t be operational until 2027, that strategy means your AI roadmap stalls while competitors move forward. The hardware-first instinct is deeply ingrained—I understand it. When you can see the constraint, buying your way out feels like control. But the math no longer works.
Shift to software-defined approaches that transform underutilized resources in existing GPU servers into high-performance, memory-class infrastructure. Co-located deployment eliminates separate storage procurement entirely, activating NVMe and CPU already in the servers you’re buying for your GPUs anyway. This isn’t theory; it’s deployable in weeks using capacity you already own.
During shortages, “reclaimed” drives can seem like an attractive stopgap to increase raw capacity. Resist the temptation. These drives are approaching end-of-life and introduce unpredictable latency spikes that drag down GPU productivity.
Focus on workload density—the amount of consistent, reliable work you can extract per drive—rather than grabbing cheap gigabytes. Aging hardware masks underlying throughput problems that will eventually force you back into the strained memory market.
Most storage tiering strategies were configured once, years ago, and never revisited. The result: cold data clogs expensive flash while frequently accessed datasets get demoted to slow storage, forcing manual copies and redundant data to keep workflows moving. AI workloads make this exponentially worse, with checkpoints sitting idle until they’re suddenly needed at full speed.
Deploy automated, behavior-based tiering that continuously adapts to actual access patterns. Systems that transparently move data between tiers without blocking workloads can turn object storage into a viable overflow valve during shortages.
The organizations that thrive through this shortage won’t be the ones who bought the most HBM. They’ll be the ones who asked harder questions while competitors were treading water.
• Is your storage actually fast enough to feed your compute?
• Can you deploy on the infrastructure you have instead of waiting for what you can’t get?
• Are you optimizing for capacity or utilization?
• Does your architecture extend GPU memory, or does it simply manage storage?
The memory crisis didn’t create a new set of problems. It revealed the one problem the industry has avoided: we’ve been buying our way around an architectural flaw rather than fixing it. The shortage is the forcing function. The solution has been available all along.
Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。