With markets staying muted over the Christmas holiday, NVIDIA made an unexpected move, agreeing to spend about $20 billion in cash to license Groq’s LPU (Language Processing Unit) technology. As highlighted by the Commercial Times, the deal signals that the AI chip leader is accelerating its push beyond traditional GPU architectures, with implications extending to how SRAM (Static Random Access Memory) is being rethought for edge AI.

While speculation continues over how NVIDIA will ultimately deploy the technology, Commercial Times reports—citing SRAM EDA tool provider iSTART—that Computing-in-Memory (CIM) chips are increasingly replacing portions of DRAM with SRAM, a design approach already adopted by Groq’s LPU. Notably, the report also flags MediaTek as another player to watch in the SRAM game, noting that its flagship Dimensity 9500 smartphone chip adopts a processing-in-memory NPU architecture.

Why SRAM Matters

As Commercial Times explains, SRAM is typically integrated within logic processes (on-chip) and supplied by foundries such as TSMC and Samsung. While it delivers superior performance and ultra-low latency, SRAM comes with trade-offs: lower integration density, higher power consumption than DRAM, and substantial die area requirements. For this reason, SRAM is often used as cache memory between processors (CPUs) and main memory, providing rapid response times for applications like L1 and L2 CPU caches.

SRAM’s advantages also extend to AI applications. According to the Economic Daily News, Groq has designed its chips without relying on external HBM, shielding them from memory supply shortages. By integrating SRAM directly on-chip, the architecture helps accelerate interactions between chatbots and other AI models, the report adds.

SRAM’s Expanding Role in AI Inference Chips

A growing number of AI inference chips and CIM architecture processors are now substituting SRAM for portions of DRAM to unlock faster access speeds and dramatically lower refresh latency, according to Commercial Times. Groq’s LPU stands as a prime example, reportedly packing a massive 230MB of SRAM onto a single chip with on-chip memory bandwidth reaching up to 80TB/s—effectively sidestepping the bandwidth bottlenecks that plague traditional DRAM and external memory.

MediaTek has taken a similar approach with the Dimensity 9500’s ultra-efficient NPU, which debuts a processing-in-memory architecture enabling AI models to run persistently. Industry analysts cited by the report note that while SRAM carries a higher total cost of ownership, it slashes data transfer latency in AI inference workloads. For edge AI devices in particular, this translates into fewer DRAM modules required and more compact system designs, the report suggests.

According to Commercial Times, these developments signal a broader shift: ASIC architectures embedding massive amounts of SRAM are gradually emerging, prompting a fundamental reassessment of the once-unchallenged HBM pathway.

Read more

• [News] Samsung Emerges as Potential Second Foundry for NVIDIA Alongside TSMC After Groq Licensing Deal
• [News] NVIDIA Reportedly Denies USD 20B Groq Acquisition Rumors

(Photo credit: MediaTek)