GPUs and other big AI chips just can’t get enough memory. Today’s datacenter computers stack DRAM 12 dies high. But as memory makers try to build higher stacks to pack in more bits and bandwidth, experts worry this highbandwidth-memory (HBM) will trap enough heat to cook itself into oblivion. That will be especially true if GPU-makers choose to start stacking HBM atop their already hot processors to boost bandwidth instead of placing them beside the processor.

Engineers around the world are working on an alternative: Instead of stacking the DRAM dies atop each other, why not stack them side by side? That way they hope to turn a future towering inferno into a cube of cool silicon fins.

Last month at IEEE VLSI Symposium, two research groups showed different paths to such a chip. One possible side-stacked chip, called V-Die, would deliver an 82 percent speed boost compared to today’s most advanced memory, HBM4, South Korean researchers predict. Engineers in Japan say their version, MOSAIC, should deliver twice the memory capacity of HBM4 without raising the peak temperature much more than 1℃.

HBM and its problems

Today’s HBM consists of multiple layers of silicon dies of DRAM stacked on another chip, called the base die. The base die delivers power to the stack and coordinates communications with the processor. Bits and power reach into the stack by vertical connections that pass through each die, called through-silicon vias (TSVs). The dies connect to each other and to the base die through microscopic bumps of solder that link to the TSVs.

A typical AI accelerator, such as the Nvidia B300, is flanked by eight HBM stacks that are each 12-dies high, delivering 36 gigabytes per stack. Each stack is placed on the same substrate as the GPU, so they can bridge the millimeters-wide gap via 2048 micrometers-wide lines. This arrangement, for HBM4, transmits 2800 GB per second to and from the GPU.

But even that won’t be enough in the future. “The size of AI models is growing explosively,” Heesoo Yang, a doctoral student at Ulsan National Institute of Science and Technology (UNIST) in South Korea, told engineers at VLSI Symposium. “But memory capacity and bandwidth are struggling to keep up, creating a massive bottleneck.”

One of HBM’s most pressing problems is how hot they get. The material that fills the gaps between the dies is about 100 times more thermally resistant than the silicon substrate, keeping heat from flowing upwards to where the processor-package’s heat sink can remove it. There are potential ways of mitigating the problem, but it’s likely to grow worse as chipmakers seek to add capacity by stacking more dies.

Another future problem is that as chipmakers seek taller stacks, trade-offs between storage capacity and bandwidth creep up, Yang, a student in the laboratory of UNIST’s Jimin Kwon pointed out. Higher stacks need more TSVs to deliver the data, eating into the area of silicon needed to store it.

V-Die memory

Last month, Kwon’s team, in collaboration with Seongju Kim at Hanbat National University in Daejon, presented its solution to HBM’s future ills. Called V-die, it will stack DRAM vertically and include microfluidic cooling channels between the dies to keep them at 45℃, which is quite cool relative to the typical 80℃-plus peak.

The dies themselves would be different, Yang explained. Without the need for vertical connections, there would be no TSVs, freeing up area for more memory cells. In addition, each die would have its own I/O systems, eliminating the need for a base die. These systems would sit all along the bottom edge of the die and connect to the silicon substrate, on which the GPU sits, via links every 20 micrometers. Such an arrangement allows for four times as many connections as HBM4 and reduces the time it takes to read from memory by 37 percent, the team calculates, although some data does have to travel a few millimeters farther to reach the processor.

The team simulated how a 16-die stack would alter the performance of AI computers, such as one made up of Nvidia H100 GPUs. Tested using a workload representing a GPT3-sized large-language model, the V-die system delivered 540 tokens per second versus 296 for HBM4 with the same memory capacity. It also reduced latency (the time it takes to deliver the first token) by 32 percent or about 24 milliseconds.

A prototype device, which will be used to validate thermal and electrical characteristics, is in the works.

Side-stacking connections

These DRAM schemes, sometimes called volumetric DRAM, are actually constructed first by adding one die atop another and then turning the whole stack on its side to connect it to the substrate or another chip. That could lead to some tricky integration problems, says James Myers, a program director at the Belgium-based microelectronics research center Imec. Myers’ team previously worked out solutions to the thermal problems of stacking DRAM on GPUs and is now looking at using vertical dies in that situation. “You have to get the thickness exactly right,” he says. Even if there’s only a few micrometers difference among the DRAM dies, it can add up to a problem connecting to the substrate. Put enough irregular dies in the stack, and “you’ll miss the bond pads.”

A team from University of Tokyo, Tohoku University, and the Japanese national research institute Riken, explained a novel way to ease this problem at IEEE VLSI Symposium. Instead of directly electrically connecting the bottom of the dies to the substrate, the team tested an inductive coupling transceiver system. On one side of the memory die, the team formed oblong-shaped inductive coils, about 80 micrometers by 240 micrometers. They placed a corresponding set at a right angle on the substrate. Current through one coil induces a magnetic field in the other, transmitting a data signal. And because the coils don’t have to overlap exactly, there’s a lot of leeway in how precisely the memory sits on the substrate.

Power connections, which are fewer in number and take up more surface area, are placed on the sides of the memory cube, University of Tokyo doctoral student Yuki Mitarai said at the symposium.

MOSAIC, which is designed to attach to the top of a GPU, fits 98 dies per cube, delivering 294 GB of memory. While it doesn’t have a microfluidic cooling structure, the fact that heat can rise through the silicon fins themselves should keep such a structure to 81.3℃, near today’s typical 80℃ limit. What’s more, thinning the DRAM dies by two-thirds to 100 micrometers would allow a MOSAIC cube to integrate 294 dies in the same volume, reaching 882 GB, Mitarai said.