Exclusive If you thought Nvidia or AMD's 72-GPU rack systems were enormous, silicon Ayar Labs has something much bigger in the works.

On Wednesday, the silicon photonics startup revealed it was working with ODM partner Wywinn to develop a new rack-scale reference platform capable of stitching more than 1,024 GPUs into a single unified system.

More importantly for datacenter operators, the optical interconnects used to connect these GPUs together won't require powering and cooling 600+ kilowatt racks to do it.

Instead, the reference design is expected to pull somewhere between 100 and 200 kilowatts per rack, putting them in line with contemporary rack systems. To reach their 1,024 accelerator target, the duo will simply add more racks, something that's not possible today using the copper interconnects found in AMD and Nvidia's latest system.

"Looking at the current racks, you're forced to have everything in that one rack. You're forced to have GPUs there, you're forced to have CPUs there. You're forced to have switches, just because copper doesn't take you that far," Ayar CTO Vladimir Stojanovic told El Reg in an exclusive interview.

Copper is one of the reasons that racks, like Nvidia's 600 kilowatt Vera Rubin Ultra systems, keep getting hotter. At the speed those interconnects operate, copper can only reach a few feet without signal degradation, while conventional pluggable optics are too inefficient to be viable. 

Ayar aims to sidestep this issue by co-packaging the optics with the compute, dramatically reducing the power consumption compared to pluggables while boosting reach and bandwidth by a factor of up to 3x.

At Super Computing last fall, we got our best look yet at what these accelerators might look like. On display was a prototype designed in collaboration with Alchip which featured eight of Ayar's TeraPHY optical engines boasting more than 100 Tbps of bandwidth.

But before you can deploy a chip like this, you need somewhere to put it.

"We want to iron out every single detail here, because, at the end of the day, a hyperscaler's unit of purchase so to speak is a rack, and I would say even maybe more, a cluster," Stojanovic said.

Compared to your average 19-inch server chassis, rack scale systems are a fair bit more complex. Systems like Nvidia's liquid-cooled NVL72 contain miles of cabling and dozens of blind mate connectors for networking, power, and cooling. Packaging all of that in a chassis that's reliable and serviceable is hard enough before you add photonics to the mix.

The introduction of photonics introduces new design decisions, like how you route the liquid cooling, Stojanovic explained. "As the cold water is coming in, what do you cool first? What do you cool second?"

"If you look at the original ELSFP [laser module] specification, it was never meant to work in a liquid-cooled environment." 

Another consideration is software management and monitoring. One of the barriers to adoption for co-packaged optics has been the increased blast radius. With pluggable transceivers, a bad optic can be swapped, but if an optical engine goes bad, it'll take the entire chip with it.

This makes software monitoring and telemetry essential for narrowing down whether potential problems are related to the optics or if another factor may be at play. Along with the mechanical design, Ayar is also working with Wiwynn to tap into the embedded firmware on its chips for this reason.

At the Optical Fiber Communciation (OFC) Conference next week, Ayar and Wiwynn will unveil the fruits of their labor, a rack scale reference design that integrates the photonics startup's optical engines and pluggable laser modules with the server makers expertise in mechanical and systems engineering.

The reference design, which El Reg has gotten a sneak peak of ahead of the event, appears to sport two optically interconnected accelerators, a single CPU. At the front of the system are 16 of Ayar's user-serviceable SuperNova laser modules, along with a slew of hyperscale-style front-mount network interfaces.

• Nvidia-backed photonics startup Ayar Labs fills its wallet to mass-produce CPO chiplets
• Nvidia burns $4B to light up American photonics manufacturing
• Bill Gates-backed startup aims to revive Moore's Law with optical transistors
• Copackaged optics have officially found their killer app - of course it's AI

Compared to the compute blades we've seen from Nvidia and AMD, the reference design is about half as dense, but because it uses optical interconnects it doesn't have to be.

Rather than 18 blades connected by super-short run copper cables, Ayar can connect hundreds of these systems to form a single enormous logical server. What's more, because the company isn't constrained by copper interconnects, those racks don't have to have all the switching, compute, and storage on board.

"This allows you to do a disaggregated architecture, where you build a rack monolithically as a compute rack, and then you have another rack that's a switch rack, and of course, you could have an extended memory rack," Stojanovic said.

Wednesday's announcement comes barely a week after Ayar closed a $500 million Series E funding round to accelerate the mass production of its co-packaged optics, and nearly six months after the startup tapped Global Unichip Corp (GUC) to develop reference designs based on its optical I/O chiplets.

The startup's work with Wiwynn now addresses the other side of the equation: where to put those chips when ready. ®