One of the things I like to do in my twenty-eight minutes of spare time per week is play Battletech. It’s a table top wargame that involves big robots and lots of weapons. Some of them are familiar, like missiles and artillery. Because it’s science fiction there are also lasers and other crazy stuff. It’s a game of resource allocation. Can my ammunition last through this fight? You might be asking yourself “why not just carry lots of lasers?” After all, they don’t need ammo. Except the game designers thought of that too. Lasers produce heat. And heat, like ammunition, must be managed. Generate too much and you will shut down. Or boil your pilot alive in the cockpit. Rewind a thousand years and the modern network in a data center is facing a similar issue.

Watt Are You Talking About?

The average AI rack is expected to consume 600 kilowatts of power by next year. GPUs and CPUs are hungry beasts. They need to be fed as much power as possible in order to do whatever math makes AI happen. They have to come up with creative ways to cool those devices as well. We’re quickly reaching the limits of air cooling, with new designs using liquid cooling and even immersion in mineral oil to keep these machines from frying everything.

Networking is no slouch in the energy consumption department either. The latest generation of networking devices are ramping up to 800 GbE and consuming a significant amount of energy to keep the bandwidth flowing. One of the biggest consumers of power in these racks is the digital signal processor (DSP) that are clustered inside fiber optic modules. You have to have a lot of DSPs to condition the light that allows for operations at the top end of the scale. Additionally, the CPUs and ASICs themselves inside the units are running at peak performance to provide the resources AI clusters need to return value.

Worse yet, the very design of the network switch works against traditional air cooling. Servers have pretty face plates with air channels that can pull in air and vent it out the back. Nothing in the way except for maybe a USB stick when you’re loading the operating system. Switches, on the other hand, plug in all the cables up front. Those cables and modules impede air flow and reduce the amount of cooling the switch is capable of producing because of simple physics. If you’ve ever put your hand in front of a modern switch you know that it can pull in a lot of air. But every connection reduces that and creates a feedback loop.

Chill Out

Just like with the AI cluster servers, we have to deal with the heat generation. Unlike the AI servers, we can’t just dunk them in oil or build a bigger chassis with more fans to bleed it off. I mean, we technically could if necessary. But when is the last time you saw a top-of-rack (ToR) switch bigger than 2U? That’s because the real value in the rack isn’t the network. It’s the server that the network connects. Every rack unit of space you give to the networking gear or anything that isn’t a server is a wasted RU that could have been generating revenue.

Modern networking devices have hit 800 GbE but AI wants more. There are designs that are looking to move to 1.6 TbE in the near future. But can they handle the load of DSPs that will be required to condition the laser light? Will they be able to reduce the size of this switches down to accommodate rack economics? Can they keep the whole mess cool enough that it won’t melt anyone that walks into the hot aisle to check on a cable before getting roasted by a flame throwing?

Plug It In

One of the solutions that looks to reduce this is Linear Pluggable Optics (LPO). LPO works to solve these issues by moving the DSPs out of the pluggable module itself and into the switch ASIC. This does do a good job of reducing the power draw of the optical module itself while moving the complexity of signal conditioning into the switch. The net result is less power consumption but the tradeoff is more complexity. To quote Battletech YouTuber Mechanical Frog, “Opportunity cost spares no one.”

Likewise, a competing solution like Co-packed Optics (CPO) trades the module itself for wiring the optical portion of connection right into the switch. This eliminates the need for DSPs to condition the signal and significantly reduces power and heat generation but at the cost of the flexibility of using modules in the first place. I’ll have more to say on CPO in the future.

Tom’s Take

The thing to think about is that the technology we’re using creates the bounds that we have to work within for other technology to build on. We have to deliver networking that doesn’t stop for AI to function the way the designers want. To them, the value of AI is not being more power efficient or generating less heat. Instead, it’s about crunching numbers faster or making better pictures that people want to generate. Data centers have an unlimited power budget and heat removal capabilities as far as AI companies are concerned. Reality dictates that nothing is unlimited but it’s up to the manufacturers to scale those limits before someone hits them. Otherwise, the network is just going to get blamed again. And the heat really gets turned up.