HPC

Omni-Path lights up Lawrence Livermore system at 400 Gbps

When it comes to networking supercomputers, Nvidia's InfiniBand rules the roost, but a new competitor is sneaking into the space with its own solution. This week the Department of Energy powered on a new cluster at Lawrence Livermore National Laboratory, and gluing it all together is Intel spinoff Cornelis Network’s Omni-Path interconnect tech.

Lynx is a relatively modest bit of iron, at least as DoE supers go, packing 952 Dell Technologies PowerEdge nodes powered by Intel’s aging 4th-gen Xeon Scalable processors, codenamed Sapphire Rapids. The system, commissioned by the National Nuclear Security Administration (NNSA) will provide additional compute capacity for some of America’s most secretive workloads. 

But what sets the machine apart isn’t the compute, but rather its choice of interconnect. Most DoE systems today either use HPE Cray’s proprietary Slingshot 11 or Nvidia’s InfiniBand networking. Lynx uses neither, instead opting for Cornelis Network’s CN5000-series Omni-Path switches and NICs.

“The collaboration between the NNSA ASC program and Cornelis has been rooted in a shared commitment to advance high-performance computing. Lynx reflects the results of that public-private R&D investment and will support the modeling, simulation, and analysis capabilities that underpin the modern NNSA complex,” Matt Leininger, a senior principal HPC strategist at LLNL, said in a statement.

If Omni-Path sounds familiar, that’s because it’s been around in one shape or form for the better part of a decade. Originally developed by Intel in 2015 for HPC applications, the lossless interconnect is similar in many respects to InfiniBand.

Several DoE Labs were early adopters, including Los Alamos National Lab’s Trinity super and the Cori machine, before Intel pulled the plug in 2019. The division was eventually spun off in 2020.

For many, this is where the story ended, but in 2025, the company unveiled its CN5000 family of NICs and switches to the world, promising 400 Gbps connectivity with near linear performance scaling.

The tech quickly attracted the attention of the DoE which tapped the niche networking startup’s tech for its Lynx system last summer. Omni-Path not only offers the agency an alternative to InfiniBand for non-Cray systems, but is now one of the fastest interconnects at their disposal.

The majority of the Cray systems deployed by the DoE labs operate at 200 Gbps. InfiniBand technically can accommodate higher port speeds, but is in extremely high demand for AI compute clusters.

For Cornelis, the deployment represents a significant proof point for the company’s next-generation Omni-Path protocol and networking systems.

“It's laying that foundational proof point for the industry to see that the most demanding customers out there have run it through its paces and are seeing really good results,” Cornelis CEO Lisa Spelman told El Reg.

In particular, Spelman says the deployment allowed Cornelis to demonstrate the scaling efficiency of its CN5000 portfolio.

As compute clusters grow larger, network interconnects can quickly become a bottleneck. “We were able to show a 91% network scaling efficiency, which is great for this size of cluster,” she said.

This scaling is so good, in fact, that Spelman expects to see Lynx outperform similarly sized clusters using more modern processors simply because the interconnects are more efficient. 

Lynx won’t be the last supercomputer Omni-Path finds its way into. The company is working on additional systems, including some, we’re told, that will make use of some non-traditional accelerators.

“We're looking forward to the next chance to prove it at 2,000, 5,000, 10,000 and just keep going up from there,” Spelman said.

Cornelis is also working to bring faster 800 Gbps equipment to market later this year, timed with the release of PCIe Gen 6.0-compatible CPUs from Intel, AMD, and others.

PCIe 5.0 connectivity effectively caps conventional NICs at 400 Gbps. Nvidia and some others have side stepped this problem by integrating large PCIe switches into their NICs which offers additional bandwidth, but adds cost and complexity that Spelman says Cornelis would prefer to avoid.

CN6000 is expected to launch in the second half of this year, and is expected to bring with it support for Ethernet connectivity allowing for greater cross compatibility with existing networks.®