Extreme network speeds on macOS/iOS devices using ConnectX network cards via USB4/Thunderbolt

macOS speed test

iOS speed test

Background info

Modern versions of macOS and iOS (starting with macOS Ventura/13) ship with NVIDIA/Mellanox ConnectX network drivers. AppleEthernetMLX5 is included by default and can be used with any NVIDIA/Mellanox ConnectX-4, 5 and 6 network card:

ConnectX-4 cards provide dual port 25 GBit/s connectivity (SFP28) and newer cards can go up to 200 GBit/s.
The maximum throughput is limited by the Thunderbolt/USB4 interface (to around 40 Gbit/s).

We were even successful in connecting a ConnectX-4Lx NIC to an iPad Pro (M1):

Hardware setup

Thunderbolt/USB4 NVMe storage enclosures can be (ab)used as an external PCIe enclosure using a M.2 to PCIE x4 adapter. These enclosures are typically used to connect fast NVMe SSDs to a computer, but the used Thunderbolt 3 controller ICs (like Intels JHL7440) will just accept Thunderbolt and turn it into 4 PCIe lanes. These lanes can then be connected to any PCIe device.

Our tests were done using a TBU401 enclosure (~100$):

Several vendors (on AliExpress, etc.) also offer little PCIe docks, like this “TH3P4 Lite GPU Dock”:

Full eGPU-enclosures (typically used for graphics cards) will also work well (but are large and very expensive).

Power supply

PCIe slots provide 2 voltage rails: 12V and 3.3V.
USB-C (without PD) provides 5V.

NVMe enclosures will convert the 5V supply down to 3.3V, but the 12V supply is still missing (and required for ConnectX cards).

Maximum available power on a USB-C port: 15W (5V/3A).

• ConnectX-4 card: ~11W
• Thunderbolt chipset: ~3W
• Conversion losses (5V->12V step-up): ~2W

The total power of this setup will exceed the maximum available power on a USB-C port.

Powering such a setup without using an external power brick won’t be easily possible.
We used an external 12V/2.5A power supply for our tests.

GPUs on a Mac?

This same setup could also be used to connect an external GPU to a computer.
This will work well on Intel/x86 Windows and Linux machines.

There is no hope of GPU support on (current-gen) Apple Silicon, though. Apart from the fact that no drivers are available on macOS, even projects like Asahi Linux run into a hardware limitation with the way memory mapping works on M-series chips.

This is the same limitation also present on other Arm platforms like the Raspberry Pi. See PCIe problems on embedded systems.