Flipping a working VMware vSphere vSAN to Proxmox 9.1 and Ceph !

Everything was working on vSphere (ESXi 8 - networking), so no issues with physical switches, vDS, SFP+ etc

but the Broadcom BCM57412 10G SFP Dual Port PCIE adapters are not initialising correctly - firmware - 36.11.73.00

[ 56.425484] bnxt_en 0000:5e:00.0 nic0: NIC Link is Up, 10000 Mbps (NRZ) full duplex, Flow control: none
[ 56.425491] bnxt_en 0000:5e:00.0 nic0: FEC autoneg off encoding: None
[ 56.589621] bnxt_en 0000:5e:00.1 nic1: NIC Link is Up, 10000 Mbps (NRZ) full duplex, Flow control: none
[ 56.589627] bnxt_en 0000:5e:00.1 nic1: FEC autoneg off encoding: None
[ 57.131786] bnxt_en 0000:5e:00.0 nic0: entered allmulticast mode
[ 57.131793] bnxt_en 0000:5e:00.1 nic1: entered allmulticast mode
[ 57.284982] bnxt_en 0000:5e:00.0 nic0: entered promiscuous mode
[ 57.285012] bnxt_en 0000:5e:00.1 nic1: entered promiscuous mode
[ 107.131965] bnxt_en 0000:5e:00.0 nic0: left allmulticast mode
[ 107.131969] bnxt_en 0000:5e:00.1 nic1: left allmulticast mode
[ 107.131977] bnxt_en 0000:5e:00.0 nic0: left promiscuous mode
[ 107.132002] bnxt_en 0000:5e:00.1 nic1: left promiscuous mode
[ 422.776053] bnxt_en 0000:5e:00.0 nic0: Discarding out of seq response: 0x46e for msg {0xd 0x46f}
[ 431.936157] bnxt_en 0000:5e:00.0 nic0: Error (timeout: 5000015) msg {0x27 0x581} len:0
[ 431.936173] bnxt_en 0000:5e:00.0 nic0: failed to update link (rc: fffffff0)
[ 431.936180] bnxt_en 0000:5e:00.0 nic0: failed to update phy settings
[ 437.057744] bnxt_en 0000:5e:00.0 nic0: failed to update phy settings after maximum retries.
[ 440.992789] bnxt_en 0000:5e:00.0 nic0: Error (timeout: 5000015) msg {0x27 0x582} len:0
[ 444.803406] bnxt_en 0000:5e:00.0 nic0: Discarding out of seq response: 0x581 for msg {0x27 0x583}
[ 449.915155] bnxt_en 0000:5e:00.0 nic0: Error (timeout: 5000015) msg {0x27 0x583} len:104
[ 449.915161] bnxt_en 0000:5e:00.0 nic0: failed to update link (rc: fffffff0)
[ 449.915164] bnxt_en 0000:5e:00.0 nic0: update phy settings retry failed
[ 458.797887] bnxt_en 0000:5e:00.0 nic0: Failed setting MC filters rc: -16, turning on ALL_MCAST mode
[ 459.895899] bnxt_en 0000:5e:00.1 nic1: Error (timeout: 5000015) msg {0x27 0x57d} len:0
[ 459.895906] bnxt_en 0000:5e:00.1 nic1: failed to update link (rc: fffffff0)
[ 459.895908] bnxt_en 0000:5e:00.1 nic1: failed to update phy settings
[ 464.828587] bnxt_en 0000:5e:00.0 nic0: Discarding out of seq response: 0x583 for msg {0x93 0x585}
[ 465.026752] bnxt_en 0000:5e:00.1 nic1: failed to update phy settings after maximum retries.
[ 468.866939] bnxt_en 0000:5e:00.1 nic1: Error (timeout: 5000015) msg {0x27 0x57e} len:0
[ 473.743478] bnxt_en 0000:5e:00.0 nic0: Error (timeout: 5000015) msg {0x27 0x586} len:0
[ 473.743484] bnxt_en 0000:5e:00.0 nic0: failed to update link (rc: fffffff0)
[ 473.743488] bnxt_en 0000:5e:00.0 nic0: update phy settings retry failed
[ 482.670548] bnxt_en 0000:5e:00.1 nic1: Failed setting MC filters rc: -16, turning on ALL_MCAST mode
[ 490.860281] bnxt_en 0000:5e:00.1 nic1: Discarding out of seq response: 0x57e for msg {0x93 0x580}
[ 499.795684] bnxt_en 0000:5e:00.1 nic1: Error (timeout: 5000015) msg {0x2a 0x581} len:0
[ 508.725545] bnxt_en 0000:5e:00.1 nic1: Error (timeout: 5000015) msg {0x27 0x582} len:0
[ 508.725559] bnxt_en 0000:5e:00.1 nic1: SP task can't update link (rc: fffffff0)
[ 512.887637] bnxt_en 0000:5e:00.1 nic1: Discarding out of seq response: 0x581 for msg {0x27 0x583}
[ 517.652209] bnxt_en 0000:5e:00.1 nic1: Error (timeout: 5000015) msg {0x27 0x583} len:40
[ 517.652215] bnxt_en 0000:5e:00.1 nic1: failed to update link (rc: fffffff0)
[ 517.652219] bnxt_en 0000:5e:00.1 nic1: update phy settings retry failed

I'm suspecting Firmware, which is 36.11.73.00, we have other new/older servers with Proxmox using same cards on 23.31.18.10, and they seem to be working, so is it firmware, so either we've got to downgrade, or find a new driver!

dmesg | grep -i bnxt
[ 3.723217] bnxt_en 0000:5e:00.0 (unnamed net_device) (uninitialized): Device requests max timeout of 100 seconds, may trigger hung task watchdog (kernel default 120s)
[ 4.051107] bnxt_en 0000:5e:00.0 eth5: Broadcom BCM57412 NetXtreme-E 10Gb Ethernet found at mem b8a10000, node addr b0:26:28:d6:ea:a0
[ 4.051115] bnxt_en 0000:5e:00.0: 63.008 Gb/s available PCIe bandwidth (8.0 GT/s PCIe x8 link)
[ 4.051481] bnxt_en 0000:5e:00.1 (unnamed net_device) (uninitialized): Device requests max timeout of 100 seconds, may trigger hung task watchdog (kernel default 120s)
[ 4.320599] bnxt_en 0000:5e:00.1 eth7: Broadcom BCM57412 NetXtreme-E 10Gb Ethernet found at mem b8a00000, node addr b0:26:28:d6:ea:a1
[ 4.320605] bnxt_en 0000:5e:00.1: 63.008 Gb/s available PCIe bandwidth (8.0 GT/s PCIe x8 link)
[ 4.428338] bnxt_en 0000:5e:00.0 ens2f0np0: renamed from eth5
[ 4.429370] bnxt_en 0000:5e:00.1 ens2f1np1: renamed from eth7
[ 6.772269] bnxt_en 0000:5e:00.1 nic1: renamed from ens2f1np1
[ 6.772937] bnxt_en 0000:5e:00.0 nic0: renamed from ens2f0np0
[ 11.133291] bnxt_en 0000:5e:00.0 nic0: NIC Link is Up, 10000 Mbps (NRZ) full duplex, Flow control: none
[ 11.133298] bnxt_en 0000:5e:00.0 nic0: FEC autoneg off encoding: None
[ 11.289875] bnxt_en 0000:5e:00.1 nic1: NIC Link is Up, 10000 Mbps (NRZ) full duplex, Flow control: none
[ 11.289883] bnxt_en 0000:5e:00.1 nic1: FEC autoneg off encoding: None
[ 11.807301] bnxt_en 0000:5e:00.0 nic0: entered allmulticast mode
[ 11.807304] bnxt_en 0000:5e:00.1 nic1: entered allmulticast mode
[ 11.807401] bnxt_en 0000:5e:00.0 nic0: entered promiscuous mode
[ 11.807429] bnxt_en 0000:5e:00.1 nic1: entered promiscuous mode

rolled back driver to
23.31.18.10
29 Jul 2025

So mlatest firmware from Dell
36.11.56.00
03 Apr 2026

borks Proxmox VE 9.1 !

Hi everyone, I was wondering if problem got solved, if so, can you please show what actions you took.

Thank you
bujar