Weve got two clusters of 3 servers, each with a BCM57504 100 GBe Dual Port in it and it is used for the ceph network with full mesh routing (no switches).
I think we started with PVE 8.3 and are now up to 9.1.4 and never had any problems.

we had a full kernel panic (eventually) on pve 9.1.x on kernel 7.0.0-3-pve due to wedged driver on a similar card, BCM57508.

topology: each node has two dual 100g port cards, and each node has two lacp bonds, one for front of house networking, one for backend (ceph, migrations, etc).

failure mode: basically you can see bnxt_en panic each cpu until it gets all the way around and then the computer hardlocked

May 05 13:49:23 proxmox7 kernel: bnxt_en 0000:11:00.0 enhe0p0: NETDEV WATCHDOG: CPU: 35: transmit queue 16 timed out 5081 ms
May 05 13:49:23 proxmox7 kernel: bnxt_en 0000:11:00.0 enhe0p0: TX timeout detected, starting reset task!
May 05 13:49:23 proxmox7 kernel: bnxt_en 0000:11:00.0 enhe0p0: [0.0]: tx{fw_ring: 1025 prod: 44f0 cons: 44f0}
May 05 13:49:23 proxmox7 kernel: bnxt_en 0000:11:00.0 enhe0p0: [0]: rx{fw_ring: 2 prod: 75b} rx_agg{fw_ring: 3 agg_prod: e204 sw_agg_prod: 204}
May 05 13:49:23 proxmox7 kernel: bnxt_en 0000:11:00.0 enhe0p0: [0]: cp{fw_ring: 0 raw_cons: 1344945}
May 05 13:49:23 proxmox7 kernel: bnxt_en 0000:11:00.0 enhe0p0: [0.0]: cp{fw_ring: 49 raw_cons: 17be4c0}
May 05 13:49:23 proxmox7 kernel: bnxt_en 0000:11:00.0 enhe0p0: [0.1]: cp{fw_ring: 33 raw_cons: a9199d}
May 05 13:49:23 proxmox7 kernel: bnxt_en 0000:11:00.0 enhe0p0: [1.0]: tx{fw_ring: 1026 prod: 6caa cons: 6caa}

info:

from dmesg:

[    2.323354] bnxt_en 0000:11:00.0 eth0: Broadcom BCM57508 NetXtreme-E 10Gb/25Gb/50Gb/100Gb/200Gb Ethernet found at mem 70490010000, node addr <mac addr>

ethtool -i enhe0p0:
driver: bnxt_en
version: 6.17.13-12-pve
firmware-version: 233.0.152.6/pkg 233.1.135.7
expansion-rom-version:
bus-info: 0000:11:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no

on kernel 6.17 under proxmox 9.1.x (and previously proxmox 9.0.x under its various 6.x kernels) we've had no issues. We did blacklist the bnxt_re module after the issue but, that probably wont be enough to stop it happening again. In googling around i did find suggestion that kernel 7 has some bnxt_en jank.

In googling around i did find suggestion that kernel 7 has some bnxt_en jank.

Could you post your full network configuration? I've seen several reports now and usually they were using some BCM cards with bonds and VLANs, but I couldn't yet tie them together...

cat /etc/network/interfaces

/etc/network/interfaces contents:

Basic gist is: two LACPs, each containing two of the ports, each nic has one port for each of the two LACPs (thus a nic card dying entirely should be ignored)

of those two bonds:

po7 / bond0 is for use by VMs via individiual subnets in SDN, which are built atop vmbr0

the other bond1 / po17 is for ceph and any back of house proxmox comms, including migration, etc.

IPs mildly redacted

lmk if you have any other questions.

auto lo
iface lo inet loopback

auto enhe0p0
iface enhe0p0 inet manual
        mtu 9000
#Port 1B - Onboard

auto enhe0p1
iface enhe0p1 inet manual
        mtu 9000
#Port 2B - Onboard

auto enhe1p0
iface enhe1p0 inet manual
        mtu 9000
#Port 1A - PCIe

auto enhe1p1
iface enhe1p1 inet manual
        mtu 9000
#Port 2A - PCIe

iface enipmi0p0 inet manual
#unused hw bond with IPMI

auto bond0
iface bond0 inet manual
        bond-slaves enhe0p0 enhe1p0
        bond-miimon 100
        bond-mode 802.3ad
        bond-xmit-hash-policy layer3+4
        mtu 9000
#bond for usernets on po7

auto bond1
iface bond1 inet manual
        bond-slaves enhe0p1 enhe1p1
        bond-miimon 100
        bond-mode 802.3ad
        bond-xmit-hash-policy layer3+4
        mtu 9000
#bond for admin/storage on po17

auto bond1.1549
iface bond1.1549 inet manual
        mtu 9000
#storage ip interface base bond slice (ceph)

auto bond1.1508
iface bond1.1508 inet manual
        mtu 9000
#admin ip interface base bond slice (admin/migration)

auto vmbr0
iface vmbr0 inet manual
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
        mtu 9000
        bridge-disable-mac-learning 1
#base switch for usernets (used as basis for SDN individual vlans listed in pve)

auto vmbr1v1508
iface vmbr1v1508 inet static
        address 10.y.y.y/23
        gateway 10.y.y.254
        bridge-ports bond1.1508
        bridge-stp off
        bridge-fd 0
#admin interface ( migration, pve login, etc)

auto vmbr1v1549
iface vmbr1v1549 inet static
        address 10.x.x.x/24
        bridge-ports bond1.1549
        bridge-stp off
        bridge-fd 0
        mtu 9000
#storage interface (ceph)

also: yes, all 4 100g nics are the same broadcom model (but two are part of a supermicro "onboard" AOM card, and the other two are on a standard broadcom PCIe card. all 4 are running the same firmware listed in the first post.)

Greetings.

We got the same issue on some an old supermicro-board with an Intel X550 2-port 10G-base-T NIC. These two ports bonded with LACP.

Error: "bnxt_en <pci:id> <nic-name> NETDEV WATCHDOG: CPU: <cpu-id>: transmit queue <queue-id> timeout <time> ms"

Tip: By reloading the NIC-driver the network-connection can "sometimes" be restored without a reboot:

ifdown -a
modprobe -r bnxt_en
modprobe bnxt_en
ifup -a

. The same issue without a bond configured (single Port).

Is there something interesting in the journal (journalctl -b)? You could also try to disable offloading as that has caused some issues in the past:
post-up ethtool -K <nic> rx-vlan-offload off

Happened to us again on the 'onboard' Supermicro AOC version of the BCM57508, PCIe version of the NIC in the same host still apparently unaffected.

We were at ~ 14 days uptime. I think it was the machine with the most uptime that went down. Rebooted most of the nodes left one to see if i can get another node to do this, because today's downtime was the same node in the cluster I posted about above.

This time, happened on kernel 6.17.13-13, which until today we thought 6.17 was bulletproof, so news to me.

let me know if there's any additional diag info I can provide.

I did switch the nic to manually configured link speed rather than autodetect since It's unclear if the flapping is causing bnxt_en to tweak out or the other way around.

-Dave

Items we changed today to try to see if this will stop happening:

configured 100g eth with reed solomon FEC in the bios, disabled autonegotiate.

using niccli changed the following nvm variables on all the cards to disable as much unused features as possible, and also inherit the PCIe variant of the cards default values as much as possible in the case that something wasnt being disabled. we're still on v233 of the firmware since so far, this has only happened on a single machine of 5 supposedly identical machines. keep in mind i'm only listing one example of each variable, not all 4 to 8 scopes per node:

disabled/constrained:
niccli -i 1 nvm -setoption support_rdma -scope 0 -value 0x00;    #disable RDMA support
niccli -i 1 nvm -setoption bw_limit -scope 0 -value 0x64  
#limits bandwidth used for ...some feature i've forgotten. not general traffic (64 was default on pcie nic)
niccli -i 1 nvm -setoption spdm_mctp_pcievdm -value 0      #disable signing comms to bmc/bios/etc
niccli -i 4 nvm -setoption mctp_discovery -value 0      #disable bios/bmc discovery (though ipmi still seems to discover it)
niccli -i 1 nvm -setoption enable_wol_on_acpi_pattern -scope 0 -value 0  # disable wake on lan patern
niccli -i 1 nvm -setoption magic_packet_wol -scope 0 -value 0    #disable wake on lan magic packet

enabled (to match PCIe variant of BCM57508):
niccli -i 1 nvm -setoption enable_sriov -value 1    #enable sr-iov to match PCIe card default
niccli -i 1 nvm -setoption enable_adapter_error_recovery -value 1  #enable adapter error recovery to match PCIe card default
niccli -i 1 nvm -setoption enable_crash_dump_to_host_ddr -value 1  #enable crash dump to host to match PCIe card default

Host was powered off for two minutes and then powered back on to ensure nvm defaults moved from set to active state.

also upped the fan speed on the chassis that was having the crash issues (default was set to optimal not optimal for heavy IO like the other nodes in the cluster.

at this point, now i guess we just run low criticality VMs on that node for a few weeks and see if anything changes with regards to reliability.

reminder that bnxt_re is still also denied at the modprobe level.

if anyone else has any suggestions for useful bits to twiddle, etc, let me know.

thanks,
-Dave