I have an odd problem since upgrading my Proxmox cluster from PVE 9.1 to PVE 9.2.3. It consists of three physical proxmox nodes joined together in a cluster. On proxmox I host 3 talos linux control planes and 3 workers. After the update I'm no longer able to reach anything over the pod and service cidr's when the workloads are hosted on a Talos node that's on a different physical proxmox node then the one I'm testing from. Once I migrate the talos nodes together on the same physical proxmox, then traffic on the overlay network starts working again. To rule out kubernetes or the cni's, I've installed different Talos clusters with Cilium and kube-proxy and without kube-proxy and even reinstalled it once with just the standard Flannel cni. To further rule out Talos I've installed a k3s cluster with the built-in flannel cni. Still the same, no traffic over the pod and service cidr's unless the nodes are on the same physical proxmox node.

So today I decided to reinstall proxmox from the 9.1 release iso and reinstalled the Talos nodes, ran my tests and traffic in the overlay network just works. To test even further I upgraded again to 9.2.3 and after that, overlay networking is broken again.

Any help would be greatly appreciated.

@gaetanc

Excellent find! I didn't have such luck when asking copilot and resorted to doing a lot of packet captures. Interestingly the traffic still reaches the virtio nics and the cillium_vxlan interface (where vxlan encapsulation is stripped and the actual UDP packet with pod ip's passes through). I had gotten as far as confirming that the traffic does not reach the pod itself using a netcat listener.

To confirm your ethtool fix works, I had to work around the fact that there's no SSH or shell on Talos Linux:

kubectl run tmp-shell --rm -i --tty --image nicolaka/netshoot -n kube-system --overrides='{"spec": {"hostNetwork": true, "nodeName": "talos-w1", "containers": [{"name": "tmp-shell1", "image": "nicolaka/netshoot", "stdin": true, "tty": true, "securityContext": {"privileged": true}}]}}'

This runs a privileged network troubleshooting container and from there I could run the ethtool against the host NIC ens18. After that, it works like you said; my netcat listener received whatever keystrokes I sent from the pod on the other node.

Hi @fiona,

I've tested it on the latest 7.0.6-2 kernel as well as the older 6.x kernel present using the proxmox boot tool; both did not work.

lshw gives me the following info on the physical nics:

 *-network
       description: Ethernet interface
       product: RTL8111/8168/8211/8411 PCI Express Gigabit Ethernet Controller
       vendor: Realtek Semiconductor Co., Ltd.
       physical id: 0
       bus info: pci@0000:02:00.0
       logical name: nic0
       version: 15
       serial: [redacted]
       size: 1Gbit/s
       capacity: 1Gbit/s
       width: 64 bits
       clock: 33MHz
       capabilities: pm msi pciexpress msix bus_master cap_list ethernet physical tp mii 10bt 10bt-fd 100bt 100bt-fd 1000bt-fd autonegotiation
       configuration: autonegotiation=on broadcast=yes driver=r8169 driverversion=7.0.6-2-pve duplex=full firmware=rtl8168h-2_0.0.2 02/26/15 latency=0 link=yes multicast=yes port=twisted pair speed=1Gbit/s
       resources: irq:16 ioport:3000(size=256) memory:80804000-80804fff memory:80800000-80803fff

I downgraded the pve-qemu-kvm packages and tested everyting in order. This one pve-qemu-kvm=10.2.1-2 was not working for me and after installing pve-qemu-kvm=10.1.2-7 and rebooting the hosts and vm's, it starting working correctly. With pve-qemu-kvm=10.1.2-7 vxlan overlay networking still works.

Thank you for testing! Between QEMU 10.1 and 10.2 support for negotiating extended VirtIO features was added, so that is likely related to the regression. Unfortunately, we were not yet able to reproduce the issue.

Could you go to the VM monitor qm monitor ID with the numerical ID of the VM and run the following commands

qm> info virtio
/machine/peripheral/net1/virtio-backend [virtio-net]
/machine/peripheral/virtio0/virtio-backend [virtio-blk]
/machine/peripheral/balloon0/virtio-backend [virtio-balloon]
qm> info virtio-status /machine/peripheral/net1/virtio-backend
/machine/peripheral/net1/virtio-backend:
  device_name:             virtio-net (vhost)
  device_id:               1
...

Once with QEMU 11 or 10.2 and once with QEMU 10.1 so that the outputs can be compared.

If you see the VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO_CSUM with QEMU 11.0, see also the following post:

@shanreich was finally able to reproduce the issue. When the physical NIC does not support the

Code:

tx-udp_tnl-segmentation: off [fixed]
tx-udp_tnl-csum-segmentation: off [fixed]

features, the issue can happen. When the physical NIC does support them, it works. The issue seems to be that the feature negotiation between QEMU/host/guest (AFAICT wrongly) decides to turn on these features for the vNIC even if the underlying physical NIC does not support them. Although they are also turned on for the bridge and tap interfaces. We'll investigate further where exactly the negotiation/setting of...

Please see the attached file for virtio output. I also ran my end to end tests; even with the patch it's still broken.

Additionally see the output below:

root@pve1:~# qm showcmd 101 --pretty | grep net
  -netdev 'type=tap,id=net0,ifname=tap101i0,script=/usr/libexec/qemu-server/pve-bridge,downscript=/usr/libexec/qemu-server/pve-bridgedown,vhost=on' \
  -device 'virtio-net-pci,mac=BC:24:11:D1:ED:48,netdev=net0,bus=pci.0,addr=0x12,id=net0,rx_queue_size=1024,tx_queue_size=256,bootindex=102,host_mtu=1500,guest_tunnel_csum=off,host_tunnel_csum=off' \

EDIT: I saw in the other thread that the cause has been found. So the problem gets triggered when the underlying physical nic does not support checksum offloading.

I have the same issue with my Proxmox 9.2.3.

I also run Talos 1.13.3(Issue persists with older version of Talos eg 1.12.8) in Proxmox 9.2.3.
I have 2 identical environments: Proxmox 9.2.3(broken) and Proxmox 9.1.6(working just fine). 3 host nodes, hardware Dell R740/R750/R340.
Both running 6 nodes Talos 1.13.3 with k8s 1.36.1 distributed across nodes: 1 cp per host node and 1 worker per host node.

K8S pods refuse to communicate with each other and corresponding services eg coredns if placed on different nodes in Proxmox 9.2.3. If I migrate all K8S nodes to same Proxmox host - everything starts working.
Environment that is running older version of Proxmox 9.1.6 has no issues. Everything is working. Pods can talk to each other and their services.

Also tried Flannel and Cilium to make sure it is not a CNI configuration issue with the same results - Both CNIs are working on Proxmox 9.1.6 and stopped working in Proxmox 9.2.3

Both K8S clusters are created using same IaC Terraform code. Both Proxmox environments has been configured manually with very basic settings (NTP, Bond). Proxmox 9.2.3 environment known to be working before the upgrade.

Attached requested `qm` debug outputs for both Proxmox 9.2.3(qemu11) and 9.1.6(qemu10)

Please let me know if you need more debug info

We are still investigating the root cause. Another workaround should be using machine version 10.1 for the affected VMs.

Thanks all for the fix ! ran into this on monday when i tried upgrading our cluster...

i did this (i run only q35 VM, take care to not break i440fx vms !)

qm list | grep -v VMID | awk '{ print $1; }' | xargs -I '{}' qm set '{}' --machine pc-q35-10.1

and then proceeded to reboot the VMs (qm shutdown xxx; qm start xxx) one by one (same context: talos v1.13.3 nodes losing cross-pve-host EVPN networking, biggest pain was longhorn volumes broken)

Thanks again