Asymmetric Tailscale speed issue with Kernel 7 and multip...

Trying here as my report in the kernel thread is not getting much attention. I know of at least 5 users reporting this issue here and on Reddit - but it definitely also doesn't affect everyone.

The issue is easily tested and consistently reproducible by me with Kernel 7 and using iperf3 across Tailscale. I have reproduced it both within my local network and across interstate connections.

The specific tests below tests are run with exactly the same hardware, same network, same Tailscale version - i.e the _only_ change is the kernel version.

Kernel 7.0.2-2-pve - clear, massive regression:

Code:

10:20 user@samba:~ > iperf3 -c 100.93.240.XX -t 30
Connecting to host 100.93.240.XX, port 5201
[  5] local 100.125.133.YY port 50050 connected to 100.93.240.XX port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   384 KBytes  3.14 Mbits/sec   52   2.40 KBytes
[  5]   1.00-2.00   sec   640 KBytes  5.24 Mbits/sec   40   2.40 KBytes
[  5]   2.00-3.00   sec   640 KBytes  5.24 Mbits/sec   46   2.40 KBytes
[  5]   3.00-4.00   sec   256 KBytes  2.10 Mbits/sec   29   1.20 KBytes
[  5]   4.00-5.00   sec   128 KBytes  1.05 Mbits/sec   29   2.40 KBytes
...
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-30.00  sec  16.0 MBytes  4.47 Mbits/sec  1200            sender
[  5]   0.00-30.00  sec  16.0 MBytes  4.47 Mbits/sec                  receiver
iperf Done.
10:20 user@samba:~ > iperf3 -c 100.93.240.XX -t 30 -R
Connecting to host 100.93.240.XX, port 5201
Reverse mode, remote host 100.93.240.XX is sending
[  5] local 100.125.133.YY port 45092 connected to 100.93.240.XX port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   150 MBytes  1.26 Gbits/sec
[  5]   1.00-2.00   sec   148 MBytes  1.24 Gbits/sec
[  5]   2.00-3.00   sec   153 MBytes  1.28 Gbits/sec
[  5]   3.00-4.00   sec   146 MBytes  1.22 Gbits/sec
[  5]   4.00-5.00   sec   155 MBytes  1.30 Gbits/sec
...
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-30.00  sec  4.27 GBytes  1.22 Gbits/sec   16            sender
[  5]   0.00-30.00  sec  4.26 GBytes  1.22 Gbits/sec                  receiver
iperf Done.

Vs. Kernel 6.17.13-7-pve results:

Code:

10:49 user@samba:~ > iperf3 -c 100.93.240.XX -t 5
Connecting to host 100.93.240.XX, port 5201
[  5] local 100.125.133.YY port 57468 connected to 100.93.240.XX port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  64.2 MBytes   539 Mbits/sec  152    562 KBytes
[  5]   1.00-2.00   sec  64.0 MBytes   537 Mbits/sec    1    631 KBytes
[  5]   2.00-3.00   sec  58.9 MBytes   494 Mbits/sec    0    690 KBytes
[  5]   3.00-4.00   sec  61.4 MBytes   515 Mbits/sec   61    534 KBytes
[  5]   4.00-5.00   sec  51.6 MBytes   432 Mbits/sec    2    597 KBytes
...
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-5.00   sec   300 MBytes   503 Mbits/sec  216            sender
[  5]   0.00-5.01   sec   298 MBytes   498 Mbits/sec                  receiver

iperf Done.
10:49 user@samba:~ > iperf3 -c 100.93.240.XX -t 5 -R
Connecting to host 100.93.240.XX port 5201
Reverse mode, remote host 100.93.240.XX is sending
[  5] local 100.125.133.11 port 54136 connected to 100.93.240.XX port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   150 MBytes  1.26 Gbits/sec
[  5]   1.00-2.00   sec   150 MBytes  1.26 Gbits/sec
[  5]   2.00-3.00   sec   138 MBytes  1.16 Gbits/sec
[  5]   3.00-4.00   sec   104 MBytes   870 Mbits/sec
[  5]   4.00-5.00   sec   153 MBytes  1.28 Gbits/sec
..
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-5.00   sec   698 MBytes  1.17 Gbits/sec   50            sender
[  5]   0.00-5.00   sec   695 MBytes  1.17 Gbits/sec                  receiver

iperf Done.

However tests between two Tailscale LXCs on the same machine show great performance:

Code:

9:42 user@samba:~ > iperf3 -c ts.ip.same.machine1 -t 5
Connecting to host ts.ip.same.machine1, port 5201
[  5] local ts.ip.same.machine2 port 59294 connected to ts.ip.same.machine1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.38 GBytes  11.9 Gbits/sec    0   4.16 MBytes
[  5]   1.00-2.00   sec  1.40 GBytes  12.0 Gbits/sec    0   4.16 MBytes
[  5]   2.00-3.00   sec  1.39 GBytes  11.9 Gbits/sec    0   4.16 MBytes
[  5]   3.00-4.00   sec  1.39 GBytes  12.0 Gbits/sec    0   4.16 MBytes
[  5]   4.00-5.00   sec  1.38 GBytes  11.8 Gbits/sec    0   4.16 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-5.00   sec  6.94 GBytes  11.9 Gbits/sec    0            sender
[  5]   0.00-5.00   sec  6.94 GBytes  11.9 Gbits/sec                  receiver

iperf Done.
09:43 user@samba:~ > iperf3 -c ts.ip.same.machine1 -t 5 -R
Connecting to host ts.ip.same.machine1, port 5201
Reverse mode, remote host ts.ip.same.machine1 is sending
[  5] local ts.ip.same.machine2 port 59310 connected to ts.ip.same.machine1 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  1.39 GBytes  11.9 Gbits/sec
[  5]   1.00-2.00   sec  1.40 GBytes  12.0 Gbits/sec
[  5]   2.00-3.00   sec  1.41 GBytes  12.2 Gbits/sec
[  5]   3.00-4.00   sec  1.42 GBytes  12.2 Gbits/sec
[  5]   4.00-5.00   sec  1.41 GBytes  12.1 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-5.00   sec  7.03 GBytes  12.1 Gbits/sec    0            sender
[  5]   0.00-5.00   sec  7.03 GBytes  12.1 Gbits/sec                  receiver

iperf Done

Issue affects multiple NIC types with at least these reported so far:

AQtion AQC113CS
Aquantia Corp. AQC113C NBase-T/IEEE 802.3an Ethernet Controller [Marvell Scalable mGig] (rev 03)
& mine:

Code:

Device Type: ConnectX4LX
Part Number: MCX4121A-ACA_Ax
Description: ConnectX-4 Lx EN network interface card; 25GbE dual-port SFP28; PCIe3.0 x8; ROHS R6
PSID: MT_2420110034
12:22 root@pve-homeserver25:~/manually_installed/Mellanox $ ./mlxup --query
Querying Mellanox devices firmware ...

Device #1:
----------

  Device Type:      ConnectX4LX
  Part Number:      MCX4121A-ACA_Ax
  Description:      ConnectX-4 Lx EN network interface card; 25GbE dual-port SFP28; PCIe3.0 x8; ROHS R6
  PSID:             MT_2420110034
  PCI Device Name:  /dev/mst/mt4117_pciconf0
  Base MAC:         0c42a12d0cd2
  Versions:         Current        Available
     FW             14.32.1912     14.32.1010
     PXE            3.6.0502       3.6.0502
     UEFI           14.25.0017     14.25.0017

  Status:           Up to date

(this is the latest Mellanox FW from: https://network.nvidia.com/products/adapter-software/firmware-tools/ and https://network.nvidia.com/support/firmware/connectx4lxen/)

Network configuration:

Proxmox host uses a standard Linux bridge (vmbr0) — no SR-IOV, no VLAN-aware bridge, etc
Physical NIC (nic2, Mellanox ConnectX-4 Lx, mlx5_core) is a bridge member of vmbr0
LXC containers connect via veth pairs through the bridge, with Proxmox firewall enabled (fwbr/fwln/fwpr chain)
Tailscale runs inside the LXC (not on the Proxmox host), so WireGuard UDP packets egress via: tailscale0 (LXC) → veth → fwbr → vmbr0 → nic2 (mlx5_core) → physical
lxc.mount.entry: /dev/net/tun passthrough (required for Tailscale in LXC)
Proxmox firewall enabled on the LXC interface (firewall=1 in LXC config) - issue still occurs with firewall disabled, though
LXCs are unprivileged, Debian 13, running basically nothing but Tailscale and samba (also effects my Jellyfin container and iperf3 performance as above!).

The issue is present and reproducible with at least these kernels:
7.0.0-3-pve
7.0.2-2-pve

The issue is NOT present with all prior kernels including (i.e. the workaround is to pin to an older kernel):
6.17.13-6-pve
6.17.13-7-pve

It would be great to get some eyes on this, and I am happy to run tests/supply logs etc.

Also reported to Tailscale at: https://github.com/tailscale/tailscale/issues/19777

Last edited: May 20, 2026

tests between two Tailscale LXCs on the same machine show great performance

HW NIC is not used in this test.

As Tailscale uses wireguard, have you tried a pure wireguard ?

I have just done a test with pure WireGuard, and it does not show the issue:

Code:

10:58 user@samba:~ > iperf3 -c 192.168.99.2 -t 10
iperf3 -c 192.168.99.2 -t 10 -R
Connecting to host 192.168.99.2, port 5201
[  5] local 192.168.99.1 port 60466 connected to 192.168.99.2 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   137 MBytes  1.15 Gbits/sec  171    585 KBytes
[  5]   1.00-2.00   sec   134 MBytes  1.12 Gbits/sec   22    521 KBytes
[  5]   2.00-3.00   sec   134 MBytes  1.12 Gbits/sec    0    680 KBytes
[  5]   3.00-4.00   sec   128 MBytes  1.07 Gbits/sec    4    632 KBytes
[  5]   4.00-5.00   sec   135 MBytes  1.13 Gbits/sec   60    566 KBytes
[  5]   5.00-6.00   sec   118 MBytes   987 Mbits/sec    0    701 KBytes
[  5]   6.00-7.00   sec   122 MBytes  1.02 Gbits/sec   28    615 KBytes
[  5]   7.00-8.00   sec   133 MBytes  1.12 Gbits/sec   48    574 KBytes
[  5]   8.00-9.00   sec   134 MBytes  1.12 Gbits/sec   24    522 KBytes
[  5]   9.00-10.00  sec   130 MBytes  1.09 Gbits/sec    0    680 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.27 GBytes  1.09 Gbits/sec  357            sender
[  5]   0.00-10.01  sec  1.27 GBytes  1.09 Gbits/sec                  receiver

iperf Done.
Connecting to host 192.168.99.2, port 5201
Reverse mode, remote host 192.168.99.2 is sending
[  5] local 192.168.99.1 port 50946 connected to 192.168.99.2 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   123 MBytes  1.03 Gbits/sec
[  5]   1.00-2.00   sec   122 MBytes  1.02 Gbits/sec
[  5]   2.00-3.00   sec   123 MBytes  1.03 Gbits/sec
[  5]   3.00-4.00   sec   121 MBytes  1.01 Gbits/sec
[  5]   4.00-5.00   sec   115 MBytes   968 Mbits/sec
[  5]   5.00-6.00   sec   104 MBytes   871 Mbits/sec
[  5]   6.00-7.00   sec   103 MBytes   864 Mbits/sec
[  5]   7.00-8.00   sec   120 MBytes  1.01 Gbits/sec
[  5]   8.00-9.00   sec   124 MBytes  1.04 Gbits/sec
[  5]   9.00-10.00  sec   121 MBytes  1.02 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.15 GBytes   988 Mbits/sec    0            sender
[  5]   0.00-10.00  sec  1.15 GBytes   986 Mbits/sec                  receiver

iperf Done.

Some more info that might be useful:

Code:

ip -d link show tailscale0
3: tailscale0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1280 qdisc fq_codel state UNKNOWN mode DEFAULT group default qlen 500
    link/none  promiscuity 0 allmulti 0 minmtu 68 maxmtu 65535
    tun type tun pi off vnet_hdr on persist off addrgenmode random numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 tso_max_size 65536 tso_max_segs 65535 gro_max_size 65536 gso_ipv4_max_size 65536 gro_ipv4_max_size 65536

vs.

Code:

ip -d link show wg-test
4: wg-test: <POINTOPOINT,NOARP> mtu 1420 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/none  promiscuity 0 allmulti 0 minmtu 0 maxmtu 2147483552
    wireguard addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 tso_max_size 524280 tso_max_segs 65535 gro_max_size 65536 gso_ipv4_max_size 65536 gro_ipv4_max_size 65536

Have you tried with VM instead LXC ?
Have you tried directly on the host ?

VM - same results (as others have also reported).

Host - I do not want Tailscale on the host (but iperf3 performance without Tailscale is fine).

we also tried to reproduce with plain wireguard and similar hardware but couldn't. could you please report this to tailscale? it sounds like their userspace wireguard implementation hits a pathological case here that regressed..

Thanks - yes, I can not reproduce with plin WireGuard either.

As per the original message, I have also reported to Tailscale here:
Also reported to Tailscale at: https://github.com/tailscale/tailscale/issues/19777

A user there has noted they have hit the same issue with Kernel 7 on Ubuntu 26.04

sorry, missed that. subscribed to the issue - if you find information elsewhere that points at specific kernel changes (causing the regression, or fixing it ) please add it either here or there and I can take a closer look!

Thank you, I definitely will!

that does look like a potential cherry-pick candidate, yes.

Will you perhaps pop a note here if you do cherry-pick this fix?

Unfortunately I have just tested and it's not working.

Code:

18:04 root@pve-homeserver25:~ $ uname -a
Linux pve-homeserver25 7.0.6-2-pve #1 SMP PREEMPT_DYNAMIC PMX 7.0.6-2 (2026-05-29T11:08Z) x86_64 GNU/Linux
18:03 root@pve-homeserver25:~ $ ethtool -K nic2 tx-udp-segmentation on generic-segmentation-offload on

Code:

18:11 user@samba:~ > iperf3 -c <REMOTE_IP> -t 10
Connecting to host <REMOTE_IP>, port 5201
[  5] local <LOCAL_IP> port 54482 connected to <REMOTE_IP> port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   512 KBytes  4.19 Mbits/sec   45   2.40 KBytes
[  5]   1.00-2.00   sec   256 KBytes  2.10 Mbits/sec   44   2.40 KBytes
[  5]   2.00-3.00   sec   384 KBytes  3.15 Mbits/sec   50   2.40 KBytes
[  5]   3.00-4.00   sec   896 KBytes  7.34 Mbits/sec   46   2.40 KBytes
[  5]   4.00-5.00   sec   768 KBytes  6.29 Mbits/sec   50   2.40 KBytes
[  5]   5.00-6.00   sec  1.25 MBytes  10.5 Mbits/sec   54   2.40 KBytes
[  5]   6.00-7.00   sec   640 KBytes  5.24 Mbits/sec   44   2.40 KBytes
[  5]   7.00-8.00   sec   640 KBytes  5.24 Mbits/sec   50   2.40 KBytes
[  5]   8.00-9.00   sec  1.00 MBytes  8.39 Mbits/sec   56   2.40 KBytes
[  5]   9.00-10.03  sec  1.00 MBytes  8.17 Mbits/sec   52   2.40 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.03  sec  7.25 MBytes  6.07 Mbits/sec  491            sender
[  5]   0.00-10.03  sec  7.25 MBytes  6.06 Mbits/sec                  receiver

iperf Done.
18:11 user@samba:~ > iperf3 -c <REMOTE_IP> -t 10 -R
Connecting to host <REMOTE_IP>, port 5201
Reverse mode, remote host <REMOTE_IP> is sending
[  5] local <LOCAL_IP> port 47058 connected to <REMOTE_IP> port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   135 MBytes  1.13 Gbits/sec
[  5]   1.00-2.00   sec   154 MBytes  1.29 Gbits/sec
[  5]   2.00-3.00   sec   151 MBytes  1.27 Gbits/sec
[  5]   3.00-4.00   sec   148 MBytes  1.25 Gbits/sec
[  5]   4.00-5.00   sec   121 MBytes  1.02 Gbits/sec
[  5]   5.00-6.00   sec   123 MBytes  1.03 Gbits/sec
[  5]   6.00-7.00   sec   148 MBytes  1.24 Gbits/sec
[  5]   7.00-8.00   sec   156 MBytes  1.31 Gbits/sec
[  5]   8.00-9.00   sec   151 MBytes  1.27 Gbits/sec
[  5]   9.00-10.00  sec   154 MBytes  1.29 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.01  sec  1.41 GBytes  1.21 Gbits/sec    3            sender
[  5]   0.00-10.00  sec  1.41 GBytes  1.21 Gbits/sec                  receiver

iperf Done.

I think perhaps only one part of the fix was cherry-picked?

Per this message, there are two parts: https://github.com/tailscale/tailscale/issues/19777#issuecomment-4577851874
Consolidated to: https://github.com/torvalds/linux/commit/9a8e01c50093e6fc6569aa8353f856f1b6097189

Apologies if I am reading the diff wrong.

I think perhaps only one part of the fix was cherry-picked?

Looks like it, we'll check and report back!

the kernel patches should be pulled in with the next 7.0 stable update on our end.

推荐订阅源

Proxmox Support Forum