Three or four days ago I deployed a Gitea runner, tested it, closed the terminal and moved on. Today, for some reason I don't even remember, I tried to SSH into the server. Nothing.
Opened the browser to check if the server was alive — course project running, Gitea up, Nextcloud up, everything in the k3s cluster fine. Went into VMmanager, rebooted. Still couldn't get in. Closed The Boys and went to fix it through VNC.
nmap first
nmap -p 22 2.27.42.100
PORT STATE SERVICE
22/tcp filtered ssh
filtered — not closed, not refused. Packets reaching the server and getting silently dropped. Firewall somewhere.
Checked the INPUT chain:
Chain INPUT (policy DROP)
1 KUBE-ROUTER-INPUT
2 ACCEPT tcp dpt:22
3 KUBE-PROXY-FIREWALL
...
8 ufw-before-input
ACCEPT rule for port 22 at position 2. Looked fine.
Wasn't fine.
Where it actually breaks
nft list ruleset | grep -E "22|drop|DROP"
type filter hook input priority filter; policy drop;
...
tcp dport 22 counter packets 0 bytes 0 accept
Zero packets on the accept rule. Connections weren't reaching it at all.
The server has both iptables-nft (kube-router, UFW) and native nftables running at the same time. They fight. kube-router constantly reconciles and overwrites stuff. The output was already warning about it:
# Warning: table ip filter is managed by iptables-nft, do not touch!
The Gitea runner deploy from a few days ago triggered a reconciliation that messed up the rule ordering. I just hadn't tried SSH since then.
The fix
Native nftables table at priority -150. Runs before any filter chains (priority 0), before kube-router, before everything. kube-router only manages the iptables-nft layer — doesn't touch native nftables tables.
nft add table inet ssh_rescue
nft 'add chain inet ssh_rescue input { type filter hook input priority -150; }'
nft add rule inet ssh_rescue input tcp dport 22 accept
SSH came back. Wrote to my friend: "fixed". He said "eee". Made it a systemd service. Two minutes later wrote to him: "SSH died again".
Round two
Back in VNC. Table still there, rules still there, zero packets on the accept rule again.
systemctl start nftables had loaded the saved /etc/nftables.conf which had kube-router's warning comments in it and corrupted the nftables state. Re-added the table manually. SSHed from the server to itself — worked. sshd on 0.0.0.0:22 — fine. Still couldn't connect from outside.
Then checked fail2ban, which had installed itself as a dependency somewhere during all this. It had banned two IPs — bots hammering port 22 the whole time. My repeated failed attempts got caught in the same drop. Ran fail2ban-client unban --all, connected immediately.
Then SSH died again. Disabled fail2ban completely — still nothing.
The actual reason
I was in the middle of debugging nftables chains when I remembered — my VPN is self-hosted on a server in Finland. The day before I'd been complaining to my friend that the VPN was slow. The provider had sent an email:
Due to an increase in brute-force complaints via SSH (port 22), outgoing traffic on port 22 has been blocked for VPS servers located in Finland.
I had read it. I knew about it. And I spent three hours in VNC chasing nftables, fail2ban and kube-router while the answer was sitting in my inbox the whole time.
Turned off the VPN. SSH connected immediately.
What to do from day one
Add the ssh_rescue table before you need it:
nft add table inet ssh_rescue
nft 'add chain inet ssh_rescue input { type filter hook input priority -150; }'
nft add rule inet ssh_rescue input tcp dport 22 accept
Make it a service so it survives reboots and kube-router reconciliation:
cat > /usr/local/bin/ssh-rescue.sh << 'EOF2'
#!/bin/bash
nft list table inet ssh_rescue 2>/dev/null || {
nft add table inet ssh_rescue
nft add chain inet ssh_rescue input { type filter hook input priority -150; }
nft add rule inet ssh_rescue input tcp dport 22 accept
}
EOF2
chmod +x /usr/local/bin/ssh-rescue.sh
After any deploy that touches networking — run nmap -p 22 <ip> immediately.
And read your damn emails.