Hello everyone,
I would like to add my observations as I am experiencing the exact same problem.
Since last night (August 28, 2025, starting around 3:00 AM), my Proxmox host becomes completely unresponsive during the scheduled backup jobs that write to an NFS share. The Web-UI is mostly down, and SSH connections are possible.
Interestingly, the host seems to "freeze" only partially. The backup jobs themselves continue to run and complete successfully. As soon as the last backup job is finished, the host immediately becomes fully accessible again, as if nothing happened. I can reproduce this behavior reliably by manually starting a backup to the NFS share.
While the system is in this "frozen" state, I managed to keep an SSH session open. My observations are:
- Simple commands like
uptimeordfstill work and return output instantly. - However, any command that tries to read process states, such as
top,htop, orps -ef, hangs indefinitely and produces no output even after several minutes. - The system load is extremely high during this period.
uptimeshows a load average of around21, 20, 15.
What might be particularly relevant for troubleshooting:
The issue first occurred last night while the host was still running Proxmox VE 8. I had performed the preparatory steps from thepve8to9 guide the evening before (August 27), but the actual distribution upgrade to PVE 9 was only done this morning at 10:00 AM, after the problem had already appeared for the first time.Here is the history from the evening before the first freeze. The night from the 26th to the 27th was completely fine.
Code:
Start-Date: 2025-08-27 20:23:16
Commandline: apt upgrade
Upgrade: pve-manager:amd64 (8.4.11, 8.4.12)
End-Date: 2025-08-27 20:23:22
Start-Date: 2025-08-27 20:32:56
Commandline: apt remove systemd-boot
Remove: systemd-boot:amd64 (252.38-1~deb12u1)
End-Date: 2025-08-27 20:32:57
Start-Date: 2025-08-27 20:34:12
Commandline: apt install amd64-microcode
Install: amd64-microcode:amd64 (3.20240820.1~deb12u1)
End-Date: 2025-08-27 20:34:34
Start-Date: 2025-08-27 20:35:07
Commandline: apt install --reinstall grub-efi-amd64
Reinstall: grub-efi-amd64:amd64 (2.06-13+pmx7)
End-Date: 2025-08-27 20:35:13
Start-Date: 2025-08-27 21:04:34
Commandline: apt full-upgrade
Upgrade: librados2:amd64 (17.2.8-pve2, 18.2.7-pve1), ceph-fuse:amd64 (17.2.8-pve2, 18.2.7-pve1), python3-ceph-common:amd64 (17.2.8-pve2, 18.2.7-pve1), librbd1:amd64 (17.2.8-pve2, 18.2.7-pve1), librgw2:amd64 (17.2.8-pve2, 18.2.7-pve1), ceph-common:amd64 (17.2.8-pve2, 18.2.7-pve1), python3-cephfs:amd64 (17.2.8-pve2, 18.2.7-pve1), libcephfs2:amd64 (17.2.8-pve2, 18.2.7-pve1), libradosstriper1:amd64 (17.2.8-pve2, 18.2.7-pve1), python3-rbd:amd64 (17.2.8-pve2, 18.2.7-pve1), python3-rgw:amd64 (17.2.8-pve2, 18.2.7-pve1), python3-ceph-argparse:amd64 (17.2.8-pve2, 18.2.7-pve1), python3-rados:amd64 (17.2.8-pve2, 18.2.7-pve1)
End-Date: 2025-08-27 21:04:40
Start-Date: 2025-08-27 21:05:47
Commandline: apt full-upgrade
Upgrade: librados2:amd64 (18.2.7-pve1, 19.2.2-pve1~bpo12+1), ceph-fuse:amd64 (18.2.7-pve1, 19.2.2-pve1~bpo12+1), python3-ceph-common:amd64 (18.2.7-pve1, 19.2.2-pve1~bpo12+1), librbd1:amd64 (18.2.7-pve1, 19.2.2-pve1~bpo12+1), librgw2:amd64 (18.2.7-pve1, 19.2.2-pve1~bpo12+1), ceph-common:amd64 (18.2.7-pve1, 19.2.2-pve1~bpo12+1), python3-cephfs:amd64 (18.2.7-pve1, 19.2.2-pve1~bpo12+1), libcephfs2:amd64 (18.2.7-pve1, 19.2.2-pve1~bpo12+1), libradosstriper1:amd64 (18.2.7-pve1, 19.2.2-pve1~bpo12+1), python3-rbd:amd64 (18.2.7-pve1, 19.2.2-pve1~bpo12+1), python3-rgw:amd64 (18.2.7-pve1, 19.2.2-pve1~bpo12+1), python3-ceph-argparse:amd64 (18.2.7-pve1, 19.2.2-pve1~bpo12+1), python3-rados:amd64 (18.2.7-pve1, 19.2.2-pve1~bpo12+1)
End-Date: 2025-08-27 21:05:53Although no kernel update is explicitly listed in this log, one of the package updates from that evening must have introduced this behavior. My symptoms align perfectly with a kernel or storage subsystem issue related to NFS.
I hope this information helps narrow down the cause.
UPDATE 2025-08-29 16:16
Code:
Aug 28 21:47:51 proxmox pvedaemon[588804]: INFO: Starting Backup of VM 111 (qemu)
Aug 28 21:47:51 proxmox kernel: sdc: sdc1
Aug 28 21:47:51 proxmox kernel: vfio-pci 0000:06:00.0: resetting
Aug 28 21:47:51 proxmox kernel: sdb: sdb1
Aug 28 21:47:51 proxmox kernel: vfio-pci 0000:06:00.0: reset done
Aug 28 21:47:51 proxmox kernel: sda: sda1
Aug 28 21:47:51 proxmox kernel: vfio-pci 0000:07:00.0: resetting
Aug 28 21:47:51 proxmox kernel: vfio-pci 0000:07:00.0: reset done
Aug 28 21:47:51 proxmox systemd[1]: Started 111.scope.
Aug 28 21:47:52 proxmox kernel: tap111i0: entered promiscuous mode
Aug 28 21:47:52 proxmox kernel: vmbr0: port 2(fwpr111p0) entered blocking state
Aug 28 21:47:52 proxmox kernel: vmbr0: port 2(fwpr111p0) entered disabled state
Aug 28 21:47:52 proxmox kernel: fwpr111p0: entered allmulticast mode
Aug 28 21:47:52 proxmox kernel: fwpr111p0: entered promiscuous mode
Aug 28 21:47:52 proxmox kernel: vmbr0: port 2(fwpr111p0) entered blocking state
Aug 28 21:47:52 proxmox kernel: vmbr0: port 2(fwpr111p0) entered forwarding state
Aug 28 21:47:52 proxmox kernel: fwbr111i0: port 1(fwln111i0) entered blocking state
Aug 28 21:47:52 proxmox kernel: fwbr111i0: port 1(fwln111i0) entered disabled state
Aug 28 21:47:52 proxmox kernel: fwln111i0: entered allmulticast mode
Aug 28 21:47:52 proxmox kernel: fwln111i0: entered promiscuous mode
Aug 28 21:47:52 proxmox kernel: fwbr111i0: port 1(fwln111i0) entered blocking state
Aug 28 21:47:52 proxmox kernel: fwbr111i0: port 1(fwln111i0) entered forwarding state
Aug 28 21:47:52 proxmox kernel: fwbr111i0: port 2(tap111i0) entered blocking state
Aug 28 21:47:52 proxmox kernel: fwbr111i0: port 2(tap111i0) entered disabled state
Aug 28 21:47:52 proxmox kernel: tap111i0: entered allmulticast mode
Aug 28 21:47:52 proxmox kernel: fwbr111i0: port 2(tap111i0) entered blocking state
Aug 28 21:47:52 proxmox kernel: fwbr111i0: port 2(tap111i0) entered forwarding state
Aug 28 21:47:52 proxmox kernel: vfio-pci 0000:06:00.0: resetting
Aug 28 21:47:52 proxmox kernel: vfio-pci 0000:06:00.0: reset done
Aug 28 21:47:52 proxmox kernel: vfio-pci 0000:07:00.0: resetting
Aug 28 21:47:52 proxmox kernel: vfio-pci 0000:07:00.0: reset done
Aug 28 21:47:52 proxmox kernel: vfio-pci 0000:07:00.0: resetting
Aug 28 21:47:52 proxmox kernel: vfio-pci 0000:07:00.0: reset done
Aug 28 21:47:52 proxmox kernel: vfio-pci 0000:06:00.0: resetting
Aug 28 21:47:52 proxmox kernel: vfio-pci 0000:06:00.0: reset done
Aug 28 21:47:52 proxmox pvedaemon[588804]: VM 111 started with PID 588840.
Aug 28 21:47:53 proxmox systemd[1]: Started check-mk-agent@895-1235-999.service - Checkmk agent (PID 1235/UID 999).
Aug 28 21:47:54 proxmox pveproxy[557805]: worker exit
Aug 28 21:47:54 proxmox pveproxy[1516]: worker 557805 finished
Aug 28 21:47:54 proxmox pveproxy[1516]: starting 1 worker(s)
Aug 28 21:47:54 proxmox pveproxy[1516]: worker 588999 started
Aug 28 21:47:55 proxmox systemd[1]: check-mk-agent@895-1235-999.service: Deactivated successfully.
Aug 28 21:47:55 proxmox systemd[1]: check-mk-agent@895-1235-999.service: Consumed 1.513s CPU time, 48.4M memory peak.
Aug 28 21:47:56 proxmox pvedaemon[514612]: <root@pam> successful auth for user 'checkmk@pve'
Aug 28 21:48:53 proxmox pvedaemon[504642]: <root@pam> successful auth for user 'root@pam'
Aug 28 21:49:43 proxmox pvestatd[1476]: status update time (72.651 seconds)
Aug 28 21:49:57 proxmox pvedaemon[504642]: <root@pam> successful auth for user 'root@pam'
Aug 28 21:50:27 proxmox pveproxy[573689]: proxy detected vanished client connection
Aug 28 21:50:59 proxmox pveproxy[583729]: proxy detected vanished client connection
Aug 28 21:51:00 proxmox pveproxy[583729]: proxy detected vanished client connection
Aug 28 21:51:07 proxmox pveproxy[573689]: proxy detected vanished client connection
Aug 28 21:51:17 proxmox pveproxy[573689]: proxy detected vanished client connection
Aug 28 21:51:25 proxmox pveproxy[588999]: proxy detected vanished client connection
Aug 28 21:51:47 proxmox pveproxy[583729]: proxy detected vanished client connection
Aug 28 21:51:59 proxmox pveproxy[583729]: proxy detected vanished client connection
Aug 28 21:51:59 proxmox pveproxy[588999]: proxy detected vanished client connection
Aug 28 21:52:29 proxmox pveproxy[573689]: proxy detected vanished client connection
Aug 28 21:52:39 proxmox sshd-session[590216]: Accepted publickey for root from 192.168.137.150 port 63955 ssh2: RSA SHA256:sOb7KQbZ2UbuUUSlRmMOqxThh+9/VfoiSoK8ki6xnxY
Aug 28 21:52:39 proxmox sshd-session[590216]: pam_unix(sshd:session): session opened for user root(uid=0) by root(uid=0)
Aug 28 21:52:39 proxmox systemd-logind[1102]: New session 29 of user root.
Aug 28 21:52:39 proxmox systemd[1]: Started session-29.scope - Session 29 of User root.
Aug 28 21:53:00 proxmox pveproxy[588999]: proxy detected vanished client connection
Aug 28 21:53:57 proxmox kernel: INFO: task ksmd:99 blocked for more than 122 seconds.
Aug 28 21:53:57 proxmox kernel: Tainted: P O 6.14.8-2-pve #1
Aug 28 21:53:57 proxmox kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Aug 28 21:53:57 proxmox kernel: task:ksmd state:D stack:0 pid:99 tgid:99 ppid:2 task_flags:0x200040 flags:0x00004000
Aug 28 21:53:57 proxmox kernel: Call Trace:
Aug 28 21:53:57 proxmox kernel: <TASK>
Aug 28 21:53:57 proxmox kernel: __schedule+0x466/0x13f0
Aug 28 21:53:57 proxmox kernel: ? srso_alias_return_thunk+0x5/0xfbef5
Aug 28 21:53:57 proxmox kernel: ? finish_task_switch.isra.0+0x9c/0x340
Aug 28 21:53:57 proxmox kernel: schedule+0x29/0x130
Aug 28 21:53:57 proxmox kernel: schedule_preempt_disabled+0x15/0x30
Aug 28 21:53:57 proxmox kernel: rwsem_down_read_slowpath+0x230/0x460
Aug 28 21:53:57 proxmox kernel: ? schedule_timeout+0x92/0x110
Aug 28 21:53:57 proxmox kernel: down_read+0x48/0xc0
Aug 28 21:53:57 proxmox kernel: ksm_scan_thread+0x16e/0x26a0
Aug 28 21:53:57 proxmox kernel: ? __pfx_ksm_scan_thread+0x10/0x10
Aug 28 21:53:57 proxmox kernel: kthread+0xfc/0x230
Aug 28 21:53:57 proxmox kernel: ? __pfx_kthread+0x10/0x10
Aug 28 21:53:57 proxmox kernel: ret_from_fork+0x47/0x70
Aug 28 21:53:57 proxmox kernel: ? __pfx_kthread+0x10/0x10
Aug 28 21:53:57 proxmox kernel: ret_from_fork_asm+0x1a/0x30
Aug 28 21:53:57 proxmox kernel: </TASK>
Aug 28 21:53:57 proxmox kernel: INFO: task worker:589731 blocked for more than 122 seconds.
Aug 28 21:53:57 proxmox kernel: Tainted: P O 6.14.8-2-pve #1
Aug 28 21:53:57 proxmox kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Aug 28 21:53:57 proxmox kernel: task:worker state:D stack:0 pid:589731 tgid:2931 ppid:1 task_flags:0x84000c0 flags:0x00000002
Aug 28 21:53:57 proxmox kernel: Call Trace:
Aug 28 21:53:57 proxmox kernel: <TASK>
Aug 28 21:53:57 proxmox kernel: __schedule+0x466/0x13f0
Aug 28 21:53:57 proxmox kernel: ? __set_task_blocked+0x29/0x80
Aug 28 21:53:57 proxmox kernel: ? srso_alias_return_thunk+0x5/0xfbef5
Aug 28 21:53:57 proxmox kernel: ? __x64_sys_rt_sigprocmask+0xd9/0x160
Aug 28 21:53:57 proxmox kernel: schedule+0x29/0x130
Aug 28 21:53:57 proxmox kernel: schedule_preempt_disabled+0x15/0x30
Aug 28 21:53:57 proxmox kernel: rwsem_down_read_slowpath+0x230/0x460
Aug 28 21:53:57 proxmox kernel: ? srso_alias_return_thunk+0x5/0xfbef5
Aug 28 21:53:57 proxmox kernel: down_read+0x48/0xc0
Aug 28 21:53:57 proxmox kernel: do_madvise+0x11f/0x480
Aug 28 21:53:57 proxmox kernel: ? switch_fpu_return+0x4f/0xe0
Aug 28 21:53:57 proxmox kernel: __x64_sys_madvise+0x2b/0x40
Aug 28 21:53:57 proxmox kernel: x64_sys_call+0x21a9/0x2310
Aug 28 21:53:57 proxmox kernel: do_syscall_64+0x7e/0x170
Aug 28 21:53:57 proxmox kernel: ? srso_alias_return_thunk+0x5/0xfbef5
Aug 28 21:53:57 proxmox kernel: ? futex_wake+0x8a/0x1a0
Aug 28 21:53:57 proxmox kernel: ? srso_alias_return_thunk+0x5/0xfbef5
Aug 28 21:53:57 proxmox kernel: ? do_futex+0x18e/0x260
Aug 28 21:53:57 proxmox kernel: ? srso_alias_return_thunk+0x5/0xfbef5
Aug 28 21:53:57 proxmox kernel: ? __x64_sys_futex+0x128/0x200
Aug 28 21:53:57 proxmox kernel: ? srso_alias_return_thunk+0x5/0xfbef5
Aug 28 21:53:57 proxmox kernel: ? srso_alias_return_thunk+0x5/0xfbef5
Aug 28 21:53:57 proxmox kernel: ? arch_exit_to_user_mode_prepare.isra.0+0x22/0xd0
Aug 28 21:53:57 proxmox kernel: ? srso_alias_return_thunk+0x5/0xfbef5
Aug 28 21:53:57 proxmox kernel: ? syscall_exit_to_user_mode+0x38/0x1d0
Aug 28 21:53:57 proxmox kernel: ? srso_alias_return_thunk+0x5/0xfbef5
Aug 28 21:53:57 proxmox kernel: ? do_syscall_64+0x8a/0x170
Aug 28 21:53:57 proxmox kernel: ? irqentry_exit_to_user_mode+0x2d/0x1d0
Aug 28 21:53:57 proxmox kernel: ? srso_alias_return_thunk+0x5/0xfbef5
Aug 28 21:53:57 proxmox kernel: ? irqentry_exit+0x43/0x50
Aug 28 21:53:57 proxmox kernel: ? srso_alias_return_thunk+0x5/0xfbef5
Aug 28 21:53:57 proxmox kernel: ? common_interrupt+0x64/0xe0
Aug 28 21:53:57 proxmox kernel: entry_SYSCALL_64_after_hwframe+0x76/0x7e
Aug 28 21:53:57 proxmox kernel: RIP: 0033:0x786b7871ebb7
Aug 28 21:53:57 proxmox kernel: RSP: 002b:00007869c1ff6dd8 EFLAGS: 00000206 ORIG_RAX: 000000000000001c
Aug 28 21:53:57 proxmox kernel: RAX: ffffffffffffffda RBX: 00007869c1ffbcdc RCX: 0000786b7871ebb7
Aug 28 21:53:57 proxmox kernel: RDX: 0000000000000004 RSI: 00000000007f7000 RDI: 00007869c17fb000
Aug 28 21:53:57 proxmox kernel: RBP: 00007869c17fb000 R08: 00007869c1ffb6c0 R09: 0000000000000000
Aug 28 21:53:57 proxmox kernel: R10: 0000000000000008 R11: 0000000000000206 R12: 0000000000801000
Aug 28 21:53:57 proxmox kernel: R13: 000000000000000b R14: 0000786b749d5880 R15: 00007869c17fb000
Aug 28 21:53:57 proxmox kernel: </TASK>
Aug 28 21:53:57 proxmox kernel: INFO: task worker:589734 blocked for more than 122 seconds.
Aug 28 21:53:57 proxmox kernel: Tainted: P O 6.14.8-2-pve #1
Aug 28 21:53:57 proxmox kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Aug 28 21:53:57 proxmox kernel: task:worker state:D stack:0 pid:589734 tgid:2931 ppid:1 task_flags:0x84000c0 flags:0x00000002
Aug 28 21:53:57 proxmox kernel: Call Trace:
...
Aug 28 21:56:05 proxmox pveproxy[583729]: proxy detected vanished client connection
Aug 28 21:56:16 proxmox pveproxy[583729]: proxy detected vanished client connection
Aug 28 21:56:16 proxmox pveproxy[573689]: proxy detected vanished client connection
Aug 28 21:56:20 proxmox pveproxy[588999]: proxy detected vanished client connection
Aug 28 21:56:37 proxmox pveproxy[573689]: proxy detected vanished client connection
Aug 28 21:56:50 proxmox pveproxy[588999]: proxy detected vanished client connection
Aug 28 22:05:23 proxmox pveproxy[573689]: worker exit
Aug 28 22:05:23 proxmox pveproxy[1516]: worker 573689 finished
Aug 28 22:05:23 proxmox pveproxy[1516]: starting 1 worker(s)
Aug 28 22:05:23 proxmox pveproxy[1516]: worker 592104 started
Aug 28 22:10:49 proxmox pveproxy[583729]: proxy detected vanished client connection
Aug 28 22:12:58 proxmox pveproxy[583729]: worker exit
Aug 28 22:12:58 proxmox pveproxy[1516]: worker 583729 finished
Aug 28 22:12:58 proxmox pveproxy[1516]: starting 1 worker(s)
Aug 28 22:12:58 proxmox pveproxy[1516]: worker 593233 started
Aug 28 22:17:01 proxmox CRON[593860]: pam_unix(cron:session): session opened for user root(uid=0) by root(uid=0)
Aug 28 22:17:01 proxmox CRON[593862]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Aug 28 22:17:01 proxmox CRON[593860]: pam_unix(cron:session): session closed for user root
Aug 28 22:18:18 proxmox pvestatd[1476]: status update time (1705.407 seconds)
Aug 28 22:18:25 proxmox pvedaemon[516904]: <root@pam> end task UPID:proxmox:0008FC04:004C5139:68B0B267:vzdump:111:root@pam: unexpected status
...
Aug 28 22:18:36 proxmox systemd[1]: Stopping user@0.service - User Manager for UID 0...UPDATE 2025-09-26
Once fixed, the backup process should no longer be able to be canceled via the GUI.
Code:
ps -ef | grep -ie vzdump -ie zstd
kill <zstd-pid>