if that happens, then your backupserver or your connection to backupserver may be too slow and your IO inside VM too high during backup window. VM is getting throttled when there is too much IO during backup. please check backup throughput and VM io throughput. please open your own thread for this, i think it's not related here.

if that happens, then your backupserver or your connection to backupserver may be too slow and your IO inside VM too high during backup window. VM is getting throttled when there is too much IO during backup. please check backup throughput and VM io throughput. please open your own thread for this, i think it's not related here.

But I think that's exactly what it is, because that's how it behaves here in a test environment.
And there is absolutely no load on the VM. they are just running along.
My observation so far is that it only happens on file servers with a second disk.

this are my settings

EDIT:
you have switched Async IO to threads? @RolandK

please better post contents of conf file instead screenshots, as this gives a closer picture. yes, i have switched.

agent: 1
balloon: 8192
bios: ovmf
boot: order=scsi0;ide0;ide2;net0
cores: 8
cpu: host
efidisk0: Data:vm-103-disk-0,efitype=4m,pre-enrolled-keys=1,size=1M
ide0: local:iso/virtio-win-0.1.240.iso,media=cdrom,size=612812K
ide2: local:iso/SW_DVD9_Win_Server_STD_CORE_2022__64Bit_English_DC_STD_MLF_X22-74290.ISO,media=cdrom,size=5420564K
machine: pc-q35-8.1
memory: 16384
meta: creation-qemu=8.1.2,ctime=1703680088
name: ENTFERNT
net0: virtio=BC:24:11:CE:59:76,bridge=vmbr0,firewall=1,tag=ENTFERNT
numa: 0
onboot: 1
ostype: win11
scsi0: Data:vm-103-disk-1,discard=on,iothread=1,size=120G,ssd=1
scsi1: Data:vm-103-disk-3,discard=on,iothread=1,size=500G,ssd=1
scsihw: virtio-scsi-single
smbios1: uuid=ENTFERNT
sockets: 1
startup: order=2
tpmstate0: Data:vm-103-disk-2,size=4M,version=v2.0
vmgenid: 94dd905c-49da-495e-9bfb-ae92ba2807e6

EDIT:

I have only changed the second disk configuration for the time being.

scsi1: Data:vm-103-disk-3,aio=threads,discard=on,iothread=1,size=500G,ssd=1

If it's occuring during the backup, it's possible that is because your backup storage is too slow.

Proxmox use copy-before-write technique, that mean that if the vm need to overwrite a not yet backuped block, it must first backup the block then ack before the guest os is able to write. (for the first write on this block only).

Depending on the workload, it can impact performance.

Not in my case. 40Gbe link. There is powerful CPU and HW RAID with SSD disks on PBS server
CEPH datastore performance is a key limited factor (of backup speed)

And one more thing:
VM "hangs" (with different event to windows syslog + ID 129 as well) when another VM is backing up (that VM is located on another node)

P.S. This happens not every night (~10 VM in that cluster and backup is one time a day) and different VMs affected
The common this is that it always happens during and only! the backup job period

P.S.S. "Hangs" - means there are a lot error messages in windows syslog (all red) and some services (like RDS) becomes unavailable (user fails to log in) how ever login from console sometimes works

Hi,
did any of you already try the workaround suggested in the linked GitHub issue, i.e. increase the miniport IoTimeOutValue registry parameter?

Otherwise, I'd suggest to reduce the number of (concurrent) backups to your PBS storage. The most likely explanation is that the IO issued by the guest has to wait too long for PBS to complete (before a guest write happens, old data needs to be written to the backup target first if it wasn't already backed up). I am able to get the same error messages by simply cutting the connection to the PBS for two minutes during backup.

the vm is not yet a productive machine. nobody is working on it. it just runs with an empty second disk. The backup is long finished according to the logs. as you can see in my post above, minutes after the backup the virtio error appears in the log. I had switched from io_uring to Native as a test. The error also occurred then. Since yesterday I changed it to threads as Roland recommended. So far no log entry in the machine.

EDIT: Here is a screenshot of a VM with an empty second disk when the errors occurred every minute. No more IO is possible on the disk. I have tried to create a new folder.

@fiona

Is there any way to limit read/write bandwidth of PBS client (something like bwlimit in vdump.conf) ?

there is a suggestion from virtio devs:

by vrozenfe
I would try to reduce the transfer size ("PhysicalBreaks") first to see if it helps to solve the problem.
Btw, there is a different option to adjust the transfer size without touching the Windows Registry.
Upstream and RHEL QEMU have "max_sectors" parameters for that.
"-device virtio-scsi-pci,id=scsi-vioscsi1,max_sectors=63" will have the same effect as "PhysicalBreaks = 0x3f (63)".
Unfortunately, I don't know if it works for Proxmox or not. You should probably clarify it with Proxmox support team.

How should I change my Vm config to incorporate this tune?

P.S. there is another thread: https://github.com/virtio-win/kvm-guest-drivers-windows/issues/756