Any ideas what to look into?

I would think the steps are the same as troubleshooting any generic slow application whose job is to transfer data:
- record/analyze CPU load on both side
- record/analyze disk load
- record/analyze network load, errors

You may want to install something like Grafana to point at PVE/PBS so you can correlate pretty graphs and have continues monitoring.
There is nothing you can do without first collecting the data.

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox