I built my Proxmox homelab on Z890 chipset with Intel Core Ultra 5 235 and 64GB of RAM a year ago.
OS and VMs are on enterprise SSD with single disk ZFS configured (RAID1 would be great, but I don't have it), and cheap storage consists of five WD Red Plus 8TB disks with ZFS raid-z2 configured. I use cheap storage for all my media files and as destination for VM backups.

I finally decided to backup those VMs and media files properly, and in order to do that I acquired ASUS Prime N100I-D D4-CSM with 32GB of RAM, NVMe for OS and three SATA disks for backup storage (two WD Red Plus 8TB and one WD Red Plus 4TB disk). Motherboard has only one PCIe 3.0 x1 slot which I slotted with ASM1064 on which I connected those three hard drives.
I'm not interested in parity since this should be for backup and I want to maximize usable space. And now I don't know what to do next...
First on my mind was choosing PBS, but what file system should I choose for those three WD Red drives and how should I configure backup from Proxmox's ZFS media storage? I would avoid spending more money if that is possible.

I assume you have your main unit on 24/7 and turn on the backup unit manually from time to time to do backups of the main unit?

PBS has a lot of benefits for backing up and restoring VMs/LXCs over your current local backups. This is even already the case without the backup unit in place and you could consider this a separate step to transition from local backups to a local PBS server on the main unit.
If you go that way, from my experience you really want the PBS datastore on SSD. Assuming your enterprise SSD is large enough to hold a PBS datastore for your deduplicated VM/LXC backups with some retention, you can install PBS in an LXC on your main unit for regular backups of all VMs/LXCs except PBS itself. Create a separate backup scheme for only the PBS LXC on the local storage, outside of PBS as you do now.
You probably already make regular zfs snapshots of the datasets on your 'cheap storage' zpool. If not, consider the excellent sanoid tool by Jim Salter. It's in the default Debian repo's.

Next up is your backup unit. Install PBS on the NVME and you ideally also want the datastore on the NVME again. Create a pull sync job that drags in the PBS backups from the main unit. Additionally, setup a zpool for your other data. Ideally you want a 3x8TB zpool alongside the nvme where you can sync the data from the 'cheap storage'. So sell the 4TB and get an extra 8TB disk. ;-) Or use the 4TB as a 3rd off-site external drive backup of your most important files in case your house burns down.

Create a script that runs the PBS sync pull job from the main PBS unit, a sync of the 'cheap storage' pool and you can also sync important folders of the proxmox host to the backup unit, including the aforementioned backups of the PBS LXC. Syncoid again is great for incrementally syncing zfs datasets to another zpool but you can also use rsync and create zfs snapshots manually after each sync as you prefer. You can either run the script manually or automated on boot of the backup unit.

You can obviously also go without PBS and just sync your current VM/LXC backups to the backup unit but PBS definitely has its advantages.

My main unit is up 24/7 and I currently I do manual backups from time to time to my ZFS raid-z2 pool with cheap disks (which are also on my main machine). I never thought of setting up PBS as LXC on my Proxmox, but it sounds like an interesting idea, but SSDs are too damn expensive to put backups on them.
I'll set up PBS on machine I acquired for backup, set stripe zpool with existing disks (if it dies, it dies, it's backup anyway) and try Syncoid as soon as possible.

So standard Proxmox backup doesn't do incremental backups and PBS does?
If that is the case PBS in LXC container makes sense for local backups to another zfs pool. But do I really need backups on the same machine where my Proxmox is when I have PBS on the separate machine? I don't really need 24/7 scheduled backups and if my Proxmox machine dies, backups would die with it anyway.

Also, I set up sanoid and syncoid for zfs pool (with cheap disks) that contains data and I want to thank daanw for recommending it to me, it is a great solution.

Exactly. Separate zpool in the same machine holding the PBS datastore and backups of the PBS LXC.

In case the Proxmox rpool dies:
- reinstall Proxmox,
- restore the last backup of the PBS LXC from the separate zpool in the same machine.
- restore all the other VMs and LXCs from PBS.

All should be back online in less than 30 min.

In case the second zpool dies, have a second PBS on another machine that does a regular sync of all VMs and LXCs from the first PBS and a copy of the PBS LXC backups.

So difference between restoring from local and remote PBS is just the restore speed?

Now that I have separate PBS machine working I see that it is consuming 16GB of ram (of max 32GB). How common is setting up second Proxmox with LXC PBS instead of standalone PBS?