With the endless pain of only a few companies taking all inventory from the normal people... The hard question has to be asked in a day when HDD are mostly archive or cold tier. I haven't used a HD outside of backup targets in years... It's a bad time to be in hardware refresh cycle.

Proxmox 9.2.x
Ceph - Current enterprise repos.

I've read the forums and most are around older versions of Ceph largely based on Proxmox 7 or early 8 versions. I know considerable changes have taken place with Ceph and it is a lot better over the years with smaller cluster setups. I have it on several small 3 node ones and have no problems for the workload it delivers. Latency is a little higher then ZFS which isn't a surprise as it's network based not local storage access.

Looking to do a ceph cluster. 5 nodes dedicated network 25Gb or 100Gb for ceph alone.
2 SSD OSD enterprise drives
6HDD OSD sas based
Ensure code for pool 8/3 should work.

Primary workload is massive file shares and low utilization VM's and a few DB servers. Current environment we generally transfer a little over 200Tb a day, lots of people working on the same projects. Not a lot overall but not a small number either. What are the general limits with HDD with Ceph? Latency and througput are large consideration as currently our other ageing vendor setup it a little lack luster but it hasn't had a problem delivering maxed 1Gb connections to several users for the past few years its just slow to start which isn't ideal. We still run several database servers on prim and with that 3ms letancy target is generally considerd max letancy for access and aiming to stick or lower if possible. Not sure how Ceph handles tierd approch as I woudnt have any db or wal drives if I wanted to use the ssds as the main tier. Tiering with ceph is like voodoo to me it's something I just haven't done.

Not likely that I'll need 100Gb nic with HD's in the mix and 25 bonded should be more then fine for 5 nodes. Math says I wouldn't hit the limit of a single 25Gb port in real world use cases.

Honest descussion as versions change over time. Last time I used HDD with ceph I swore off Proxmox and Ceph all togther for years becuase of just how bad it was. Using SSD I don't have to many issues... Not none just not many. With current pricing on SSD's I'm into the 7 figure mark to get the capacity I would need which just a year ago when I did the same quote it was sub 350k all in. This s*** has gotten out of hand which is why HDD are back in the mix.