FWIW we're running a smallish (~200TB gross) Garage cluster for a year or two, initially backed by a replication setup that fails over to AWS and this year slowly migrating workloads completely off AWS. IIRC there have been a couple of controlled node replacements that have gone well, though I wasn't directly involved. Heavy caching keeps IOPS low so can't really say much about performance on that front.

No complete disk or node failures so can't talk about those, but last month we had a situation where a node hit the LMDB memory limit and due to a monitoring fail on our side spent a few weeks in that state. The node stayed up and kept serving its data and resynced everything without problems afterwards.

The surprising part was that the metadata databases can have _huge_ size differences between nodes, in the range of 50-1200GB (the 1T+ databases have something like 300-400G actual data but likely ballooned due to some temporary-data buckets that have come and gone). Because LMDB compactions aren't run automatically, you may need to do that manually in some circumstances.

A couple of simple API compatibility issues we reported were fixed quickly.