The PVE 9.0 Beta Release Notes states: As GlusterFS is no longer maintained upstream, Proxmox VE 9 drops support for GlusterFS storages.

But this is simply not true. Glusterfs is actively maintained. The last commit and release (11.2) happened just three weeks ago! Furthermore, Debian Trixie has recent packages for gluster server, client, and cli.

While I understand that glusterfs may be in an awkward spot for enterprise since Red Hat EoL-ed Red Hat Gluster Storage, that happened in 2024, and it has not stopped development of the open source project.

It is well known that ceph does not perform well with consumer hardware, so glusterfs fills a much needed use case for homelabs and small scale users.

Please don't drop support for glusterfs!

Their github not even close to be "active maintained". Some bugfixes and nothing more.

If it's not broke, even as low as a few commits a year can be considered actively maintained. It might not be active development, but that's not the same as not being maintained.

That said, I wouldn't think glusterfs to be good for vms, except maybe a few low I/O ones... what kind of performance (ie: fio benchmark) can you get inside of a vm? Maybe I am under estimating it for random I/O....

It appears (based on the path, but maybe I glanced too quick) that was tested based on the performance of the host. I would be more interested in seeing what the performance would look like to a vm running in proxmox... that could give significantly different characteristics of qcow2 vs block layers.

Or was that script run from inside a VM on proxmox? It's not completely clear.

There are a bunch of different tests there. Most of them were on the Proxmox host. A few (which are noted as such) were run in a VM with virtiofsd mapping the directory into the VM.

Checking the host is best case scenario, though, right? Aside from caching reads in memory, how could performance in the VM be better than on the host?

I consider checking the VM is closer to real world. I don't care what the host sees, I want to know what it translates to the vms. I don't expect it to be better then the host, but more importantly I also don't expect the VM to scale in performance with the host when comparing completely different filesystems like block vs file. Who cares if the host can do a million operations a second if from within the VM it can only do 10. Who knows, the way caches interact, it's possible that you could get better performance inside a VM. You could say I want to know worse case scenario, not best...

I consider checking the VM is closer to real world. I don't care what the host sees, I want to know what it translates to the vms. I don't expect it to be better then the host, but more importantly I also don't expect the VM to scale in performance with the host when comparing completely different filesystems like block vs file. Who cares if the host can do a million operations a second if from within the VM it can only do 10. Who knows, the way caches interact, it's possible that you could get better performance inside a VM. You could say I want to know worse case scenario, not best...

Fair enough. I agree with you in principle, though I do think that in this particular case, especially with virtiofsd, there's no way to make up the performance gap on cephfs. Nonetheless, I'm willing to try the benchmark. I just need to tear down my running glusterfs volumes, so it'll be a little project.

Releases are slower, but still pretty consistent: https://github.com/gluster/glusterfs/tags

These results are too slow to be of any production value. on the basis of your benchmarks it wouldnt be worth pursuing.

It is well known that ceph does not perform well with consumer hardware, so glusterfs fills a much needed use case for homelabs and small scale users.

It really doesnt. Gluster doesnt have much value under a certain critical hardware deployment, and its only really suitable for file store- which at the "homelab" use case is much more effectively served with a NAS.

These results are too slow to be of any production value. on the basis of your benchmarks it wouldnt be worth pursuing.

You mean the benchmark results are too slow to be valuable in production? That's somewhat irrelevant to a homelab.

It really doesnt. Gluster doesnt have much value under a certain critical hardware deployment, and its only really suitable for file store- which at the "homelab" use case is much more effectively served with a NAS.

Your first point may be true. I don't really have much visibility beyond my own use case and what I see online. I disagree with the second, though. With a proxmox cluster, having a separate NAS is not only inefficient from a power standpoint, it also introduces a single point of failure, which I'm trying to avoid. It may be less common than a simple NAS and nfs, smb/cifs, etc, but I know I'm not the only one looking at hyperconverged homelabs - I got the idea from reading about others who've done it.

I only seen reference to one little announcement, that they might possibly drop support, not that they will or have dropped support. Anyways, it does sound like those interested in glusterfs for proxmox should lobby qemu to remove it from it's list of depreciated features. I assume that would help proxmox to continue to support it if qemu continued to.

On the page of the qemu project dont tell about the release 10 but "in a future release" and they write "Unless the development gains momentum again"
maybe the support can continue after the new release

Yeah, QEMU dropping support was actually what put GlusterFS in the spotlight for things to reconsider for this release.
Maintaining downstream QEMU support is a huge amount of extra work that would need good justification, but we do not see the usage numbers for GlusterFS in our enterprise support evaluations that would justify that effort.

Showing some lightweight development activity after years of slowing down to a crawl is better than nothing, but evaluating the commits it's rather still a bit far away from what we need for enterprise support, that's why the decision to drop built-in support for Proxmox VE 9 will be upheld.

But note that Proxmox VE 8 will be supported for about a year after the final PVE 9.0 release, and GlusterFS will keep working there.
Additionally, you can still mount a GlusterFS storage manually and add it as directory storage to Proxmox VE 9.

Thanks for at least taking a look and considering it.

I have not used glusterfs "recently" (within the last few years), but I have used it in the past, and the results were unstable at best, abysmal at worst, and the management was miserable. We had consistent crashes of the daemons under benchmark loads, but even for backup storage it was finnicky and we found it unreliable and fragile.

The gluster mailing list is utterly dead, the website is dead, and the repo is more or less dead. it's in "critical maintenance fix" mode, which redhat said would happen.

I don't think ceph is unusably slow, but one critical issue that I see repeatedly is that people always set it up with very few OSDs, which is not what it was designed for.

In my homelab (or more accurately, home prod), I get very usable performance out of 24x 500gb 2.5" laptop SATA drives, paired with 8x 500gb SATA SSDs for DB/WAL across 4 hosts - which is not an especially unreasonable number of disks imo, and someone running fewer or less demanding workloads would likely be fine with half as many disks.

The RBD volumes on this ceph cluster back an elasticsearch + graylog cluster, with the bulk of the message volume being netflow from a pair of opnsense router VMs, so the workload is about 6-10x more write than read most of the time. I'm not really using cephfs as RBD volumes are more performant to begin with for backing VMs and containers.