ceph / cephfs feature level mismatch

I experience the same issue mentioned here:

Hello,

I am trying to put the balancer mode into upmap-read which requires to run: ceph osd set-require-min-compat-client reef
I can't do so because I have a single client running on luminous compat version:

Code:

# ceph features
{
    "mon": [
        {
            "features": "0x3f03cffffffdffff",
            "release": "squid",
            "num": 1
        }
    ],
    "osd": [
        {
            "features": "0x3f03cffffffdffff",
            "release": "squid",
            "num": 8
        }
    ],
    "client": [
        {
            "features": "0x2f018fb87aa4aafe"...

and here:

I have installed cpeh 19.2 squid release and would like to use some new features.
Best I can tell erasure coding has 2 new features that look useful to me.
crush-num-failure-domains=
crush-osds-per-failure-domain=

I can't currently find how I learned about these but I think it may do some sort of custom crushmap rule when used.
When I used it to create an erasure coded pool
ceph osd pool create newpool erasure newprofile
Error EINVAL: new crush map requires client version squid but require_min_compat_client is luminous
I set
ceph osd set-require-min-compat-client squid...

(both of which seem to have remained unanswered)

the kernel-driver for cephfs seems to be using a feature level of luminous while the rest of ceph is at squid release.
Running ceph features without cephfs mounted:

JSON:

{
    "mon": [
        {
            "features": "0x3f03cffffffdffff",
            "release": "squid",
            "num": 5
        }
    ],
    "mds": [
        {
            "features": "0x3f03cffffffdffff",
            "release": "squid",
            "num": 5
        }
    ],
    "osd": [
        {
            "features": "0x3f03cffffffdffff",
            "release": "squid",
            "num": 47
        }
    ],
    "client": [
        {
            "features": "0x3f03cffffffdffff",
            "release": "squid",
            "num": 297
        }
    ],
    "mgr": [
        {
            "features": "0x3f03cffffffdffff",
            "release": "squid",
            "num": 5
        }
    ]
}

and with cephfs mounted:

JSON:

{
    "mon": [
        {
            "features": "0x3f03cffffffdffff",
            "release": "squid",
            "num": 5
        }
    ],
    "mds": [
        {
            "features": "0x3f03cffffffdffff",
            "release": "squid",
            "num": 5
        }
    ],
    "osd": [
        {
            "features": "0x3f03cffffffdffff",
            "release": "squid",
            "num": 47
        }
    ],
    "client": [
        {
            "features": "0x2f018fb87aa4aafe",
            "release": "luminous",
            "num": 1
        },
        {
            "features": "0x3f03cffffffdffff",
            "release": "squid",
            "num": 297
        }
    ],
    "mgr": [
        {
            "features": "0x3f03cffffffdffff",
            "release": "squid",
            "num": 5
        }
    ]
}

Since I would like to make use of squid features (read balancer), I wonder if it is at all possible to use cephfs (which I use to provide ISO / templates across the cluster) and the new features at the same time? At the very least, the documentation might benefit from a note that using cephfs will (as of now) prevent use of some current ceph features.

Last edited: Apr 29, 2026

the documentation might benefit from a note that using cephfs will (as of now) prevent use of some current ceph features.

agreed. the kernel cephfs does not advertise the reef features. because the pg-upmap-primary needs the cooperation of client side, one cannot just set it by forcing it. otherwise a kernel client that hits pg-upmap-primary entries routes reads to the wrong OSD silently, and we'd expect I/O hangs on the affected PGs.

probably you could use the userspace ceph-fuse client? something like

Code:

pvesm set <your-cephfs-storatge> --fuse 1

then unmount the kernel mount. remount with the fuse.

the documentation might benefit from a note that using cephfs will (as of now) prevent use of some current ceph features.

indeed, probably you could check out the upstream doc at https://docs.ceph.com/en/latest/rados/operations/require-min-compat-client/kernel ?

Thank you for the suggestion. I was not aware of the fuse variant. That works and resolves my problem!
Where did you get the link to the upstream doc? It points to an error page.
This still works, though:
https://docs.ceph.com/en/latest/rados/operations/require-min-compat-client
The hint about the kernel client not supporting pg-upmap-primary can be found in the warning at the bottom.

glad it helps =)

i copied it from my browser. i just managed to paste another path when pasting it to the forum. =(

推荐订阅源

Proxmox Support Forum