[SOLVED] - Cluster 2 node + qdevice, different pvecm status

I have a test Cluster with 2 nodes + qdevice.
Everything seems to work, but i got different outputs of "pvecm status" on both nodes, is this normal?

Code:

pve2:~#  pvecm status
Cluster information
-------------------
Name:             PVE-CLUSTER
Config Version:   5
Transport:        knet
Secure auth:      on

Quorum information
------------------
Date:             Tue Jun  2 10:39:48 2026
Quorum provider:  corosync_votequorum
Nodes:            2
Node ID:          0x00000001
Ring ID:          1.39
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   3
Highest expected: 3
Total votes:      2
Quorum:           2 
Flags:            Quorate

Membership information
----------------------
    Nodeid      Votes    Qdevice Name
0x00000001          1         NR 192.168.2.2 (local)
0x00000002          1    A,V,NMW 192.168.2.3
0x00000000          0            Qdevice (votes 1)


--------------------------------------------------------------------


pve3:~#  pvecm status
Cluster information
-------------------
Name:             PVE-CLUSTER
Config Version:   5
Transport:        knet
Secure auth:      on

Quorum information
------------------
Date:             Tue Jun  2 10:40:43 2026
Quorum provider:  corosync_votequorum
Nodes:            2
Node ID:          0x00000002
Ring ID:          1.39
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   3
Highest expected: 3
Total votes:      3
Quorum:           2 
Flags:            Quorate Qdevice

Membership information
----------------------
    Nodeid      Votes    Qdevice Name
0x00000001          1         NR 192.168.2.2
0x00000002          1    A,V,NMW 192.168.2.3 (local)
0x00000000          1            Qdevice

Hi @Martin.B.

thanks for posting in the forum!

In reference to the docs [1] the flag "NR" on pve2 indicates that the Qdevice is not registered with this node and such it only has two votes instead of three.
In this case the cluster would still fail if pve3 fails, since pve2 seems to be unable to communicate with the Qdevice.

How did you install and configure the Qdevice?
Were there any error messages during the install?

Yours sincerely
Jonas

[1] https://pve.proxmox.com/pve-docs/chapter-pvecm.html#pvecm_qdevice_status_flags

For installation i followed the steps in https://pve.proxmox.com/wiki/Cluster_Manager Topic: QDevice-Net Setup

I got no error messages and i am pretty sure i did the " pvecm qdevice setup <QDEVICE-IP>" on node pve2.

The network 192.168.2.0/24 is a separate 1GB network where only the 2 PVE nodes and the qdevice are connected.
Ping from each node to qdevice and vice versa works. Ping latency is below 0.25ms.

Anything i can do to repair this? Can i safely issue the setup command again?

Edit: 1 day after seting this up i simulated an outage and removed power from one of the nodes. HA did work and start the machines of the failed node. But i do not remember which node i shut down.

Last edited: Tuesday at 11:32

Did you apt install corosync-qdevice on both pve2 and pve3?
Anything relevant in the logs / journalctl?

Last edited: Tuesday at 11:46

yes, i did install it on both nodes.

I now did:

Then uninstalled all corosync packages with purge on the qdevice.
Re-installed those packages and did another setup (this time from node pve3). Got no error message. Status is now the same on both nodes:

Code:

pve2:~# pvecm status
Cluster information
-------------------
Name:             PVE-CLUSTER
Config Version:   7
Transport:        knet
Secure auth:      on

Quorum information
------------------
Date:             Tue Jun  2 11:58:52 2026
Quorum provider:  corosync_votequorum
Nodes:            2
Node ID:          0x00000001
Ring ID:          1.39
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   3
Highest expected: 3
Total votes:      2
Quorum:           2
Flags:            Quorate Qdevice

Membership information
----------------------
    Nodeid      Votes    Qdevice Name
0x00000001          1   A,NV,NMW 192.168.2.2 (local)
0x00000002          1   A,NV,NMW 192.168.2.3
0x00000000          0            Qdevice (votes 1)

------------------------------------------------------

pve3:~# pvecm status
Cluster information
-------------------
Name:             PVE-CLUSTER
Config Version:   7
Transport:        knet
Secure auth:      on

Quorum information
------------------
Date:             Tue Jun  2 12:41:11 2026
Quorum provider:  corosync_votequorum
Nodes:            2
Node ID:          0x00000002
Ring ID:          1.39
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   3
Highest expected: 3
Total votes:      2
Quorum:           2
Flags:            Quorate Qdevice

Membership information
----------------------
    Nodeid      Votes    Qdevice Name
0x00000001          1   A,NV,NMW 192.168.2.2
0x00000002          1   A,NV,NMW 192.168.2.3 (local)
0x00000000          0            Qdevice (votes 1)

So: i do not know why this issue occured, but it looks better ( to me) now.

Edit: Only thing that confuses me a bit is the expected votes is 3 but the total votes are only 2.

Last edited: Tuesday at 12:46

Last edited: Tuesday at 13:23

corosync.conf:

Code:

pve3:~# cat /etc/pve/corosync.conf
logging {
  debug: off
  to_syslog: yes
}

nodelist {
  node {
    name: cdppve2
    nodeid: 1
    quorum_votes: 1
    ring0_addr: 192.168.2.2
    ring1_addr: 172.16.5.95
  }
  node {
    name: cdppve3
    nodeid: 2
    quorum_votes: 1
    ring0_addr: 192.168.2.3
    ring1_addr: 172.16.5.96
  }
}

quorum {
  device {
    model: net
    net {
      algorithm: ffsplit
      host: 192.168.2.10
      tls: on
    }
    votes: 1
  }
  provider: corosync_votequorum
}

totem {
  cluster_name: PVE-CLUSTER
  config_version: 7
  interface {
    linknumber: 0
  }
  interface {
    linknumber: 1
  }
  ip_version: ipv4-6
  link_mode: passive
  secauth: on
  token_coefficient: 125
  version: 2
}

----------------------------------------------------

pve2:~# cat /etc/pve/corosync.conf
logging {
  debug: off
  to_syslog: yes
}

nodelist {
  node {
    name: cdppve2
    nodeid: 1
    quorum_votes: 1
    ring0_addr: 192.168.2.2
    ring1_addr: 172.16.5.95
  }
  node {
    name: cdppve3
    nodeid: 2
    quorum_votes: 1
    ring0_addr: 192.168.2.3
    ring1_addr: 172.16.5.96
  }
}

quorum {
  device {
    model: net
    net {
      algorithm: ffsplit
      host: 192.168.2.10
      tls: on
    }
    votes: 1
  }
  provider: corosync_votequorum
}

totem {
  cluster_name: PVE-CLUSTER
  config_version: 7
  interface {
    linknumber: 0
  }
  interface {
    linknumber: 1
  }
  ip_version: ipv4-6
  link_mode: passive
  secauth: on
  token_coefficient: 125
  version: 2
}

looks pretty much the same to me.
Well, journalctl says there is a problem with some certificate:

Code:

Jun 02 13:17:56 cdppve2 corosync-qdevice[1986900]: Server certificate verification failure. (-8182): Peer's certificate has an invalid signature.

Jun 02 13:18:38 cdppve3 corosync-qdevice[1941324]: Server certificate verification failure. (-8182): Peer's certificate has an invalid signature.

Both nodes have certificates from our internal CA installed. This was done after creatiing the cluster, as certificates are lost during cluster creation. Those certificates contain all dns names but only the management IP as alternate names.
The qdevice did not get any certificate from us, so i assume it is using a self signed one.

Edit: `pvecm updatecerts --force` did not change anything

Last edited: Tuesday at 13:31

reading a few more threads to crosync with qdevice and removing/setting it up some more times, it is still not voting.

collected some more information. so on the pve nodes i have the follwing packages installed:

Code:

pve3:~# dpkg -l | grep corosy
ii  corosync                             3.1.10-pve2                          amd64        cluster engine daemon and utilities
ii  corosync-qdevice                     3.0.3-2                              amd64        cluster engine quorum device daemon
ii  libcorosync-common4:amd64            3.1.10-pve2                          amd64        cluster engine common library

on the qdevice:

Code:

pveq:~# dpkg -l | grep corosy
ii  corosync                               3.1.6-1ubuntu1.2                                 amd64        cluster engine daemon and utilities
ii  corosync-qnetd                         3.0.1-1                                          amd64        cluster engine quorum device network daemon
ii  libcorosync-common4:amd64              3.1.6-1ubuntu1.2                                 amd64        cluster engine common library

output of the pvecm qdevice setup does not contain any warning, error or failure.
Status of qdevice service (on both nodes):

Code:

pve3:~# systemctl status corosync-qdevice.service
● corosync-qdevice.service - Corosync Qdevice daemon
     Loaded: loaded (/usr/lib/systemd/system/corosync-qdevice.service; enabled; preset: enabled)
     Active: active (running) since Tue 2026-06-02 13:53:44 CEST; 7min ago
 Invocation: f2f00a9bd87749ba88fbaba48f849c90
       Docs: man:corosync-qdevice
   Main PID: 1994592 (corosync-qdevic)
      Tasks: 2 (limit: 34840)
     Memory: 2.8M (peak: 3.1M)
        CPU: 82ms
     CGroup: /system.slice/corosync-qdevice.service
             ├─1994592 /usr/sbin/corosync-qdevice -f
             └─1994594 /usr/sbin/corosync-qdevice -f

Jun 02 14:00:24 cdppve3 corosync-qdevice[1994592]: Connect timeout
Jun 02 14:00:24 cdppve3 corosync-qdevice[1994592]: Can't connect to qnetd host. (-5986): Network addres>
Jun 02 14:00:32 cdppve3 corosync-qdevice[1994592]: Connect timeout
Jun 02 14:00:32 cdppve3 corosync-qdevice[1994592]: Can't connect to qnetd host. (-5986): Network addres>

ssh from both pve nodes to qdevice is working (without password).
ping from/to qdevice and pve is working.

qdevice is an ubuntu 22.04 LTS on a normal PC hardware. running nothing more than caorosync-qdevice.
It is connected to the corosync network (192.168.2.0/24) and the management network (172.16.0.0/16).

setup of qdevice:

Code:

pve3:~# pvecm qdevice setup 192.168.2.10
/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed

/bin/ssh-copy-id: WARNING: All keys were skipped because they already exist on the remote system.
                (if you think this is a mistake, you may want to use -f option)


INFO: initializing qnetd server
Certificate database (/etc/corosync/qnetd/nssdb) already exists. Delete it to initialize new db

INFO: copying CA cert and initializing on all nodes

node 'pve2': Creating /etc/corosync/qdevice/net/nssdb
password file contains no data
node 'pve2': Creating new key and cert db
node 'pve2': Creating new noise file /etc/corosync/qdevice/net/nssdb/noise.txt
node 'pve2': Importing CA
node 'pve3': Creating /etc/corosync/qdevice/net/nssdb
password file contains no data
node 'pve3': Creating new key and cert db
node 'pve3': Creating new noise file /etc/corosync/qdevice/net/nssdb/noise.txt
node 'pve3': Importing CA
INFO: generating cert request
Creating new certificate request


Generating key.  This may take a few moments...

Certificate request stored in /etc/corosync/qdevice/net/nssdb/qdevice-net-node.crq

INFO: copying exported cert request to qnetd server

INFO: sign and export cluster cert
Signing cluster certificate
Certificate stored in /etc/corosync/qnetd/nssdb/cluster-PVE-CLUSTER.crt

INFO: copy exported CRT

INFO: import certificate
Importing signed cluster certificate
Notice: Trust flag u is set automatically if the private key is present.
pk12util: PKCS12 EXPORT SUCCESSFUL
Certificate stored in /etc/corosync/qdevice/net/nssdb/qdevice-net-node.p12

INFO: copy and import pk12 cert to all nodes

node 'pve2': Importing cluster certificate and key
node 'pve2': pk12util: PKCS12 IMPORT SUCCESSFUL
node 'pve3': Importing cluster certificate and key
node 'pve3': pk12util: PKCS12 IMPORT SUCCESSFUL
INFO: add QDevice to cluster configuration

INFO: start and enable corosync qdevice daemon on node 'pve2'...
Synchronizing state of corosync-qdevice.service with SysV service script with /usr/lib/systemd/systemd-sysv-install.
Executing: /usr/lib/systemd/systemd-sysv-install enable corosync-qdevice
Created symlink '/etc/systemd/system/multi-user.target.wants/corosync-qdevice.service' -> '/usr/lib/systemd/system/corosync-qdevice.service'.

INFO: start and enable corosync qdevice daemon on node 'pve3'...
Synchronizing state of corosync-qdevice.service with SysV service script with /usr/lib/systemd/systemd-sysv-install.
Executing: /usr/lib/systemd/systemd-sysv-install enable corosync-qdevice
Created symlink '/etc/systemd/system/multi-user.target.wants/corosync-qdevice.service' -> '/usr/lib/systemd/system/corosync-qdevice.service'.
Reloading corosync.conf...
Done

even though ufw is not configured, i completly disabled the ufw deamon on qdevice. But no luck...
getting short of further ideas.

corosync-qnetd service is running (without problems) on 192.168.2.10?
Can you telnet 192.168.2.10 5403 from the nodes?

Last edited: Tuesday at 15:16

There is no corosync-qdevice service, only corosync-qnetd
But looks like this is not running.

Code:

pveq:/etc/corosync/qnetd# systemctl status corosync-qnetd.service
× corosync-qnetd.service - Corosync Qdevice Network daemon
     Loaded: loaded (/lib/systemd/system/corosync-qnetd.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Tue 2026-06-02 13:17:07 UTC; 2s ago
       Docs: man:corosync-qnetd
    Process: 7676 ExecStart=/usr/bin/corosync-qnetd -f $COROSYNC_QNETD_OPTIONS (code=exited, status=1/F>
   Main PID: 7676 (code=exited, status=1/FAILURE)
        CPU: 5ms

Jun 02 13:17:07 pveq systemd[1]: Starting Corosync Qdevice Network daemon...
Jun 02 13:17:07 pveq corosync-qnetd[7676]: Can't open NSS DB directory (13): Permission denied
Jun 02 13:17:07 pveq systemd[1]: corosync-qnetd.service: Main process exited, code=exited, status=1/>
Jun 02 13:17:07 pveq systemd[1]: corosync-qnetd.service: Failed with result 'exit-code'.
Jun 02 13:17:07 pveq systemd[1]: Failed to start Corosync Qdevice Network daemon.

if i am not wrong, this is the nssdb dir:

Code:

~# ll /etc/corosync/qnetd/
total 12
drwxr-xr-x 3 root root 4096 Jun  2 09:54 ./
drwxr-xr-x 4 root root 4096 Jun  2 13:10 ../
drwxrwx--- 2 root root 4096 Jun  2 09:56 nssdb/

~# ll /etc/corosync/qnetd/nssdb/
total 112
drwxrwx--- 2 root root  4096 Jun  2 09:56 ./
drwxr-xr-x 3 root root  4096 Jun  2 09:54 ../
-rw-rw---- 1 root root 28672 Jun  2 09:54 cert9.db
-rw-r----- 1 root root   679 Jun  2 12:49 cluster-PVE-CLUSTER.crt
-rw-rw---- 1 root root 53248 Jun  2 09:54 key4.db
-rw-rw---- 1 root root    41 Jun  2 09:54 noise.txt
-rw-rw---- 1 root root   432 Jun  2 09:54 pkcs11.txt
-rw-rw---- 1 root root     0 Jun  2 09:54 pwdfile.txt
-rw-r--r-- 1 root root  4272 Jun  2 09:54 qnetd-cacert.crt
-rw-rw---- 1 root root     4 Jun  2 12:49 serial.txt

Last edited: Tuesday at 15:22

Code:

chown root:coroqnetd /etc/corosync/nssdb/
chown root:coroqnetd /etc/corosync/nssdb/cert9.db
chown root:coroqnetd /etc/corosync/nssdb/key4.db
chmod 750 /etc/corosync/nssdb/
chmod 640 /etc/corosync/nssdb/*.db /etc/corosync/nssdb/*.txt
systemctl restart corosync-qnetd

Last edited: Tuesday at 15:31

Looks like i got it working, even though, i do not know how.
This is what i did (again):
on pve:
pvecm qdevice remove

on qdevice:
apt purge corosyc corosync-qnetd
check that /etc/corosync is gone
apt install corosync-qnetd
check that /etc/corosync is back
check that corosync-qnetd is running

on pve:
pvecm qdevice setup 192.168.2.10
got same messages as last time!
checked corosync-qdevice.service is running

Now i have the following status:

Code:

pve3:~# pvecm status
Cluster information
-------------------
Name:             PVE-CLUSTER
Config Version:   13
Transport:        knet
Secure auth:      on

Quorum information
------------------
Date:             Tue Jun  2 15:26:19 2026
Quorum provider:  corosync_votequorum
Nodes:            2
Node ID:          0x00000002
Ring ID:          1.4b
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   3
Highest expected: 3
Total votes:      3
Quorum:           2
Flags:            Quorate Qdevice

Membership information
----------------------
    Nodeid      Votes    Qdevice Name
0x00000001          1    A,V,NMW 192.168.2.2
0x00000002          1    A,V,NMW 192.168.2.3 (local)
0x00000000          1            Qdevice

Thanks all for your help and input. As i said, i did this multiple times , but it did not work until now.

Last edited: Tuesday at 15:46

推荐订阅源

Proxmox Support Forum