Good morning Proxmox Community!

I'm currently planing to reinstall one node in a working cluster because of a completely new disk-setup. (I'll be swapping all disks of the node.)

As far as I've researched there are two working ways:

1. like documented here: https://pve.proxmox.com/wiki/Upgrade_from_5.x_to_6.0#New_installation
So I would backup all files and reinstall it (without leaving the cluster). The question is: Do I then need to rejoin the cluster, or does the other nodes think the node which was newly installed with all old config-files restored was only shutdown and is now online again?

My second way I thought of is:
Remove the node from the cluster (as documented here: https://pve.proxmox.com/wiki/Cluster_Manager#_remove_a_cluster_node).
Then reinstall the node (With same name & ip-address - so I would need to swap the certificates and the SSH fingerprint after rejoining).

Has anyone done that so far? Is there a better way to do that?

Best regards,
Floh

PS: After reinstalling this one node, I have to reinstall every node one after another.

Hi,

So I would backup all files and reinstall it (without leaving the cluster). The question is: Do I then need to rejoin the cluster, or does the other nodes think the node which was newly installed with all old config-files restored was only shutdown and is now online again?

If you restore the whole /etc/corosync directory including it's authkey and the local corosync.conf the node can directly join, as it never left. Copy the corosync.conf also to /etc/pve and restart both corosync and pve-cluster service in one go: systemctl restart corosync pve-cluster

The configs from /etc/pve are all restored if that works, but other local configs or disks are not.

Additionally it would be good to also restore the root SSH key pair under /root/.ssh/id* as else the node has changed access and migration or the like will give you SSH host indentity changed errors. Alternatively you could run a "pvecm updatecerts" after the node rejoined the cluster.

Has anyone done that so far? Is there a better way to do that?

Both work fine. You could try out with a virtual node or cluster (PVE as PVE VMs) and play your scenario through. This ensures you have a feeling for it and gives a chance to ask about problems, if one arises, while not doing the change in production.

Thanks @t.lamprecht!

I'll try that and will update this thread when I've completed the tasks successfully.

Update:
I reinstalled the Node and restored: /etc/pve/* + /etc/corosync/* + /etc/passwd + /root/.ssh/*
Then I had to update the certs with "pvecm updatecerts" and everything seems working fine.

The only thing is that the Subscription-Key isn't working anymore. Status: Invalid: Invalid Server ID.

Is there a way to restore the subscription key from the old backup-files? (I have a backup from /etc from the node before reinstall)
Or is the only way to reactivate the key at shop.maurer-it.com?

Just to clarify:
(1) backup ~/.ssh, /etc/passwd, /etc/shadow, /etc/pve and /etc/corosync
(2) reinstall without leaving the cluster
(3) install re-activated license key
(4) restore from (1)
(5) reboot

is this correct?

shadow and passwd are normally not required, at least if you did not add extra user you'd like to conserve.
I'd also recommend avoiding doing cluster-wide changes during the reinstallation, or at least ensure that you really only restore the affected node relevant configuration files in /etc/pve (most gets synced anyway after the reboot). IOW. backing up the /etc/pve part is mostly done out of safety if anything goes wrong with rejoining or the like.
So .ssh and corosync are the most important directories to restore.

Ensure also the VM disks ain't on local storage (e.g., local-lvm) which gets destroyed if its disk is selected as target in the ISO installer for reinstallation.

That's a followup error, the journal further up would be more interesting, e.g. the whole output of journalctl -b -u pve-cluster

This is the output of journatctl -b -u pve-cluster (the same block appears over and over, only the counter increases)

Feb 08 12:24:51 qonos systemd[1]: pve-cluster.service: Scheduled restart job, restart counter is at 3.
Feb 08 12:24:51 qonos systemd[1]: Stopped The Proxmox VE cluster filesystem.
Feb 08 12:24:51 qonos systemd[1]: Starting The Proxmox VE cluster filesystem...
Feb 08 12:24:51 qonos pmxcfs[1706]: fuse: mountpoint is not empty
Feb 08 12:24:51 qonos pmxcfs[1706]: fuse: if you are sure this is safe, use the 'nonempty' mount option
Feb 08 12:24:51 qonos pmxcfs[1706]: [main] crit: fuse_mount error: File exists
Feb 08 12:24:51 qonos pmxcfs[1706]: [main] notice: exit proxmox configuration filesystem (-1)
Feb 08 12:24:51 qonos pmxcfs[1706]: [main] crit: fuse_mount error: File exists
Feb 08 12:24:51 qonos pmxcfs[1706]: [main] notice: exit proxmox configuration filesystem (-1)
Feb 08 12:24:51 qonos systemd[1]: pve-cluster.service: Control process exited, code=exited, status=255/EXCEPTION
Feb 08 12:24:51 qonos systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
Feb 08 12:24:51 qonos systemd[1]: Failed to start The Proxmox VE cluster filesystem.
Feb 08 12:24:52 qonos systemd[1]: pve-cluster.service: Scheduled restart job, restart counter is at 4.
Feb 08 12:24:52 qonos systemd[1]: Stopped The Proxmox VE cluster filesystem.
Feb 08 12:24:52 qonos systemd[1]: Starting The Proxmox VE cluster filesystem...
Feb 08 12:24:52 qonos pmxcfs[1708]: fuse: mountpoint is not empty
Feb 08 12:24:52 qonos pmxcfs[1708]: fuse: if you are sure this is safe, use the 'nonempty' mount option
Feb 08 12:24:52 qonos pmxcfs[1708]: [main] crit: fuse_mount error: File exists
Feb 08 12:24:52 qonos pmxcfs[1708]: [main] crit: fuse_mount error: File exists
Feb 08 12:24:52 qonos pmxcfs[1708]: [main] notice: exit proxmox configuration filesystem (-1)
Feb 08 12:24:52 qonos pmxcfs[1708]: [main] notice: exit proxmox configuration filesystem (-1)
Feb 08 12:24:52 qonos systemd[1]: pve-cluster.service: Control process exited, code=exited, status=255/EXCEPTION
Feb 08 12:24:52 qonos systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
Feb 08 12:24:52 qonos systemd[1]: Failed to start The Proxmox VE cluster filesystem.
Feb 08 12:24:52 qonos systemd[1]: pve-cluster.service: Scheduled restart job, restart counter is at 5.
Feb 08 12:24:52 qonos systemd[1]: Stopped The Proxmox VE cluster filesystem.
Feb 08 12:24:52 qonos systemd[1]: pve-cluster.service: Start request repeated too quickly.
Feb 08 12:24:52 qonos systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
Feb 08 12:24:52 qonos systemd[1]: Failed to start The Proxmox VE cluster filesystem.

Feb 08 12:24:51 qonos pmxcfs[1706]: fuse: mountpoint is not empty
Feb 08 12:24:51 qonos pmxcfs[1706]: fuse: if you are sure this is safe, use the 'nonempty' mount option

Obiges ist der Fehler, da wurde wohl das backup nach /etc/pve enpackt als pve-cluster (also pmxcfs) nicht rannte, nun kann es nicht mehr mounten.

mv /etc/pve /etc/pve-OLD
mkdir /etc/pve
chown root:www-data /etc/pve
systemctl restart pve-cluster

Danke, das hat mich deutlich wetergebracht. Allerdings zeigt der Node immer noch ein graues Fragezeichen im Cluster.
Habe ich was übersehen?

Das kann heißen, dass der pvestatd service nicht (mehr) rennt oder sich aufgehängt hat, evtl. mal probieren den zu restarten

systemctl restart pvestatd

Danke, daran hatte ich auch schon gedacht.
Erstaunlicherweise hat er jetzt, nach dem gefühlt fünften Mal, wirklich durchgestartet und den Status aktualisiert.
TAL!
Ich werde das gleich in unsere Doku einfügen, damit das nicht nochmal passiert.

Geht das? Oder wird irgendwo der Key/Fingerprint des alten Servers in Verbindung mit seiner IP gespeichert?

Der public Key des Servers wir zwar in /etc/ssh/ssh_known_hosts was ein Link ins Cluster Filesystem ist (/etc/pve/priv/known_hosts) gespeichert, aber beim joinen einer node wird das Keyfile gemerged, dabei werden die RSA public Keys mit gleichen Hostnamen von der "joining node" immer überschrieben.

Also ja der Ansatz sollte so funktionieren.