I'm really at a loss. Since yesterday, my Proxmox server has crashed, and after a reboot the server is no longer reachable.
I can no longer log in through the console either. The only way I can access a console is by booting from the USB stick (that contains the Proxmox installation files.)
Please see the error message below.
Can anyone help me? The files stored on the server are very important to me, and I'm worried about losing them.
Thank you in advance for any assistance.
----------------------------
Welcome to the Proxmox Virtual Environment. Please use your web browser to
configure this server - connect to:
Then you need something like clonezilla, bootable usb, and clone the corrupt physical drive.
Once you have that drive cloned and the pve folder all backed up.
From that point on you then need to just try any number of routes that will result in getting back to a running system, first off a new drive is needed, keep the clone safe until you are happy that you got everything up and running.
I was installing a truenas at the proxmox, but too late with the backup
It's never too late unless the drive is totally dead. Clone the drive the first chance you get, do NOT disable RO mode, it does that to protect the data, at this point you have no idea what could have caused the ext4 journalling issue, hence cloning it at this stage is paramount, so that you have a point you can keep going back to until you recover.
I've recovered from far worse situations, the main rule is to get a clone before you do anything else!
journalctl -kr
ls -l /dev/mapper/
lsblk -o+FSTYLE,LABEL,MODEL
lsblk -o+FSTYPE,LABEL,MODEL
Please post the requested info.
Were on your Proxmox server are the files you are worried about?
Depending on your setup they may be on another filesystem than the one that is currently troubled.
After backing up you should check the filesystem on /dev/dm-1 when unmounted.
You can do this by booting rescue mode from the installer USB and issue: fsck.ext4 -n /dev/dm-1
This performs a read-only check of the ext4 filesystem without making any changes or repairs (yet).
Got this problem after an update couple of days ago. Spent 12 hours trying to figure out what it was. Downgrading pve-firmware and loading an older kernel worked and everything was accessible again. Just downgrading one of these did not do anything. I could still chroot to pve-root from liveusb with all data and VMs being in mint condition. This had nothing to do with data corruption, the disk was clean.
The filesystem journal had errors because the kernel communicates incorrectly with the Kingston NVME (in my case) and causes this catastrophic failure, later it simply remounts the root file system as read-only and starts spamming journal log with inability to write to the journal. This is a bug.