I reinstalled a few nodes this week-end (Dell R630) from scratch. I didn't bother wiping the disks, thinking (wrongly or not?) the regular Proxmox install would. I mention this because I am not sure if it's related. So I went through the "pvecm delnode" method, etc.

I am seeing a weird error now, not all the time, but when it happens I need to manually reboot the machine, and either immediately or after a few attempts it works.

The install is done using ZFS Mirrored (2 x SSD drives) that both show good health.

(text in red)

../src/boot/boot.c:2576@image_start: Error loading \EFI\proxmox\7.0.2-6-pve\vmlinuz-7.0.2-6-pve: Not found
Boot failed: Integrated Storage Controller 1: Linux Boot Manager

The usually suspects would probably be hardware fault, but this was a reinstall on a previously working machine running the latest 9.1 (I just wanted a more coherent naming scheme).

I have no idea where to start. I think my efibootmgr -v output might be relevant, so there it is:

BootCurrent: 0008
BootOrder: 0008,0007
Boot0000* Hard drive C: VenHw(d6c0639f-c705-4eb9-aa4f-5802d8823de6)feff0000000011000000040000000106780100c902000000a10500c98d0100c90000000000000000000000000000000000000000000000000000000000000000000000000012000000000000000000200002010c00d041030a0000000001010600041103120a000000000000007fff0400530053004400530043003200420042003900360030004700370052000000
      dp: 01 04 14 00 9f 63 c0 d6 05 c7 b9 4e aa 4f 58 02 d8 82 3d e6 / 7f ff 04 00
    data: fe ff 00 00 00 00 11 00 00 00 04 00 00 00 01 06 78 01 00 c9 02 00 00 00 a1 05 00 c9 8d 01 00 c9 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 12 00 00 00 00 00 00 00 00 00 20 00 02 01 0c 00 d0 41 03 0a 00 00 00 00 01 01 06 00 04 11 03 12 0a 00 00 00 00 00 00 00 7f ff 04 00 53 00 53 00 44 00 53 00 43 00 32 00 42 00 42 00 39 00 36 00 30 00 47 00 37 00 52 00 00 00
Boot0001* IBA 40G Slot 0100 v1031       BBS(128,IBA 40G Slot 0100 v1031,0x0)feff01000000000000000000000002009b0000c8800000007d0b00c8dc0000c800000000000000000000000000000000000000000000000000000000000000000000000000110000020000000000001c0002010c00d041030a000000000101060000030101060000007fff04004900420041002000340030004700200053006c006f007400200030003100300030002000760031003000330031000000
      dp: 05 01 20 00 80 00 00 00 49 42 41 20 34 30 47 20 53 6c 6f 74 20 30 31 30 30 20 76 31 30 33 31 00 / 7f ff 04 00
    data: fe ff 01 00 00 00 00 00 00 00 00 00 00 00 02 00 9b 00 00 c8 80 00 00 00 7d 0b 00 c8 dc 00 00 c8 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 11 00 00 02 00 00 00 00 00 00 1c 00 02 01 0c 00 d0 41 03 0a 00 00 00 00 01 01 06 00 00 03 01 01 06 00 00 00 7f ff 04 00 49 00 42 00 41 00 20 00 34 00 30 00 47 00 20 00 53 00 6c 00 6f 00 74 00 20 00 30 00 31 00 30 00 30 00 20 00 76 00 31 00 30 00 33 00 31 00 00 00
Boot0007* Linux Boot Manager    HD(2,GPT,8ac6fa31-12d7-480f-ae36-7ef2a7bec092,0x800,0x200000)/File(\EFI\systemd\systemd-bootx64.efi)
      dp: 04 01 2a 00 02 00 00 00 00 08 00 00 00 00 00 00 00 00 20 00 00 00 00 00 31 fa c6 8a d7 12 0f 48 ae 36 7e f2 a7 be c0 92 02 02 / 04 04 46 00 5c 00 45 00 46 00 49 00 5c 00 73 00 79 00 73 00 74 00 65 00 6d 00 64 00 5c 00 73 00 79 00 73 00 74 00 65 00 6d 00 64 00 2d 00 62 00 6f 00 6f 00 74 00 78 00 36 00 34 00 2e 00 65 00 66 00 69 00 00 00 / 7f ff 04 00
Boot0008* Linux Boot Manager    HD(2,GPT,0f3f42b9-0991-4242-a57a-d7fa9ad45d46,0x800,0x200000)/File(\EFI\systemd\systemd-bootx64.efi)
      dp: 04 01 2a 00 02 00 00 00 00 08 00 00 00 00 00 00 00 00 20 00 00 00 00 00 b9 42 3f 0f 91 09 42 42 a5 7a d7 fa 9a d4 5d 46 02 02 / 04 04 46 00 5c 00 45 00 46 00 49 00 5c 00 73 00 79 00 73 00 74 00 65 00 6d 00 64 00 5c 00 73 00 79 00 73 00 74 00 65 00 6d 00 64 00 2d 00 62 00 6f 00 6f 00 74 00 78 00 36 00 34 00 2e 00 65 00 66 00 69 00 00 00 / 7f ff 04 00

I don't use an R630, but if I did, this is what I would probably do:

1. I'd reset/clear the BIOS & clear all UEFI data + keys & update it to latest for that R630 (2.19.0 ?).
2. Ensure UEFI is selected & SB is disabled.
3. Wipe the disks.
4. Reinstall.

(Since it seems your problem is hit & miss, it may also be that the controller/cables that controls the 2 drives is failing or not seated properly.)

What do your EFI partitions look like?
If it works sporadically maybe you have duplicate IDs and/or one of EFI partitions didn't update correctly.

Can you run:
cat /etc/kernel/proxmox-boot-uuids
lsblk -f |grep vfat

I don't use an R630, but if I did, this is what I would probably do:

1. I'd reset/clear the BIOS & clear all UEFI data + keys & update it to latest for that R630 (2.19.0 ?).
2. Ensure UEFI is selected & SB is disabled.
3. Wipe the disks.
4. Reinstall.

(Since it seems your problem is hit & miss, it may also be that the controller/cables that controls the 2 drives is failing or not seated properly.)

Yes the Dell server has it's bios all updated. I did all except the start from scratch and wipe the disk. I guess I'll do that. Thank you.

What do your EFI partitions look like?
If it works sporadically maybe you have duplicate IDs and/or one of EFI partitions didn't update correctly.

Can you run:
cat /etc/kernel/proxmox-boot-uuids
lsblk -f |grep vfat

8AD9-ED70
8ADA-A25D

lsblk -f | grep pvfat
|-sda2 vfat FAT32 8AD9-ED70
|-sdb2 vfat FAT32 8ADA-A25D

I'm definitely no expert on EFI, but it feels like this should work

please try and post the output of the following commands:

Code:

proxmox-boot-tool status
proxmox-boot-tool refresh

thanks!

I did this already (previously) but no harm in doing it again I guess

8AD9-ED70 is configured with: uefi (versions: 7.0.2-6-pve)
8ADA-A25D is configured with: uefi (versions: 7.0.2-6-pve)

Running hook script 'proxmox-auto-removal'..
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
Copying and configuring kernels on /dev/disk/by-uuid/8AD9-ED70
Copying kernel and creating boot-entry for 7.0.2-6-pve
Copying and configuring kernels on /dev/disk/by-uuid/8ADA-A25D
Copying kernel and creating boot-entry for 7.0.2-6-pve

thanks. could you also add the output of "blkid"?

/dev/sdb2: UUID="8ADA-A25D" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="0f3f42b9-0991-4242-a57a-d7fa9ad45d46"
/dev/sdb3: LABEL="rpool" UUID="6840547563085811721" UUID_SUB="10113094323433538383" BLOCK_SIZE="4096" TYPE="zfs_member" PARTUUID="638c0f3e-4436-4b97-838f-914fd8d6e689"
/dev/sda2: UUID="8AD9-ED70" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="8ac6fa31-12d7-480f-ae36-7ef2a7bec092"
/dev/sda3: LABEL="rpool" UUID="6840547563085811721" UUID_SUB="10043213790492623468" BLOCK_SIZE="4096" TYPE="zfs_member" PARTUUID="767198b2-adf8-4053-86e1-1368f95f1d4a"
/dev/sdb1: PARTUUID="0520ff72-8471-4466-8872-290fd0dd433e"
/dev/sda1: PARTUUID="d9758b30-5514-45e9-b8dd-97e1a28f4c3b"

that looks good.. maybe the issue was temporary? you could also try mounting both ESPs manually and comparing their contents..

In the end, there must have been a weird partition somewhere. I used ShredOS to wipe both disks and, finally, Proxmox installed with no issue.