How a Kubernetes worker VM with a passed-through AMD GPU sent my entire home lab into an infinite crash loop — and the GRUB-level trick that saved it.

What I Was Trying to Do

My home lab runs on a Mini PC with Proxmox as the hypervisor. The setup hosts a mix of LXC containers and KVM virtual machines — Jellyfin for media, a Pi-hole, a Kubernetes cluster (k3s), Cloudflare tunnels, and a bunch of other self-hosted services.

I was upgrading the hardware configuration of VM 104 (k3s-vm-worker) — one of the worker nodes in my k3s Kubernetes cluster. The goal was straightforward: pass through the host's AMD GPU directly into the VM so the Kubernetes worker could handle GPU-accelerated workloads.

In Proxmox, GPU passthrough (PCIe passthrough) requires a few things:

• IOMMU enabled in BIOS (AMD-Vi for AMD platforms)
• amd_iommu=on iommu=pt kernel parameters in GRUB
• Machine type set to q35 (required for PCIe passthrough)
• The PCI device was added as hostpci0 in the VM's hardware config

I configured all of this. The hardware tab showed the PCI device (0000:e6:00, highlighted in orange — a warning sign I probably should have paid more attention to). I saved the config, felt good about it, and rebooted.

That was my mistake.

What Happened — The Crash Loop

The Mini PC came back up, Proxmox started loading, and then — nothing. The host became completely unresponsive. No web UI. No SSH. Nothing.

I power-cycled it. Same thing. It would power on, start booting, and crash before I could even get to the Proxmox web interface.

Here's what was actually happening, which I only understood after I could finally get in and check the logs:

VM 104 had "Start at boot" set to Yes.

The moment Proxmox finished loading, it triggered a "Bulk Start VMs" task — its automatic process for starting all VMs flagged to auto-start. VM 104 was in that queue. The second that task reached VM 104, Proxmox tried to initialise the PCIe passthrough, the VM grabbed the GPU from the host, and the entire system crashed — hard.

I panicked and quickly got back into the web UI during a narrow window and changed "Start at boot" to No. Thought that would fix it.

It didn't.

Proxmox had already queued the bulk start task the moment it turned on. Changing the setting mid-flight did nothing. The next boot, same crash. The boot after that, same crash.

I was locked in an auto-start crash loop with no way out through the web UI.

Understanding the Problem — Why It Was So Hard to Escape

This is the part that makes this specific failure mode so nasty:

Boot → Proxmox loads → "Bulk Start VMs" fires instantly →
VM 104 grabs GPU → Host kernel panics → System crashes →
Reboot → repeat forever

The window between "Proxmox is up enough to serve the web UI" and "VM 104 has already been queued and grabbed the GPU" was too small to intervene through normal means. Every time I managed to get in and change something, the crash had already been scheduled.

The only way out was to break the loop before Proxmox had any chance to load its VM management services at all — which means intervening at the GRUB level, before the operating system even finishes booting.

The Solution — Masking the VM Start Service at Boot

This required plugging a physical keyboard and monitor into the Mini PC. No SSH, no remote — physical access only.

Step 1: Stop the autoboot timer at GRUB

Power on the machine and watch the screen. The moment the blue Proxmox GRUB menu appears, press the Down Arrow key immediately. This stops the autoboot countdown timer and lets you stay at the GRUB menu.

Step 2: Edit the boot command

Highlight the top "Proxmox VE" entry and press e to edit the boot command. You'll see the full kernel command line. Navigate to the end of the linux line.

Remove amd_iommu=on iommu=pt (these were the IOMMU parameters I'd added earlier that enabled the passthrough).

Then, at the very end of the linux line, add:

systemd.mask=pve-guests.service

So the line ends with:

ro quiet systemd.mask=pve-guests.service

Press Ctrl + X to boot with this modified command.

What systemd.mask=pve-guests.service actually does

pve-guests.service is the systemd unit responsible for the "Bulk Start VMs" behaviour in Proxmox. Masking it via the kernel command line at boot tells systemd to completely skip that service for this boot session only — it doesn't persist across reboots.

The result:

• ✅ Proxmox boots normally
• ✅ Web UI comes up
• ✅ Network is available
• ✅ Zero VMs start automatically
• ✅ Host does not crash

Step 3: Fix the VM config via web UI

Once safely in the web UI with no VMs running:

1. VM 104 → Options → "Start at boot" → No
2. VM 104 → Hardware → PCI Device (hostpci0) → Remove
3. VM 104 → Hardware → Machine → change from q35 back to i440fx
4. Reboot normally — don't touch the GRUB menu this time

Plot Twist — The GPU Renamed Itself

After fixing the crash loop and getting the host stable, I started my LXC containers — CT 204 (jellyfin-arr) and CT 208 (linkwarden) — and hit a completely different error:

Error: Device /dev/dri/card0 ...

Both containers refused to start. I checked the DRI devices on the host:

ls -la /dev/dri/

Output:

crw-rw---- 1 root video  226,   1 May 13 13:51 card1
crw-rw---- 1 root render 226, 128 May 13 13:51 renderD128

The AMD GPU loaded perfectly (Kernel driver in use: amdgpu). But Linux had decided to name it card1 instead of card0.

This is a surprisingly common Linux behaviour: when you change BIOS IOMMU settings, or reboot without an HDMI cable connected to a GPU, the kernel can enumerate DRM devices in a different order and assign a different cardN number. The GPU is fine. The driver is fine. The device node just got a different name.

My LXC containers were hardcoded to pass through /dev/dri/card0. The fix was trivial once I understood it:

1. CT 204 → Resources → Device (dev1) — change /dev/dri/card0 to /dev/dri/card1
2. CT 208 → Resources → Device (dev1) — same change

Both containers started instantly. Hardware transcoding in Jellyfin came back up immediately.

Root Cause Analysis

Looking back, there were actually three separate issues stacked on top of each other:

The orange 0000:e6:00 text in the VM Hardware tab was the visual warning I missed — in Proxmox, orange on a PCI device entry typically indicates the device may not be properly isolated in its IOMMU group, or that there's a configuration issue worth investigating before saving and booting.

Lessons Learned

1. Never enable "Start at boot" on a VM with PCIe passthrough until you've verified the passthrough works correctly in a manual start first.

Test the VM with a manual start. Confirm the host stays stable. Confirm the VM boots correctly with the passed-through device. Only then enable auto-start.

2. systemd.mask=<service> at GRUB is your emergency brake.

Any time Proxmox is in a state where it crashes before you can intervene through the web UI, this technique gives you a clean boot with surgical control over what services start. pve-guests.service is the specific one to mask for VM auto-start loops.

3. GPU device node names are not stable across reboots in Linux.

/dev/dri/card0 is not guaranteed to be your GPU after every reboot, especially when BIOS settings change. Instead of hardcoding card0, consider:

• Using udev rules to create a stable symlink to your specific GPU by PCI ID
• Or checking ls /dev/dri/ after any BIOS/IOMMU change before starting dependent containers

# Find which card corresponds to your GPU by PCI ID
ls -la /dev/dri/by-path/
# or
lspci -k | grep -A 3 VGA

4. PCIe passthrough machine type matters — and q35 has implications.

q35 is required for proper PCIe passthrough but behaves differently from i440fx in several ways. Switching machine types mid-configuration without a working baseline is risky. Start with a known-good VM on i440fx, verify it's stable, then migrate to q35 and add the PCI device.

5. Physical access to your home lab server is non-negotiable.

If you're running a home lab with GPU passthrough or any kind of hardware-level config, have a keyboard and monitor you can plug in. SSH and the web UI are great until they're not. The GRUB console saved this entire setup.

The Final State

After all of this:

• ✅ Proxmox host boots cleanly and stays stable
• ✅ All LXC containers (jellyfin-arr, linkwarden) running with GPU access on card1
• ✅ k3s Kubernetes cluster (VMs 103, 104, 105) running normally
• ✅ All other services (Plex, Pi-hole, Cloudflare, Pulse, Prowlarr, Pendela) unaffected
• ⚠️ GPU passthrough into k3s-vm-worker: parked for now, to be re-approached correctly

The GPU passthrough into the Kubernetes worker is still on the roadmap — I'll approach it differently: test in a throwaway VM first, verify IOMMU grouping, confirm stability with manual start, and only then wire it into the k3s node.

Quick Reference — Commands Used

# Check GPU device assignment on host
ls -la /dev/dri/
lspci -k | grep -A 3 -i vga

# Check IOMMU groups (run as root on Proxmox host)
for d in /sys/kernel/iommu_groups/*/devices/*; do
  n=${d#*/iommu_groups/*}; n=${n%%/*}
  printf 'IOMMU Group %s ' "$n"
  lspci -nns "${d##*/}"
done

# GRUB emergency boot (type this at the end of linux line in GRUB edit)
systemd.mask=pve-guests.service

# Find stable GPU device path by PCI address
ls -la /dev/dri/by-path/

Running a home lab means breaking things and learning from them. This one taught me more about Proxmox internals, systemd masking, and Linux DRM device enumeration than any documentation would have. Hope it saves someone else the same panic.

Feel free to drop questions in the comments — happy to help if you're stuck in a similar loop.