Hello Arch community,

I am getting freezes/DE crashes where the system becomes unresponsive. Audio and video might keep playing for a moment or so, but shortly after the freeze, the screen will go black. Sometimes my cursor is still responsive, but if I attempt to switch to another tty by CTRL+ALT+F6 then the system will be entirely frozen.  I have the magic sysrq keys enabled, so at that point I usually do an ALT + SYSRQ+r,e,i,s,u,b

Alas, I have not been successful in collecting logs from these crashes. I would appreciate guidance on resolving this issue. I have gathered the relevant information below in a way that I hope is useful. I am trying to determine if it is a filesystem corruption, hardware/kernel issue, or something else.

Timeline / wall of text

I had a previous desktop computer on the Fedora KDE spin (and then briefly CachyOS) which this current machine has replaced. It had occasional stability issues and, at the time, I chalked most of it up to the Nvidia 1080ti card. The only hardware carried over to the new system is my monitor, keyboard, mouse, and usb/wireless headset.

On the new desktop machine, mistakes were made, and I got some pretty bad file corruption. I think this was due to an unsafe shutdown after a freeze during a package install. It was bad enough that I reformatted the drive, and started over. On the second install, I made sure to enable the magic sysrq keys in the kernel parameters.

Things went well until I was trying to build a container with  Podman to run a Mindwtr sync service. I was getting crashes which, at the time, I thought were related to podman. I eventually gave up on Podman and built the container and ran it in Docker. Shortly afterwards, when I joined some friends for a movie night via Discord, I ran into another freeze. I was up using the AUR Vesktop package instead of the official Discord client, and it worked great, until everything froze about 30 minutes into the movie.

I don't remember exactly what I did afterwards, but the problem got worse. GNOME was freezing immediately after login. I tried restoring to some older btrfs snapshots, but it did not help. I was able to switch to tty from gdm instead of starting a gnome session. and disable the Docker service, but that did not fix the problem. I then uninstalled all of my GNOME packages, and related packages, and installed KDE Plasma. Since then, the system has been stable unless I have some media playing. e.g.after 10 to 60 minutes with youtube playing in Firefox, it will freeze as described above.

I took an embarrassingly long time typing this up, all without any chill beats to relax/chill to. And it has not crashed during this time. It really seems correlated with playing videos or perhaps having my wireless headset powered on and connected.

Troubleshooting attempts

Ran memtest86 and an nvme test from the motherboard's UEFI settings. No errors detected.

I booted to my live archiso USB and ran

# btrfs check --readonly /dev/nvme0n1

It did find some issues. I will have to boot to that environment again to get those specific error logs. btrfs scrub, on the other hand, detected no issues. (I will edit this post shortly once I get that output saved)
Edit: Darn, after this last boot to get the btrfs check output, KDE Plasma is also crashing right after login, just like GNOME was.

I wanted to get journal logs from when these freezes happen, to see if it is indeed a filesystem issue, or if it is something else. So, I tried a couple of AUR kdump solutions from the wiki. kdumpst didn't work, so I tried simple-kdump, and I was able to generate crash logs from a test crash via

# sync; echo c > /proc/sysrq-trigger

Unfortunately, today when it froze, I did not get a kdump log. After it was frozen at the black screen for a while, I attempted ALT+SYSRQ+r,s,c but It did not automatically reboot into the crashkernel, or if it did, something went wrong, as all I had was an unresponsive system, so I held down the power button to power it off and then booted it back up.

System Info

inxi -ez

simple-kdump

$ cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-linux root=UUID=acc44320-1a27-4adb-a8ec-80c1071e995f rw rootflags=subvol=@ crashkernel=512M loglevel=3 sysrq_always_enabled=1 quiet splash

$ cat /etc/conf.d/simple-kdump.conf 
# Kernel and initramfs for the kexec environment.
# Recommended to use the linux or linux-lts kernel with 'default' preset.
KERNEL=/boot/vmlinuz-linux
INITRAMFS=/boot/initramfs-linux.img

# No crashkernel= option for the kexec kernel cmdline.
# Just regular boot options, the extra needed ones will be added by
# simple-kdump-setup service automatically.
BOOT_OPTIONS=root=UUID=acc44320-1a27-4adb-a8ec-80c1071e995f rw rootflags=subvol=@ loglevel=5 video=2560x1440@59.94e

Logs

journalctl -b -1 -n 400 -W
journalctl -b -1 --dmesg -n 400 -W
cat /var/log/pacman.log

EDIT2:

I tried to pipe the output from btrfs check --readonly /dev/nvme0n1p2 to a file, but it didn't give all the output

Opening filesystem to check...
Checking filesystem on /dev/nvme0n1p2
UUID: acc44320-1a27-4adb-a8ec-80c1071e995f
found 205445181440 bytes used, error(s) found
total csum bytes: 188301852
total tree bytes: 3488694272
total fs tree bytes: 3162406912
total extent tree bytes: 124125184
btree space waste bytes: 506990402
file data blocks allocated: 490399457280
 referenced 349471985664

I feel like my fstab may be relevant too:

# Static information about the filesystems.
# See fstab(5) for details.

# <file system> <dir> <type> <options> <dump> <pass>
# /dev/nvme0n1p2 LABEL=ArchLinuxFS
UUID=af67c079-2f5a-49f5-b4b0-309cf5357017	/         	btrfs     	rw,relatime,ssd,discard=async,space_cache=v2,subvol=/@	0 0

# /dev/nvme0n1p1
UUID=1DD0-54C5      				/boot     	vfat      	rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro	0 2

UUID=af67c079-2f5a-49f5-b4b0-309cf5357017	/home         	btrfs     	rw,relatime,ssd,discard=async,space_cache=v2,subvol=/@home	0 0

UUID=af67c079-2f5a-49f5-b4b0-309cf5357017	/swap        	btrfs     	rw,relatime,ssd,discard=async,space_cache=v2,subvol=/@swap	0 0


/swap/swapfile					none      	swap      	defaults 	0 0

I am starting to think maybe btrfs was a bad choice for me. But it works so nicely on my laptop. I have Arch on there and it has been a good time. I am at least able to get to a tty, so I am willing to try to find out what went wrong instead of wiping and starting over again. If you are reading all my nonsense here, let me know if you have any constructive thoughts

Last edited by SamuraiGelato (Today 01:10:12)