Hello.
I need some help with the above mentioned issue. I have a simple Proxmox installation on a single Mini PC. Yesterday, I spotted some unusual problems, like not getting into the web UI. After several reboots, the system seems to work again and is accessible. But there are still some issues. I guess, the cause is "no space left on device", as the title says. The problem might already exist for several days. I noticed, that the last automatic backup is five days ago.
Ok. Jumping into the problem: When trying to do a simple apt update, I get the following output:
Code:
root@pve:/# apt update
Hit:1 http://deb.debian.org/debian trixie InRelease
Get:2 http://security.debian.org/debian-security trixie-security InRelease [43.4 kB]
Ign:2 http://security.debian.org/debian-security trixie-security InRelease
Get:3 http://download.proxmox.com/debian/pve trixie InRelease [3,534 B]
Ign:3 http://download.proxmox.com/debian/pve trixie InRelease
Get:4 http://deb.debian.org/debian trixie-updates InRelease [47.3 kB]
Ign:4 http://deb.debian.org/debian trixie-updates InRelease
Get:2 http://security.debian.org/debian-security trixie-security InRelease [43.4 kB]
Ign:2 http://security.debian.org/debian-security trixie-security InRelease
Get:4 http://deb.debian.org/debian trixie-updates InRelease [47.3 kB]
Ign:4 http://deb.debian.org/debian trixie-updates InRelease
Get:3 http://download.proxmox.com/debian/pve trixie InRelease [3,534 B]
Ign:3 http://download.proxmox.com/debian/pve trixie InRelease
Get:2 http://security.debian.org/debian-security trixie-security InRelease [43.4 kB]
Ign:2 http://security.debian.org/debian-security trixie-security InRelease
Get:3 http://download.proxmox.com/debian/pve trixie InRelease [3,534 B]
Ign:3 http://download.proxmox.com/debian/pve trixie InRelease
Get:4 http://deb.debian.org/debian trixie-updates InRelease [47.3 kB]
Ign:4 http://deb.debian.org/debian trixie-updates InRelease
Get:2 http://security.debian.org/debian-security trixie-security InRelease [43.4 kB]
Err:2 http://security.debian.org/debian-security trixie-security InRelease
Error writing to file - write (28: No space left on device) [IP: 146.75.122.132 80]
Get:3 http://download.proxmox.com/debian/pve trixie InRelease [3,534 B]
Err:3 http://download.proxmox.com/debian/pve trixie InRelease
Error writing to file - write (28: No space left on device) [IP: 185.219.221.167 80]
Get:4 http://deb.debian.org/debian trixie-updates InRelease [47.3 kB]
Err:4 http://deb.debian.org/debian trixie-updates InRelease
Error writing to file - write (28: No space left on device) [IP: 146.75.118.132 80]
118 packages can be upgraded. Run 'apt list --upgradable' to see them.
Warning: Failed to fetch http://deb.debian.org/debian/dists/trixie-updates/InRelease Error writing to file - write (28: No space left on device) [IP: 146.75.118.132 80]
Warning: Failed to fetch http://security.debian.org/debian-security/dists/trixie-security/InRelease Error writing to file - write (28: No space left on device) [IP: 146.75.122.132 80]
Warning: Failed to fetch http://download.proxmox.com/debian/pve/dists/trixie/InRelease Error writing to file - write (28: No space left on device) [IP: 185.219.221.167 80]
Warning: Some index files failed to download. They have been ignored, or old ones used instead.As I am a typical Windows user, I somewhat struggle with Linux systems, and do not know how to handle this. Googling the phrase "no space left on device" reveals a ton of articles and forum posts. But I fear, that they do not help me. Most of them explain, how to find out, which directories and/or files take that space, and to delete them. Here my problem: I do not find them.
So I will post here as much information as possible, so that you could possibly help me.
Maybe I first explain, what disks I have attached to that PC and how I use them. I have put a 1TB SATA SSD and a 4TB M.2 (NVMe) SSD into the PC. Additionally, I have four 4TB SATA HDDs inside a USB enclosure, which itself is attached to the PC. All 4TB disks (SSD and HDDs) are used inside ZFS pools for various purposes. Proxmox itself was installed with the standard installation procedure onto the 1 TB SATA SSD. This uses LVM. Sic:
Code:
root@pve:~# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sda 8:0 0 931.5G 0 disk
├─sda1 8:1 0 1007K 0 part
├─sda2 8:2 0 1G 0 part /boot/efi
└─sda3 8:3 0 930G 0 part
├─pve-swap 252:0 0 8G 0 lvm [SWAP]
├─pve-root 252:1 0 96G 0 lvm /
├─pve-data_tmeta 252:2 0 8.1G 0 lvm
│ └─pve-data-tpool 252:4 0 793.8G 0 lvm
│ ├─pve-data 252:5 0 793.8G 1 lvm
│ ├─pve-vm--100--disk--0 252:6 0 4G 0 lvm
│ ├─pve-vm--120--disk--0 252:7 0 4G 0 lvm
│ ├─pve-vm--300--cloudinit 252:8 0 4M 0 lvm
│ ├─pve-vm--300--disk--0 252:9 0 8G 0 lvm
│ ├─pve-vm--301--cloudinit 252:10 0 4M 0 lvm
│ ├─pve-vm--301--disk--0 252:11 0 8G 0 lvm
│ ├─pve-vm--302--cloudinit 252:12 0 4M 0 lvm
│ ├─pve-vm--302--disk--0 252:13 0 8G 0 lvm
│ ├─pve-vm--510--disk--0 252:14 0 128G 0 lvm
│ ├─pve-vm--199--disk--0 252:15 0 4G 0 lvm
│ ├─pve-vm--510--disk--1 252:16 0 4M 0 lvm
│ └─pve-vm--590--disk--0 252:17 0 8G 0 lvm
└─pve-data_tdata 252:3 0 793.8G 0 lvm
└─pve-data-tpool 252:4 0 793.8G 0 lvm
├─pve-data 252:5 0 793.8G 1 lvm
├─pve-vm--100--disk--0 252:6 0 4G 0 lvm
├─pve-vm--120--disk--0 252:7 0 4G 0 lvm
├─pve-vm--300--cloudinit 252:8 0 4M 0 lvm
├─pve-vm--300--disk--0 252:9 0 8G 0 lvm
├─pve-vm--301--cloudinit 252:10 0 4M 0 lvm
├─pve-vm--301--disk--0 252:11 0 8G 0 lvm
├─pve-vm--302--cloudinit 252:12 0 4M 0 lvm
├─pve-vm--302--disk--0 252:13 0 8G 0 lvm
├─pve-vm--510--disk--0 252:14 0 128G 0 lvm
├─pve-vm--199--disk--0 252:15 0 4G 0 lvm
├─pve-vm--510--disk--1 252:16 0 4M 0 lvm
└─pve-vm--590--disk--0 252:17 0 8G 0 lvm
sdb 8:16 0 3.6T 0 disk
├─sdb1 8:17 0 3.6T 0 part
└─sdb9 8:25 0 8M 0 part
sdc 8:32 0 3.6T 0 disk
├─sdc1 8:33 0 3.6T 0 part
└─sdc9 8:41 0 8M 0 part
sdd 8:48 0 3.6T 0 disk
├─sdd1 8:49 0 3.6T 0 part
└─sdd9 8:57 0 8M 0 part
sde 8:64 0 3.6T 0 disk
├─sde1 8:65 0 3.6T 0 part
└─sde9 8:73 0 8M 0 part
nvme0n1 259:0 0 3.6T 0 disk
├─nvme0n1p1 259:1 0 3.6T 0 part
└─nvme0n1p9 259:2 0 8M 0 partNow to the file system space:
Code:
root@pve:/# df -h
Filesystem Size Used Avail Use% Mounted on
udev 32G 0 32G 0% /dev
tmpfs 6.3G 13M 6.3G 1% /run
/dev/mapper/pve-root 94G 94G 0 100% /
tmpfs 32G 19M 32G 1% /dev/shm
efivarfs 128K 25K 99K 20% /sys/firmware/efi/efivars
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 1.0M 0 1.0M 0% /run/credentials/systemd-journald.service
tmpfs 32G 0 32G 0% /tmp
/dev/sda2 1022M 8.8M 1014M 1% /boot/efi
cpool/media 6.0T 256K 6.0T 1% /zfs/cpool/media
cpool/backup 6.2T 240G 6.0T 4% /zfs/cpool/backup
cpool/eon 6.0T 4.2G 6.0T 1% /zfs/cpool/eon
cpool/documents 6.0T 96M 6.0T 1% /zfs/cpool/documents
cpool/misc 6.0T 256K 6.0T 1% /zfs/cpool/misc
cpool/entertainment/video 6.0T 256K 6.0T 1% /zfs/cpool/entertainment/video
cpool/entertainment/lego 6.0T 256K 6.0T 1% /zfs/cpool/entertainment/lego
cpool/software 6.0T 21G 6.0T 1% /zfs/cpool/software
hpool/apps/kubernetes 3.6T 3.8G 3.6T 1% /zfs/hpool/apps/kubernetes
cpool/cloud/sarah_jonas 6.0T 256K 6.0T 1% /zfs/cpool/cloud/sarah_jonas
cpool/temp 11T 4.3T 6.0T 42% /zfs/cpool/temp
cpool/cloud/jenny_mirko 6.0T 256K 6.0T 1% /zfs/cpool/cloud/jenny_mirko
cpool/entertainment/audio 6.0T 256K 6.0T 1% /zfs/cpool/entertainment/audio
cpool/cloud/ronja 6.0T 256K 6.0T 1% /zfs/cpool/cloud/ronja
hpool/apps/postgres 3.6T 15M 3.6T 1% /zfs/hpool/apps/postgres
hpool/apps/nexus 3.6T 2.1G 3.6T 1% /zfs/hpool/apps/nexus
cpool/studies 6.0T 1010M 6.0T 1% /zfs/cpool/studies
/dev/fuse 128M 32K 128M 1% /etc/pve
bkp.fritz.box:/mnt/HD/HD_a2/Store 15T 4.7T 9.8T 33% /mnt/bkp.fritz.box/store
tmpfs 1.0M 0 1.0M 0% /run/credentials/getty@tty1.service
tmpfs 6.3G 4.0K 6.3G 1% /run/user/0This shows that the root filesystem (correct wording?) seems to be 100% full. But where? Disk usage reports:
Code:
root@pve:/# du -sh /*
0 /bin
309M /boot
19M /dev
5.7M /etc
4.0K /home
0 /lib
0 /lib64
16K /lost+found
4.0K /media
4.7T /mnt
4.0K /opt
du: cannot access '/proc/58210/task/58210/fd/4': No such file or directory
du: cannot access '/proc/58210/task/58210/fdinfo/4': No such file or directory
du: cannot access '/proc/58210/fd/3': No such file or directory
du: cannot access '/proc/58210/fdinfo/3': No such file or directory
0 /proc
72K /root
13M /run
0 /sbin
4.0K /srv
0 /sys
0 /tmp
5.4G /usr
1.2G /var
4.6T /zfs
root@pve:/# du -sh /mnt/*
4.7T /mnt/bkp.fritz.box
4.0K /mnt/vzsnap0
Some explanation to the /zfs directory: All ZFS pools and datasets are mounted here. Remember: These are the other disks. A similar fact for the /mnt directory: I have mounted an NFS share from a NAS device into /mnt/bkp.fritz.box. This is, why you see such high disk usage here.
As you can see, space consumption of all other directories does not sum up to the nearly 100GB of the root filesystem. So ... How can I narrow this down? How can I find out, what actually takes up the space? How can I solve my problem(s)?
Let me know, what else I should provide, or what steps I should do for analyzing and troubleshooting.
Many thanks.
Mirko