Some memory leaks are straightforward to detect. A process allocates memory, forgets to free it, and grows until something breaks. This was not that kind of leak.
This is the story of how Canonical Support helped a global retail organization trace the cause for an unusual memory leak originating in PID 1, the very first process started by the kernel during the system boot sequence. The memory footprint was 10 times higher than expected. By investigating the issue across three separate system layers – a misconfigured storage orchestrator, a kernel race condition, and glibc’s allocator – our team was able to identify the source and fast-track a patch.
The customer was the innovation and engineering arm of a global retail company. They were responsible for driving the company’s retail transformation – operating a fleet of servers running as part of a Ceph storage cluster. Across several of these cluster nodes, they began observing a pattern that made no immediate sense: /sbin/init (a symlink to systemd, which was running as PID 1) was consuming over 50% of available system memory. On servers with 125 GB of RAM, that meant PID 1 alone was holding approximately 70 GB, when only 8 GB were expected. The systems were experiencing out-of-memory kills, with the kernel forcibly terminating processes to free up available memory to survive.
Nothing in the obvious diagnostic places explained it. Memory consumption continued to grow, and was never released. The investigation didn’t point to any obvious application-level cause. The customer had ruled out the workloads running on the servers as being the issue. The problem was somewhere deeper, below the application layer.
The support team at Canonical’s first move was to request sos-reports from the affected nodes, the standard starting point for system-level investigation. But sos-report output, comprehensive as it is, didn’t contain anything that explained a 70 GB footprint on PID 1. The next step was more direct: the engineer requested a core dump of PID 1 itself, generated with gcore 1.
This is not a routine ask. Generating a core dump of the init process (PID 1) on a running production system is the technical equivalent of performing open-heart surgery while the patient is running a marathon. PID 1 is not just any process; it is the root of the entire user-space ecosystem. If it stumbles, the whole system collapses. Because the resulting file is massive – In this case the core dump compressed from 70 GiB to less than 10 GiB, a major red flag that there was significant repetition in the memory of the process –, transferring, downloading, and analyzing it can significantly delay the start of troubleshooting, and the investigation itself can take an unpredictable amount of time. The customer uploaded it, and that artifact is what cracked the case open.
Analyzing the core dump, the Canonical engineer could see directly what was occupying PID 1’s memory. The footprint was tied to mount table entries. Specifically, entries from runs of a ceph-volume Docker container. The mount table, which systemd maintains and reparses every time changes are made, had grown to millions of entries.
The behavior of systemd here is by design: it watches /proc/self/mountinfo and reparses the full table on every change. Under normal conditions, reparsing is not an issue, but because the table was now very large, continuously regenerating it caused serious issues. Each time systemd parsed it, glibc’s allocator retained the memory rather than returning it to the kernel. The memory accumulated but was never freed.
That explained the mechanism, but it didn’t yet explain why the mount table had grown so large, or why ceph-volume containers were running so frequently.
This is where the investigation crossed layers.
Tracing the frequency of ceph-volume container activity, the engineer identified the actual source: a cephadm process – part of the Ceph orchestration layer – was repeatedly attempting to “zap” (a destructive operation that wipes all data and metadata from a disk to reset it to a clean state) an OSD (Object Storage Daemon) that no longer existed. Because there was no Logical Volume associated with it, the operation failed every time. And every time it failed, it retried. The ceph-volume Docker container was spinning up and tearing down in a continuous loop.
Nobody had flagged this because the orchestration layer in Ceph was silently failing, and the only visible symptom, the one that brought the customer to Canonical Support, was 70 GB of memory consumed by PID 1 on nodes that had nothing obviously wrong with them.
Stopping the loop was the workaround. Once cephadm was corrected and the repeated zap attempts ceased, the mount/unmount thrashing stopped, the race condition stopped triggering, and memory usage stabilized. The customer confirmed that the correction resolved the immediate problem.
Once a workaround was implemented, the Canonical team was able to shift their attention to finding a permanent fix.
Canonical’s support engineers traced the bug to the upstream code and regularly engaged with the open source community to implement a fix that extended beyond the immediate customer. By providing Linux kernel maintainers with a fully characterized bug and a verified trigger, they secured a resolution that benefited the entire open source ecosystem.
By shrinking the problem to a minimal reproducer, mounting and unmounting a tmpfs in a tight loop, the team isolated the precise, reproducible mechanism: a kernel race condition. This churn caused mount table entries to intermittently spike to millions of lines, triggering continuous systemd reparsing and glibc memory retention, which resulted in the 70 GB footprint on PID 1.
The patch was merged into the mainline kernel. After passing through the Stable Release Update (SRU) process, the fix landed in several Ubuntu kernel releases.
What looked like a memory leak wasn’t quite the whole story. While the memory was being properly freed by systemd and tracked by glibc, it became unusable to the rest of the system, effectively starving the node. The real issue was that a silent misconfiguration in a storage orchestration layer, cephadm looping on a non-existent OSD, was the trigger for a kernel-level race condition that manifested as an inexplicably large footprint. No single layer told the full story:
That chain, from production symptom to cross-layer diagnosis to mainline kernel patch, is what enterprise Linux support looks like when the problem isn’t immediately obvious and doesn’t fit a known category.
If you want to understand what support from Canonical can bring to complex infrastructure problems, visit our support page, or, get in touch.
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。