TL;DR: We identified and fixed an extreme performance bottleneck in our Sealos DevBox platform where container commits took up to 15 minutes. By replacing an inefficient O(n²) filesystem walk in containerd's diff service with a filesystem-aware method that reads changes directly from the OverlayFS upperdir, we cut incremental commit times by over 98%—from 39 seconds to just 0.46 seconds.
Sealos DevBox, our cloud development environment platform serving tens of thousands of developers, hit a critical performance wall. The container commit feature—a cornerstone of a developer's workflow for saving their workspace state—became painfully slow. As usage scaled, commit times ballooned, leading to frustrating delays and workflow disruptions.
Our monitoring revealed some alarming metrics under load:
For a feature designed to be nearly instantaneous, these latencies were a significant roadblock to productivity and a poor user experience for every developer on the DevBox platform.
In Sealos DevBox, a "container commit" captures the current state of a running development environment and saves it as a new OCI image layer. This mechanism is what allows developers to persist work, share reproducible environments, and resume sessions instantly. This entire process runs within an isolated Kubernetes Pod, giving developers a consistent and powerful environment.
The performance of this commit operation is fundamental to the DevBox value proposition, which stands apart from other development models.
| Feature | Traditional Local Dev | Cloud IDEs | Sealos DevBox |
|---|---|---|---|
| Setup Time | Hours to days | Minutes | Minutes |
| Environment Consistency | Poor | Good | Perfectly Consistent (via image layers) |
| Resource Requirements | High local specs | Browser only | Browser only |
| State Persistence | Manual | Varies | Automated & Instant (via optimized commits) |
| IDE Support | Native | Limited | Any IDE (via remote connection) |
| Environment Isolation | Docker/VM | Container | Secure Kubernetes Pod |
Slow commits directly undermined the "instant" and "fluid" experience we promise our users.
To pinpoint the bottleneck, we used pprof to generate flame graphs of the containerd process during a commit. We designed two specific test cases to replicate the user-reported issues.
Test 1: Large Initial Commit Simulates the first save of a large project.
Test 2: Small Incremental Update Simulates a common developer workflow: saving a minor code change.
The initial test results confirmed our monitoring data:
| Test Scenario | Commit Time | Expected Time |
|---|---|---|
| Test 1: 10GB Initial Commit | 846.99s | ~60s |
| Test 2: 1KB Incremental Commit | 39.14s | <1s |
The flame graphs for both tests were nearly identical, pointing to a massive CPU-bound operation deep within containerd's diff service, regardless of the change size. This was our smoking gun.
DevBox Commit Performance Flame Graph Before Optimization - Test 001
DevBox Commit Performance Flame Graph Before Optimization - Test 002
Our code analysis led us to containerd's default diff service, which used a function called doubleWalkDiff. This function calculates changes by performing a full, recursive walk of two directory trees: the base image's filesystem (lowerdir) and the container's merged view.
This approach has a time complexity of O(n²), as it compares every file and directory from the source with the target. This meant that even for a tiny 1KB change, the algorithm was wastefully traversing and comparing the entire 10GB of existing data.
This discovery was the key: the tool was working as designed, but the design was not optimized for our high-performance DevBox use case.
The solution came from a deeper understanding of OverlayFS. An OverlayFS mount consists of a read-only lowerdir and a writable upperdir. Crucially, all modifications within the container—creations, modifications, and deletions—are recorded exclusively in the upperdir.
The upperdir itself is the diff. The doubleWalkDiff function was redundantly recalculating what the filesystem had already tracked.
The fix was to bypass this generic walk and generate the diff by reading only the upperdir. We found that containerd's continuity library already supported this via fs.DiffDirChanges when used with an fs.DiffSourceOverlayFS flag.
This simple change transformed the operation's complexity from O(n²) to O(m), where m is the size of the modifications, not the entire filesystem.
We rolled out the optimized containerd binary across our Sealos Cloud.
We activated the new diff plugin by editing /etc/containerd/config.toml:
The impact was immediate and transformative for the Sealos DevBox platform.
| Test Scenario | Before | After | Improvement |
|---|---|---|---|
| Test 1: 10GB Initial Commit | 846.99s | 266.83s | 3.17x faster |
| Test 2: 1KB Incremental Commit | 39.14s | 0.46s | 85.08x faster |
Across our production clusters serving over 10,000 active developers:
This optimization proves the immense value of using filesystem-aware algorithms over generic ones. While containerd's default diff mechanism ensures portability, it carries a severe performance penalty when a specialized method is available.
It's important to note that this solution is specific to environments using containerd with the OverlayFS snapshotter. Systems relying on other snapshotters (e.g., ZFS, Btrfs) would not benefit from this patch and would require their own purpose-built diff implementations.
With the diffing bottleneck eliminated, our new flame graphs show that tar and gzip operations are the next performance frontier, especially for large initial commits. Our future work for DevBox performance will focus on:
zstd.Our implementation is open-source and can be reviewed in our containerd repository fork. This performance boost is now standard in all Sealos DevBox environments.
A: DevBox now uses a custom-patched containerd with an OverlayFS-aware diff algorithm. Instead of inefficiently comparing the entire filesystem (an O(n²) task), it intelligently reads changes directly from the OverlayFS upperdir layer. This reduces the work to O(m)—proportional to the change size—making incremental commits exceptionally fast.
A: This specific fix is for container runtimes using containerd with the OverlayFS snapshotter, which is a common setup in modern Kubernetes environments like the one DevBox is built on. The underlying principle—using filesystem-specific logic—can be applied elsewhere, but the code is specific to OverlayFS.
A: Faster commits create a seamless, uninterrupted workflow. Developers on DevBox can now save their environment state frequently without a second thought, which encourages experimentation, reduces the risk of lost work, and speeds up CI/CD pipelines that rely on creating image snapshots.
A: We used pprof to collect CPU profiles from the live containerd process and generated flame graphs to visualize the performance hotspots. This quickly and clearly identified the doubleWalkDiff function as the primary bottleneck in our DevBox environment.
A: After solving the diffing issue, the main bottlenecks are now the archiving (tar) and compression (gzip) stages. These processes are the most time-consuming parts of a commit, especially when dealing with large new files.
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。