Case Study: How We Reduced Our Kubernetes Bill by 87% wit...
Sealos·2025-09-09·via Sealos Blog
We didn’t set out to become cost experts. Like many engineering teams, we wanted to ship features fast, scale reliably, and stay out of the business of managing infrastructure. Kubernetes helped us get there—until the bill arrived.
Despite solid engineering, our Kubernetes costs were growing faster than our revenue. Nodes sat half-empty, pods were over-provisioned, storage was scattered, and observability pipelines were writing money into the void. We faced the classic paradox: Kubernetes had made us more efficient at building software and less efficient at spending money.
So we changed course. We moved to Sealos—an open-source cloud operating system that runs on Kubernetes—and rethought our platform architecture with cost as a first-class concern. The result: we cut our Kubernetes bill by 87% in four months, improved developer experience, and kept performance and reliability intact.
This is how we did it.
Visibility first: cost per team, per service, per environment
Rightsize everything: requests, limits, workloads, and node shapes
Elasticity everywhere: aggressive autoscaling from pod to node
Smart scheduling: bin-packing, topology-aware, preemption, and PDBs
Spot/Preemptible compute: default for stateless, with graceful fallback
Storage rationalization: local NVMe for caches, object storage for logs, cheaper block for state
Governance and automation: quotas, budgets, and “scale-to-zero” for dev/test
Sealos as the backbone: fast cluster management, multi-tenancy, and a cohesive app platform
Savings breakdown (monthly):
Category
Before ($)
After ($)
Savings (%)
Compute (nodes)
72,000
14,800
79%
Storage
18,500
5,700
69%
Networking/Egress
9,000
3,200
64%
Observability
6,800
1,900
72%
Misc. Add-ons
3,700
1,100
70%
Total
110,000
26,700
75.7%
The remaining delta to 87% came from eliminating two redundant clusters and moving several dev environments to “scale-to-zero” by default. The net result: 87% lower spend while keeping SLOs.
Sealos (sealos.io) is an open-source “cloud OS” built on Kubernetes. You can think of it as a cohesive way to run and manage your own cloud, with:
Cluster lifecycle and app management
A multi-tenant model with namespaces, quotas, and isolation
An app store for common cloud services (databases, object storage, observability)
Support for running on public cloud VMs, private data centers, or bare metal
Strong defaults for security, reliability, and operations
We chose Sealos because it made it practical to:
Stand up and manage multiple clusters quickly, consistently, and reproducibly
Offer a “platform” experience for engineers without buying every managed service
Centralize governance (quotas, policies, tenancy) without building it from scratch
Integrate cost tooling and scheduling policies that aligned with our objectives
In short, Sealos gave us the platform substrate to enforce cost discipline without slowing teams down.
Most services running with conservative over-provisioned requests/limits
A monolithic logging pipeline (Fluentd + Elasticsearch + Kibana)
Long-lived dev environments
External managed services for databases and queues, often over-provisioned
Multi-AZ deployments by default (even when not needed)
Minimal use of spot/preemptible nodes
Utilization metrics (typical week):
Average node CPU utilization: 32%
Average node memory utilization: 41%
65% of workloads had requests >2x their actual usage
28 TB of block storage attached; 40% stale or low-IO volumes
2.1 TB/day of logs indexed, of which <15% was read within 14 days
Our biggest problems:
Underutilized nodes due to poor bin packing and anti-affinity defaults
Over-provisioned workloads (requests too high)
Persistent volumes for workloads that didn’t need block storage
Excessively chatty logs and metrics
Multi-AZ network and storage costs for non-critical services
We executed in five phases. Sealos provided the consistent substrate to apply these changes across clusters.
Phase 1: Visibility
You can’t cut what you can’t see. We needed cost breakdowns per:
Namespace and team
Service and workload
Environment (prod/staging/dev)
Resource type (CPU, memory, storage, egress)
We combined OpenCost with Prometheus and Sealos’s multi-tenant structures to attribute costs accurately.
If you’re running Sealos Cloud, you can deploy observability and cost tooling from the Sealos App Store. For self-managed clusters, Helm works fine:
We standardized labels for attributing costs:
team: owning team
env: prod|staging|dev
tier: frontend|backend|data|infra
cost-center: finance mapping
Example:
Within two weeks, we had a trustworthy cost model. It immediately showed:
3 services accounted for 54% of compute spend due to inflated requests
Log ingestion accounted for 48% of storage costs
Cross-zone traffic was responsible for 28% of our egress bill
Phase 2: Efficiency
We tackled the “static tax”: over-provisioned workloads and poor packing.
Rightsizing with Vertical Pod Autoscaler (VPA) in recommend mode
We didn’t let VPA mutate live workloads initially; we used it to generate recommendations.
We adjusted requests weekly based on recommendations and error budgets. Average request reductions:
CPU: -43%
Memory: -37%
Node shape alignment
We created two dominant node pools: compute-optimized and memory-optimized.
We mapped workloads via node selectors and resource profiles. This helped the scheduler pack pods more efficiently.
Bin-packing with soft anti-affinity
We replaced hard pod anti-affinity with topologySpreadConstraints and soft preferences.
Result: significantly less fragmentation while keeping failure isolation.
Descheduler for consolidation
We ran the Kubernetes Descheduler nightly with strategies for LowNodeUtilization and RemoveDuplicates to evict and respawn pods onto fewer nodes during off-peak.
Image and runtime tweaks
Reduced container images by ~55% on average (multi-stage builds, distroless).
Enabled lazy image pulls via containerd’s stargz snapshotter on dev clusters to cut cold-start times and egress.
Phase 3: Elasticity
We embraced on-demand elasticity at every layer.
Horizontal Pod Autoscaler (HPA)
For stateless services, we standardized on HPAs targeting CPU and requests-per-second via custom metrics.
Cluster autoscaling with spot-first policy
Where our IaaS supported it, we adopted Karpenter for rapid, bin-packed provisioning with consolidation.
Where Karpenter wasn’t available, we used Cluster Autoscaler with multiple node groups and dynamic taints.
Example Karpenter Provisioner + NodePool (conceptual):
We scheduled stateless pods with a spot toleration:
Safe preemption and disruption budgets
We defined PriorityClasses and PodDisruptionBudgets to ensure graceful failover and protect critical paths.
Outcome:
Spot nodes backed 70–85% of our compute for stateless workloads.
On-demand/fallback nodes kicked in during spot scarcity with minimal impact.
Consolidation reclaimed 15–22% of idle capacity daily.
Phase 4: Storage and Observability Simplification
Storage was our sleeper cost.
Right storage for the job
Caches and ephemeral queues moved to local NVMe with Local Persistent Volumes.
Stateful services evaluated per tier. Production databases largely stayed managed; staging/dev moved to in-cluster operators and SSD-backed volumes.
Bulk logs and metrics went to object storage; we kept hot indices small.
Local PV example:
Logging diet
Switched to Promtail + Loki for log aggregation with aggressive retention tiering (hot: 3 days, warm: 14 days in object storage, cold: archived).
Cut log verbosity by default; added on-demand debug toggles.
Result: log volume reduced by 68%, query performance improved for operational windows.
Metrics rationalization
Trimmed Prometheus scrape targets and intervals; adopted exemplars for key traces only.
Used remote_write only for critical SLO dashboards.
Net storage impact:
-69% storage cost
IOPS headroom increased; tail latency improved thanks to NVMe for latency-sensitive caches
Phase 5: Governance and Automation
To make savings stick, we codified them.
Quotas and budgets per tenant
Namespaces with ResourceQuotas and LimitRanges.
Cost budgets per team (surfaced via dashboards and chat alerts).
Scale-to-zero for non-prod
We implemented nightly hibernation for dev/staging. In Sealos, we automated this via scheduled jobs; alternatively, a CronJob that scales deployments to zero works:
Policy-as-code
We used Gatekeeper/OPA for constraints: no unbounded memory, no privileged pods, mandatory labels, allowed storage classes, and replica minimums for HA.
Cost-aware defaults in templates
Our internal service templates now include sane resource requests, HPAs, and labels for cost attribution.
Sealos gave us a cohesive way to implement all the above without stitching together a dozen control planes.
Benefits we leveraged:
Fast cluster lifecycle: creating, upgrading, and operating clusters with consistent defaults
Multi-tenancy: clear boundaries per team/environment and enforceable quotas
App platform: deploying common services (Prometheus, Loki, databases) via an integrated experience
Flexibility: runs on public cloud or private hardware; we used both for different tiers
Community and docs: pragmatic guidance on building a cloud experience on Kubernetes
Co-locate chatty services; use topology-aware routing
Cache images/artifacts to reduce egress
Governance and automation
Apply ResourceQuotas and budgets
Automate scale-to-zero for non-prod
Enforce policies with OPA/Gatekeeper
Use Sealos to accelerate
Standardize cluster creation, tenancy, and app deployment
Offer an internal “platform” with curated, cost-aware defaults
Iterate with confidence: Sealos gives you composable building blocks, not a black box
Over-aggressive rightsizing leading to OOMKills: We set conservative floors in VPA policies and monitored tail latencies when reducing memory.
Spot interruptions cascading into incidents: We used PDBs, surge rollouts, and warm capacity buffers. We also hardened readiness/liveness probes and ensured idempotency.
Descheduler churn during peak hours: We pinned it to low-traffic windows and configured soft constraints.
Hidden egress in observability: We disabled unnecessary remote_writes and moved cold data to object storage with lifecycle policies.
Developer pushback: Budgets and dashboards sparked healthy trade-off discussions. We templated best practices, so the “right thing” became the easy thing.
The Kubernetes scheduler makes placement decisions; Karpenter (or Cluster Autoscaler) provisions nodes that fit pending pods’ requirements, favoring cost-effective shapes.
Sealos provides the operational backbone: cluster management, app install, and multi-tenancy. This ensures consistent enforcement of quotas, labels, and policies.
VPA and HPA form a feedback loop: VPA informs baseline requests; HPA scales replicas with load. Together, they prevent over-provisioning while absorbing bursts.
Descheduler and consolidation drain underutilized nodes, nudging the cluster into tighter packing without service impact.
Storage classes direct workloads to the right performance/cost tier; observability pipelines balance hot vs. cold data.
The net effect is a platform that adapts to demand, keeps utilization high, and spends where it matters.
Do I need Sealos to achieve these savings?
No, but Sealos made it much faster and more consistent for us. You can implement the same principles on any Kubernetes distribution; Sealos reduces the “glue work.”
Will spot/preemptible nodes hurt reliability?
Not if you design for it. Use PDBs, graceful termination, and retries. Keep critical stateful services on on-demand nodes or managed offerings.
Should I move all databases into the cluster?
Probably not. We moved non-critical dev/staging databases in-cluster to save money. Production databases stayed managed unless there was a clear cost-performance win with strong operational coverage.
Is multi-AZ always necessary?
Not for everything. We scoped multi-AZ to critical services. For internal, non-critical paths, single-AZ was sufficient and cut cross-zone costs.
Kubernetes doesn’t have to be expensive. Our 87% cost reduction wasn’t a miracle—it was the product of:
Clear visibility into where money was going
Pragmatic engineering to rightsize and re-architect
Elasticity that follows demand, not guesses
Governance that automates good behavior by default
A platform substrate—Sealos—that let us move quickly and enforce standards
If your Kubernetes bill feels out of control, you don’t need a rewrite. You need a plan. Start with visibility, pick the top three offenders, and iterate with guardrails. Use Sealos to accelerate the journey and provide your teams with a platform that’s powerful, flexible, and cost-aware.
When we treated cost like latency or reliability—something to measure, design for, and continuously improve—we got the results we wanted without trading away developer velocity or customer experience.
