惯性聚合 高效追踪和阅读你感兴趣的博客、新闻、科技资讯
阅读原文 在惯性聚合中打开

推荐订阅源

Jina AI
Jina AI
T
The Blog of Author Tim Ferriss
B
Blog
L
LangChain Blog
Y
Y Combinator Blog
美团技术团队
博客园 - 三生石上(FineUI控件)
钛媒体:引领未来商业与生活新知
钛媒体:引领未来商业与生活新知
G
Google Developers Blog
量子位
博客园_首页
让小产品的独立变现更简单 - ezindie.com
让小产品的独立变现更简单 - ezindie.com
C
Check Point Blog
D
Docker
小众软件
小众软件
The Cloudflare Blog
大猫的无限游戏
大猫的无限游戏
T
Tailwind CSS Blog
Apple Machine Learning Research
Apple Machine Learning Research
博客园 - 聂微东
Blog — PlanetScale
Blog — PlanetScale
GbyAI
GbyAI
Google DeepMind News
Google DeepMind News
IT之家
IT之家

The Practical Developer

The Libuv Thread Pool Trap: Why Node.js Async APIs Stall Under Load Postgres Covering Indexes with INCLUDE: Eliminate Heap Fetches on Read-Heavy Workloads Postgres DISTINCT ON: The Fastest Way to Get the Latest Row Per Group Postgres Transaction Isolation: The Anomalies Your App Actually Faces in Production Linux TCP Tuning for Node.js Microservices: The Kernel Settings That Stop Silent Connection Drops Under Load Postgres HOT Updates and Fillfactor: Why Not All Writes Are Created Equal Database Connection Pool Leaks: Finding the Promise That Never Returns Its Seat Linux OOM Killer in Production: Why Your Node.js Containers Die Without a Stack Trace Postgres Materialized Views: Refresh Strategies That Do Not Lock Your Dashboards API Dependency Health Checks: Why /health Is Not Enough Authorization with Zanzibar Tuples: How Google Manages Permissions and How To Build the Same Check in Node.js Postgres Advisory Locks: The 20-Character Primitive That Replaces Redis for Coordination Dead Letter Queues: The Message Queue Pattern That Saves You at 2 a.m. File Descriptor Exhaustion: The Kernel Limit That Silently Drops Node.js Connections Graceful Degradation: The Pattern That Turns Total Outages into Partial Success PostgreSQL Full-Text Search: Dropping Elasticsearch for 90% of Use Cases S3 Presigned Multipart Uploads: Stop Your API Server from Being a File Upload Bottleneck MessagePack vs JSON: The Binary Serialization Switch That Cut Our Internal RPC Overhead by 40% DNS Caching in Node.js: The Silent Cause of Production Latency Spikes Reliable Cron Jobs: The Pattern That Stops Double Runs, Missed Executions, And The 2 AM Page GraphQL Query Complexity: Stop the OOM Query Before It Reaches Your Resolver Node.js Event Loop Lag: The Hidden Metric Behind Random Latency Spikes API Request Validation with Zod: The Schema That Catches Bad Input Before It Corrupts Your Database Load Shedding in Node.js: How to Reject Traffic Before You Drown Request Hedging: Cut Tail Latency In Half Without Overprovisioning Git Bisect: The Automated Binary Search That Finds Breaking Commits in Minutes Node.js Garbage Collection Tuning: Stop Letting V8 Pause Your Event Loop Node.js Server Timeouts: The Settings That Stop Slow Clients from Holding Sockets Hostage Postgres BRIN Indexes: The Time-Series Secret That Shrinks Indexes by 99% Event Sourcing with PostgreSQL: The Pragmatic 80% Solution
Pod Disruption Budgets: The K8s Object That Keeps Your Se...
The Practica · 2024-01-05 · via The Practical Developer

Cluster autoscaler drains a node. It does the polite thing: sends SIGTERM, waits for graceful shutdown. Except: 8 of your 10 API pods happen to be on that node. They all drain at once. The two remaining pods cannot serve the traffic. 502s for two minutes while replacements come up on other nodes.

This is the Pod Disruption Budget (PDB) gap. Kubernetes’ rolling deploy logic respects maxUnavailable, but voluntary disruptions (node drains, autoscaler-driven evictions, cluster upgrades) do not, unless you tell it. PDB is the manifest that says “never voluntarily evict more than N pods of this deployment at once.” Three lines of YAML. The default behavior is “no protection.”

What a PDB is

A Pod Disruption Budget specifies the minimum number of pods (or maximum number of disrupted pods) that must remain available for a labeled set of pods. Voluntary disruptions (node drains, evictions, scale-downs by autoscaler, cluster upgrades) respect PDBs. Involuntary ones (a node dying, OOM killer) do not.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api-pdb
spec:
  minAvailable: 80%       # never have less than 80% of api pods available
  selector:
    matchLabels:
      app: api

Or specify max unavailable:

spec:
  maxUnavailable: 1

minAvailable and maxUnavailable are mutually exclusive. Pick one based on which is more meaningful for the workload.

For a 3-replica deployment with minAvailable: 80%, that rounds up to 3, meaning no pods can be disrupted simultaneously. So percentages are clean for larger deployments; integers are cleaner for small ones.

When PDBs apply

PDBs gate voluntary disruptions:

  • kubectl drain node (manual node maintenance).
  • Cluster upgrades that drain nodes one at a time.
  • Cluster autoscaler scaling down nodes.
  • API-initiated evictions (kubectl evict).

PDBs do not affect involuntary disruptions:

  • Node hardware failure.
  • OOM kills.
  • kubectl delete pod (this is direct deletion, not eviction).
  • Application crashes.

For involuntary disruptions, your replicas, anti-affinity, and graceful shutdown are what protect you. PDBs are specifically for the cluster-management side.

A working example

For a 5-replica API deployment, you want:

  • During node drains: at most one pod down at a time (so other pods can absorb traffic).
  • During deploys: rolling update with maxUnavailable: 1 (configured on the Deployment).
# pdb.yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api-pdb
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: api
# deployment.yaml (excerpt)
spec:
  replicas: 5
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1
  selector:
    matchLabels:
      app: api

Now the PDB and the rolling-update strategy are aligned. Cluster operations and deploys both respect the same constraint.

The classic mistake: setting min higher than replicas

spec:
  minAvailable: 3
  selector:
    matchLabels:
      app: api  # but this deployment only has 2 replicas

PDB says “at least 3 must be available.” Deployment runs 2. Cluster operations are stuck; they cannot drain any pod because doing so would violate the PDB.

A node drain hangs forever. Cluster upgrades fail. The team finds out at 3 a.m. when an emergency upgrade is blocked.

The fix is to make sure minAvailablereplicas - 1, or use percentages so it scales with the deployment.

PDBs and StatefulSets

PDBs work with StatefulSets too. They are more important for stateful workloads because:

  • Restarting a stateful pod takes longer (volume reattach, leader election).
  • Quorum-based services (etcd, Kafka, ZooKeeper) need a majority, and PDB enforces it.
# PDB for a 3-node Kafka cluster
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: kafka

You always need at least 2 of 3 Kafka brokers up to maintain quorum. The PDB makes that explicit.

When disruptionsAllowed is 0

kubectl get pdb shows the current state:

NAME      MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS
api-pdb   N/A             1                 1
db-pdb    2               N/A               0

ALLOWED DISRUPTIONS = 0 means voluntary disruptions are blocked right now. Common reasons:

  • A pod is already disrupted (rolling deploy in flight, recent eviction).
  • Pods are not yet ready (just created, still starting up).

If you see this consistently, voluntary operations cannot proceed. Investigate before scheduling node maintenance.

Combining PDB with topology spread

A PDB caps the number of pods that can be down. Topology spread constraints (or pod anti-affinity) ensure pods are distributed across nodes/zones in the first place, so a node drain naturally affects only one or two pods.

# In the deployment
spec:
  template:
    spec:
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: ScheduleAnyway
        labelSelector:
          matchLabels:
            app: api

This says “spread pods across zones so no zone has more than one extra than another.” Combined with PDB, you get both “spread out” and “don’t disrupt too many at once.”

For services that should not lose more than 1 zone at a time:

- maxSkew: 1
  topologyKey: topology.kubernetes.io/zone
  whenUnsatisfiable: DoNotSchedule

Stronger guarantee: pods will not be co-located if it violates the spread.

What about Jobs?

For Jobs and CronJobs, PDBs are less useful. Jobs are short-lived and meant to be disrupted. Don’t put PDBs on them.

Testing your PDB

Before relying on a PDB in production, test it:

# Simulate a drain on the node where pods are running.
kubectl get pods -l app=api -o wide
kubectl drain node-name --ignore-daemonsets --delete-emptydir-data

Watch what happens. Pods should evict one at a time. The drain command should respect the PDB and proceed slowly.

If it doesn’t proceed at all (disruptionsAllowed: 0), the PDB is too strict.

When NOT to use PDBs

A few cases:

  • Single-replica services. A PDB on a 1-replica deployment is “you can’t restart this pod ever,” which usually breaks operations. Run more replicas.
  • Best-effort batch workloads. A worker pool that can lose 50% of its capacity for an hour is fine without a PDB.
  • DaemonSets. PDBs on DaemonSets rarely make sense. They have one pod per node, and node drains specifically delete those pods.

For everything else, even a permissive PDB (maxUnavailable: 50%) is better than none.

Operational pattern

The full reliability stack for a stateless web service:

  1. 3+ replicas, distributed via topology constraints.
  2. PDB: maxUnavailable: 1 (or 25% for larger deployments).
  3. Liveness/readiness probes correctly configured (see the probes post).
  4. Graceful shutdown in the application (see the graceful-shutdown post).
  5. Rolling-update strategy matching the PDB.

With all five, voluntary disruptions are non-events. Without any one of them, you get random outages during cluster maintenance.

The takeaway

Pod Disruption Budgets are three lines of YAML that prevent the most embarrassing kind of outage: “we drained a node and our service went down.” Match maxUnavailable (or minAvailable) to your replica count and rolling-update strategy. Combine with topology spread for full resilience. Test by simulating a drain.

The next time the cluster admin says “we’re upgrading next week,” the answer is “fine, our PDBs are set,” instead of “let me know when so I can stand by.”


A note from Yojji

The kind of platform-engineering detail that turns “the cluster upgrade went smoothly” from luck into design (PDBs, topology spread, probes, graceful shutdown working together) is the kind of long-haul DevOps work Yojji’s teams build into the Kubernetes deployments they ship for clients.

Yojji is an international custom software development company founded in 2016, with teams across Europe, the US, and the UK. They specialize in the JavaScript ecosystem, cloud platforms (AWS, Azure, GCP), and Kubernetes operations, including the disruption-budget and topology work that decides whether your service stays up during cluster maintenance.