The team’s PagerDuty page count is growing. The on-call goes home Friday tense. Somebody suggests “we should do chaos engineering” and the room nods, but nobody knows what it means in practice for a 20-engineer team that does not have Netflix’s SRE org.
Chaos engineering is not “randomly break production.” It is “deliberately introduce a known failure mode and verify the system handles it as expected.” Done well, the failure modes you care about are the ones that paged you last quarter: DB hiccups, downstream timeouts, deploy interruptions. You don’t need a fancy framework. You need a list of five drills, an afternoon, and willingness to find out something embarrassing.
This post is those five drills, the playbook for running them safely, and the metric that tells you whether you actually got better.
Three prerequisites:
The five drills below are ordered by complexity. Do them in order.
The simplest chaos action: pick a pod, kill it. Validate that the system recovers.
kubectl get pods -l app=api -o name | shuf -n 1 | xargs kubectl delete podWatch:
Common findings: readiness probe is checking the wrong thing (pod stays in LB after kill), graceful shutdown is broken (in-flight requests dropped instead of completing), startup probe is too slow (replacement pod takes minutes).
This is the drill that earns a proper graceful shutdown setup and the right probe configuration.
Pick a non-critical downstream service. Make it return errors or timeouts.
If the dependency is internal, scale it to zero or block it with a NetworkPolicy:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: chaos-block-email }
spec:
podSelector: { matchLabels: { app: email-service } }
policyTypes: [Ingress]
ingress: [] # deny all ingressIf the dependency is third-party, use a toxiproxy sidecar to inject latency or errors:
toxiproxy-cli toxic add stripe -t latency -a latency=5000Watch:
Common findings: the team thought they had circuit breakers but discovered they only have retries. Cascading failures show up here that don’t show up in load tests.
Promote the read replica to primary. Or, less aggressively, restart the primary.
# AWS RDS: forced failover (Multi-AZ)
aws rds reboot-db-instance --db-instance-identifier prod-db --force-failoverWatch:
The RDS Multi-AZ failover is typically 30-60s of downtime for the database. The application should recover automatically; if it does not, you have a connection pool problem. (See pgbouncer post.)
Inject 500ms or 2000ms of latency on a network path. Toxiproxy or Linux tc:
# Add 500ms to all traffic to the database.
sudo tc qdisc add dev eth0 root netem delay 500ms
# Remove
sudo tc qdisc del dev eth0 rootOr via a service mesh (Istio, Linkerd) which can inject faults via configuration without touching pods.
Watch:
Common findings: timeouts are configured at one layer (HTTP client) but not another (database connection). A 500ms downstream latency causes 30s end-to-end latency because connection pool is starving and queueing.
Fill up disk to 95% and watch what happens:
# Inside the pod (only on a chaos test, never on prod):
fallocate -l $(df --output=avail / | tail -1 | awk '{print int($1*0.95)}')K /tmp/chaos-fillWatch:
Common findings: nothing alerts on disk-full because disks have been over-provisioned for years. When the alert finally fires, the on-call learns there is no runbook for clearing space and three services break.
Every drill follows the same six-step structure:
The hypothesis step is the highest leverage. It forces you to articulate your model of the system before testing it. Most “chaos engineering finds bugs” stories are actually “the team’s mental model didn’t match reality.”
Most teams never do production chaos. The progression is:
Don’t jump to step 3 without having lived through 1 and 2 for at least a few quarters. The cost of a bad chaos drill in production is real; the team has to be confident the system is resilient before introducing automated chaos.
Pick a single number: “MTTR for the last 10 incidents.” Mean time to recovery. As your chaos drills surface and fix issues, MTTR should drop.
Other useful metrics:
A team that does chaos drills consistently has incidents that are short and uneventful. A team that doesn’t has incidents that produce 4-hour postmortems.
For most teams’ first 6 months of chaos work, you don’t need a tool. kubectl delete pod, tc, iptables, scaling a deployment to 0: these are enough.
Three cases:
If any of those is true, work on monitoring or the immediate fires first.
Chaos engineering is a practice, not a tool. Five practical drills (kill a pod, break a downstream, fail over the database, inject latency, fill the disk) surface most of the resilience gaps a typical mid-size system has. Run them in staging first; promote to production drills with the team standing by once you trust the system.
The team that runs one of these drills per month is dramatically better at handling real incidents six months later. The drill is not the point; the muscle memory and documented runbook gaps are.
The kind of reliability practice that turns a fragile system into one that survives its bad days (chaos drills, runbook gaps, MTTR tracking) is the kind of long-haul engineering Yojji’s teams put into the platforms they ship for clients.
Yojji is an international custom software development company founded in 2016, with teams across Europe, the US, and the UK. They specialize in the JavaScript ecosystem, cloud platforms (AWS, Azure, GCP), and full-cycle product engineering, including the resilience and reliability work that decides whether your service is robust or just unbroken so far.
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。