Postgres has a brief blip. Connections fail for 30 seconds. The application’s health endpoint, which queries the database, starts returning 500. Kubernetes’ liveness probe sees the 500 and restarts every pod. Now you have zero healthy pods, every pod is restarting at once, and the brief blip has become a 10-minute outage.
This is the classic liveness-probe death spiral, and it happens because most teams treat liveness and readiness as the same thing. They are not. They have different responsibilities, different failure semantics, and need different endpoint shapes. Getting the difference right is one of those Kubernetes details that turns an outage into a non-event.
Liveness probe. “Is this process alive, or is it stuck and needs a kill?” A failed liveness probe causes Kubernetes to restart the pod. The right reason for liveness to fail is something a restart will fix: a deadlock in the application, a stuck event loop, a process that has wedged itself.
Readiness probe. “Is this pod ready to receive traffic?” A failed readiness probe causes Kubernetes to remove the pod from the service’s load balancer, but the pod keeps running. The right reasons for readiness to fail are transient: starting up, dependency unavailable, paused for graceful shutdown.
Startup probe. “Is this pod still booting?” Used in front of liveness so that a slow-starting pod does not get killed before it finishes initializing. Configurable per-app.
The death spiral happens when teams point liveness at an endpoint that checks the database. Database has a blip → liveness fails → pod restarts → restart does not fix the database blip → next pod’s liveness fails → cascade.
Two endpoints, two responsibilities.
/health/live, liveness:
app.get('/health/live', (req, res) => {
// Only check things a restart can fix.
// Not: "can I reach the database?"
// Yes: "is the event loop responsive?"
res.status(200).send('ok');
});For most apps, returning 200 ok is enough. The probe answers “is the process responsive?” If the HTTP server can respond, the answer is yes. If you have a worker that can deadlock, add an in-process heartbeat counter and check it has incremented recently.
/health/ready, readiness:
app.get('/health/ready', async (req, res) => {
try {
// Check that downstream dependencies the pod cannot serve traffic without are reachable.
await db.query('SELECT 1', { timeout: 2000 });
await redis.ping();
if (state.shuttingDown) {
return res.status(503).send('shutting down');
}
res.status(200).send('ready');
} catch (err) {
res.status(503).send('not ready');
}
});Readiness can fail freely. Failing readiness pulls the pod out of the load balancer; it does not kill it. When the database recovers, the next probe succeeds, the pod re-joins the LB. No restart.
spec:
containers:
- name: api
livenessProbe:
httpGet:
path: /health/live
port: 8080
initialDelaySeconds: 30
periodSeconds: 10
timeoutSeconds: 2
failureThreshold: 3
readinessProbe:
httpGet:
path: /health/ready
port: 8080
periodSeconds: 5
timeoutSeconds: 2
failureThreshold: 1 # readiness flaps quickly
successThreshold: 1
startupProbe:
httpGet:
path: /health/live
port: 8080
periodSeconds: 5
failureThreshold: 60 # 60 × 5s = 5min to startA few non-obvious choices:
failureThreshold: 3 on liveness. Three consecutive failures, 10s apart = 30 seconds of grace. A transient hiccup does not kill the pod.failureThreshold: 1 on readiness. One failure pulls the pod out of the LB. Readiness should be jumpy; that is the point.startupProbe. Lets initialDelay be small for liveness and lets a slow-starting app have a long grace before liveness starts checking.timeoutSeconds: 2. If your liveness endpoint takes >2s to respond, the probe fails. Keep it cheap.When Kubernetes terminates a pod (rolling deploy, scale-in, evicted), it sends SIGTERM. Your app should:
state.shuttingDown = true).The terminationGracePeriodSeconds: 30 field gives you 30s before SIGKILL. The preStop hook can run a sleep:
lifecycle:
preStop:
exec:
command: ["sh", "-c", "sleep 10"]This buys time for the readiness probe to mark the pod down and the LB to stop sending traffic before your app starts dropping connections. Without it, you get 502s during every deploy. (See the graceful-shutdown post for the Node.js side.)
A non-exhaustive list of things teams put in liveness that they shouldn’t:
The single rule: if the failure cause is external, do not check it from liveness. External failures should fail readiness, not liveness.
If your application can wedge itself (a deadlocked promise, a stuck synchronous loop, a worker that has stopped processing) liveness needs to detect it. The pattern: an in-process heartbeat that the main loop updates, and a liveness handler that checks it.
let lastHeartbeat = Date.now();
// Main loop / event handler increments the heartbeat.
setInterval(() => { lastHeartbeat = Date.now(); }, 1000);
app.get('/health/live', (req, res) => {
if (Date.now() - lastHeartbeat > 30_000) {
return res.status(500).send('event loop stalled');
}
res.status(200).send('ok');
});If the event loop is responsive, the interval runs and lastHeartbeat is recent. If it has stalled (CPU-bound code, deadlock), lastHeartbeat ages and liveness fails. Restart fixes the wedge.
For worker processes that don’t have an HTTP server, write a heartbeat file and have a sidecar HTTP server check its mtime. Same shape.
When liveness keeps failing, kubectl describe pod shows it:
Warning Unhealthy ... Liveness probe failed: HTTP probe failed with statuscode: 500
Normal Killing ... Container api failed liveness probe, will be restartedIf you see a series of these, two questions:
timeoutSeconds: 1 will fail a slow-but-alive pod)Most “we are restart-looping” incidents are one of those two.
A correctly configured probe stack makes rolling deploys boring:
If any of these stages is wrong (probe shape, timing, graceful shutdown) you get 502s, dropped connections, or restart loops. The whole system has to fit together.
Kubernetes’ two-probe model is more sophisticated than most teams use. Liveness restarts; readiness drains. Liveness should check only what a restart fixes; readiness can check everything that affects whether traffic should be routed here. The death-spiral pattern of pointing liveness at the database is one of the most common production misconfigs, and getting it right is a four-line change to the manifest.
Spend an afternoon refactoring your /health endpoints into /health/live and /health/ready. Set failureThreshold correctly for each. Test what happens when the database is paused. The next dependency outage you have will be a 30-second non-event instead of a 10-minute outage.
The kind of platform-engineering detail that turns a downstream blip into a non-event (probe shape, graceful shutdown, restart semantics) is the unglamorous DevOps work that decides whether a deploy is uneventful or front-page. It is the kind of work Yojji’s teams build into the Kubernetes platforms they ship for clients.
Yojji is an international custom software development company founded in 2016, with offices across Europe, the US, and the UK. Their teams specialize in the JavaScript ecosystem, cloud platforms (AWS, Azure, GCP), and Kubernetes-based deployments, including the probe and rollout configuration that decides whether your service stays up during a database hiccup.
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。