The team reads the Google SRE book and decides to “do SLOs.” Three months later, every service has 8 SLIs, dashboards everywhere, and nobody has actually used the SLO data to make a decision. The error budget exists in a spreadsheet that gets updated quarterly. Reliability has not measurably improved.
SLOs done well are not bureaucratic. They are two numbers per service and one rule that applies them: when the error budget is gone, stop shipping features and fix reliability instead. That is the entire mechanism. Everything else is optional.
This post is the version that works for normal-sized teams: how to pick the SLI, set the SLO, calculate the error budget, and tie it to actual decisions.
SLI (Service Level Indicator). A measurement. “Percentage of HTTP requests that returned 2xx within 500ms.”
SLO (Service Level Objective). The target. “99.5% of requests over the trailing 30 days.”
Error budget. The gap between perfection and the SLO. If your SLO is 99.5%, your error budget is 0.5%. Over 30 days, that’s 216 minutes of allowed badness. If you’ve used 200 minutes, you have 16 minutes of error budget left.
SLA (Service Level Agreement). A contract. “We refund customers if we drop below 99%.” Looser than the SLO; SLO is internal target, SLA is what you promise outside.
Most teams confuse SLO and SLA. The SLA exists for sales / legal; the SLO is the internal target. SLO is always tighter than SLA.
The biggest mistake is having too many SLIs. The right number is two. Pick one availability SLI and one latency SLI:
Availability SLI: percentage of valid requests that succeeded.
SLI = (valid_responses_2xx_3xx + valid_responses_4xx_excluding_429) / total_valid_responsesThe 4xx exclusion is important. A user sending a malformed request is not a service failure. 429 (rate limited) is included because it’s the service deciding to fail; 400 / 401 / 403 / 404 are typically excluded.
Latency SLI: percentage of valid requests served within a threshold.
SLI = (requests_under_500ms) / (total_valid_requests)500ms is a starting point. Pick a number based on user expectations for your service.
For services that aren’t HTTP (queue consumers, batch jobs), the SLIs are different but the same idea: one “did it work” SLI, one “did it work fast enough” SLI.
A common trap: setting the SLO at “the current observed level.” If the service is currently at 99.97% availability, setting an SLO of 99.95% means the team is below SLO before they start.
The right SLO is the level at which users start to notice and complain. For most consumer-facing APIs:
99.9% is the right SLO for most teams. Higher requires actual engineering investment in resilience; you can’t get there by copying a number.
Here is the entire mechanism: when you have used your error budget, stop shipping features and fix reliability.
If your SLO is 99.9% and you’ve spent 50 minutes of downtime in the last 28 days (out of 40 minutes allowed), you are over budget. The next sprint is not a new feature; it is the work that prevents the next 50 minutes.
This rule is what makes SLOs valuable. Without it, the SLO is decoration, interesting numbers that don’t change anyone’s behavior. With it, the SLO is a forcing function for engineering investment.
Once a quarter, review the SLOs:
The right cadence is quarterly. Monthly is too noisy; annual is too slow.
A working SLO measurement stack is Prometheus collecting metrics and a small set of dashboards. The availability burn-rate query:
1 - (
sum(rate(http_requests_total{job="api", status!~"5.."}[28d]))
/
sum(rate(http_requests_total{job="api"}[28d]))
)For burn-rate alerts, Sloth or similar generators turn an SLO definition into Prometheus alert rules:
# sloth definition
service: "api"
slos:
- name: "availability"
objective: 99.9
sli:
events:
error_query: 'sum(rate(http_requests_total{status=~"5.."}[{{.window}}]))'
total_query: 'sum(rate(http_requests_total[{{.window}}]))'
alerting:
page_alert: { labels: { severity: page } }
ticket_alert: { labels: { severity: ticket } }Sloth generates the alert rules: page on a fast burn (using budget too quickly), ticket on a slow burn. The alerts say “you’re going to be out of budget in X hours if this continues,” which is meaningful.
A naive “alert when SLI is below SLO” fires on every blip. The better pattern is multi-burn-rate: alert when you’re burning budget too fast relative to the time window.
The math: if your SLO is 99.9% (0.1% error budget), a burn rate of 14× means you’d consume the entire 30-day budget in ~50 hours. That’s the threshold for a fast-burn page.
Sloth, Pyrra, Nobl9 all generate these alerts from a simple SLO definition.
For queue consumers:
For batch jobs:
For a database:
The pattern is the same: pick “did it work” and “fast enough.”
1. Tracking too much. 30 SLIs across 10 services, nobody knows which to look at first. Two SLIs per service. Roll up across services for an org-level view.
2. SLO too tight. “We aim for 99.99%” but the team is paged every 3 days. The SLO is wrong; nobody trusts it; it gets ignored. Set the SLO at “what we can credibly hit,” not “what we wish we could.”
3. Ignoring the rule. SLO violated, error budget gone, the team ships a new feature anyway because there’s a deadline. Now the SLO is theatre. The rule is the value; if you can’t enforce it, the SLO is decoration.
4. SLO based on infrastructure, not user impact. “99.9% pod uptime” is not what users care about. Measure user-facing requests, not internal pod health.
A common question: how big should the team be before SLOs are useful? Roughly: when there are enough services that prioritization between “feature work” and “reliability work” needs a process. Below that, say 5 engineers, SLOs are overkill.
For a 20-engineer org, SLOs help. For a 100-engineer org, they’re essential. For a 5-engineer startup, just have good monitoring and trust.
Two SLIs per service. SLO at “users notice and complain” level. Error budget determines what the team works on next. Quarterly review. Multi-burn-rate alerts so you know when budget is being spent fast.
That is the whole system. Anything more is process the team will eventually drop. Anything less is dashboards without consequence. The middle is the version that pays back.
The kind of reliability process that prevents “we hope it’s reliable” from masking a real problem (SLOs, error budgets, the cadence that ties them to engineering investment) is the kind of long-haul engineering culture Yojji’s teams put into the platforms they ship for clients.
Yojji is an international custom software development company founded in 2016, with teams across Europe, the US, and the UK. They specialize in the JavaScript ecosystem (React, Node.js, TypeScript), cloud platforms (AWS, Azure, GCP), and full-cycle product engineering, including the SRE-adjacent practices that decide whether reliability is measured or just hoped for.
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。