The team had an outage Tuesday. The postmortem doc was opened Wednesday, written Thursday, presented Friday, and forgotten by the following Monday. Three months later, the same incident happens with the same root cause, because the action items lived in a Google Doc nobody read.
A useful postmortem is not the document. It is the process around the document: the questions asked, the system-level conclusions drawn, the action items that get owned and shipped. Most teams have the document right and the process wrong, which is why the same incidents keep happening.
This post is the template that has worked for me on multiple teams, plus the six rules that distinguish “incident analysis that improves the system” from “an artifact for the records.”
# Incident Postmortem: <short summary>
## Status
- **Severity:** SEV1 / SEV2 / SEV3
- **Duration:** 14:32 - 15:47 UTC (1h15m)
- **Customer impact:** ~3% of users saw 5xx on /api/checkout for 35 min
- **Author:** @engineer (incident commander during)
- **Reviewers:** @teamlead, @sre
## Summary
Two paragraphs max. What happened, what was the impact, what was the trigger.
## Timeline
| Time (UTC) | Event |
|---|---|
| 14:25 | Deploy of api-svc v2.43 begins |
| 14:30 | First 5xx spike observed in dashboards |
| 14:32 | PagerDuty fires; @oncall acknowledges |
| 14:38 | @oncall identifies recent deploy as likely cause |
| 14:42 | Rollback initiated |
| 14:48 | Rollback complete; 5xx rate normalizing |
| 15:47 | All metrics back to normal; incident declared resolved |
## Root cause
What changed in the system that caused this. Not "human error."
Be specific: which line, which config, which assumption.
## Why we didn't catch it sooner
Why did monitoring not fire earlier? Why did testing not catch this?
Why did the deploy succeed when it should have failed?
## What went well
Yes, really. The team responded fast, the rollback was clean, etc.
Most postmortems skip this and miss reinforcing what worked.
## What didn't go well
Concrete failures. The dashboard for X took 10 minutes to load.
The runbook for Y was outdated. Etc.
## Lessons learned
The system-level takeaways. What does this incident reveal about
our architecture, processes, or assumptions?
## Action items
| # | Action | Owner | Due | Status |
|---|---|---|---|---|
| 1 | Add unit test covering the bad config path | @alice | 2024-04-30 | Open |
| 2 | Update deploy gate to verify config before rollout | @bob | 2024-05-15 | Open |
| 3 | Add SLO alert for 5xx ratio > 0.5% (currently > 1%) | @carol | 2024-05-08 | Open |
## Related incidents
Links to similar past incidents.The format is not magic. The discipline of filling each section honestly is what matters.
1. Blameless means systems-focused, not “no consequences.”
A blameless postmortem doesn’t mean “Alice merged the bad PR” is forbidden. It means the question is “why did our system make it possible for Alice to merge the bad PR,” not “Alice should be more careful.” Action items target the system, not the person.
The right question is “what would have made it impossible for anyone to do this?”
The wrong question is “who is responsible?”
2. Root cause is the change, not the symptom.
“Database was slow” is a symptom. “Connection pool was exhausted” is closer. “We released a feature that opens a connection per request and never returns it” is root cause.
Keep asking “and why did that happen” until you hit a system property that can change. Often three to five whys deep.
3. Action items must have owners and dates.
A postmortem with action items like “improve testing” and no owner is a postmortem that produced nothing. Each action item needs:
If you can’t write all three, the action is too vague. Refine.
4. Action items get tracked like any other work.
After the postmortem, action items go into the team’s normal task tracker (Linear, Jira, GitHub Issues). Not into a “postmortem follow-ups” graveyard that nobody opens. They are reviewed in the next sprint planning like any other work.
The metric: how many postmortem action items were completed within their due date. A healthy team is above 80%. Below 50%, postmortems are theatre.
5. Share the postmortem widely.
The team that had the incident learns. But the next team has similar systems and similar risks. Share postmortems org-wide, in a channel everyone reads, with clear summaries.
This is also how cultural norms spread. New teams see what a good postmortem looks like by reading existing ones.
6. Read your old postmortems.
Once a quarter, scan your incident history. Are the same root causes recurring? Are action items shipping? Has anything actually improved?
If you cannot point to a specific reliability win in the last quarter that came from postmortem follow-ups, the process is broken. Probably action items aren’t being completed or aren’t ambitious enough.
A real example of drilling down:
Q: Why did the API return 5xx? A: Database connections were exhausted.
Q: Why were connections exhausted? A: A new endpoint we deployed opens a connection per request and never closes it.
Q: Why did that ship to prod? A: Code review missed it; there’s no automated test for connection lifetime.
Q: Why is there no test? A: We don’t have an integration-test setup that catches resource leaks.
Q: Why don’t we have one? A: Building it would take a week and we’ve been deferring it.
Action items emerge naturally from each “why”:
Outages are also detection failures. Always include a “why didn’t we catch this sooner?” section. Common findings:
These are typically the easiest, highest-leverage action items. Fixing detection is cheap. Fixing the underlying system is harder.
Three antipatterns to call out and avoid:
“We learned a lot from this.” A postmortem that identifies no concrete actions has not learned anything. The action items are the learning.
“We need to improve communication.” Vague. What specifically needs to change? Who owns it?
“This was a freak occurrence; nothing to do.” Almost never true. The Swiss-cheese model: every incident is multiple holes lining up. Even rare incidents tell you about the holes.
A weekly schedule that prevents postmortem rot:
Don’t let the document drag on for two weeks. The longer it sits, the less anyone remembers, the less useful it becomes.
A postmortem is a tool for systemic improvement, not an artifact. The template is straightforward; the discipline around it is what matters. Blameless means systems-focused. Root cause is the change, not the symptom. Action items have owners and dates and get tracked. Share widely. Re-read quarterly.
The team that does this well shows up at the next incident response with the muscle memory built from previous postmortems. The team that doesn’t has the same incident every quarter.
The kind of incident-management discipline that turns “we had an outage” from a recurring drama into a cycle of measurable improvement (postmortems with owned action items, shared learnings, quarterly reviews) is the kind of long-haul engineering culture Yojji’s teams put into the products they ship for clients.
Yojji is an international custom software development company founded in 2016, with teams across Europe, the US, and the UK. They specialize in the JavaScript ecosystem, cloud platforms (AWS, Azure, GCP), and full-cycle product engineering, including the operational practices that decide whether reliability improves over time or stays the same.
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。