What follows is the incident-management model as I learned it from Jesse Robbins, written down from memory before I lose any more of it. Any omissions or confusions are mine. I learned it at the feet of the master, but that doesn’t mean I grokked everything he was laying down.
I have run a lot of incidents, I have made every mistake described below at least once, and I have watched this model — applied with discipline — turn 3 AM chaos into something that survives shift change and produces a usable postmortem. It is opinionated. It is rigid. The rigidity is the feature.
The framework described here did not start in software. It started in the US fire service, as the Incident Command System (ICS) — developed in the 1970s after a series of catastrophic California wildfires in which the failure was not the fire itself but the coordination between agencies responding to it. Different departments arrived with different vocabularies, different chains of command, and no shared notion of who was in charge. People died because of organizational confusion, not because the fire was unfightable.
ICS was the response. It is a standardized, role-based command structure that scales from a single-engine response up to a multi-agency, multi-jurisdiction disaster. It is now codified federally as part of the National Incident Management System (NIMS), taught through FEMA’s ICS-100 / 200 / 300 / 400 course series, and used by every fire department, EMS service, and emergency management agency in the United States.
Jesse Robbins brought ICS into web operations during his time at Amazon, where his title was “Master of Disaster” and he was responsible for the availability of every property bearing the Amazon brand. Robbins is a volunteer firefighter and emergency manager — he deployed on the federal task force to Hurricane Katrina — and he distilled the operational discipline of the fire service into a model that the technology industry has been adopting ever since. He went on to co-found the Velocity Conference with Tim O’Reilly, where this model was popularized, and to found Chef (the configuration-management company). His framework is the upstream source of the incident response processes you now see at PagerDuty, Google SRE, Atlassian, and most mature operations organizations.
What follows is that model. It is opinionated. It is deliberately rigid. The rigidity is the feature. What’s more, it works.
The role list below is the web-ops adaptation. The full ICS structure includes additional sections (Planning, Logistics, Finance/Admin) that rarely map to technology incidents — those exist for multi-day, multi-agency physical-world responses.
The single accountable leader of the response. The IC owns the call, sets objectives, allocates resources, makes decisions, and decides when the incident is over.
The IC does not fix the problem. This is the most important and most-violated rule in the entire framework. The moment the IC starts typing into a terminal, opening a dashboard, or running a query, they have stopped being IC and no one is. The IC’s job is coordination, not execution. They direct; they do not do.
The IC is also responsible for declaring the incident, for calling severity, and for calling resolution. These are not group decisions. They are IC decisions. The IC may consult, but the IC decides.
The IC role rotates by shift. A long incident has multiple ICs handing off to each other. The hand-off is formal: the outgoing IC briefs the incoming IC on state, open questions, and current objectives. Nothing about the response should depend on a specific human being remaining awake.
The IC’s backup and shadow. The Deputy:
In smaller incidents, the Deputy role may be folded into the Scribe. In larger incidents it should be its own person.
Maintains the authoritative timeline of record for the incident. Everything else downstream — the postmortem, customer communications, regulatory reporting, legal review — flows from this artifact.
The Scribe records:
The Scribe does not interpret. The Scribe records. Interpretation happens later, in the postmortem, with the timeline as evidence.
The Scribe role is often given to a newer engineer as a way to expose them to incident response without putting them on the hook for decisions. This is a legitimate use of the role, but the Scribe must still be trained — bad timekeeping during an incident produces a useless postmortem.
The person actually working on the problem. The hands on the keyboard. There can be multiple SMEs in a single incident, one per affected domain (database, network, application, third-party, etc.).
The SME’s responsibilities:
SMEs should not be the IC simultaneously. The cognitive load of fixing the problem is incompatible with the cognitive load of coordinating the response.
External-facing communications. Owns the status page, customer email notifications, social media posts, and any other channel by which customers and the public learn what is happening.
The Communications Lead:
This role exists so that the IC and the SMEs do not have to context-switch between fixing the problem and explaining the problem.
Internal-facing communications. Updates executives, posts in internal channels, pages additional responders when the IC asks, and mobilizes adjacent teams.
Like the Comms Lead, the Internal Liaison posts on a defined cadence — typically every hour, set by the IC, more often during active escalation. The audience is different (executives, support, adjacent engineering teams rather than customers) but the discipline is the same: regular heartbeat updates so the audience does not need to ping the response team for status.
The Internal Liaison’s most important function: insulating the IC from “what’s going on?” pings. When a VP DMs the IC during a Sev-1, the IC should not be answering. The Internal Liaison handles it, refers the VP to the appropriate channel or status update, and lets the IC keep coordinating.
Three of the roles above involve writing things down during an incident, and the distinction between them is deliberate and important. They are not the same person and they are not the same artifact.
Same act in the abstract — write things down — but three different audiences, three different artifacts, three different people. Collapsing any two of them into one role is a common antipattern: the Scribe ends up drafting customer messaging instead of recording the timeline, or the Comms Lead gets pulled into briefing an executive and stops posting to the status page. Keep them separate.
The six-role model is what you grow into. A team of five engineers cannot field six distinct responders on a Tuesday afternoon, and a Sev-3 doesn’t require it. This section is placed here because “how do we do this with our small team?” is the first question every team asks after seeing the roles.
The severity terms used below (Sev-1 through Sev-4) are formally defined in the next section. For now, read them as a rough scale: Sev-1 is catastrophic, Sev-4 is minor.
A team starting from zero rarely lands all six roles on day one. Trying to is the surest way to get the framework rejected as “too heavy.” Roll out in stages:
Each stage builds on the previous one. Skipping ahead — declaring on day one that the team will adopt all six roles — produces a framework that exists on paper but isn’t followed in practice, which is worse than no framework at all.
The first failure of most teams new to incident management is not the response — it’s the declaration. People work a problem in their team channel for two hours without ever acknowledging that an incident is in progress. No IC, no Scribe, no cadenced updates, no externally-visible response. By the time anyone formally declares, the response has already failed: the timeline is incomplete, customers have been silent for hours, executives find out from outside the company.
Declaration is cheap. Not declaring is expensive. The default is to declare and downgrade if wrong, not to wait and upgrade when forced.
The IC sets severity at declaration and updates it as scope clarifies. Use the highest severity supported by current evidence; downgrade as facts emerge, not as hope emerges. A team-specific severity rubric is mandatory — without one, the IC has no anchor and severity calls become political.
A starting template, to be customized to your business:
| Sev | Trigger | Response posture |
|---|---|---|
| Sev-0 | Company-existential event. Active data breach, ransomware, or exfiltration in progress. Total platform outage with no workaround and no ETA. Imminent regulatory disclosure clock running. Active safety event affecting customers or employees. | Full company mobilization. CEO, Legal, and PR engaged from minute one. Every external statement legal-reviewed. Regulatory and disclosure clocks tracked explicitly by the IC. Engineering is one function in a multi-function response, not the lead. |
| Sev-1 | Major customer-visible outage. Significant revenue impact. Multiple regions or business units affected. Material but not company-existential. | Full role assignment, all six roles distinct people. Executive notification immediately. Round-the-clock response until resolved. |
| Sev-2 | Significant customer-visible degradation affecting a meaningful share of users, OR a critical internal system down. | Full role assignment, some combinations acceptable. Executive notification within an hour. Business-hours-plus response. |
| Sev-3 | Limited customer impact (specific feature, geography, or customer segment) OR significant internal degradation not blocking customer-facing systems. | IC + SME minimum, plus Scribe. Executive notification at next regular cadence. Business-hours response. |
| Sev-4 | Minor degradation, mostly internal, low or no customer impact. | IC + SME. Async response. Documented and tracked, but no urgent paging. |
The severity table is the most context-dependent artifact in the framework. What counts as Sev-1 at a payments company is different from what counts as Sev-1 at an internal tooling company. Set yours explicitly, document it, post it where on-call can see it without searching.
On Sev-0 specifically. The defining feature of Sev-0 is not “really bad” — Sev-1 already covers really bad. It is that the response posture is qualitatively different: CEO, Legal, and PR are engaged from minute one; external statements are legal-reviewed before they go out; regulatory disclosure clocks (GDPR breach notification, SEC material-event disclosure, contractual customer notification windows) are running and tracked. Engineering is no longer driving the response — it is one team in a multi-function company-wide response, with the IC coordinating against a designated executive sponsor. Sev-0 exists as its own tier because the playbook is fundamentally different from any other severity, not because the impact is fractionally larger. If your org doesn’t have customer data, regulatory exposure, or safety obligations that would trigger this posture, you may not need Sev-0 — but most production businesses do.
Declare an incident when any of the following is true:
The first responder declares. Anyone can declare. Declaring is not an admission of fault and does not require IC approval — you declare because you need an IC.
Bug reports that don’t meet the criteria above are bugs, not incidents. Performance reports without measurable impact are tickets, not incidents. A single customer complaint about an isolated issue is a support case, not an incident. The discipline of not declaring for non-incidents is what keeps the team responsive when a real one hits — incident fatigue is real, and declaring too liberally trains people to ignore the page.
Where does the incident actually happen? This is the most boring part of the discipline and the one that determines whether the framework survives contact with reality. A team without explicit operational mechanics will try to run a Sev-1 in their normal #engineering channel with PMs asking unrelated questions and executives interrupting — and the response will fail accordingly.
#inc-YYYY-MM-DD-short-description. Examples: #inc-2026-05-20-db-degraded, #inc-2026-05-20-auth-outage. Dated and descriptive names are searchable and unambiguous later.If those ten steps happen in the first ten minutes, the response is on rails. If they don’t, the response is improvising under pressure — which is exactly the failure mode the framework exists to prevent.
The framework assumes a baseline of scaffolding exists. A team starting from zero should put these in place in parallel with adopting the role model — without them, the framework cannot function:
You do not need expensive tooling. You do need some tooling. The minimum viable stack — Slack workspace, a paging tool, a status page service — costs a few hundred dollars a month for a small team. The cost of not having it is measured in incidents that go undeclared, customers who learn about outages from Twitter, and postmortems that nobody can write because the timeline is gone.
There is exactly one Incident Commander at any moment. Not two. Not “co-ICs.” If the IC needs to hand off, the hand-off is explicit and announced: “Acknowledging hand-off. I am now IC. Confirm.” The previous IC confirms, and from that moment forward the new IC is in charge.
This sounds bureaucratic. It exists because in real incidents, fatigue, time zones, and escalation paths mean ICs do change hands, and ambiguity about who is in charge is a known cause of bad outcomes. The fire service learned this lesson at terrible cost. The discipline transfers directly.
The IC commands. The SMEs execute. The IC does not execute and the SMEs do not command. This separation is what allows the response to scale beyond a single very-smart person heroically holding everything in their head.
When command and execution collapse into one person, two failure modes become inevitable:
Neither of these is what you want during a real outage.
If it is not in the Scribe’s timeline, it did not happen. This is a forcing function. It pushes responders to surface their actions through the chain — “telling the Scribe” is functionally the same as “telling the IC” — and it produces an artifact that survives the incident.
Postmortems written from chat-log scrollback are bad postmortems. Postmortems written from a Scribe’s timeline are good postmortems. The difference is the quality of the next incident’s prevention.
Cadenced posting is a discipline that applies to every role whose job is keeping an audience informed — not just the Comms Lead. The IC sets the cadence for each audience, and the cadences are different:
None of these posters stop posting when “nothing has changed.” A heartbeat of “still investigating, no new information, next update in 30 minutes” is itself a piece of information: it tells the audience that the response is active and they do not need to ping the response team for status. The Scribe is the only documentation role that does not operate on a cadence — they write continuously as events occur, because their job is the timeline of record, not audience messaging.
Severity is not a vote. Resolution is not a vote. The IC declares them based on impact and information. The IC may be wrong; the IC is still the one who declares. Disagreement gets surfaced to the IC, who decides. This is faster than consensus and almost always at least as accurate.
Every cadenced update and every status report — from an SME to the IC, from the outgoing IC to the incoming IC at hand-off, from the IC to the response channel at the top of the hour — should follow the CAN format. CAN is another direct lift from the fire service: it stands for Conditions, Actions, Needs, and it forces a complete, concise status into three slots.
Conditions — what is the situation right now, in observed facts? Not what you think is happening. Not what might be happening. What is measurably true at this moment. Metrics, error rates, customer reports, affected systems, scope of impact.
Actions — what are you currently doing about it? Not what you plan to do, not what you considered. What is in flight right now. What has already been completed.
Needs — what do you need to keep going? Resources, decisions, access, information, additional responders, IC approval for a specific action. This is the slot responders most often skip, and it is the slot that most often determines whether the response moves forward or stalls.
Each slot exists because it addresses a known failure mode in status reporting:
The three slots together produce a status report that the IC can act on immediately — they know the situation, they know what’s being done, and they know what decisions they’re being asked to make.
C: Database primary CPU at 95% sustained, replication lag at 4 minutes and growing, p99 latency 8 seconds against 200ms baseline. Customer-facing impact confirmed in EU region; US still nominal. Started 14:32 UTC, correlates with deploy at 14:30.
A: Provisioning a read replica, ETA 6 minutes. Pulling query log for the window around the deploy to identify the offender. Paged the application team for a deploy-side rollback option.
N: Need IC approval to fail over to the secondary if the read replica does not reduce primary load. Need someone from the application team to confirm whether the 14:30 deploy added new query patterns — Jeff is on the page but hasn’t responded yet.
This takes about ninety seconds to deliver verbally. The IC now has everything they need: situational awareness, knowledge of work in flight, and a clear list of decisions they own.
CAN reports are the default format for:
The discipline is small. The payoff is that every status update is immediately actionable, every hand-off is complete, and the Scribe’s timeline captures structured information rather than ad-hoc chatter.
These are the failure modes you’ll see most often in organizations that haven’t adopted ICS — and, frankly, in many that have.
The IC is also the best debugger, so the IC is debugging. Already covered. This is the #1 violation. The fix is cultural: senior engineers must be willing to step back from the keyboard during an incident and let less-senior people drive the actual fix, while they coordinate. This is harder than it sounds and is the single biggest skill that distinguishes mature incident response from immature.
No declared IC. A bunch of engineers in a channel typing furiously, with no one explicitly in charge. Symptoms: duplicated investigation, missed escalations, no timeline, no external communication, a postmortem written from chat scrollback. Fix: the first responder declares themselves IC (“I’m IC for this incident”) and immediately starts assigning roles. Anyone can be IC; the role just has to exist.
The IC is taking notes themselves. No Scribe assigned, so the IC tries to do both. The notes are bad and the coordination is bad. Fix: assign a Scribe within the first five minutes, even if it’s the most junior person on the call.
Engineers freelancing. SMEs investigating independently, not surfacing findings to the IC, running fixes without authorization. Symptoms: contradictory mitigations, services restarted twice, hypothesis disproven by one engineer being re-investigated by another an hour later. Fix: IC explicitly recognizes each SME, asks for explicit status, and routes work.
Executives in the response channel. VP shows up in the incident channel and starts asking questions. The IC tries to answer and stops being IC. Fix: Internal Liaison redirects the executive to an exec-updates channel or to the cadenced status updates. The IC is unreachable to anyone except the response team.
No formal hand-off. Shift change happens by osmosis. The new on-call shows up and starts firefighting; the old on-call drifts away. State is lost. Fix: explicit hand-off briefing, even if it takes ten minutes. Outgoing IC briefs incoming IC on state, open actions, current objectives, and pending decisions.
“All-clear” called too early. Symptoms go away, IC declares resolution, problem recurs an hour later, second incident is opened. Fix: the IC distinguishes between mitigation (the immediate symptom is gone) and resolution (the root cause is understood and addressed). Mitigation may end the customer-facing incident; resolution is what ends the work.
The incident response framework is half of incident management. The other half is the learning loop: every incident produces a postmortem, every postmortem produces action items, and every action item gets tracked to completion. A team that responds well to incidents but never learns from them will keep incidenting on the same root cause indefinitely.
A postmortem is a structured document and review process that turns the incident into organizational learning. Blameless is the operating principle: the goal is to understand what happened and prevent recurrence, not to identify whom to punish.
Blameless does not mean “no accountability.” Engineers remain accountable for their work and for following process. Blameless means that the postmortem assumes good faith and competence on the part of every responder, and asks what about the system — tools, alerts, runbooks, documentation, process, training — allowed the incident to occur or escalated its impact. The question is “why did this make sense to a competent person at the time?” not “who screwed up?”
The reasoning is operational, not sentimental. In an organization where postmortems are punitive, people hide information, contest findings, and refuse to participate honestly. In an organization where postmortems are blameless, people surface near-misses, admit confusion, and contribute fully. The blameless organization learns faster and incidents less. This is well-established in the SRE literature and is not optional in any team serious about reliability.
A postmortem document contains, at minimum:
The Scribe’s timeline and the postmortem are different artifacts and the distinction matters. The timeline is evidence — comprehensive, factual, untouched by interpretation. The postmortem is interpretation — selective, analytical, drawing conclusions from the timeline. A good postmortem cites the timeline; it does not replace it. Both artifacts get preserved. If the postmortem is ever contested — by Legal, by a customer, by a regulator — the timeline is what defends it.
Postmortems produce action items. Action items go to die.
This is the single most common failure mode of mature-looking incident response programs: the response is well-run, the postmortem is well-written, and then the action items languish in a backlog that nobody owns until the next incident proves they were never done. The discipline of follow-through is what closes the loop, and without it, the framework is theater.
The mechanics:
A team that runs three good incident responses and lets the action items rot has gained nothing. A team that runs three mediocre responses and closes every action item from the postmortems will, in six months, be running good responses and incidenting less.
Adoption is not a tool purchase. It is a cultural change, and it takes practice. The fire service spends a significant portion of its training time drilling on ICS specifically because the discipline does not survive contact with a real fire unless it has been rehearsed.
The technology analog is Game Days — Robbins’ other contribution to the field. A Game Day is a planned, controlled incident. You break something in a non-production environment (or a carefully chosen production component) and you run the full response — IC, Deputy, Scribe, SMEs, Comms, Internal Liaison — as if it were real. You write a real timeline. You write a real postmortem. You evaluate the response, not the technical fix.
Game Days are how the muscle gets built. Without them, the first time the team uses ICS will be at 3 AM during a real Sev-1, which is the worst possible time to be learning role discipline. Run Game Days. Run them often. Rotate roles so that everyone has been IC and everyone has been Scribe.
Incident Management for Operations — Schnepp, Vidal, Hawley (O’Reilly, 2017). Robbins wrote the foreword. This is the closest thing to a definitive written treatment of his approach. If you read one thing, read this.
Google SRE Book, Chapter 14: Managing Incidents — Free online. Same lineage, Google’s naming and operational specifics. Worth reading alongside the O’Reilly book to see how the model adapts to different organizational shapes.
Google SRE Book, Chapter 15: Postmortem Culture: Learning from Failure — The canonical statement of blameless postmortem culture in the SRE tradition. Read this alongside the postmortems section above.
John Allspaw — “Blameless PostMortems and a Just Culture” — The 2012 Etsy essay that established blameless postmortems as a discipline in web operations. Short, foundational, still relevant. Allspaw and Robbins were both at Velocity during the same window and the two threads (incident response + blameless review) developed in parallel.
PagerDuty Incident Response Documentation — PagerDuty was an early adopter of the Robbins-flavored model and open-sourced their full process. This is the most accessible public codification.
Atlassian: Incident Response Roles and Responsibilities — Atlassian’s variation, also Robbins-influenced.
FEMA ICS-100 Course Material (PDF) — The upstream source. Free, federal, comprehensive. Worth understanding if you want to know why the roles are shaped the way they are.
Incident Command System — Wikipedia — Overview of the full ICS structure, including the sections (Planning, Logistics, Finance) that the web-ops adaptation typically omits.
Jesse Robbins’s site and his Topics page — Robbins’s personal site (bio, portfolio, mentions) and his topical writing on Master-of-Disaster work, Game Days, DevOps history, and the ops culture lineage this framework comes from.
Jesse Robbins — Wikipedia — Background on Robbins himself: Amazon “Master of Disaster,” Chef co-founder, Velocity co-founder, volunteer firefighter, Katrina task force.
Robbins’ Velocity Conference talks — Available through the O’Reilly Velocity channel. His talks on operations culture and the fire-service-to-IT translation are the original source material in spoken form.
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。