惯性聚合 高效追踪和阅读你感兴趣的博客、新闻、科技资讯
阅读原文 在惯性聚合中打开

推荐订阅源

V
Visual Studio Blog
OSCHINA 社区最新新闻
OSCHINA 社区最新新闻
N
Netflix TechBlog - Medium
博客园 - 叶小钗
大猫的无限游戏
大猫的无限游戏
S
SegmentFault 最新的问题
V
V2EX
IT之家
IT之家
J
Java Code Geeks
Hacker News - Newest:
Hacker News - Newest: "LLM"
cs.AI updates on arXiv.org
cs.AI updates on arXiv.org
GbyAI
GbyAI
D
Docker
S
Secure Thoughts
Recent Announcements
Recent Announcements
Webroot Blog
Webroot Blog
Application and Cybersecurity Blog
Application and Cybersecurity Blog
云风的 BLOG
云风的 BLOG
博客园_首页
cs.CV updates on arXiv.org
cs.CV updates on arXiv.org
Cyber Security Advisories - MS-ISAC
Cyber Security Advisories - MS-ISAC
Security Archives - TechRepublic
Security Archives - TechRepublic
酷 壳 – CoolShell
酷 壳 – CoolShell
让小产品的独立变现更简单 - ezindie.com
让小产品的独立变现更简单 - ezindie.com
N
News | PayPal Newsroom
S
Security @ Cisco Blogs
I
InfoQ
Last Week in AI
Last Week in AI
SecWiki News
SecWiki News
钛媒体:引领未来商业与生活新知
钛媒体:引领未来商业与生活新知
W
WeLiveSecurity
T
Troy Hunt's Blog
Recent Commits to openclaw:main
Recent Commits to openclaw:main
H
Hackread – Cybersecurity News, Data Breaches, AI and More
Attack and Defense Labs
Attack and Defense Labs
美团技术团队
T
The Blog of Author Tim Ferriss
Google DeepMind News
Google DeepMind News
Martin Fowler
Martin Fowler
B
Blog
CTFtime.org: upcoming CTF events
CTFtime.org: upcoming CTF events
Scott Helme
Scott Helme
T
Tor Project blog
Know Your Adversary
Know Your Adversary
有赞技术团队
有赞技术团队
Hugging Face - Blog
Hugging Face - Blog
Recorded Future
Recorded Future
C
Cyber Attacks, Cyber Crime and Cyber Security
AI
AI
G
Google Developers Blog

DEV Community

Authentication Security Deep Dive: From Brute Force to Salted Hashing (With Java Examples) Why AI Systems Don’t Fail — They Drift Spilling beans for how i learn for exam😁"Reinforcement Learning Cheat Sheet" I Replaced Chrome with Safari for AI Browser Automation. Here's What Broke (and What Finally Worked) How Python Borrows Other People's Work The $40 Architecture: Processing 1 Billion API Requests with 99.99% Uptime Vibe Coding: A Workflow Guide (From Zero to SaaS) Most webhook security guides protect the wrong side. The scary part is delivery. Headless CMS for TanStack Start: Build a Blog with Cosmic EU Age Verification App "Hacked in 2 Minutes" — What Actually Happened Comfy Cloud’s delete function does not actually remove files Running AI Models on GPU Cloud Servers: A Beginner Guide Event-driven media intelligence with AWS Step Functions and Bedrock I scored 500 AI prompts across 8 quality dimensions — here's what broke How to Call Google Gemini API from Next.js (Free Tier, No Backend Needed) The Portal Protocol: Reclaiming Human Connection in the Age of AI How to Fix Your Team's Scattered Knowledge Problem With a Self-Hosted Forum Intro to tc Cloud Functors: A Graph-First Mental Model for the Modern Cloud Designing Multi-Tenant Backends With Both Ownership and Team Access I Built a Neumorphic CSS Library with 77+ Components — Here's What I Learned PostgreSQL Performance Optimization: Why Connection Pooling Is Critical at Scale Cómo construí un SaaS multi-rubro para gestionar expensas en Argentina con FastAPI + Vue 3 🚀 I Built an Ethical Hacking Scanner Tool – Open Source Project I Replaced /usage and /context in Claude Code With a Single Statusline A Pythonic Way to Handle Emails (IMAP/SMTP) with Auto-Discovery and AI-Ready Design I Collected 8.9 Million Polymarket Price Points — Here's What I Found About How Markets Really Move EcoTrack AI — Carbon Footprint Tracker & Dashboard Everyone's Using AI. No One Agrees How. 5 self-hosted ebook managers worth trying in 2026 Building Your First AI Agent with LangChain: From Chatbot to Autonomous Assistant Common SOC 2 Failures (Real World) Stop Vibe-Checking Your AI App: A Practical Guide to Evals How to Use SonarQube and SonarScanner Locally to Level Up Your Code Quality Your Next To-Do App Is Dead — I Replaced Mine with an OpenClaw AI Sign a Nostr event in 60 lines of Python using coincurve — no nostr-sdk, no nbxplorer, no rust toolchain ITGC Audit Explained Like You’re in Big 4 Patch Tuesday abril 2026: Microsoft parcha 163 vulnerabilidades y un zero-day en SharePoint Stop scraping everything: a better way to track competitor price changes Listing on MCPize + the Official MCP Registry while routing payments OUTSIDE the marketplace — how I kept 100% of my x402 revenue Building an AI-Powered Risk Intelligence System Using Serverless Architecture Why We Ripped Function Overloading Out of Our AI Toolchain Testing AI-Generated Code: How to Actually Know If It Works SaaS Churn Is Killing Your Business. Here Is What to Do About It (Without a Support Team) The Speed of AI Is No Longer Linear - And Self-Improving Models Are Why How to Implement RBAC for MCP Tools: A Practical Guide for Engineering Teams From Standard Quote to Persuasive Proposal: AI Automation for Arborists I built a CLI that scaffolds complete multi-tenant SaaS apps Axios CVE-2025–62718: The Silent SSRF Bug That Could Be Hiding in Your Node.js App Right Now The dashboard that ended our friendship Data Pipelines Explained Simply (and How to Build Them with Python) The Hidden Cost of AI Systems Nobody Talks About. undefined vs undeclared, and how typeof behaves Switching from file-based jobs to NATS/Kafka in Rust without changing code io_uring Adventures: Rust Servers That Love Syscalls Why Agentic AI is Killing the Traditional Database The POUR principles of web accessibility for developers and designers Quantum Neural Network 3D — A Deep Dive into Interactive WebGL Visualization How To Install Caveman In Codex On macOS And Windows Automation Pipeline Reliability: Why Your Workflow Breaks When Nobody Is Watching I Built an 'Open World' AI Coding Agent — It Works From ANY Folder From Freelancing to Product: A Tech Service Company's SaaS Transformation China's AI Giants: Adding Tencent Hunyuan & ByteDance Doubao to AI University (74 Providers) On the Vibe Coders and Their Lies clerk: Auto-Summarize Your Claude Code Sessions AI Weekly — 2026/04/10–04/17 | The Model Lockdown Is Here, but the Toolchain Is the Real Battleground AI 週報 — 2026/04/10–2026/04/17 模型封鎖潮來了,但工具鏈才是真戰場 Maybe this is how Open-Source apps are born... 🚀 Fine-Tune LLMs with LoRA and QLoRA: 2026 Guide tRPC v11 + Next.js App Router: End-to-End Type Safety Without the Boilerplate ShadCN UI in 2026: Why I Stopped Installing Component Libraries and Started Owning My Components SaaS Billing in React Server Components: Stripe + Supabase Without a Single `useEffect` Join our DEV Weekend Challenge — $1,000 in Prizes Across TEN winners! Submissions Due April 20 at 6:59 AM UTC. Implementing FSRS Spaced Repetition in Flutter + Supabase — Adding Memory Science to an AI Learning App "I Texted My Localhost From the Train — Claude Code Fixed the Bug Before I Got Home" I Built a Sales Prep AI and It Went Deeper Than Expected Design to Code #2: One JSON, Eleven Outputs Solving the 100M-Row Problem: A Summary Table Pattern for High-Volume Push Notification Logs Flutter Web With Wasm: What Actually Changes For Developers I Built 50 Royalty-Free Soundtracks for My Side Project in a Weekend Using AI Music Generation The Vibe Coding Security Checklist: 7 Things to Check Before You Ship Stop Letting Googlebot Guess Fix Your React App's SEO Right Desconstruindo o Streaming do LinkedIn: Como Criar um Engine de Extração de Vídeo de Alta Performance com HLS e FFmpeg (EDA Part-1) EDA (Exploratory Data Analysis) Explained With Real Life — Why Looking at Your Data Is the Most Important Step in Machine Learning Brand Relationship Management at Scale: Our 4-Touch Outreach System for 200+ Brands Why String.fromEnvironment() Might Return an Empty String in Dart JGuardrails 1.0.0 — Hardening Java LLM Apps Against Jailbreaks, Toxicity, and Prompt Injection Plan and Schedule a Full Week of Threads Content From One Claude Conversation Coding Cat Oran Ep3, Five Tables Changed Everything Updated: BFF Pattern I'm done watching freelancers get buried by 200 proposals. So I'm building the alternative. This is my first post BFS Algorithm in Java Step by Step Tutorial with Examples Tracking LLM Pricing Monthly: An Open Dataset for 22 AI Models How We Measure Content ROI on a Comparison Site: Revenue Attribution Without Perfect Data Introducing Nova AI Ops: The AI-Native Operating System for SRE Teams I built a free desktop video downloader for Windows — Grabbit How Talkie OCR Helps Vision-Impaired & Dyslexic Users Read the World Around Them VRCFaceTracking安装和iPhone面捕配置教程,有bug Even CrowdStrike Can't See Your Agents The Automation Gold Rush: What n8n Workflows and Claude Are Opening Up for Developers Right Now
Your Engineers Aren't Slow. Your incident response is. Here's Where the First 20 Minutes Actually Go
Jay Saadana · 2026-04-29 · via DEV Community

Your last P0 incident probably took 50 minutes to resolve. The fix itself? Likely under 10 minutes. A config rollback. A connection pool bump. A single kubectl command.

So where did the other 40 minutes go?

Not to engineering. To coordination. Jumping between tools, paging the right people, checking what changed, and trying to piece together the context from six different dashboards before anyone even starts debugging.

The data backs this claim up. An incident.io analysis of real-world P0 incidents found a typical MTTR breakdown of 12 minutes assembling the team and gathering context, 20 minutes troubleshooting, 4 minutes on actual mitigation, and 12 minutes on cleanup, meaning coordination consumes roughly 70% of the total resolution window while the actual repair takes a fraction of it. Separately, the Catchpoint SRE Report 2025 found that and operational toil rose to 30% of engineering time, up from 25% the first increase in five years. Splunk's State of Observability 2025 reported that 73% of organisations experienced outages related to ignored or suppressed alerts because their teams were drowning in noise.

The pattern is consistent across the industry and matches what we've seen firsthand: roughly 70% of incident response time goes to coordination, not engineering. Whether it's a PagerDuty report showing customer-impacting incidents increased 43% year-over-year, or incident.io's data showing that team assembly and cleanup alone consume half the resolution window, the bottleneck isn't your engineers. It's everything they have to do before they can start fixing.


Key Takeaways

  • ~70% of incident response time is coordination, not engineering. The fix is usually immediate. Getting to the solution takes 50 minutes.
  • The first 20 minutes are almost entirely logistics. Detection, assembly, and context gathering before a single engineer has looked at a log line with intent.
  • MTTR is a misleading metric. A 50 minute MTTR doesn't tell you if your team spent 40 minutes coordinating and 10 debugging, or the other way around. Same number, entirely different problems. Track where the time actually goes.
  • The highest-ROI improvements target coordination, not debugging. If 70% of your incident time is spent on people jumping between tools and paging each other, buying a faster APM will not help. Automate the assembly, pre-wire the context, and let your engineers go straight to the problem.
  • On-call burnout is a coordination problem. Your senior engineers aren't experiencing burnout due to the difficulty of the fixes. They're burning out because they're the only ones who can navigate across the tools effectively.

The Real Anatomy of a P0 Incident

So what does that 70% actually look like in practice? Here's the minute-by-minute breakdown of a typical P0 incident. The pattern was remarkably consistent across every team we spoke to.

  • Minutes 0–4: Detection. The alert fires. The on-call engineer acknowledges. If they're in a meeting or away from their desk, the delay alone eats four minutes.
  • Minutes 4–20: The Assembly Phase. This is where time goes away. The engineer opens Slack and posts in the incidents channel, but then they remember that they don't know who owns the checkout service. They have Datadog open in one tab and the deployment dashboard in another, and they're looking through GitHub commits to see if anyone pushed anything in the last hour. They haven't even started debugging yet with six tools open. They're just trying to figure out what's going on.
  • Minutes 20–34: Investigation. The actual diagnostic work begins, but it is hindered by coordination issues. Two engineers independently check if a recent config change caused the issue. One checks Elasticsearch logs, while the other checks Datadog logs, as they didn't coordinate. Meanwhile, Slack is buzzing with questions: "Is the issue related to the deploy we did at 2:30?" "Should we roll back?" "Do we need to update the status page?" A focused investigation of about five minutes reveals the actual engineering insight: "The connection pool size was reduced in the 2:30 config push." But that five minutes is buried inside fourteen minutes of tool-hopping and duplicated effort.
  • Minutes 34–40: The Fix. Almost always fast. Roll back the config. Bump the pool size. Push the change. Verify.
  • Minutes 40–50: Cleanup. Update the status page. Close PagerDuty. Post a summary. Create the post-mortem ticket. More coordination, zero engineering.

Here's what that looks like when you map every minute:

Why This Should Concern Engineering Leaders

The obvious cost is downtime. According to ITIC's 2024 Hourly Cost of Downtime survey, over 90% of mid-size and large enterprises report that a single hour of downtime costs more than $300,000, with 41% putting it between $1 million and $5 million. Gartner puts the average for Fortune 500 companies at $500,000 to $1 million per hour. But there's a quieter cost.

If your team handles 15 incidents per month with an average of 3 engineers per incident, and each one carries 39 minutes of coordination overhead, that's roughly 29 engineer-hours per month nearly four full engineering days spent not on diagnosis, not on the fix, but on opening Slack channels, paging people, and checking who deployed what.

And that calculation doesn't include context-switching costs. Each incident interruption costs 15–25 minutes to return to deep work afterward. The real productivity loss is multiples higher.

This cost falls disproportionately on your most experienced engineers the ones who know which signals matter, who own which service, and where to look first. When those engineers burn out and leave, they take that institutional knowledge with them. The next incident takes longer because the coordination phase expands.

MTTR hides all of these issues. A 50-minute MTTR doesn't tell you whether you spent 40 minutes on coordination and 10 on the fix or 10 on coordination and 40 on a genuinely challenging problem. These require entirely different solutions.

What You Can Do About It

The 70/30 split tells you exactly where to focus.

Pre-wire your incident response. Most coordination in the first 20 minutes comes from answering questions that should already have answers: Who owns this service? Who's on call? What changed recently? Where's the dashboard? A well-maintained service catalogue eliminates the "who do I page?" and "where do I look?" questions that consume the opening minutes.

Eliminate parallel tool-hopping. If your engineers are independently querying three different observability tools during an incident, you have a coordination problem. Assign roles explicitly: one person investigates logs, one checks deploys, one handles communication.

Automate the coordination layer. Creating channels, paging owners, and pulling context are almost entirely automated. Every minute your engineers spend on logistics during an active incident is a minute they're not diagnosing the problem.

Automate the investigation layer. This area is the frontier. The investigation phase remains time-consuming because it requires connecting the dots across tools mapping an error spike to a recent deploy, linking a latency increase to a config change, and grouping 30 cascading alerts into a single root cause. This kind of cross-tool correlation is exactly what AI is adept at.

At Steadwing, this type of cross-tool correlation is the problem we solve. When an alert fires, we pull context from your logs, metrics, traces, and recent code changes, connect the dots across your whole stack, and give you a full root cause analysis with automatable fixes on the code level, around deployment, and infra. The RCA investigation is over by the time the on-call person opens their laptop.

We handle the 70%, so your engineers can focus on the 30% that actually requires their expertise.

Frequently Asked Questions

Where does the 70% coordination figure come from?
We timed real incidents across multiple engineering teams and categorized every minute as either coordination (team assembly, tool-switching, communication, cleanup) or engineering (diagnosis, root cause identification, fix). The split consistently landed between 65–80% coordination. This aligns with publicly available incident data incident.io's MTTR breakdown shows coordination and investigation phases consume the majority of resolution time, while the Catchpoint SRE Report 2025 and Splunk State of Observability 2025 confirm that operational toil and alert noise continue to rise across the industry.

What's the business case for fixing this?
A mid-stage SaaS company handling 15 incidents per month with 3 engineers per incident and 39 minutes of coordination overhead per incident loses roughly 29 engineer-hours per month to non engineering work. At a fully loaded cost of $150/hour, that's about $52,000/year in direct labor before accounting for context-switching costs and the attrition risk of burned out on-call engineers.

How does Steadwing specifically address this?
When an alert fires, Steadwing takes info from your logs, metrics, traces, and codebase, connects the dots across your whole stack, and gives the on-call engineer a full root cause analysis with automatable fixes on code level, around deployment, and infra in under 5 minutes. The RCA investigation is over by the time the on-call person opens their laptop. Your engineers still make the decisions but they start with a diagnosis and solution, not a blank screen and six browser tabs.

Steadwing is an autonomous on-call engineer. It connects the dots across your stack and gives you a full RCA with fixes before your team starts the manual scramble. Start free →