The Runbook That Lied to Me at 3am

The pager went off at 3:14am for a wedged OpenStack Neutron agent. I did what any tired engineer does: I opened the runbook. It told me to restart a service that had been renamed eighteen months earlier, pointed at a Grafana dashboard that 404'd, and assumed a network topology we'd migrated off of two quarters back. The runbook wasn't just unhelpful. It was actively lying to me, and I burned twenty minutes trusting it before I gave up and went to read the source.

That's the real problem with documentation. It isn't that we don't write it. It's that the moment we finish writing it, it starts rotting, and the cost of keeping it fresh is high enough that nobody pays it until the document has already betrayed someone at 3am. A runbook your team doesn't trust is worse than no runbook, because no runbook at least forces you to think.

This is where AI actually earns its keep in a platform org, and not in the way the marketing decks suggest. AI is not going to own your documentation. It's going to do the tedious first-draft labor — turning a resolved incident, a chunk of shell history, or a deploy diff into a structured skeleton — so a human engineer can spend their scarce attention on the part that matters: verifying the commands, marking what's unproven, and editing the robotic tone out so the team actually reads it. AI drafts. You verify and sign off. That distinction is the whole game.

Why "Humanizing" AI Is the Job, Not a Slogan

Let me be precise about what I mean by "humanizing AI," because the phrase gets abused. I don't mean making AI sound human to fool a reader. I mean keeping a human in the loop as the editor and owner of record, and doing the unglamorous work of turning a competent-but-soulless machine draft into something a colleague trusts.

Two things break trust in AI-drafted docs, and both are fixable by a human pass:

1. Unverified claims stated with confidence. An LLM will happily tell you to run systemctl restart neutron-l3-agent whether or not that's the actual unit name on your boxes. It doesn't know. It's pattern-matching. So the human's first job is to run every command, in a safe environment, and confirm it does what the draft claims.

2. The robotic tone. Machine drafts read like a compliance memo: hedged, repetitive, weirdly formal, full of "it is important to note that." Engineers smell that instantly and stop reading. Editing for voice and concision isn't vanity — a doc people skim past doesn't get used. That cleanup pass is legitimate, valuable work, and it's exactly the "humanizing" angle that matters.

If you keep those two responsibilities firmly with a person, AI becomes a force multiplier instead of a liability generator. I've written more about the philosophy of runbooks engineers actually trust at 3am, but the short version is: trust is built by verification, and verification is a human act.

Troubleshooting Guides: Draft From the Incident You Just Resolved

The best time to write a troubleshooting guide is in the hour after you've fixed the thing, while the diagnostic path is still warm in your head. The worst time is never, which is the default. AI closes that gap because the raw material already exists in your terminal.

Here's the prompt pattern I use. I dump my actual shell history and the incident timeline at the model and constrain it hard:

You are drafting an internal troubleshooting guide for a platform team. I'll give you my shell history and a rough incident timeline. Produce a guide with these sections: Symptoms, Prerequisites/Access, Diagnosis Steps, Resolution, Rollback, Verification. Use the exact commands from my history — do not invent flags or paths. Wherever you infer something I didn't explicitly state, prefix that line with [ASSUMPTION] so I can verify it. Keep the prose tight; no filler.

That [ASSUMPTION] instruction is the single most important line in the prompt.

Pro Tip: Always force the model to explicitly tag anything it couldn't verify from your inputs. An LLM's failure mode is confident gap-filling. When you make it mark its own guesses, you convert an invisible risk into a visible checklist — every [ASSUMPTION] tag is a thing you personally confirm before the doc ships. The tags get deleted in editing, but they drive the verification pass.

Feed that prompt a history | grep neutron plus a three-line timeline, and you get back something like this — which is a genuinely good starting point, not a finished doc:

Diagnosis Steps

1. Confirm the agent is actually down, not just flapping: openstack network agent list --agent-type l3 --host compute-04
2. [ASSUMPTION] The L3 agent unit is neutron-l3-agent. Check status: systemctl status neutron-l3-agent
3. Tail the agent log for the RPC timeout signature: journalctl -u neutron-l3-agent --since "10 min ago" | grep -i timeout

Resolution

Restart the agent and confirm it re-registers with the server:
systemctl restart neutron-l3-agent && sleep 5 && openstack network agent list --host compute-04

Rollback

[ASSUMPTION] No rollback required for a restart. If the agent fails to re-register, fail the router over to a standby node before deeper debugging.

Notice what the AI did well: it structured the chaos of my history into a clean diagnostic flow and pulled the real commands I ran. Notice what it flagged: it didn't know my unit name, so it tagged it. When I verified, the unit was actually devstack@q-l3.service on the affected host — a thing I'd never have caught if the draft had just stated it confidently. That one assumption tag is the difference between a guide that helps and one that lies. I go deeper on this workflow in my piece on drafting runbooks from resolved incidents with AI.

Postmortems: AI Handles the Timeline, You Own the Blamelessness

Postmortems are where AI's drafting strength and its judgment weakness are both on full display. The mechanical parts — assembling a coherent timeline from Slack threads, alert timestamps, and deploy logs, then drafting an impact summary — are exactly the tedious work that delays postmortems for weeks. AI eats that for breakfast.

But the blameless part is not a tone setting you toggle. It's an editorial and cultural stance that a human has to own. I give the model the raw timeline and explicitly instruct it:

Draft a blameless postmortem. Describe what the system did and what signals were available, never what a person "failed" to do. Frame every human action as a reasonable decision given the information available at the time. Sections: Summary, Impact, Timeline, Contributing Factors, What Went Well, Action Items. Mark any causal claim you can't support from the inputs with [UNVERIFIED].

The model will get you 80% of the way to neutral language. The remaining 20% — catching the sentence that subtly implies the on-call engineer should have known better — is human work, every time. Blameless writing is a skill, and the editing pass is where you apply it. If you haven't internalized what separates a postmortem people read from one that gets filed and forgotten, this breakdown of blameless postmortems people actually read is worth your time.

Pro Tip: Never let AI write your Action Items unsupervised. The model loves to generate plausible-sounding remediation ("add more monitoring," "improve documentation") that sounds responsible and commits no one to anything. Real action items have an owner, a due date, and a verifiable definition of done. That's a leadership decision, not a text-generation task — strike every vague item the draft produces and replace it with something a person actually agreed to do.

The payoff is real, though. When the timeline and impact summary are drafted in ten minutes instead of taking the better part of a day, postmortems actually get written while the details are fresh, instead of being abandoned because everyone moved on to the next fire.

SOPs: Encode the Tribal Knowledge Before It Walks Out the Door

Standard operating procedures are the documents most likely to live entirely in one senior engineer's head — how we rotate the cluster certs, how we drain a node for maintenance, the precise dance to cut a new Terraform workspace without orphaning state. That knowledge is a bus-factor risk, and writing it down has always lost to more urgent work.

AI lowers the activation energy enough that it stops losing. I'll narrate the procedure out loud into a rough text file — half sentences, command fragments, the gotchas I remember — and have the model turn that mess into a structured SOP with numbered steps, prerequisites, and explicit "you are done when..." verification criteria.

The thing to watch here is that an SOP encodes policy, not just commands. The AI can format your steps beautifully and still produce something that violates your change-management rules because it doesn't know them. So the human pass on an SOP is checking two layers: are the commands correct, and does the procedure comply with how we're actually supposed to operate? The model can't see your org chart or your change board. You can.

I keep a small library of these drafting prompts so I'm not rewriting the scaffolding each time — collecting and reusing the prompts that work is half the productivity gain, because the quality of the draft is downstream of the quality of the prompt.

Deployment and Release Docs: Draft Straight From the Diff

Deployment documentation has a unique advantage: the source of truth is structured and machine-readable. A PR diff, a Terraform plan, a Helm values change — these are precise artifacts an AI can read directly, which means the draft starts from facts rather than recollection.

My workflow for release notes and deploy runbooks: pipe the diff or the terraform plan output to the model and ask for a deployment guide that includes pre-flight checks, the apply procedure, the blast radius, the rollback, and the post-deploy verification. Because the input is concrete, the hallucination rate drops sharply. The model isn't guessing what changed — it's reading it.

Pre-flight Checks

• Confirm the target workspace: terraform workspace show returns prod-us-east.
• [ASSUMPTION] This plan adds 2 nodes and modifies the ASG launch template; no destroys. Re-run terraform plan and confirm zero resources to destroy before applying.

Rollback

The launch template change is versioned. To roll back, point the ASG at the previous template version and trigger an instance refresh — do not terraform destroy.

Even here, the human verifies the blast radius claim. "No destroys" is the kind of statement that's true until a provider upgrade quietly makes a field force-new, and a misread there is how you turn a routine deploy into an outage. The AI gets you a structured, mostly-correct draft fast; you confirm the dangerous parts with your own eyes. For a fuller treatment of wiring this into a real pipeline, my 2026 guide to runbook automation with AI walks through the tooling end to end.

The Editing Pass Is Where Trust Is Built

I want to close on the part that gets skipped, because it's the part that actually decides whether your docs get used. Once the commands are verified and the assumptions are resolved, you still have a machine-shaped document on your hands — competent, structured, and slightly lifeless. It hedges too much. It repeats the obvious. It has that flat, over-formal register that makes an engineer's eyes glaze.

Editing that out is not cosmetic. A runbook nobody can stand to read is a runbook nobody reads, which puts you right back at 3am reading source code. So I do a final pass for voice: cut the filler, sharpen the warnings, add the one line of context only a human who lived the incident can add ("the agent flaps for about thirty seconds after restart — don't panic and restart it again"). That sentence is worth more than the entire generated scaffold, and only a person can write it.

That's the humanizing loop in full: AI drafts the structure from real artifacts, a human verifies every command and resolves every assumption, and then a human edits the robotic tone into something a tired colleague will actually trust and follow. Keep ownership with the person at every step and AI becomes the best documentation intern you've ever had — fast, tireless, and entirely supervised. Skip the human steps and you're just generating the next lie waiting to fire at 3am.

---

James Joyner IV runs devopsaitoolkit.com, where he writes about running production OpenStack, Kubernetes, and observability with AI in the loop. If your AI-drafted docs read like a robot wrote them, his Writing Humanizer pack is a toolkit for making machine drafts read like a human actually sat down and wrote them.