The 2 a.m. compute runaway is the canonical FinOps incident. A Spark job is misconfigured to provision new EMR nodes every minute it cannot find a leader. A test agent left running on a developer's laptop loops infinite Claude calls against the prod API key. An autoscaling group's max gets bumped from 20 to 2000 in a Terraform plan that nobody reviewed at the right line number. Everything is asleep. The hourly spend goes from $63 to $830 to $4,200. By 9 a.m. the team gets a Slack ping from finance asking why yesterday's bill spiked $47,000.

AWS Budgets fires a soft alert when daily spend crosses a threshold. The alert goes to an SNS topic that emails a distribution list and pings a Slack channel. Nobody reads the channel at 2 a.m. The on-call engineer is paged for production outages, not budget overages. By the time someone sees the alert, the damage is hours old and the runaway has either burned itself out or kept running because the alert did not actually stop anything.

The structural fix is to replace the email with an action. A closed-loop budget brake fires a remediation playbook when a hard daily cap is crossed: stop non-prod EC2 launches, pause non-prod autoscaling groups, freeze agent provisioning, throttle batch jobs, page the on-call. The 5-minute detect-decide-act-verify shape from the closed-loop FinOps work applies directly, with the cap value as the signal and the playbook as the action.

The piece composes with the closed-loop trust score (deciding which playbook tiers auto-fire) and runs alongside cost anomaly detection (which catches longer-horizon structural drift the brake cannot see).

The 2 a.m. runaway and why email alerts fail

Look at how the four common detection mechanisms catch a 2 a.m. runaway, and what they cost in dollars by the time someone acts.

The dollar gap between the soft alert and the brake is the case for the brake. Soft alerts are real signals, but they are signals that go to humans who are not actively monitoring at 2 a.m. The brake removes the "wait for a human" step from the loop.

The other detectors are not redundant. Hourly cost alarms catch slower-building issues the brake's daily cap might miss within a single day. Cost anomaly detection catches structural shifts (a new feature with legitimate higher spend, a pricing change, a seasonal pattern) over multi-day windows. The brake handles the within-day catastrophic runaway. Three detectors, three different horizons.

The brake: short-circuit, not email

The brake's shape is the same four-stage loop as any other closed-loop FinOps system, with the cap as the input signal.

Detect samples live spend at a 5-minute interval from Cost Explorer or the equivalent on GCP / Azure. Five minutes is the floor: the billing APIs are eventually consistent and finer-grained polling produces false negatives (real spend that has not yet shown up in the API). Decide compares the running daily total against the cap. If crossed, the brake fires a tiered playbook. Act runs the playbook. Verify samples spend again 15 minutes after the playbook fires, confirming the runaway has stopped.

The playbook is per-account and lives in version control. A typical Tier 1 payload contains six actions: stop all env=non-prod EC2 instances launched in the last 60 minutes, pause non-prod autoscaling group scale-outs by setting max_size = current_size, freeze agent provisioning by revoking the agent service role's ec2:RunInstances, throttle non-prod batch queues to zero concurrency, snapshot the spend-by-service breakdown to an S3 bucket for postmortem, and notify the FinOps channel with the breakdown.

The brake does not delete anything. Everything Tier 1 does is reversible in seconds. Spend stops; nothing breaks. The on-call wakes up to a "brake fired" page, reads the breakdown, decides whether the spend was legitimate (and reverses the playbook) or a runaway (and starts the postmortem).

Sizing the cap: three inputs, one formula

The cap value is not a guess. It is computed from three inputs that already exist in the cost data.

The formula: cap = max(typical * variance, emergency_floor + 20%).

Worked example for three account profiles:

The 20% buffer on the emergency floor matters. A legitimate spike that happened once (a launch event, a load test, an unusual data migration) might not happen the same day next month, but the cap has to be high enough that the same pattern would not trip the brake if it recurs. Without the 20% buffer, the brake fires on every recurrence of every legitimate pattern, and the on-call learns to ignore it.

Cap recomputation happens monthly. The typical daily spend drifts as the company grows. The emergency floor may shift if a new legitimate workload pattern emerges. A static cap that lasts more than a quarter starts to over-fire or under-fire because the inputs moved.

Tier the action: composing with the trust score

The brake's playbook is tiered by what the trust score allows. Without the trust score, the brake either over-reaches (touches production resources, causes a customer-facing incident, gets disabled forever) or under-reaches (only pages, doesn't actually stop the spend).

Tier 1 is always-on because the actions are low-blast-radius and fully reversible. Stopping a non-prod EC2 instance launched in the last 60 minutes affects only the developer who launched it, and they can re-launch in 90 seconds. Pausing a non-prod ASG scale-out blocks new capacity but does not terminate existing capacity.

Tier 2 needs the trust score because the actions have wider blast radius. Throttling a training job interrupts the team running it; downscaling notebook compute kicks people out of their analyses. The trust score asks: is the spend signal high-confidence enough (cap crossed by 2x+, multiple services contributing) to justify the disruption? If yes, fire. If no, page only.

Tier 3 always pages. The brake does not touch production resources, ever. The math is simple: a production-impacting incident caused by the brake costs more than any single-day cost runaway. The brake's job is to give the human enough time to fix the runaway before the bill is catastrophic, not to be the fix itself.

Cap vs anomaly detection: different time horizons

A common mistake is treating the brake as a replacement for cost anomaly detection. They are not the same system. They run on different signals and they fire on different timescales.

The brake answers "is something burning right now?" The signal is daily-spend-rate exceeds cap. The action is immediate playbook.

Anomaly detection answers "is the spending pattern different from what we expect?" The signal is statistical: spend by service over a multi-day window deviates from forecasted baseline. The action is queued analyst review, often with a recommendation to update budgets or investigate a new workload.

Both run. The brake catches the rare catastrophic runaway. Anomaly detection catches the steady-state drift (a new feature legitimately moving the spend curve, a pricing change at a vendor, a regional capacity shift). Each is useless for the other's job: anomaly detection cannot stop a 2 a.m. runaway because the analyst is not online; the brake cannot tell you that your spend has structurally shifted because it only sees the daily total.

The first-week tuning ritual

The brake gets tuned in its first two weeks. The cap value comes from the formula, but the formula's inputs are estimates. The actual firing rate in week one tells you whether the cap is right.

The classification at each firing matters. The on-call writes a one-paragraph postmortem: was this a runaway (which workload, what was the root cause, how long would it have run unchecked) or a legitimate spike (which team, why, was it within budget, why did the cap not anticipate it). False positives raise the cap. Real positives keep the cap and become input to the trust score weights.

The discipline that makes the brake trusted is that every firing produces a postmortem. Without postmortems, the team starts to debate "the brake fires too much," lower the cap to make it stop firing, and the brake silently becomes a $50k cap that catches nothing. With postmortems, the cap value is defensible and the trust accumulates.

The exempt-tag escape valve

Some workloads legitimately spike beyond the cap. ML training that goes from $200/day to $8,000/day during a three-day training run looks identical to a runaway under a daily cap. Ad-hoc analytics that spin up 50 BigQuery slots for a quarterly report look identical. The right fix is not "raise the cap to absorb all training spikes" because that defeats the brake. The right fix is to exempt the workload and route it through a different cap.

The pattern: any resource tagged brake_exempt=true is excluded from the daily-cap calculation. Exempt resources go into a separate weekly cap (typically 3x the equivalent daily cap times 7) that catches truly anomalous training or analytics spend.

The exempt tag has to be opt-in and reviewed. A team that wants to exempt their workload submits a one-page rationale to the FinOps team. The exemption is granted with an expiry date and a recompute schedule. Without that discipline, every team eventually tags their workload exempt and the brake erodes back to nothing.

The dollar math

A 2 a.m. runaway on a mid-market AWS account typically costs $30,000 to $80,000 by the time anyone notices. Larger accounts can hit $200,000+ before the morning Slack ping. The frequency is low but not negligible: ZopDev customer audits show one runaway every 4-7 months on mid-market accounts, more frequent during periods of rapid infrastructure change.

The brake pays back after the first prevented incident. After that, it is one of the highest-ROI items on a FinOps roadmap, ahead of right-sizing and behind only the cost-allocation work that lets you see where the spend goes in the first place.

The brake is not a substitute for budgets, anomaly detection, or right-sizing. It is the layer that catches what those other systems are not designed to catch: the within-day catastrophic spend event. Email alerts go to inboxes. The brake fires a playbook. The 2 a.m. runaway becomes a 5-minute incident with a $1,500 ceiling instead of an 8-hour incident with an $80,000 ceiling. Set the cap, write the playbook, watch the first two weeks of firings, and stop arguing with the morning bill.