A ghost lock blocked every cron trigger for 6 hours after a Korean ad-network API took 45 seconds to respond — 15 seconds past the Workers wall clock limit.
The Workers runtime doesn't queue a second invocation and wait for the first to finish. It fires the next cron tick regardless. So when my D1 write job occasionally crept past 30 seconds, two instances ran simultaneously, both inserting aggregated impression rows faster than my UNIQUE constraint could reject them. The 3am wake-up call was D1_ERROR: UNIQUE constraint failed: impressions.event_id.
The obvious fix is a distributed lock with a TTL — so even if the Worker dies mid-run without releasing, the lock eventually clears itself. I tried KV first because it's simpler. The code looks correct:
const existing = await env.KV.get(lockKey);
if (existing) return; // skip this tick
await env.KV.put(lockKey, "1", { expirationTtl: 55 });
It works — until it doesn't. KV is eventually consistent across Cloudflare PoPs. During a spike to ~12K writes/minute on an ad-server stress test, Cloudflare routed two consecutive triggers to different PoPs in the same minute. Both read null, both wrote the lock, both ran the full aggregation. Duplicate rows, again. KV locks are survivable if your job is idempotent. Mine wasn't, and I'd bet most non-trivial jobs aren't either.
The fix that actually held was a Durable Object acting as the lock. A single DO instance is strongly consistent — there's no eventual propagation window to race against. The DO stores a locked_until timestamp, and acquire returns HTTP 423 if that timestamp is still in the future. TTL is set to 55 seconds: just above the 30s wall clock, so a genuine overrun also self-clears within a minute without blocking the rest of the day's triggers.
The part that tripped me up in practice wasn't the lock logic — it was the [[migrations]] block in wrangler.toml. Skip it and you get a runtime binding error that's easy to misread as a code bug. I've hit it twice copy-pasting configs between projects.
I wrote up the full breakdown — including the complete DO class, the wrangler.toml wiring, and the specific failure scenario where even the DO approach has an edge case — over on dailymanuallab.com.