[14:15:00] *** PagerDutyBot: SEV-0: GLOBAL CONTROL PLANE UNREACHABLE. API ERROR RATE > 95%.

[14:15:05] *** sev_manager has joined

[14:15:10] <sev_manager> Status?

[14:16:12] <net_ops_jen> It’s bad. Traffic to the auth service just verticalized. We’re seeing 50M RPS.

[14:17:00] <sev_manager> DDoS? Scrubbing center active?

[14:17:30] <net_ops_jen> That’s the weird part. It’s bypassing the WAF. It looks like legit traffic. TLS fingerprints are valid.

[14:18:10] <sev_manager> Did someone let Tyler push a new WAF regex? Please tell me we didn’t backpedal into 2019.

[14:19:00] <backend_bob> I’m looking at the sample logs. These are all authenticated requests.

[14:19:45] <backend_bob> They are all hitting POST /api/v4/user/token/refresh.

[14:20:10] <sev_manager> Why is everyone refreshing their token at the exact same second?

[14:21:00] <net_ops_jen> It’s not one refresh. I’m seeing the SAME user IDs hitting it 500 times per second.

[14:22:15] <frontend_felix> Uh, guys?

[14:22:30] <sev_manager> Go ahead Felix.

[14:23:00] <frontend_felix> We pushed the “Seamless Session” update to the dashboard 15 minutes ago.

[14:23:15] <frontend_felix> The goal was to silently refresh the token in the background so users don’t get logged out.

[14:24:00] <backend_bob> Felix… look at the code.

[14:25:00] <frontend_felix> I am.

[14:25:10] <frontend_felix> …

[14:25:15] <backend_bob> Felix?

[14:25:45] <frontend_felix> Oh no.

[14:26:00] <sev_manager> REPORT.

[14:26:30] <frontend_felix> Okay, so, in React… useEffect runs when a dependency changes.

[14:26:45] <frontend_felix> We have: useEffect(() => { refreshToken() }, [token])

[14:27:00] <backend_bob> And refreshToken()… updates the token?

[14:27:05] <frontend_felix> Yes.

[14:27:10] <backend_bob> Which triggers the useEffect again?

[14:27:12] <frontend_felix> Yes.

[14:27:30] <backend_bob> So you turned every single customer’s browser tab into an infinite loop cannon pointed at our auth servers?

[14:28:00] <frontend_felix> In my defense, the tokens are incredibly fresh.

[14:28:15] <sev_manager> Rollback the frontend.

[14:29:10] <sev_manager> I’ll start writing another ridiculously well edited postmortem blog post.

[14:29:41] <backend_bob> Hackernews and primeagen is going to going to love this one.