On June 26, 2025, all Clerk services were down from 6:16 UTC to 7:01 UTC, caused by an outage of our compute infrastructure that impacted all Clerk customers.
We are deeply sorry for this outage. Clerk is a critical infrastructure component for our customers, and we take our reliability and uptime seriously. We know that any amount of downtime is unacceptable, and regardless of the cause, our system’s reliability is our responsibility and we fell short of our standards and your expectations.
Graph of request throughput to our services during the outage. (GMT+3)
SIGINT shutdown signals immediately on startSIGINTs are related to the particular containerSIGINTWe architected Clerk to be resilient against failures in Google Cloud availability zones, but not entire regions. Cloud Run is stated to provide zonal redundancy, which implies that this incident was caused by a full regional failure.
On the other hand, if this was truly a regional failure, we would expect many more services to be impacted than just Clerk. While there was some discussion on Hacker News, the blast radius of this event is surprisingly small for a regional failure.
We are awaiting more information from Google about exactly which system failed, and will update this post when it’s received.
Update (June 27, 12:40AM UTC): Google has notified us that their root cause analysis would be published by June 30.
Update (July 8): Google provided an RCA Summary, titled "Cloud Run customers are experiencing an issue in “us-central1” region" (Event ID: MLFZZXV)
us-central1 region experienced downtime for a duration of 51 minutes (June 26 00:07 PDT).When incidents like this happen, we immediately turn our attention toward preventing their recurrence. Regardless of the root cause, it is our responsibility to build a service that is resilient to failures within our infrastructure providers. To that end, we are starting the following remediations:
This incident could have been mitigated with a failover that shifted our Cloud Run traffic to a different region when us-central1 began failing. Work is starting on this immediately.
Although Google Cloud Platform (GCP) was remarkably stable for Clerk’s early years, we have faced three major server disruptions since May 2025 that we attribute to GCP incidents. This shows that we need to explore additional redundancy outside of a single cloud vendor.
We will begin investigating multi-cloud redundancy for our compute infrastructure. This would make Clerk resilient to complete service failures of Cloud Run, as well as failure of Google’s Cloud Load Balancer.
Any incident in our Session Management service has an outsized impact on our customers, since it results in complete downtime of their service.
Following an incident in February, we isolated our Session Management service from our User Management service, ensuring that bugs in our User Management codebase would not impact the availability of our Session Management service.
Unfortunately, in the event of a compute outage at origin like we saw in this incident, both services still come down.
To further mitigate session management failures, we are exploring architectural changes that will allow Clerk to continue issuing session tokens for a greater variety of incidents. Though a longer-term project, this will include bringing distributed storage and compute to our Session Management service.
This list of remediations we are exploring is not exhaustive, and doesn’t represent a final state for our efforts to make Clerk as resilient as possible. We will continue to invest in stability and scalability to make sure our customers can rely on Clerk as a critical service provider.
This was a serious outage, and we know that businesses rely on Clerk. We are again deeply sorry for the impact on our customers and will continue working to improve our reliability going forward.
For any questions, please contact support.
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。