惯性聚合 高效追踪和阅读你感兴趣的博客、新闻、科技资讯
阅读原文 在惯性聚合中打开

推荐订阅源

V
V2EX
奇客Solidot–传递最新科技情报
奇客Solidot–传递最新科技情报
WordPress大学
WordPress大学
罗磊的独立博客
小众软件
小众软件
I
InfoQ
Y
Y Combinator Blog
宝玉的分享
宝玉的分享
钛媒体:引领未来商业与生活新知
钛媒体:引领未来商业与生活新知
Hugging Face - Blog
Hugging Face - Blog
MyScale Blog
MyScale Blog
博客园 - 聂微东
Microsoft Security Blog
Microsoft Security Blog
H
Help Net Security
酷 壳 – CoolShell
酷 壳 – CoolShell
博客园_首页
S
SegmentFault 最新的问题
博客园 - 三生石上(FineUI控件)
P
Proofpoint News Feed
博客园 - 司徒正美
Cyber Security Advisories - MS-ISAC
Cyber Security Advisories - MS-ISAC
Microsoft Azure Blog
Microsoft Azure Blog
Jina AI
Jina AI
N
Netflix TechBlog - Medium

Catchpoint Blog

SRE Report: AI optimism and the economics of effort SRE Report: Why fast is what users trust SRE Report 2026: What surprised us, what didn't, and why the gaps matter most The SRE Report 2026: Defensible Ns Why Synthetic Tracing Delivers Better Data, Not Just More Data A New Chapter: LogicMonitor + Catchpoint – A Personal Note from Mehdi Mezmo + Catchpoint deliver observability SREs can rely on The four pillars holding up your digital business, and what happens when they crumble When payments pause: lessons from a global payments outage Observability 2025 Decoded: What the DZone Report Means for SLO-Driven Ops The next evolution of WebPageTest has arrived, and it’s a game-changer The Monitoring Blind Spot That Could Cost You Black Friday Powering Mexico’s Digital Future: Expanded Internet Observability with Catchpoint The Next Chapter of WebPageTest: Your New Experience Starts Soon SRE Report Retrospectives — Have AIOps Predictions Held Up? When BGP becomes UX: The inside story of a SaaS routing decision gone wrong (or right) Session Replay explained: A guide to seeing digital experience through your user’s eyes Making the invisible visible: Are your cloud firewalls and DDoS protection really working? Why it’s time to move beyond APM: Monitoring from the user’s perspective When metrics mislead: Inside the 2025 Retail Web Performance Benchmark The vendor trap: why your next outage won’t be your fault—but will be your problem LLMs don’t stand still: How to monitor and trust the models powering your AI Semantic Caching: What We Measured, Why It Matters The Annual SRE Survey Is Open—We Want to Hear from You Observability isn’t about the tool. It’s about the truth Real-time detection of BGP blackholing and prefix hijacks Leading analyst firm reveals the real cost of internet disruptions The Power of Over 3000 Intelligent Observability Agents Monitoring in the Age of Complexity: 5 Assumptions CIOs Need to Rethink Why Intelligent Traffic Steering is Critical for Performance and Cost Optimization
Invisible dependencies, visible impact: Lessons from the ...
2025-06-13 · via Catchpoint Blog

in this blog post

June 12, 2025. A date most of the Internet won’t remember — but anyone relying on Google Cloud will. In the span of minutes, a routine quota update snowballed into global disruption. APIs stopped responding. Dashboards stayed green. And across continents, teams scrambled to figure out if the problem was theirs — or Google's.

It wasn’t a cyberattack. It wasn’t a datacenter fire. It was an automated quota change buried deep in Google’s infrastructure — and it was enough to ripple across the digital world.

This wasn’t just about Google. It was a stark reminder that no one is too big to go down. The systems we rely on are deeply interconnected, often veiled, and rarely as fail-safe as we imagine. And when things break, it’s not always clear who’s accountable — or even what’s broken.  

This post isn’t just about an outage; it’s about what we miss when we rely on status pages that lag by 30 minutes to an hour, what we gain when we can see independently and respond in real time — and why every digital business, no matter how large, needs a better way to see in the dark.

What happened?

At 1:49 PM ET on June 12, Google Cloud began experiencing a major service disruption. The cause: an automated quota update in Google’s global API management system that triggered widespread 503 errors and external API request failures. What followed was a cascading breakdown — one that reached deep into the infrastructure powering core Google Cloud services and beyond.

Affected services included:

  • Google Cloud Console, App Engine, Cloud DNS, and Dataflow
  • Identity and Access Management (IAM), Pub/Sub, and Dialogflow
  • Apigee API Management and other backend services
  • 30+ additional GCP products across the Americas, EMEA, APAC, and Africa

A screenshot of a computer screenAI-generated content may be incorrect., Picture

But it didn’t stop there. The outage reverberated outward, hitting major platforms like Discord, Spotify, Snapchat, Twitch, and Cloudflare — digital giants that depend on GCP under the hood.

Recovery began within hours for most regions. But in us-central1, where a quota policy database was overwhelmed, the impact lingered well into the afternoon

How was it detected?

Within a minute, Catchpoint Internet Sonar started flagging anomalies in services like Google Drive and App Engine at 1:50 PM ET

A screenshot of a phoneAI-generated content may be incorrect., Picture

Catchpoint Internet Sonar view

A map of the world with orange dotsAI-generated content may be incorrect., Picture

Apigee service disruptions detected across 109 cities worldwide

A screenshot of a mapAI-generated content may be incorrect., Picture

Google Cloud service degradation detected across 157 cities, with active incidents spanning North America, Latin America, Europe, and Asia-Pacific

Picture 1, Picture

This screenshot shows a spike in failed tests across multiple workflows. The failure pattern is sudden, sustained, and correlated across multiple test types — a classic signal of a widespread upstream infrastructure disruption.

A screenshot of a graphAI-generated content may be incorrect., Picture

This chart from one of our users shows a sharp drop in availability and a spike in checkout failures across multiple countries — confirming global impact on user-facing transactions during the outage window.

A screenshot of a computerAI-generated content may be incorrect., Picture

Catchpoint Internet Stack Map

This Internet Stack Map shows a clear breakdown in service dependencies, with Google Cloud and Apigee at the center of the disruption.

You can see how multiple layers — including analytics, cloud logging, storage, and identity — all failed in tandem. These failures directly impacted third-party tools, CDNs, and SaaS integrations downstream.

The alert indicators show transaction failures and response errors cascading outward, affecting not just infrastructure, but also the digital experience of users relying on these tools in real time.

This stack map captures what many teams experienced: when one layer goes, it doesn’t go alone.

No official word — yet.

While synthetic tests and customer data clearly showed the disruption unfolding in real time, Google Cloud’s first public acknowledgment didn’t arrive until 2:46 PM ET — nearly an hour after anomalies first appeared and services began failing globally.

A close-up of a textAI-generated content may be incorrect., Picture

Google cloud status page

During that gap, the GCP status page remained green.  

For teams on the front lines — whether SREs, DevOps, or customer support — that matters. It creates doubt. Are we at fault? Is this a local issue? Can we act, or do we wait?

This isn’t a critique of Google’s infrastructure — it’s a reality of operating at scale. Status pages are often downstream from detection. They’re built for caution, not speed. And by the time they update, most teams have already lost the window for a proactive response.

What was the impact?

The blast radius was wide — and layered.

  • Direct GCP customers lost access to core infrastructure: management consoles, APIs, storage, and authentication services.
  • SaaS and enterprise platforms saw critical user workflows stall for over two hours — with transaction-level failures visible in real-time test data.
  • Consumer platforms including Discord, Snapchat, Twitch, and Spotify suffered cascading slowdowns due to their dependencies on affected Google services.
  • Geographic scope: Failures were observed across North America, EMEA, APAC, and Latin America.

What began as a quiet configuration change rippled outward — breaking not just cloud services, but the trust and functionality layered on top of them.

Key lessons

Outages like this don’t just break systems — they expose assumptions. They challenge how we monitor, communicate, and respond. Here’s what the June 12 Google Cloud incident reinforced.

#1 You’re never too big to go down

No provider — not even Google — is immune to large-scale outages. The events of June 12 made that clear. A routine quota update cascaded into global downtime, reminding every digital business just how quickly dependencies can unravel.

“This is a timely wakeup call: even hyperscalers like Google aren’t immune to largescale outages. In today’s interconnected digital landscape, external observability via tools like Catchpoint isn’t optional — it’s essential.”
Mehdi Daoudi, CEO & Co-founder, Catchpoint

#2 The domino effect is real

When a hyperscaler like Google stumbles, it’s not just their systems that suffer — it’s everything built on top of them. That includes SaaS platforms, third-party APIs, and the end-user experiences they power.

This incident showed just how fast a single point of failure can ripple outward, breaking things that appear unrelated at first glance.

“Your system is as resilient as the weakest of its components. Which means it only takes one dependency to bring down an entire system. If the authentication service used by a critical API is down, your system is down — even if everything else is working.”
Gerardo Dada, Field CTO, Catchpoint

#3 Status pages aren’t enough

Provider dashboards serve a purpose — but they aren’t built for real-time incident response. They’re designed to be cautious, accurate, and measured — which often means they lag behind the actual impact felt by users and customers.

This isn’t a knock on any provider — it’s a reflection of operational reality at scale. The lesson isn’t to expect more from status pages, but to expect more from your own visibility.

#4 Design for resilience

Outages happen — even to the best-engineered platforms on the planet. What matters isn’t avoiding every failure, but mitigating the blast radius when one occurs.

That means building systems that assume things will break. Architecting across regions. Diversifying providers. Creating failovers that actually fail over.

“One thing we see with Internet Sonar is even the greatest companies with the most advanced tech can suffer outages. It happens to the best. All the more reason for a multi-multi architecture.”
Matt Izzo, VP Product, Catchpoint

#5 Your monitoring can’t live in the same cloud you’re trying to monitor

Outages like this raise an uncomfortable but important question: Can your monitoring still see what’s happening when your cloud provider goes dark?

Many monitoring tools rely on cloud-hosted vantage points — often within the very infrastructure they’re meant to observe. When that cloud provider has an outage, your diagnostics might vanish along with it. Synthetic tests hosted in hyperscalers rarely reflect real user environments, and they mask failures within the provider itself.

  • Synthetic tests inside the     cloud often     miss real-world issues like DNS errors, CDN disruptions, or ISP-level     failures.
  • Cloud-only testing creates a false sense of     security — you’re essentially monitoring yourself, from yourself.

True visibility comes from outside the cloud, from the edge of the Internet where real users live. Because monitoring should never go down when you need itmost.

Staying resilient when the Internet isn’t

The June 12 outage didn’t just disrupt Google Cloud — it created cascading effects across Cloudflare, CDNs, productivity platforms, and AI tools. It’s a vivid reminder of how deeply interconnected and interdependent digital systems have become.

These weren’t abstract failures — they had real consequences. Picture a hospital unable to access patient records or drug databases due to a cloud outage. This isn’t about someone being unable to order lunch. This is about critical services failing at the worst possible moment. This is life and death.

At Catchpoint, we take that responsibility seriously. That’s why during the incident:

  • Our monitoring continued from outside the public cloud
  • While some customers faced issues reaching third-party services, Catchpoint itself remained visible and stable throughout

We remain committed to resilience, transparency, and visibility — not just for our own platform, but across the entire Internet ecosystem that powers your business.

Tools that made a difference

Catchpoint users navigating the Google Cloud outage had two key advantages on their side:

  • Internet Sonar offers real-time, independent monitoring of the internet’s core services — helping you detect third-party issues early, understand their scope, and act decisively.
  • Internet Stack Map provides a live view of your service’s dependencies, making it easy to trace cascading failures and pinpoint root causes fast.

Schedule a demo to learn how Catchpoint can help your team stay ahead of the next incident.

Summary

June 12, 2025. A date most of the Internet won’t remember — but anyone relying on Google Cloud will. In the span of minutes, a routine quota update snowballed into global disruption. APIs stopped responding. Dashboards stayed green. And across continents, teams scrambled to figure out if the problem was theirs — or Google's.

It wasn’t a cyberattack. It wasn’t a datacenter fire. It was an automated quota change buried deep in Google’s infrastructure — and it was enough to ripple across the digital world.

This wasn’t just about Google. It was a stark reminder that no one is too big to go down. The systems we rely on are deeply interconnected, often veiled, and rarely as fail-safe as we imagine. And when things break, it’s not always clear who’s accountable — or even what’s broken.  

This post isn’t just about an outage; it’s about what we miss when we rely on status pages that lag by 30 minutes to an hour, what we gain when we can see independently and respond in real time — and why every digital business, no matter how large, needs a better way to see in the dark.

What happened?

At 1:49 PM ET on June 12, Google Cloud began experiencing a major service disruption. The cause: an automated quota update in Google’s global API management system that triggered widespread 503 errors and external API request failures. What followed was a cascading breakdown — one that reached deep into the infrastructure powering core Google Cloud services and beyond.

Affected services included:

  • Google Cloud Console, App Engine, Cloud DNS, and Dataflow
  • Identity and Access Management (IAM), Pub/Sub, and Dialogflow
  • Apigee API Management and other backend services
  • 30+ additional GCP products across the Americas, EMEA, APAC, and Africa

A screenshot of a computer screenAI-generated content may be incorrect., Picture

But it didn’t stop there. The outage reverberated outward, hitting major platforms like Discord, Spotify, Snapchat, Twitch, and Cloudflare — digital giants that depend on GCP under the hood.

Recovery began within hours for most regions. But in us-central1, where a quota policy database was overwhelmed, the impact lingered well into the afternoon

How was it detected?

Within a minute, Catchpoint Internet Sonar started flagging anomalies in services like Google Drive and App Engine at 1:50 PM ET

A screenshot of a phoneAI-generated content may be incorrect., Picture

Catchpoint Internet Sonar view

A map of the world with orange dotsAI-generated content may be incorrect., Picture

Apigee service disruptions detected across 109 cities worldwide

A screenshot of a mapAI-generated content may be incorrect., Picture

Google Cloud service degradation detected across 157 cities, with active incidents spanning North America, Latin America, Europe, and Asia-Pacific

Picture 1, Picture

This screenshot shows a spike in failed tests across multiple workflows. The failure pattern is sudden, sustained, and correlated across multiple test types — a classic signal of a widespread upstream infrastructure disruption.

A screenshot of a graphAI-generated content may be incorrect., Picture

This chart from one of our users shows a sharp drop in availability and a spike in checkout failures across multiple countries — confirming global impact on user-facing transactions during the outage window.

A screenshot of a computerAI-generated content may be incorrect., Picture

Catchpoint Internet Stack Map

This Internet Stack Map shows a clear breakdown in service dependencies, with Google Cloud and Apigee at the center of the disruption.

You can see how multiple layers — including analytics, cloud logging, storage, and identity — all failed in tandem. These failures directly impacted third-party tools, CDNs, and SaaS integrations downstream.

The alert indicators show transaction failures and response errors cascading outward, affecting not just infrastructure, but also the digital experience of users relying on these tools in real time.

This stack map captures what many teams experienced: when one layer goes, it doesn’t go alone.

No official word — yet.

While synthetic tests and customer data clearly showed the disruption unfolding in real time, Google Cloud’s first public acknowledgment didn’t arrive until 2:46 PM ET — nearly an hour after anomalies first appeared and services began failing globally.

A close-up of a textAI-generated content may be incorrect., Picture

Google cloud status page

During that gap, the GCP status page remained green.  

For teams on the front lines — whether SREs, DevOps, or customer support — that matters. It creates doubt. Are we at fault? Is this a local issue? Can we act, or do we wait?

This isn’t a critique of Google’s infrastructure — it’s a reality of operating at scale. Status pages are often downstream from detection. They’re built for caution, not speed. And by the time they update, most teams have already lost the window for a proactive response.

What was the impact?

The blast radius was wide — and layered.

  • Direct GCP customers lost access to core infrastructure: management consoles, APIs, storage, and authentication services.
  • SaaS and enterprise platforms saw critical user workflows stall for over two hours — with transaction-level failures visible in real-time test data.
  • Consumer platforms including Discord, Snapchat, Twitch, and Spotify suffered cascading slowdowns due to their dependencies on affected Google services.
  • Geographic scope: Failures were observed across North America, EMEA, APAC, and Latin America.

What began as a quiet configuration change rippled outward — breaking not just cloud services, but the trust and functionality layered on top of them.

Key lessons

Outages like this don’t just break systems — they expose assumptions. They challenge how we monitor, communicate, and respond. Here’s what the June 12 Google Cloud incident reinforced.

#1 You’re never too big to go down

No provider — not even Google — is immune to large-scale outages. The events of June 12 made that clear. A routine quota update cascaded into global downtime, reminding every digital business just how quickly dependencies can unravel.

“This is a timely wakeup call: even hyperscalers like Google aren’t immune to largescale outages. In today’s interconnected digital landscape, external observability via tools like Catchpoint isn’t optional — it’s essential.”
Mehdi Daoudi, CEO & Co-founder, Catchpoint

#2 The domino effect is real

When a hyperscaler like Google stumbles, it’s not just their systems that suffer — it’s everything built on top of them. That includes SaaS platforms, third-party APIs, and the end-user experiences they power.

This incident showed just how fast a single point of failure can ripple outward, breaking things that appear unrelated at first glance.

“Your system is as resilient as the weakest of its components. Which means it only takes one dependency to bring down an entire system. If the authentication service used by a critical API is down, your system is down — even if everything else is working.”
Gerardo Dada, Field CTO, Catchpoint

#3 Status pages aren’t enough

Provider dashboards serve a purpose — but they aren’t built for real-time incident response. They’re designed to be cautious, accurate, and measured — which often means they lag behind the actual impact felt by users and customers.

This isn’t a knock on any provider — it’s a reflection of operational reality at scale. The lesson isn’t to expect more from status pages, but to expect more from your own visibility.

#4 Design for resilience

Outages happen — even to the best-engineered platforms on the planet. What matters isn’t avoiding every failure, but mitigating the blast radius when one occurs.

That means building systems that assume things will break. Architecting across regions. Diversifying providers. Creating failovers that actually fail over.

“One thing we see with Internet Sonar is even the greatest companies with the most advanced tech can suffer outages. It happens to the best. All the more reason for a multi-multi architecture.”
Matt Izzo, VP Product, Catchpoint

#5 Your monitoring can’t live in the same cloud you’re trying to monitor

Outages like this raise an uncomfortable but important question: Can your monitoring still see what’s happening when your cloud provider goes dark?

Many monitoring tools rely on cloud-hosted vantage points — often within the very infrastructure they’re meant to observe. When that cloud provider has an outage, your diagnostics might vanish along with it. Synthetic tests hosted in hyperscalers rarely reflect real user environments, and they mask failures within the provider itself.

  • Synthetic tests inside the     cloud often     miss real-world issues like DNS errors, CDN disruptions, or ISP-level     failures.
  • Cloud-only testing creates a false sense of     security — you’re essentially monitoring yourself, from yourself.

True visibility comes from outside the cloud, from the edge of the Internet where real users live. Because monitoring should never go down when you need itmost.

Staying resilient when the Internet isn’t

The June 12 outage didn’t just disrupt Google Cloud — it created cascading effects across Cloudflare, CDNs, productivity platforms, and AI tools. It’s a vivid reminder of how deeply interconnected and interdependent digital systems have become.

These weren’t abstract failures — they had real consequences. Picture a hospital unable to access patient records or drug databases due to a cloud outage. This isn’t about someone being unable to order lunch. This is about critical services failing at the worst possible moment. This is life and death.

At Catchpoint, we take that responsibility seriously. That’s why during the incident:

  • Our monitoring continued from outside the public cloud
  • While some customers faced issues reaching third-party services, Catchpoint itself remained visible and stable throughout

We remain committed to resilience, transparency, and visibility — not just for our own platform, but across the entire Internet ecosystem that powers your business.

Tools that made a difference

Catchpoint users navigating the Google Cloud outage had two key advantages on their side:

  • Internet Sonar offers real-time, independent monitoring of the internet’s core services — helping you detect third-party issues early, understand their scope, and act decisively.
  • Internet Stack Map provides a live view of your service’s dependencies, making it easy to trace cascading failures and pinpoint root causes fast.

Schedule a demo to learn how Catchpoint can help your team stay ahead of the next incident.

This is some text inside of a div block.

You might also like

SRE Report: Why fast is what users trust

SRE Report 2026: What surprised us, what didn't, and why the gaps matter most

The SRE Report 2026: Defensible Ns