tl;dr When you delete a Google API key, it says it’s immediately deleted. Our testing says ~23 minutes. During that window, an attacker with a leaked key keeps access to your data and enabled APIs (including Gemini). You have no way to revoke it faster or confirm when it stops working. Google closed our report as “won’t fix”.
When you delete an API key, you expect access to end immediately.
Google API keys don’t work this way. Revocation propagates gradually across Google's infrastructure. Some servers reject the key within seconds, others keep accepting it for 23 minutes.
An attacker holding your deleted key can keep sending requests until one reaches a server that has not caught up. If Gemini is enabled on the project, they can dump files you have uploaded and exfiltrate cached conversations.
The GCP console will not show the key, and it will not tell you the key is still working. You are trusting Google's infrastructure to eventually catch up.
Many of Google Cloud's services are eventually consistent by design. In this model, updates propagate gradually across its servers rather than all at once. This tradeoff lets Google scale globally and stay fast, and for most services, the lag is invisible. But for authentication, that tradeoff is harder to justify.
Credential revocation delays are exploitable. A few months ago, Eduard Agavriloae disclosed a 4-second delay that enabled deleted AWS access keys to create new credentials. Four seconds was enough to matter on AWS.
Given recent attention to Google API keys used to access Gemini, we set out to measure how long Google's API key revocation window remains open.
The revocation window is the time between when you delete a key and the last successful authentication.
If the window is a few microseconds, the behavior matches what users expect. If it’s longer, every extra second gives attackers more time to misuse a stolen key.
To measure Google’s API key revocation window, we ran 10 trials over two days.
In each trial, we created an API key, deleted it, and sent 3-5 authenticated requests per second until no valid response came back for several minutes.
We picked that rate because we don’t control how Google routes our requests to different authentication servers across the world. The volume was meant to rotate through as many of them as possible per trial, while still respecting Google's infrastructure. We can't say whether a higher request rate would extend the observed window. An attacker has no reason to throttle the way we did, so it's worth noting our numbers may not represent the worst case.
For completeness, we also spot checked our work a few weeks later to ensure the behavior we saw was not due to transient network issues.
Across ten trials, the longest window was nearly 23 minutes! That is an incredibly long time for a deleted key to still authenticate successfully.
The shortest window was nearly 8 minutes, and the median was around 16 minutes. Across trials, the success rate was highly unpredictable: one minute after deletion, one trial saw 79% of requests succeed while another saw only 5%.
An attacker holding a stolen key does not see a clean cutoff or a predictable decay. Access keeps working, unevenly, until it eventually just stops.
To illustrate the inconsistency in Google's infrastructure, we monitored one of our trials using the "Traffic by Credential" graph in GCP. The lower (blue) line shows successful authentications and reflects the length of the revocation window.
We didn't expect to see any other data, but a second line appeared. The upper (green) line plots rejected requests and is labeled apikey:UNKNOWN. We had (wrongly) assumed that invalid requests would simply be dropped without any project attribution. Instead, GCP includes them in this graph.
To better understand the mysterious apikey:UNKNOWN value, we attempted to authenticate with a days-old deleted API key. Surprisingly, those requests showed up in the same graph, bundled into the same apikey:UNKNOWN group.
Our best guess is that Google preserves project attribution across key deletion in case users decide to restore the credential.
For IR teams investigating a leaked credential, the apikey:UNKNOWN value can be confusing. Any request made with any deleted Google API key gets bundled into the same UNKNOWN category, making it hard to tell which requests relate to a particular credential. A request in that bucket could mean a threat actor is still trying to authenticate with the deleted key, or it could be a legitimate service running on an unrelated, outdated one.
If you're inside the 23-minute revocation window, look for valid authentications using the leaked key. If you're outside that window, the risk appears extremely low.
We ran the first experiment from a residential IP address on the East Coast of the US. We hypothesized that running a similar experiment across VMs in different GCP regions might expose additional inconsistencies.
We spun up a VM in three regions: us-east1, europe-west1, and asia-southeast1. We then ran 5 trials. For each trial, we deleted a single API key and sent requests from each of the three VMs in parallel.
Two things stuck out. First, immediately after a key is deleted, VMs in different regions saw very different authentication success rates. The table below maps the percentage of valid requests in the first minute by region.
Look at trial 1: us-east1 82%, europe-west1 60%, asia-southeast1 32%. VMs further from the US picked up the deletion faster, which is the opposite of what you'd expect. We can't say exactly why from the outside. Google's request routing is more complex than "VM region equals server region," and a VM in Singapore isn't necessarily talking to servers in Singapore. But the pattern was consistent across trials, which points to something about regional infrastructure, caching, or routing affinity driving the difference.
Second, the regional gap was not just a first-minute thing. Across the full window, asia-southeast1 had a median request authentication success rate of 22%, while us-east1 and europe-west1 both came in around 49%. Asia stayed lower minute by minute, not just at the start.
Whatever's driving these differences, the server’s location clearly shapes how a deleted key behaves after deletion.
Our trials all used keys with access to Gemini, but we observed the same behavior with keys scoped to other GCP APIs, such as BigQuery and Maps. The delay is a property of the credential type, not of which APIs are enabled on the project.
For completeness, we tested two other Google credential types:
Both run at Google scale, and both revoke much faster than the 23 minutes we measured for Google API keys.
We reported this to Google. Google closed the report as “won’t fix”. The team's position, as we understand it, is that propagation delay is a known property of the system and not a security issue.
While Google publicly documents that their IAM API is eventually consistent, they don’t specifically document this behavior for Google API keys.
[Caption: Screenshot from Google’s IAM API documentation page.]
Distributed systems at Google's scale are hard, and this is not a critique of the GCP IAM team. But a 23-minute revocation window is fundamentally at odds with what users expect from a delete button. The gap between that expectation and Google's behavior highlights four problems:
1. The UI is extremely misleading.
When you delete the key, Google removes it from your view and tells you "Once deleted, it can no longer be used to make API requests." That statement is demonstrably false. The user has no way to know whether the key is still live, no way to speed up revocation, and no way to confirm when it has fully stopped working.
2. Google built faster revocation for other credential types.
Service account credential revocations propagate in about 5 seconds. Gemini's newer API key format propagates in about a minute. Both run at Google scale. Both suggest this is technically solvable for Google API keys, too.
3. Long consistency windows are not compatible with authentication.
The expectation when you delete a credential is that the credential is dead. Even a few seconds of delay matters, as Eduard's AWS research showed last year.
4. Long revocation windows break Just-in-Time credential minting.
A service provider that wants to dynamically mint Google API keys has to build in a 23-minute buffer after revocation before the key is guaranteed dead. That is incompatible with how JIT is supposed to work.
Until Google ships faster revocation, the responsibility for closing this gap falls on users. Two things help.
1. Treat key deletion as a 30-minute operation, not an instant one.
If you are responding to a leaked Google API key, assume the key is live for 30 minutes after clicking delete. That gives a bit of a buffer beyond the maximum 23 minutes we observed. Plan the rest of your incident response around that window.
2. Watch usage during the window.
Under "Enabled APIs and services" in the GCP console, review API requests by credential. If you see unexpected usage from that credential after deletion, someone could be actively exploiting it.
Large eventual consistency windows are the wrong design choice for credential revocation. Until Google changes that, treat every key deletion as a 30-minute operation and watch the window for misuse.
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。