DevOps

Plenty of time for bad actors to grab data or hit you with a giant bill

You know your Google API key has leaked so you rush to disable it before bad actors can start running up charges on your account. Bad news: According to security researchers at Aikido, people can use the API keys for up to 23 minutes after a user deletes them, creating a window of opportunity that, when combined with Google’s automatic billing tier upgrades, can devastate victims.

“We've identified a substantial window where an attacker with access to a leaked Google API key can continue to misuse that credential, after the user believes the key is revoked,” Joseph Leon, a security researcher with Aikido, told The Register. “In that window, an attacker could run up charges, pull sensitive files uploaded to Gemini, and exfiltrate cached context.” 

Aikido tested the gap during 10 trials over two days. In each trial, researchers created an API key, deleted it, and then sent three to five authenticated requests per second until no valid response came back for several minutes. 

From the time a user deletes the Google API key to when it can no longer be used propagates gradually across Google's infrastructure, he said. Some servers reject the key within seconds while others keep accepting it for 23 minutes.

What this means is that an attacker holding a deleted key can repeatedly send requests until one reaches a server that has not caught up, Leon said. If Gemini is enabled on the project, they can dump files that were uploaded and exfiltrate cached conversations.

The paper cited a similar problem researchers disclosed in December involving AWS keys. In that case, after deletion, attackers had a four-second window to exploit, and researchers showed how they could create new credentials in that time. 

“Four seconds was enough to matter on AWS,” Leon wrote in the paper. “Given recent attention to Google API keys used to access Gemini, we set out to measure how long Google's API key revocation window remains open.” 

Flaws can hit devs with huge surprise bills

The Register has reported numerous cases of Google API key abuse in which developers are suddenly hit with five figure bills after their credentials are compromised. 

The problem was compounded in April after Google reworked its billing policy to include spending tiers for users. While developers initially thought of it as a way to limit costs, Google automatically upgrades that spending tier to the next highest level without their knowledge. 

For users who have been working with Google for more than 30 days and have spent more than $1,000 over the lifetime of the account, their cap can be increased from $250 to $100,000 if their usage spikes – a windfall for crooks if the credentials fall into the wrong hands.

Developers whose Google API keys were stolen told The Register that their bills rocketed up to five figures minutes after their credentials were stolen, as bad actors loaded up on Google’s Gemini models such as Nano Banana and its video production model Veo 3. 

Google issued refunds in the three instances that The Register brought to its attention, returning $154,000 to those developers.

The victims told The Register that, during the attack, they were frantically trying to shut down the spending and turn off access to their projects even as costs climbed by thousands of dollars. Leon said in cases where a Google developer tries to shut off access to their account, deleting the API key will still give crooks time to inflict damage. 

“It's hard to put a dollar figure on it,” Leon told us. “The window averaged 16 minutes in our testing and stretched to nearly 23 at the worst. During that window, the success rate is wildly unpredictable. We saw minutes where over 90% of requests still authenticated, and others where fewer than 1% did. An attacker who knows this can send requests at high volume to maximize their odds of hitting a server that hasn't caught up. For Google API keys with Gemini access, the damage isn't just a compute bill. It's the files and cached context an attacker can exfiltrate before the key actually dies.” 

Using VMs, Aikido tested its findings across three Google Cloud regions – east coast US, western Europe, and southeast Asia – then they spot checked those results on different dates. For each trial, Aikido deleted a single API key and sent requests from each of the three VMs in parallel, Leon wrote in the paper. 

“VMs further from the US picked up the deletion faster, which is the opposite of what you'd expect. We can't say exactly why from the outside. Google's request routing is more complex than ‘VM region equals server region,’ and a VM in Singapore isn't necessarily talking to servers in Singapore,” the paper states. “But the pattern was consistent across trials, which points to something about regional infrastructure, caching, or routing affinity driving the difference.” 

The trial used keys with access to Gemini, but he observed the same behavior with keys scoped to other GCP APIs, such as BigQuery and Maps. Google has built faster revocation for other credential types, Leon said. 

He said Google’s service account API credential revocations propagate in about 5 seconds. Gemini's newer API key format – the one that starts with AQ – propagates in about a minute. 

“Both run at Google scale. Both suggest this is technically solvable for Google API keys, too,” Leon wrote. 

But Google told Aikido it has no plans to address the 23-minute gap researchers found with its other API keys. 

“After reviewing our report, they closed it as ‘Won't Fix (Infeasible)’ with the comment ‘the delay due to propagation of the deletion of these keys is working as intended,’ “ Leon told us.

The Register has reached out to Google about this research, but has not yet received a response.  ®