Australia-based AI consultant and founder of Agentic Labs Jesse Davies woke up to an unpleasant surprise earlier this month: A Google Cloud bill of $25,672.86 AUD (approximately $18,391.78 USD) — even though there was a budget of $10 AUD (approximately $7 USD) on his account. And it happened overnight.
According to Davies' account on LinkedIn, he was well-versed with Google AI Studio and had followed practices such as per-project API keys, separate billing accounts, two-factor authentication, and Cloud audit logging. However, it only took a single weak link to nullify those precautions, as evidenced by the shockingly large overnight bill. On top of that, Davies found nine Google Cloud safety features that should have prevented this incident — but that were turned off by default.
Go deeper with TH Premium: AI and data centers
"The attacker didn't steal my key. They found a Cloud Run service I'd published from AI Studio months earlier, hit the public URL, and Google's own proxy signed every request on their behalf using the API key stored as a plaintext environment variable in the container," Davies wrote in his LinkedIn post.
"Even though it was public, the link wasn't shared or indexed anywhere. By the time I got a budget alert the next morning, A$10,000 had already been charged to my credit card, now getting insufficient funds. I was still talking to Google support when A$15,000 more came through."
What’s worse was that Google automatically upgraded the tier of Davies' account without any notification. The account was initially at Tier 2, which had a $2,000 limit, but Google automatically upgraded it to the next level when the account crossed the $1,000 threshold during the incident. This increased the cap to between $20,000 and $100,000. While this is likely designed to make it easier for a service to scale, it also has the unwanted effect of costing the user more than intended, e.g. if they are the victim of an attack.
Went to bed with a $10 budget alert. Woke up to $25,672.86 in debt to Google Cloud. from r/googlecloud
Their headaches did not end here, though. It took several days before Davies was able to get through to a real human customer support. Thankfully, it seems that the charge has been waived, while the transactions that actually pushed through were credited back by their bank. Still, the issue isn’t settled, and Davies has a meeting scheduled with Google managers to talk about the case.
Article continues below
Davies also shared the experience on Reddit, on the r/googlecloud subreddit, and asked if other users had similar stories to share. It turns out they did — several other users reported getting hit with insane bills, including one commenter from Japan who said that they were hit with a $44,000 bill that ballooned to $128,000 even after they paused the API. And last month, we covered a case in which an API thief racked up $82,314.44 in charges on an account that typically saw around $180 per month.
Cybersecurity firm Truffle Security Co. has already highlighted the risks associated with Google Cloud using a single API key format. These API keys were previously used as project identifiers, but when the Gemini API is activated on any Google Cloud project, these existing API keys become Gemini credentials — allowing anyone who can copy them to rack up AI bills. So... it's likely we'll see more horror stories of shocking API bills if Google doesn't update its Gemini policies.
Follow Tom's Hardware on Google News, or add us as a preferred source, to get our latest news, analysis, & reviews in your feeds.