off-prem

Amazon asks users not to panic as it works to fix the bug

Your AWS billing estimate might look just a little inflated right now. If you woke up to find an email from Amazon Web Services this morning telling you that you’d gone over your billing threshold by a few hundred million dollars, don’t panic: Something’s gone wrong in the AWS Billing Console, the company admitted. 

An open issue on the AWS Health Dashboard (archived copy at the time of writing) popped up at 1:33 am Pacific time on Friday informing users that Cost Explorer was “reflecting inaccurate estimated billing data.” As of writing, the issue is still unresolved despite AWS trying several different things to get it fixed.

The company apparently identified the root cause within an hour and a half of beginning its investigation, only describing it as “an issue with unit pricing within the estimated billing computation subsystem.” 

AWS followed up by pausing estimated bill updates, saying customers would continue to see the inflated figures already displayed, but that those estimates would not increase further.

“The displayed billing estimates do not reflect actual usage and charges,” AWS explained, noting that customers don’t need to take any action, like, we imagine, flooding the help portal with tickets telling them what they already know, for instance. 

“Once the issue has been mitigated, we expect full resolution to take multiple hours as we work through recomputing the estimated billing data,” AWS added. 

The company said it was working through multiple mitigation paths in parallel, including “reverting to the last known good estimated bill computation,” and “rolling back a recent change to the billing computation subsystem.”

We owe HOW much?

Users took to Reddit and Hacker News this morning to report they’d received overage emails for massive amounts - we weren’t exaggerating with that hundreds of millions opening line. If anything, it was an understatement.

Screenshots posted in the Reddit thread showed one user whose AWS charges totaled just $0.19 last month receiving an estimated bill of nearly $2.5 billion. Others in the thread claimed to have received estimated monthly charges ranging from $126,000 to as much as $2.5 trillion. Hacker News users similarly reported estimates in the billions.

Amazon said the figures shown in customers' accounts were inaccurate estimates rather than actual charges. As for when users might see their billing portal reflect an accurate number, that could take a while. 

AWS’ most recent update to the thread said that the billing computation subsystem change rollback “did not resolve the issue,” and that it’s still trying to figure out what to do. 

AWS declined to explain the issue aside from pointing us to the dashboard page linked above. We'll be keeping an eye on this developing story and update it as we learn more. ®