off-prem

An expired card, overzealous spam filter, and broken authenticator create havoc with a very clear moral 

Nothing can ruin the end of a week like finding out that all of the websites and hosted email you’re responsible for are offline, which is exactly what happened to Christopher Bradbury and his web design and development firm, Digital Takumi last week. AWS has since resolved the situation, but Bradbury's mistakes can serve as a useful lesson to others of what not to do.

As Bradbury explained to The Register in an email, he noticed last Thursday, July 16, that all the websites and Google Workspace email accounts connected to domains he manages were offline. All of those sites are hosted through Route 53, AWS’ combined DNS/hosting service, and none of them were resolving. 

Bradbury went digging through his emails to figure out if there was anything to point to the failures, and sure enough: In his spam folder were numerous messages from AWS telling him a payment card on file had expired, and warning him that the account used to host all those customer websites through Route 53 was going to be suspended unless he took action. He didn’t obviously, because he didn’t realize there was an issue.

“AWS had been sending billing notifications, but unfortunately they had been filtered into a spam folder and, in some cases, were being delivered to an employee who had since left the business,” Bradbury told us. 

“I accept responsibility for missing those notifications,” he added, but that didn't help his customers' websites get back online. 

The situation could have been resolved sooner, but Bradbury had made some other mistakes as well:

“The root account had MFA enabled using a software authenticator that had been stored on an older laptop which has since suffered a motherboard failure,” Bradbury explained. Without access to that authentication code generator, he was unable to get into the account. Rather than getting the dead-laptop-with-a-critical-authenticator-on-it problem resolved, he was just relying on MFA emails instead – not the best idea.

"I've been bypassing the device key for quite a while by just using the recovery MFA via my email," Bradbury told us. "It works, I get access. However if I just had my Passkey up to date it would have let me right in and I could have solved this."

And then there was the email address where those MFA codes were going:

“The AWS recovery process required email verification using the registered root email address,” Bradbury told us. “That email address belonged to one of the domains whose DNS was hosted in the suspended AWS account, meaning I couldn't receive the verification email.”

The account recovery tango

Bradbury’s next option was contacting AWS from a different email address, which didn’t go anywhere. He created another AWS account and purchased business support access on someone’s recommendation, but the support engineers he spoke to using that method wouldn’t discuss the other account until he verified he owned the one in question. 

“Over the following days I spoke with several AWS teams, including Billing and Account Recovery. I was transferred between teams multiple times, but nobody was able to complete the ownership verification or restore access to the account,” Bradbury told us.

He wanted to pay AWS, Bradbury told us, but it took a while for them to be able to take his money. “The practical consequence is that I cannot log into the AWS account, cannot receive email at the registered root address, cannot access the MFA device, cannot update the expired payment method, and therefore cannot pay the outstanding invoices from inside the account.” 

While we were working on this story, after speaking to both Amazon and Bradbury, he contacted us to say that access to the sites had been restored, and that he had logged in, paid the back invoices, updated his payment method, reset his MFA keys, and generally taken care of all the stuff he had been putting off until all this happened. It's unfortunate that someone has to go through AWS billing hell to serve as an example to others, so let this be a warning to anyone else managing client websites through AWS.

"Firstly pay your AWS bills," Bradbury said in an email, adding that anyone running an AWS hosting account also makes sure their emergency recovery email isn't on the same domain as one of the sites they manage through that profile. That, and "stop using shortcuts" when it comes to MFA. 

"This isn't a big infrastructure account, we run a single company marketing website and some domains through Route 53," Bradbury explained. "It just shows that even the most modest instances of AWS can be absolutely business critical and you need to use proper practices and processes." ®