off-prem
Storm sends winds whipping beyond cloud giant, knocking out services including Hugging Face and the UK's National Lottery
Amazon Web Services (AWS) is experiencing another outage after a CloudFront issue began throwing 5xx errors, knocking a string of websites and online services offline across multiple regions.
According to AWS, the issue began at 0145 PDT (0945 UTC) this morning and affects CloudFront customers using VPC Origins. This is a relatively new CloudFront feature that lets customers serve applications running behind private load balancers through CloudFront without exposing their back-end infrastructure to the public internet.
The cloud giant said customers using other origin types are not impacted, and suggested that anyone who doesn't strictly need VPC Origins could switch origin types as a temporary workaround while engineers work on a fix.
“We are experiencing increased 5xx errors for CloudFront customers utilizing VPC Origins connectivity,” the cloud giant said on its service status page. “Our engineers are engaged and are actively working to mitigate impact."
At 03:18 PDT (118BST, 1018 UTC), it added: "We continue working to resolve the increased 5xx errors for CloudFront customers utilizing VPC Origins connectivity. Customers utilizing other origin types remain unaffected by this issue.
"Based on our investigation, we believe the root cause is related to a packet processing subsystem responsible for routing requests from CloudFront's edge locations to resources within customer VPCs. We continue to recommend that customers who are able to do so temporarily change their origin type to resolve the errors." It promised another update in the next hour.
Users trying to reach sites affected by the CloudFront issues are being met with an error page that reads: “We can't connect to the server for this app or website at this time. There might be too much traffic or a configuration error. Try again later, or contact the app or website”
Among the early casualties was the AI developer platform Hugging Face, which acknowledged that its service was unavailable "from most regions in the world" due to the AWS outage while it worked on mitigation. Meanwhile, the UK's National Lottery admitted in a post on X that players were unable to access its website and mobile app due to what it described as a wider AWS outage, advising hopeful millionaires to try refreshing later.
Gamers weren't spared either. Reports quickly piled up on Reddit from Fallout 76 players wondering why Bethesda's post-apocalyptic wasteland had become more inaccessible than usual, while various other threads rapidly filled with reports from disgruntled AWS customers, all describing the same symptom: CloudFront distributions returning 5xx errors while other AWS services carried on as normal.
AWS had not disclosed the cause of the borkage and didn’t immediately respond to The Register’s questions. While it said the outage was limited to one CloudFront configuration, the list of broken services made it look considerably less limited. ®