Managed Identity and virtual machine failures triggered knock-on problems throughout cloud platform
Microsoft has reported two Azure service wobbles in as many days, including a disruption affecting Virtual Machine management ops yesterday and a Managed Identity for Azure resources outage in East US and West US regions today.
According to Microsoft, today's Managed Identity for Azure resources issue impacted the East US and West US regions and lasted almost six hours, from 0015 UTC to 0605 UTC. The problem hit when users were trying to create, update, delete, or acquire tokens.
Security vulnerabilities and system downtime often stem from the manual management of secrets and certificates. Managed Identity for Azure resources is intended to removes the burden of credential management from developers.
REG AD
The outage, which has been mitigated, "impacted dependent services such as Azure Synapse Analytics, Azure Databricks, Azure Stream Analytics, Azure Kubernetes Service, Microsoft Copilot Studio, Azure Chaos Studio, Azure Database for PostgreSQL Flexible Servers, Azure Container Apps, and Azure AI Video Indexer." So rather a lot.
REG AD
The blackout followed hot on the heels of another, where service management operations for virtual machines resulted in error notifications. Microsoft gave the example of "create, delete, update, scaling, start, stop."
As with this morning's downtime, yesterday's problem was exacerbated because services had dependencies on these operations, including Azure Arc Enabled Servers, Azure Batch, Azure Cache for Redis, Azure Container Apps, Azure DevOps (ADO), Azure Kubernetes Service (AKS), Azure Backup, Azure Load Testing, Azure Firewall, Azure Search, Azure Virtual Machine Scale Sets (VMSS), and GitHub.
Microsoft acknowledged the issue last night at 1946 UTC but did not provide a timeline for mitigation. The Register asked for more details and will update this piece should the company respond.
GitHub noted degraded performance for Actions at 1903 UTC, and it took until 0056 UTC on February 3 for the code shack to state that the incident was resolved.
As for what caused the virtual machine outage, Microsoft said: "We identified the root cause as a configuration change that unintentionally restricted public access to certain Microsoft‑managed storage accounts used to host virtual machine extension packages."
Yes, Microsoft made a configuration change that inadvertently broke things for developers in multiple regions. Those same developers would likely have some advice for Microsoft on testing changes before deploying to production.
The issues highlight the interdependencies between cloud services. A faulty configuration change in one place can result in a cascade of problems elsewhere. ®