** Update ** Learn how to protect your environment in our latest post

Nearly everything we do online these days runs through applications and databases in the cloud. While leaky storage buckets get a lot of attention, database exposure is the bigger risk for most companies because each one can contain millions or even billions of sensitive records. Every CISO’s nightmare is someone getting their access keys and exfiltrating gigabytes of data in one fell swoop.

▶️Watch our ChaosDB: How We Hacked Databases of Thousands of Azure Customers BlackHat 2021 live talk

So you can imagine our surprise when we were able to gain complete unrestricted access to the accounts and databases of several thousand Microsoft Azure customers, including many Fortune 500 companies. As part of building a market-leading CNAPP, Wiz’s security research team (that’s us) constantly looks for new attack surfaces in the cloud, and two weeks ago we discovered an unprecedented breach that affects Azure’s flagship database service, Cosmos DB.

Some of the world’s biggest businesses (see their website) use Cosmos DB to manage massive amounts of data from around the world in near real-time. As one of the simplest and most flexible ways for developers to store data, it powers critical business functions like processing millions of prescription transactions or managing customer order flows on e-commerce sites.

Database exposures have become alarmingly common in recent years as more companies move to the cloud, and the culprit is usually a misconfiguration in the customer’s environment. In this case, customers were not at fault.

Rather, a series of flaws in a Cosmos DB feature created a loophole allowing any user to download, delete or manipulate a massive collection of commercial databases, as well as read/write access to the underlying architecture of Cosmos DB.

We named this vulnerability #ChaosDB. Exploiting it was trivial and required no other credentials.

Part 1: Stealing primary keys of Cosmos DB customers

First, we gained access to customers’ Cosmos DB primary keys. Primary keys are the holy grail for attackers – they are long-lived and allow full READ/WRITE/DELETE access to customer data.

In 2019, Microsoft added a feature called Jupyter Notebook to Cosmos DB that lets customers visualize their data and create customized views (see image below). The feature was automatically turned on for all Cosmos DBs in February 2021.

A series of misconfigurations in the notebook feature opened up a new attack vector we were able to exploit. In short, the notebook container allowed for a privilege escalation into other customer notebooks (we’ll share technical details on the escalation soon).

Part 2: Accessing customer data in Cosmos DB

Next, after harvesting the Cosmos DB secrets, we showed that an attacker can leverage these keys for full admin access to all the data stored in the affected Cosmos DB accounts.

We exfiltrated the keys to gain long-term access to the customer assets and data. We could then control the customer Cosmos DB directly from the internet, with full read/write/delete permissions.

Now imagine repeating this process for thousands of different customers across more than 30 regions...

Impact and Scope

Microsoft’s Security Team deserves enormous credit for taking immediate action to address the problem. We rarely see security teams move so fast! They disabled the vulnerable notebook feature within 48 hours after we reported it. It’s still turned off for all customers pending a security redesign.

However, customers may still be impacted since their primary access keys were potentially exposed. These are long-lived secrets and in the event of a breach, an attacker could use the key to exfiltrate databases. Today Microsoft notified over 30% of Cosmos DB customers that they need to manually rotate their access keys to mitigate this exposure.

Microsoft only emailed customers that were affected during our short (approximately weeklong) research period. However, we believe many more Cosmos DB customers may be at risk. The vulnerability has been exploitable for at least several months, possibly years.

As a precaution, we urge every Cosmos DB customer to take steps to protect their information.

Every Cosmos DB account that uses the notebook feature, or that was created after January 2021, is potentially at risk. Starting this February, every newly created Cosmos DB account had the notebook feature enabled by default and their Primary Key could have been exposed even if the customer was not aware of it and never used the feature.

If the customer didn’t use the feature in the first three days, it was automatically disabled. An attacker who exploited the vulnerability during that window could obtain the Primary Key and have ongoing access to the Cosmos DB account.

Wiz customer Jim Routh, CISO at MassMutual, had this to say about ChaosDB:

This discovery clearly confirms the need for enterprises to improve DevOps configuration management processes with data protection capabilities.

If you believe your organization may be impacted by ChaosDB, please follow the steps described in this article for detailed instructions on how to regenerate and rotate keys. We welcome your questions and feedback! Email us at research@wiz.io.

Nir Ohfeld and Sagi Tzadik are security researchers at Wiz.