Cloudflare uses a vendor called Verkada for cameras in our offices in San Francisco, Austin, New York, London and Singapore. These cameras are used at the entrances, exits and main thoroughfares of our offices and have been part of maintaining the security of offices that have been closed for almost a year.

Yesterday, we were notified of a breach of Verkada that allowed a hacker to access Verkada’s internal support tools to manage those cameras remotely, as well as access them through a remote root shell. As soon as we were notified of the breach, we proceeded to shut down the cameras in all our office locations to prevent further access.

To be clear: this hack affected the cameras and nothing else. No customer data was accessed, no production systems, no databases, no encryption keys, nothing. Some press reports indicate that we use a facial recognition feature available in Verkada. This is not true. We do not.

Our internal systems follow the same Zero Trust model that we provide to our customers, and as such our corporate office networks are not implicitly trusted by our other locations or data centers. From a security point of view connecting from one of our corporate locations is no different than connecting from a non-Cloudflare location.

While these camera devices were breached, it’s important to understand that inherently every employee has something like a root shell on a corporate network. It is simply not enough to rely on background checks, training on phishing and other threats or policies about not visiting questionable websites to keep malware off your network. This incident emphasizes the importance of the Zero Trust model that Cloudflare follows and provides to customers, which ensures that if any one system or vendor is compromised, it does not compromise the entire organization.

For example, a claim was made on Twitter that it might have been possible to access Cloudflare CEO Matthew Prince’s laptop.

This claim is spurious for a few reasons. Firstly, Matthew isn’t in the San Francisco office because of COVID-19. None of us are. So being on the corporate San Francisco network doesn’t get you closer to his or my machine.

Secondly, even if, somehow, an attacker were to get into his laptop they’d still have to get past the per-application, hard-key protected access control through our Zero Trust product Cloudflare Access.

And another claim was that the attackers had a “root shell inside Cloudflare’s network”.

That sounds scary but we don’t trust our corporate network; we use our products, such as Cloudflare Access, to control access to resources. The fact that the attacker had access to a machine inside the corporate network is no better than the kind of access they’d have had if they’d connected to our corporate WiFi network. The network isn't important, it's the access control that matters.

Of course, if we had been using the old castle-and-moat style of corporate networking (where anything and anyone on the corporate network are inherently trusted) the outcome could have been different.

This is why Zero Trust is so powerful. It allowed us all to work from home because of COVID-19 and it means that an attacker who got into the office network doesn’t get any further.

Internal Monitoring

Our monitoring and network intrusion detection system were able to capture the Verkada compromise and show exactly what network connections were made from the cameras. Additionally Verkada was able to provide us with a log of commands executed on our cameras.

Some press stories contain an image of the “root shell” on one of the cameras. It shows the attacker having executed commands like whoami, ifconfig and curl ifconfig.me.

Here are the commands executed by the intruder on March 8 (we are waiting for Verkada to supply us with the commands executed on March 9 but our other monitoring and logging tools give us a good sense of what those commands will be):

2358 Singapore       curl ifconfig.me

2357 New York        curl ifconfig.me

2352 San Francisco   curl ifconfig.me
2352 San Francisco   ifconfig
2352 San Francisco   whoami

1451 San Francisco   ifconfig
1451 San Francisco   whoami

Here’s a log of the traffic from the cameras. Let’s start with unusual DNS queries. This shows DNS lookups from two cameras in San Francisco, one in London and one in Singapore. Mostly the attacker was using ifconfig.me to find their external IP address.

In San Francisco they attempted to access Cloudflare’s main website, an admin panel that doesn’t exist (there is no admin.cloudflare.com) and a specific customer that I have replaced with example.com (the hacker also tried admin.example.com).

2021-03-09T00:01:15.373343Z	London	ifconfig.me
2021-03-09T00:01:15.373343Z	London	ifconfig.me
2021-03-09T00:01:15.372302Z	London	ifconfig.me
2021-03-09T00:01:15.371451Z	London	ifconfig.me

2021-03-09T00:01:00.580199Z	San Francisco	admin.example.com
2021-03-09T00:00:56.818201Z	San Francisco	admin.cloudflare.com.as13335.com
2021-03-09T00:00:56.768506Z	San Francisco	admin.cloudflare.com
2021-03-09T00:00:50.285755Z	San Francisco	ff.as13335.com
2021-03-09T00:00:45.014855Z	San Francisco	example.com
2021-03-09T00:00:45.011649Z	San Francisco	example.com
2021-03-09T00:00:33.418232Z	San Francisco	cloudflare.com
2021-03-08T23:59:49.677219Z	San Francisco	ifconfig.me
2021-03-08T23:59:49.677216Z	San Francisco	ifconfig.me

2021-03-08T23:59:27.261902Z	Singapore	cloudflare.com
2021-03-08T23:59:27.25919Z	Singapore	cloudflare.com
2021-03-08T23:58:30.274265Z	Singapore	ifconfig.me
2021-03-08T23:58:30.274155Z	Singapore	ifconfig.me

2021-03-08T23:52:23.145321Z	San Francisco	ifconfig.me

They made some of their requests using HTTP and we captured those requests

2021-03-09T00:01:15.375176Z London
ifconfig.me GET /
curl/7.71.1
200 OK

2021-03-09T00:01:00.630747Z San Francisco
admin.example.com GET /
curl/7.64.0
301 Moved Permanently

2021-03-09T00:00:45.041609Z San Francisco
example.com GET /
curl/7.64.0
301 Moved Permanently

2021-03-09T00:00:33.476684Z San Francisco
cloudflare.com GET /
curl/7.64.0
301 Moved Permanently

2021-03-08T23:59:49.708081Z San Francisco
ifconfig.me GET /
curl/7.64.0
200 OK

2021-03-08T23:58:30.289352Z Singapore
ifconfig.me GET /
curl/7.64.0
200 OK

2021-03-08T23:52:23.183201Z San Francisco
ifconfig.me GET /
curl/7.64.0
200 OK

We also have complete connection logging but in the interests of space I won’t put it all here.

The downloaded archived video

The attacker used the Verkada tools to download two archived recordings of our San Francisco office lobby. We only keep 30 days of video but when there’s a formal investigation we keep specific video for up to one year. The downloaded videos date back to July 13, 2020; we had kept them to support investigation into a theft. Here are some stills from each of those videos:

Timeline (UTC)

March 81451 - Unauthorized access to Verkada security cameras2352 - Unauthorized access is used to run curl ifconfig.me from a Cloudflare camera in San Francisco

March 90029 - Unauthorized download of an archived video from our front door camera in San Francisco0033 - Unauthorized download of an archived video from our elevator camera in San Francisco0040 - A second unauthorized download of an archived video from our elevator camera1750 - Contacted by a reporter about security incident involving cameras2107 - First internal report of a Twitter user claiming to have accessed our cameras with confirmation one of our cameras had been compromised2122 - Cloudflare disables all network connectivity of our security cameras2126 - Internal review of our corporate network security controls is completed2149 - Article is released about compromise of Verkada cameras