An extensive program at Meta to gather a wide range of data from employees to train its AI model has been frozen after employees reportedly broke through its guardrails and accessed restricted data, and then did so again after Meta claimed to have fixed the vulnerability.

Whether or not the data collection by the $201 billion owner of Facebook was a good idea, analysts argue that the data protections deployed were woefully inadequate, given the extreme sensitive nature of the collected data.

“Meta had the resources to get it right, and yet they failed exponentially,” said Karianne Michelle, a director with consulting firm Acceligence. “That is what it looks like when the policy decision and the technical execution are happening in two different rooms that are not fully in sync. It is the kind of gap you see often enough at organizations under structural strain.”

Fritz Jean-Louis, principal cybersecurity advisor at Info-Tech Research Group,  agreed.

“What we just observed from the Meta story is a classic failure mode in AI-era data strategy: collecting high-risk telemetry without equally mature access controls,” Jean-Louis said. “At that scale, a single misconfiguration turns internal data into a systemic exposure.”

The story, detailed in a Wired report, involved a program that Meta rolled out in April called the Model Compatibility Initiative (MCI), which collects computer inputs such as mouse movements, click locations, and keystrokes, as well as screen content, the story said. Meta employees were initially not allowed to opt out. 

The data collected included full prompts and transcriptions, private conversations, people and performance data, Wired said, adding, “Meta executives have repeatedly defended the data-gathering project, saying it was necessary to train AI systems to operate computer software the way humans do, and that employees were the best examples for the artificial intelligence to learn from.”

Wired also quoted Stephane Kasriel, a Meta vice president overseeing AI research, who saying that the company discovered that unauthorized employees were found to have accessed MCI data on June 18, and that the hole was closed “within four hours.” But, he added, “ the initial fix didn’t stick, and access to the data had to be further locked down.”

In an email statement, Meta confirmed that the program was being halted for the time being. “We have carefully designed this program with privacy safeguards, and while we have no indication at this time that any data was improperly accessed by Meta employees, we’re pausing it while we investigate,” Meta said. 

A ‘liability surface’

Analysts, consultants, and industry practitioners said they were more concerned about the inadequate protections than the underlying data exposure.

Carmi Levy, an independent technology analyst, said that although there should be concerns about Meta’s “Orwellian oversight of workers’ keystrokes and mouse movements,” the bigger issue is the paper-thin protections that it used to protect that data.

“As creepy as MCI was and is, the reason Meta has hit the pause button has nothing to do with the moral and ethical fuzziness of everyday employee surveillance, and everything to do with its failure to secure the data it collected in the process,” Levy said. “Conceivably, it will resume monitoring and data collection once it fully understands how highly sensitive data, such as employees’ private conversations, performance data, and transcriptions, ended up being inadvertently shared with the entire workforce.”

One of the critical background issues is that while the data collected was highly sensitive, it was not, from a strict compliance law perspective, PII (personally identifiable information). That distinction might have lulled Meta into a false sense of security and convinced it that the data did not merit strong protections. 

“I think companies can get a little too comfortable saying, ‘Well, it’s not PII,’ as if that makes the data low-risk,” said Tom Findling, CEO of Conifers.ai. “But internal prompts, transcripts, chats, data tables, and performance notes can tell you a lot about how a company works, what it’s building and where things are messy or exposed. That’s sensitive, even if it’s not someone’s Social Security number.”

Findling argued that Meta executives “wanted to pretend that they didn’t understand” how sensitive the collected data was, and that was their excuse for why they should not have to protect it sufficiently. “There is no doubt that Meta did not tag this at an appropriate risk level,” he said.

Info-Tech’s Jean-Louis took particular umbrage at the particulars of the collected data. 

“Employee behavioral data, such as keystrokes, screenshots, and usage patterns, is effectively sensitive by default. If you’re using it to train AI, you have to treat it like production secrets, not analytics exhaust,” Jean-Louis said. “When thousands of internal tables are broadly accessible, you have a liability surface rather than a data platform. Trust nowadays is a security control. Once employees believe their data is overcollected or underprotected, you introduce both insider risk and reputational damage at the same time.”

Acceligence’s Michelle echoed Jean-Louis’ concerns.

“The data Meta exposed is not the real risk. Security policy only works if people believe it, and belief is exactly what is now in question,” Michelle said. “That gap is where incidents like this one do their damage: once employees stop trusting what leadership says about their own data, the doubt follows every policy that comes next, producing workarounds, quiet noncompliance, and employees who stop raising flags.”

This article originally appeared on CSOonline.