But behind the scenes, an equally important if less dramatic AI struggle is playing out—as U.S. defense and intelligence agencies try to leverage the technology without sacrificing their need for secrecy. A small handful of AI infrastructure companies have been quietly doing complex, rarely-seen work that makes it possible for the U.S. government to securely use AI in the first place.

“It’s probably a $2 billion market right now,” says Nicolas Chaillan, founder of an AI platform called Ask Sage that’s used by thousands of teams across the Department of Defense. The opportunity these pick-and-shovel companies are chasing grows out of an extreme case of a dilemma faced by anyone looking to deploy off-the-shelf LLMs on confidential data: They’re trying to figure out how to use these powerful tools without inadvertently exposing the wrong information to the wrong people through the AI training process.

These AI infrastructure companies receive less media attention for their government work than bigger peers like Google, xAI, OpenAI, and of course Anthropic. Until the recent dispute broke out, Anthropic’s Claude model was among the only LLMs approved for use on the Defense Department’s classified networks. But this arrangement was made possible by a 2024 deal with two other firms that provided the necessary infrastructure—Palantir and Amazon Web Services (AWS)—which operated the secure software platforms and cloud services that host the AI. Imagine that large language models are a bit like the U.S. military’s newest, shiniest warplane: The infrastructure companies provide something like the radios and runways that help these new machines talk to the rest of the military, and land safely.

“There’s probably, I don’t know, a hundred people, 200 people who deeply care about this question inside the intelligence community,” says Emily Harding, a former CIA analyst who now researches defense tech at the Center for Strategic and International Studies. “I think there’s millions and millions of business people who are going to face this same problem, not with as high stakes.”

Any corporate leader sitting on a trove of proprietary information has probably run into some version of this issue with their AI strategy. Imagine training a bespoke instance of ChatGPT or Claude on all of your company’s mission-critical files: A law firm’s case documents; a drug company’s internal research reports; a retailer’s real-time supply chain data; an investment bank’s risk models or due diligence memos. Trained on such a corpus, an AI helper could speak your company’s language fluently, and reveal richly profitable connections in your files. But consider the consequences if the wrong person—say, a competitor—got access to that helper. 

“It’s kind of a Catch-22,” Harding tells Fortune. “Feed it enough, it knows too much. You don’t feed it enough and then it can’t do its job.”

With the right prompting from an outside party, the contents of any confidential file that the AI touched in training could be spilled. Which means teaching an LLM all a company’s secrets could simultaneously boost the business—and risk blowing it up. 

When secrets are a matter of national security

Now consider how much worse that problem becomes if that AI helper works for the CIA, where secrecy is a matter of national security and breaches could endanger lives. 

Intelligence agencies and the military depend on the compartmentalization of sensitive information. Human agents and analysts gain access to secrets on a strict, need-to-know basis to reduce the risk of leaks. (This may be among the reasons that a recent report stating the Pentagon was discussing training LLMs on secret data sparked immediate criticism.) So what happens if every analyst’s AI assistant suddenly knows all of an agency’s secrets?

“Compartmentalization goes out the window,” says Brian Raymond, another former CIA analyst who’s now CEO of Unstructured, an AI infrastructure company that serves both commercial and government clients. 

 “Let’s say I’m an Iraq analyst,”  Raymond explains, by way of example. “From an intel organization’s perspective, I have no business reading reports from covert assets on Chinese military technology. Everyone stays in their swim lane and that’s great security. If all of a sudden, I could start asking all sorts of questions like, ‘Tell me all the assets we have in some county in Asia and tell me all their real names’—those are our most closely guarded secrets!”

And so a small crop of AI infrastructure firms has sprung up to solve what amounts to AI’s secrecy problem. These companies build a scaffolding of software and services around commercial large language models, which allow organizations to use the AI without exposing their secrets. 

At the heart of this scaffolding is a carefully orchestrated version of technique called Retrieval Augmented Generation, or RAG. Commercial LLMs use a version of RAG whenever they look at documents you upload into the chat window. A model like Claude retrieves information from that document and then augments its responses based on its findings before generating an answer to your questions. Still, there’s often a limit to how much data you can upload. And giving a commercial LLM sensitive documents remains risky because the contents could end up being used for future training, or end up in a temporary cache that isn’t necessarily siloed from the provider’s view.  

The companies working with the U.S. government offer far more secure, managed RAG systems, in which commercial LLMs function more like a processing engine—and sensitive information stays walled off in secure libraries. These systems can be used to separate what a commercial AI model like Claude or ChatGPT “knows” from what it looks up.  

The AI equivalent of a ‘secure room’

Let’s say the Iraq analyst from Raymond’s example employs a secure, RAG-based AI assistant to put together a report on U.S. Navy assets in the Persian Gulf. The analyst types a question into this assistant’s chat window, asking for the latest count of warships there. The RAG system she’s using employs a private, secure library that, let’s say, contains some recent, classified intelligence reports about Navy deployments in the region. This library—technically a vector database, mathematically indexed for connected meanings rather than just keywords—is the first place the system looks for an answer. 

Think of this as the step where the AI assistant steps into a secure room to get briefed on a need-to-know basis. The assistant retrieves these classified details about U.S. ships and then hands them over to a commercial LLM like Gemini that’s running on secure servers. The LLM then uses the classified details to augment its response before generating it in the text window for the analyst. Secure systems like these are often set to expunge questions and answers from their memory once a session is done, so classified information is neither used for later training nor retained in any memory.

The Iraq analyst in this example would only have clearance to access a secure library of documents related to her tasks in Iraq. Out-of-scope questions about China, from Raymond’s example, wouldn’t be answerable. There’d be no classified China documents in the secure library, nor would the commercial LLM have any of that information in its training data. In short, this method creates a scaffolding that gives the AI a way to read and use sensitive data without remembering it forever or revealing it to the wrong people.  

Raymond’s company, Unstructured, works at the scaffolding’s base. His team cleans and converts messy internal files—from handwritten field notes for commercial clients to exotic classified file formats for the government—so they can be searched safely inside a secure vector database. Or as Raymond says, “We vacuum up all that data in the world, get it into book form, and to the library.”

Other companies like Berkeley-based Arize AI, which has raised more than $130 million of funding since it launched in 2020, work at the center of the structure. Arize tests and monitors RAG pipelines as well as the agents and applications built on them—debugging and hunting down errors and hallucinations.  

“Controlling these systems is hard and making sure they do the right thing is one of the most mission-critical parts of the process,” Arize CEO Jason Loepatecki tells Fortune. ”I wouldn’t deploy an AI without using one of my products or my competitors’ products.”

At the top of scaffolding you’ll find players like Ask Sage. While Unstructured and Arize serve a relatively even mix of government and commercial clients, Ask Sage is more of a Pentagon specialist, doing around 65% of its business with the Defense Department. The Virginia-based company sells a government-grade software interface where users can safely query approved commercial LLMs, run agents, and get answers drawn from their own restricted data, all without the model ever “learning” the secrets behind the scenes. 

A Pentagon in-house competitor?

In December the Defense Department announced the launch of its own internal LLM platform, called GenAI.mil. Defense Secretary Pete Hegseth introduced the rollout by way of a department-wide message that said, “I expect every member of the department to login, learn it, and incorporate it into your workflows immediately.” Afterward, Pentagon officials said, more than a million unique users signed on to the platform. 

At present, GenAI.mil offers a simple chatbot interface, allowing service members to employ a commercial LLM running on secure servers for drafting documents or analyzing files—but only for work that is unclassified.  This is among the reasons that GenAI.mil—unlike products from Ask Sage, Palantir or Scale AI—can’t do RAG on secure off-platform databases full of top-secret files. A Pentagon official told Fortune that the department is looking to deploy AI tools across “all classification levels” moving forward, but declined to answer questions about timeline, specific software architecture or upcoming changes to the GenAI.mil platform.  In its current form at least, the Pentagon’s new product can’t solve AI’s secrecy problem. 

Which is perhaps good news for products like Ask Sage. While Chaillan says new government subscriptions have leveled off since January, 14,000 teams across 27 U.S. government agencies remain subscribed to Ask Sage. On the strength of those numbers, Ask Sage was acquired in November by the defense-focused analytics company BigBear.ai in a $250 million deal. (Chaillan left the company in February.)

Raymond, of Unstructured, sees the Pentagon’s new platform as an opportunity. “With GenAI.mil making these models more available, that’s going to unlock a lot of demand for what we build,” he said.

Knowledge workers in the U.S. military and intelligence communities have reams of documents to summarize, tons of text to draft, and endless compliance tasks to carry out, all buried under a dense thicket of government acronyms. “Take an ATO in the government with FedRAMP, or you know, pick your poison of compliance nightmare,” Chaillan says. For such tasks, he adds, a platform like AskSage “really drastically reduces the human manual burden.” 

And this is likely one of many reasons why leaders like Arize’s Loepatecki see a huge opportunity solving AI’s secrecy problem both inside the government and out.  

“The vertical we’re in is probably one of the fastest growing picks-and-shovels spaces,” Loepatecki says. “The world’s data is infinite, and the pockets of data that you don’t want to be trained publicly are large.”