It’s not superintelligence or autonomous weapons. It’s a much more mundane — and in some ways more frightening — problem.

The problem with AI is “not because it’s smarter than us and going to turn us into pets,” Blankfein said in a new interview on Andreessen Horowitz’s The a16z Show, published Monday, “but because we don’t have the ability to test whether it’s right or not.” When you’re running a big institution, he explained, you can’t make mistakes and numbers really matter.

Alluding to AI in particular but technological advancement in particular, he said, “everything is whirring behind the scenes,” and you don’t really get a close look at the thought process of the technology on which you’re relying. “Now you can leave a piece of software, [and it] could go out and do 70,000 transactions,” he said, explaining that when he started on the trading floor, everyone could hear every mistake, and the room would get quiet at the smallest slip-up.

This simple explanation may be the most precise articulation yet of why Wall Street — despite spending billions deploying AI across trading, compliance, and back-office operations — remains deeply reluctant to hand autonomous agents the keys to anything that actually matters.

Speed without oversight is the real risk

The financial industry has long understood that speed creates leverage, and leverage cuts both ways. A well-timed trade amplifies gains. A mistaken one — executed at machine speed, across thousands of positions, before a human can intervene — amplifies losses just as fast.

What Blankfein is describing isn’t a hypothetical. The “flash crash” of 2010, when algorithmic trading briefly erased nearly $1 trillion in market value in minutes, offered an early preview. So did the 2012 Knight Capital disaster, in which a software glitch caused the firm to lose $440 million in 45 minutes — effectively destroying the company. Both events predate the current generation of AI agents by more than a decade.

The new generation is faster, more autonomous, and more capable of chaining decisions together without a human checkpoint between them. A March 2026 Deloitte analysis of the MIT AI Risk Database identified more than 350 distinct risks that can arise from autonomous or agentic behavior in banking alone — many of which are not addressed by existing frameworks. The firm’s researchers described the core mechanism Blankfein was warning about: a single hallucination can cascade across linked systems, a payment-routing agent can misallocate funds before any human catches it, and a recursive agent loop can drive cloud costs into six figures before anyone notices.

The American Bankers Association warned in December 2025 of a potential “737 Max moment” — where overreliance on automation collides with public trust and regulatory accountability before guardrails are in place.

The numbers behind the gut feeling

The data bears out Blankfein’s instinct in striking detail. A January 2026 Wakefield Research study found that only 14% of CFOs completely trust AI to deliver accurate accounting data on its own — yet the vast majority of those same firms are already using AI tools. Ninety-seven percent said human oversight remains critical for accuracy, and most had already encountered at least one instance of hallucinated or inaccurate AI output.

The CFA Institute’s 2025 report on explainable AI in finance put the technical problem plainly: AI-driven systems present “oversight difficulties caused by limited transparency in data sources and decision-making logic.”

A separate LinkedIn analysis from January 2026 was even blunter: “Supervisors lack consistent, granular data on where and how AI is actually being used,” and existing model risk management frameworks “challenge traditional validation, monitoring, and auditability.”

Meanwhile, deployment is racing ahead of governance. Ninety-two percent of leading fintech firms had integrated at least one autonomous agent into core production as of Q1 2026 — the same quarter that saw rushed standardization of “Guardrail Protocols” requiring human authentication for transactions over $1 million. And 70% of banking executives at firms already using agentic AI reported that governance frameworks lag far behind the pace of deployment, per a 2025 MIT Technology Review Insights survey.

Goldman’s unusual caution

Blankfein also offered a pointed observation about how Goldman historically approached system transitions: running legacy and new systems in parallel for years before making a full switch. It’s a discipline, he noted, that most technology companies don’t share — and one increasingly at odds with the “move fast” culture defining the AI deployment wave sweeping through finance.

The implicit warning: the firms most aggressively deploying AI agents are also the least likely to have stress-tested what happens when those agents are wrong.

That contrast is particularly relevant now. Goldman has rolled out its AI assistant to all 46,000-plus employees and identified six business areas “ripe for disruption” in its most recent shareholder letter. JPMorgan has more than 450 AI use cases in production, and its LLM Suite is used by 150,000 employees weekly. Citi has more than 70% of its 182,000 employees using firm-approved AI tools.

But nearly all have drawn the same line: autonomous execution above certain thresholds still requires human sign-off. The industry is racing to deploy AI everywhere except the places where Blankfein’s 70,000-transaction problem would actually materialize.

“We always had to do things twice,” Blankfein said about the old way of working. “We had to run things 50 times and be perfect the last 49 times before we could go that way.” That means it could be a long, long time before AI agents are fully trusted to get it right every time out of the gate.