Ravi Palwe, Senior Manager, AI/UX Transformation, Financial Services | Capgemini.
getty
The technology launches well. The supervision doesn't survive. Here's the failure pattern almost no one is measuring.
I deploy agentic AI systems at large U.S. financial institutions, and over the past two years, I've watched a pattern of failures emerge across engagements. The agent works. The model is within tolerance. The dashboards are green. And then, somewhere between weeks 8 and 16, the program stops working—not in a way that triggers any alarm, but in a way that surfaces months later as a customer complaint, an audit finding or a question from a regulator that nobody can answer.
I've started calling this the 2:47 a.m. failure, after a specific case at a top-10 U.S. bank where I was helping deploy an AI system.
On a Tuesday in March, at 2:47 a.m. Eastern time, an autonomous AI agent at that bank approved a $1.4 million commercial line of credit. No human reviewed it. No human knew it had happened until the morning standup six hours later, when a junior analyst flagged it. The decision was correct—the borrower met every credit criterion. She flagged it because nobody was sure exactly who was accountable for it.
Six weeks later, the agent was rolled back to a lower autonomy tier. Not for a model reason or for a regulatory one. The decision was operational: The people supervising the agent had stopped looking at the screen.
That is the 2:47 a.m. failure. Once you look for it, you see it across institutions and agent types.
In weeks 1 through 3, supervisors review the agent's outputs carefully. They expand the reasoning trace in most cases. They flag minor issues. They feel productive engagement.
By weeks 4 through 6, they're still reviewing, but faster. The minor issues have started to repeat, and supervisors are trusting the agent on the obvious decisions.
By weeks 7 through 10, they're skimming. They open each case but spend less than 15 seconds on it. The override rate, which started at maybe 8%, has dropped under 2%, and the institution reads this as the agent improving. It isn't. The supervisors are catching less.
By weeks 12 through 16, they're clicking through. They're paid to supervise, but functionally they're watching a monitor that no longer demands their attention. They'll tell you the agent is reliable. They have no real way to know whether that's true.
This is the moment when the agent does something that, six months earlier, a supervisor would have caught—and nobody catches it.
The standard agentic AI program plan does an admirable job of governing the agent. It does almost nothing to govern the supervision of the agent.
We instrument the agent's accuracy down to the decimal point, but we don't instrument supervisor attention. We publish dashboards on decision rate and error rate. We publish nothing on time-on-case or reasoning-trace-expansion rate. We schedule quarterly model reviews. We don't schedule supervisor calibration reviews.
The result is that institutions can answer detailed questions about agent performance and almost no questions about supervisor performance. When something goes wrong, the postmortem reveals what the agent did. It rarely reveals whether the supervisor was actually supervising.
In three different engagements, I've asked the same question: "What's your average time-on-case for human reviewers in Week 12, compared to Week 1?" In every case, nobody had measured it. When we instrumented it, time-on-case had dropped by 60% to 80% not because the cases got simpler, but because the supervisors had drifted into a different relationship with the work.
This is the leading indicator the industry isn't reading.
The institutions that avoid this pattern share a discipline that has little to do with the technology. They measure supervisor behavior with the same rigor they measure agent behavior. They rotate supervision assignments before fatigue sets in. And they close the override loop visibly—every override feeds into a documented review cycle, with policy changes communicated back to the supervising team.
In one wealth management implementation I led, rotating supervisors across agent workflows every six weeks improved error-catch rates by approximately 35%, with no change to the underlying agent.
This is also why 48% of financial institutions are creating new roles specifically to supervise AI agents, according to the Capgemini Research Institute. The industry recognizes that supervision is its own discipline. What it hasn't recognized is that the role existing is not the same as the role being effective. A supervisor on an org chart is not a supervisor at Week 12.
When supervisors see their input acted on, supervision behavior holds. When they don't, it decays. An open override loop ends supervision quietly.
Take 30 minutes this week and answer three questions about each agent in production.
What's the average time-on-case today, and what was it in Month 1? If you don't know, you have a measurement problem before you have a supervision problem.
What was the override rate in Month 1, and what is it now? If it's dropped substantially, decide whether that's the agent improving or the supervisor disengaging. The two look identical from the outside, and only one is good news.
When was the last time a supervisor's override actually changed the agent's policy? If the answer is "never," your override loop is open and your supervision is decaying whether you can see it on a dashboard or not.
These questions take a half day to answer and tell you whether your agent is heading toward its 2:47 a.m. moment. The institutions that ask early have time to fix the answers. The ones that don't will read about their version of the failure in a postmortem six months from now.
If you're running an agentic AI program and the supervisor experience hasn't been instrumented, stop. Instrument it. It will save you months, and it might save the program.
Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。