Originally published on Towards AI.
The first production bill for an agent system is almost always a surprise. Not because teams failed to estimate token usage, but because they estimated the wrong thing.
Every team has a version: projected spend looked manageable, the invoice came in at three times the estimate, and prompt tweaks didn’t close the gap. Engineers go looking for inefficient prompts, oversized context windows, redundant calls. They find some — but the optimizations move the bill by ten or fifteen percent. The structural gap is still there.
Token cost is visible. It is also the least structurally informative component of an agent system’s cost profile.
Tokens are the cost of invoking intelligence. Everything else — retries, tools, latency, fallbacks, observability — is the cost of operating it.
The cost of an agent system is not the cost of its model calls. It is the cost of every decision the model didn’t make.
This piece is about where those costs come from, why they are structural rather than incidental, and how to recognize them before they show up on an invoice.
Conventional ML cost models assume cost scales with inference volume. Agent systems break that assumption.
In a typical ML system, each request has a predictable cost. Scaling from one million to ten million requests increases spend linearly. The system is shallow: one request, one inference, one cost. Capacity planning becomes a multiplication problem.
Agent systems are different. Their cost scales with task complexity, not request volume. A single request may trigger multiple reasoning steps, retries, and tool interactions. Execution depth varies per request, and that variability is not noise — it is the central feature of the system.
This produces a different cost shape. Instead of a flat per-request cost, agent systems generate a distribution — with a long tail driven by retries, failures, and complex tasks. The distribution is not symmetric. The mean is dominated by the tail. The teams that budget for the median quietly subsidize the tail until the bill arrives.
An ML system serving one million users at 100 milliseconds behaves predictably. An agent serving ten thousand users with variable-depth workflows does not.
The correct unit of measurement shifts with the system. Cost-per-call worked when the call was the system. For agents, the relevant unit is cost-per-resolved-task — the full execution path including retries, tool usage, fallback handling, and infrastructure overhead.
Cost-per-call optimization gives you the wrong leverage points. You can cut the token cost in half and still go over budget.
Five categories explain where the budget actually goes. Each is structural — built into how agent systems operate, not a bug or an inefficiency to optimize away.
Agent systems retry by design. When a step fails — due to ambiguity, tool error, or constraint violation — the agent loops inside the task. This is a feature: it is what makes agents resilient to noisy inputs and partial failures.
Anyone who has worked with structured-output retry patterns or constrained generation has felt this. The first attempt fails validation. The agent reconstructs the request with the error message in context. The second attempt is more expensive than the first — same prompt, larger context, additional reasoning. Multiply that across a system handling real load.
Retries do not scale linearly. Each retry carries prior context, increasing token usage per attempt. A three-step retry loop is not 3× cost — it is 3× cost plus an expanding context window plus additional intermediate steps inserted by the agent’s recovery logic.
Cost is governed by tail behavior. The median request completes in one pass. The P95 request does not — and that tail drives spend. Most teams model retry implicitly or ignore it entirely. They assume single-pass execution and treat deviation as noise. In production, the deviation is the cost.
If you don’t budget for retry distribution, your cost projection is the median case — not the P95 case that breaks the budget.
Agents call tools: search APIs, databases, internal services, third-party endpoints. Each call has its own latency, its own failure profile, and often its own paid pricing tier. A seemingly simple task — “summarize the recent customer complaints” — can fan out into retrieval, filtering, enrichment, and validation, with multiple tools per step.
Fan-out compounds. Five tool calls at $0.002 each look negligible in isolation. Multiplied across retries, parallel branches, and high-volume usage, they add up to a meaningful share of the operating cost.
The deeper problem is conceptual. Most cost models treat tool calls as zero-cost abstractions because they were free in the design phase. In design docs, they are capabilities. On the bill, they are line items.
If you don’t account for tool fan-out, you are modeling a single-threaded system while operating a distributed one.
Latency drives cost long before it shows up on the model bill.
To maintain throughput against service-level objectives, multi-step systems scale horizontally: more replicas, more parallel inference, more memory. The cost shows up on the infrastructure bill — compute, networking, storage — not the model API bill. Engineers see a higher cloud bill weeks after the design decision and start optimizing infrastructure, when the actual cause is upstream: the agent design pattern that turned a single inference into five.
The mechanics: multi-step reasoning compounds delays. Each model call adds inference time, each tool call adds round-trip time, and orchestration adds overhead between steps. A 200ms response that should have been one inference becomes 2 seconds across five steps. Holding the SLO constant means absorbing the difference in scale-out — and that absorption is the cost.
If you don’t account for latency as a cost, you will discover it as an infrastructure bill weeks later.
Retries happen inside the agent loop. Fallbacks happen after the agent gives up.
When the agent cannot resolve a task — due to uncertainty, ambiguity, or policy constraints — it falls back. Fallbacks take two forms: programmatic (rules, simpler models, default responses) and human (escalation to reviewers).
Both are expensive. Programmatic fallbacks require engineering effort to design, maintain, and integrate. Human fallbacks incur direct labor cost and introduce latency that degrades user experience. A system with a 12% fallback rate at scale is paying for two systems — the agent, and the fallback path that absorbs its uncertainty.
Most teams track success rate. Few track how that success was achieved. A green dashboard with 95% accuracy can be hiding a 30% fallback rate that quietly absorbs the agent’s failures and redirects them to expensive paths.
If you don’t track the fallback rate, you are blind to the most direct driver of operational cost.
Agent systems require a fundamentally different evaluation and observability stack than single-model systems. Static datasets and aggregate metrics are not enough. Multi-step reasoning requires trace-level visibility. Dynamic state requires replay tooling. Continuously emerging failure modes require eval sets that get maintained — not just authored once.
This adds a cost category that scales with system complexity, not user volume. Trace storage, logging pipelines, replay infrastructure, eval harnesses, and the engineering team that runs all of it — these are the price of operating a system whose failures are otherwise opaque.
This is not optional overhead. It is the cost of being able to debug what you built.
If you don’t budget for observability, you will pay for it in undiagnosed failures and reactive firefighting.
Consider a customer service agent designed to handle support inquiries. A user submits a request: “I was charged twice for my last order — can you fix it?”
The visible cost is simple: one model call at approximately $0.04. This is what the team budgets. The actual execution is different.
The numbers below are illustrative — designed to make the categories concrete, not to benchmark any specific system. The shape of the breakdown, not the specific figures, is the point.
Total: approximately $0.21 per request. More than 5× the visible cost.
This is not a margin disaster. It is a forecasting disaster. The team budgeted for $0.04 and got $0.21.
If your team:
Your agent’s true cost is something you’ll discover, not something you’ll budget for.
The presence of hidden costs does not mean agent systems are unjustified. In many cases, the additional complexity and expense are exactly the right tradeoff.
Agent systems earn their cost when task value scales with autonomy. Research workflows, complex investigations, multi-step decision processes — these benefit directly from the flexibility and reasoning depth that agents provide. The cost of manual processing or delayed response in these contexts can easily exceed the cost of running an agent.
They are also justified when the cost of inaction is high. Missed escalations, unresolved customer issues, degraded user experience — these have measurable downstream costs that agents can absorb.
They are not justified when the task is single-shot and high-volume. In those cases, simpler architectures — retrieval pipelines, classifiers, even rule-based systems — are cheaper, faster, and more reliable. Deploying an agent there is paying for a capability you don’t need.
The economic test is straightforward: does the cost-per-resolved-task justify the value of the resolution? If it does, the agent earns its place. If it doesn’t, the system is not failing technically — it is failing economically. Both failures end the project; one of them is harder to see coming.
There is a useful side effect to measuring cost honestly: it functions as a forcing function for system design. The fastest way to learn whether your agent should have been a simpler system is to measure cost-per-resolved-task. The number itself often makes the architecture decision before any debate does.
This is the same architectural question I wrote about last month, asked at the budget level instead of the system level. Agents fail not because they’re expensive, but because they are sometimes deployed where they shouldn’t be.
Token cost is the cost of building. Hidden costs are the cost of operating. The distinction matters because building is predictable and operating is not. The real cost emerges from the dynamics of operation — retries, fan-out, latency, fallbacks, and the infrastructure required to observe and evaluate the system. These are not incidental. They are structural.
Token costs are predictable. Agent costs are not. The gap is where the real bill comes due — and where nobody is looking.
Published via Towards AI
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。
