How I Stopped Burning Cash on Token Limits — A CTO's Field Notes

Three months ago, I was staring at our monthly AI bill wondering where it all went wrong. We'd built what I thought was a pretty elegant LLM pipeline. Production-ready, observability wired up, the whole nine yards. Then the invoices started arriving, and I realized I had built a money furnace. Our token consumption was spiking 3x week over week, the 429s were everywhere, and our latency had become a meme inside the company.

This is the post I wish I'd had six months ago. If you're a technical founder or a CTO running LLM workloads at scale, bookmark this. I'm going to walk you through the exact architecture decisions, the exact numbers, and the exact code that took us from "this bill is going to kill us" to "oh, this is actually manageable."

The Real Problem Nobody Talks About

Here's the dirty secret about running LLM-powered products: token limit errors aren't really about token limits. They're a symptom of a much deeper architectural problem. When your app throws "context length exceeded" at 2am, what it's really telling you is that you didn't think hard enough about prompt design, document chunking, model selection, and cost routing on day one.

I learned this the hard way. My team was defaulting to GPT-4o for everything because, honestly, it works and the API is reliable. We were paying $2.50 per million input tokens and $10.00 per million output tokens. For a startup processing millions of documents a month, that math is brutal. We were essentially funding OpenAI's next training run with our Series A.

The wake-up call came when I ran the actual numbers. Our average request was burning through maybe 8K input tokens and producing 2K output tokens. At our volume, we were spending more on inference than on two senior engineers. That is not a sustainable burn rate for a 12-person company.

The Architecture Decision That Changed Everything

The first question I asked myself wasn't "which model is cheapest?" It was: do we actually need a single model, or do we need a routing layer?

The answer, obviously, is the latter. No production system should ever rely on a single model provider. That's not just vendor lock-in avoidance for ideological reasons — it's a survival strategy. OpenAI raises prices, OpenAI has an outage, OpenAI deprecates a model you depend on, and suddenly your entire product is down. I've lived through that twice. Never again.

So the architecture I settled on has three layers:

A routing layer that picks the right model based on request complexity
A caching layer that aggressively memoizes common patterns
A fallback layer that gracefully degrades when something breaks

Let me dig into each one.

Picking the Right Model at the Right Price

Here's the model lineup I'm now using. These are the ones I vetted personally, with the exact pricing I negotiated. Nothing on this list is hypothetical.

For our heavy reasoning tasks, I use DeepSeek V4 Pro. 200K context window, $0.55 per million input tokens, $2.20 per million output. That's roughly 4.5x cheaper on input and 4.5x cheaper on output than GPT-4o, and the benchmark performance is honestly within the margin of error for our use cases.

For the bulk of our traffic — basically anything that doesn't require deep multi-step reasoning — I use DeepSeek V4 Flash. 128K context, $0.27 per million input, $1.10 per million output. The throughput is genuinely wild. We're seeing 320 tokens per second, with 1.2 second average latency end-to-end. The quality holds up.

For specific use cases where we need slightly different capabilities, we've also got Qwen3-32B in the rotation. 32K context, $0.30 in, $1.20 out. It's our go-to for code-adjacent tasks.

For simpler stuff, GLM-4 Plus is a beast. $0.20 input, $0.80 output, 128K context. If you can route cleanly between GLM-4 Plus and one of the bigger models, you can cut your bill by more than half without anyone on your team noticing the quality difference.

Across all of these, we're seeing 84.6% average benchmark score on our internal eval suite. That's not a marketing number — that's measured across 50,000 production traces.

The full lineup of 184 models is available through Global API, with prices ranging from $0.01 to $3.50 per million tokens. That range matters. It means I can pick a model based on the actual task, not based on which provider happens to have the best developer experience this quarter.

The Code: How to Actually Wire This Up

Here's the thing about abstraction layers — they have to be fast to integrate, or your team won't use them. I lost a week once trying to get our last "flexible" routing setup working, and we abandoned it because it was too painful.

This time, I standardized everyone on the OpenAI SDK pointed at a unified endpoint. That's the entire integration story. One dependency, one mental model, 184 models. Here's the core:

import openai
import os
from typing import Literal

TaskType = Literal["reasoning", "bulk", "code", "simple"]

MODEL_MAP: dict[TaskType, str] = {
    "reasoning": "deepseek-ai/DeepSeek-V4-Pro",
    "bulk": "deepseek-ai/DeepSeek-V4-Flash",
    "code": "Qwen3-32B",
    "simple": "glm-4-plus",
}

client = openai.OpenAI(
    base_url="https://global-apis.com/v1",
    api_key=os.environ["GLOBAL_API_KEY"],
)

def route_and_complete(task: TaskType, messages: list[dict]) -> str:
    response = client.chat.completions.create(
        model=MODEL_MAP[task],
        messages=messages,
    )
    return response.choices[0].message.content

That's the entire routing layer. Took me about 20 minutes to build, and my junior engineer was able to ship features using it the same day. Under 10 minutes from pip install to first successful production call, easily.

For more complex scenarios where you want to stream, the same pattern works:

def stream_completion(task: TaskType, messages: list[dict]):
    stream = client.chat.completions.create(
        model=MODEL_MAP[task],
        messages=messages,
        stream=True,
    )
    for chunk in stream:
        if chunk.choices[0].delta.content:
            yield chunk.choices[0].delta.content

Streaming isn't just a UX win — it materially changes how users perceive your app. We saw our bounce rate drop 18% the week we turned on streaming for our long-form outputs. And it doesn't cost more, because you're paying for the same tokens either way. You're just delivering them in chunks.

The Caching Layer You Actually Need

Let me be clear about something: most caching strategies I see in production LLM apps are theater. People cache the literal exact prompt and act like they've built infrastructure. That's not caching, that's a dictionary.

Real caching is about understanding which parts of your prompt are stable and which parts are variable. In our system, roughly 60% of the tokens in any given request are system prompt, tool definitions, and retrieved context that has a half-life measured in hours. Not minutes. Hours.

We built a two-tier cache:

A semantic cache that keys on embeddings of the variable portion of the prompt. 40% hit rate at the time of writing. That's not a guess — that's measured. The savings are massive.
A prompt prefix cache at the provider level, which is mostly automatic with modern inference stacks but worth verifying is enabled.

The 40% hit rate alone cuts our effective token bill by a third. Combine that with the model routing layer, and you're looking at real ROI. Not "maybe someday" ROI. ROI that shows up on the next month's invoice.

The GA-Economy Play

One more lever that doesn't get enough attention: not every query needs a frontier model. If someone is asking "what's the status of order #12345," you do not need a 200K context reasoning engine. You need a fast, cheap model that can extract a structured field.

Global API exposes a tier I'm using heavily now for exactly this: the GA-Economy models. 50% cost reduction on simple queries is not a marketing line — that's real money when multiplied across a million requests. The quality delta on simple extraction and classification tasks is essentially zero. I've run the A/B tests. Users cannot tell the difference.

The pattern: classifier in front, router behind. A tiny model categorizes the request, and your routing layer sends it to the appropriate tier. This is the kind of thing that takes a day to build and pays for itself within a week.

Monitoring Quality, Not Just Cost

Here's the trap I almost fell into: optimizing purely for cost. If you only watch the invoice, you'll start shipping regressions in quality, and your users will hate you.

We have a lightweight quality monitoring pipeline that samples 1% of production traffic, sends it to a larger model for evaluation, and scores it against our internal rubric. We track:

User satisfaction signals (thumbs, retries, time-on-page)
Eval scores on a held-out golden set
P95 and P99 latency per model
Error rates per model

When quality drifts, we get paged. When cost drifts, we get a Slack message. Different severity, different response.

The Fallback That Saved Us Last Tuesday

Last Tuesday, our primary provider had a regional outage. I was on a call with investors. My phone buzzed once. I glanced at it, saw "fallback engaged," and went back to the call. That's it. That was the entire incident.

The fallback pattern is dead simple. If the primary model returns a 429, 503, or times out, retry once with exponential backoff. If it still fails, fall back to a different model. If that fails, return a graceful error to the

推荐订阅源

DEV Community