No fancy infrastructure. Just a 50-line Python function that picks the right model for the right query.

Last month my LLM API bill hit $340. This month: $67.

Same traffic. Same product. The only change was adding a simple router that stops sending every request to Claude Sonnet when GPT-4o mini can handle it just as well.

Here's exactly how it works.

The Problem
When you prototype, you pick one model and hardcode it everywhere. Usually something capable like GPT-4o or Claude Sonnet, because you want good results fast.

Then you ship, traffic grows, and you get a bill that makes you question your life choices.

The thing is — not all queries need a flagship model. In a typical RAG app:

"What is the return policy?" → GPT-4o mini handles this fine
"Summarize these 5 conflicting documents and identify the key disagreement" → needs Sonnet
You're paying Sonnet prices for return policy questions. That's the bug.

The Fix: A Complexity Router

import anthropic
from openai import OpenAI

openai_client = OpenAI()
anthropic_client = anthropic.Anthropic()

def classify_complexity(query: str) -> str:
"""Returns 'simple' or 'complex'."""
simple_indicators = [
len(query.split()) < 15,
query.endswith("?") and query.count("?") == 1,
not any(w in query.lower() for w in [
"compare", "analyze", "summarize", "explain why",
"difference between", "pros and cons", "evaluate"
])
]
return "simple" if sum(simple_indicators) >= 2 else "complex"

def route(query: str, context: str = "") -> str:
complexity = classify_complexity(query)

if complexity == "simple":
    # $0.15/M input — GPT-4o mini
    response = openai_client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": context},
            {"role": "user", "content": query}
        ]
    )
    return response.choices[0].message.content
else:
    # $3.00/M input — Claude Sonnet (only when needed)
    response = anthropic_client.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=1024,
        system=context,
        messages=[{"role": "user", "content": query}]
    )
    return response.content[0].text

Adding a Cache Layer
The router alone saved me ~50%. The cache pushed it to 80%.

import hashlib
import json
from functools import lru_cache

In production: use Redis. For prototyping: this works fine.

_cache: dict = {}

def get_cache_key(query: str, context: str) -> str:
payload = json.dumps({"q": query, "c": context}, sort_keys=True)
return hashlib.sha256(payload.encode()).hexdigest()

def route_cached(query: str, context: str = "") -> str:
key = get_cache_key(query, context)

if key in _cache:
    return _cache[key]  # free

result = route(query, context)
_cache[key] = result
return result

Turns out ~30% of queries in my app were near-identical. "What are your hours?" gets asked constantly. Paying for the same LLM call 200 times/day is just burning money.

Logging Costs in Real Time
You can't optimize what you don't measure. I added cost tracking so I know exactly what each call costs:

COST_PER_1K_TOKENS = {
"gpt-4o-mini": {"input": 0.000150, "output": 0.000600},
"claude-sonnet-4-6": {"input": 0.003000, "output": 0.015000},
}

def calculate_cost(model: str, input_tokens: int, output_tokens: int) -> float:
rates = COST_PER_1K_TOKENS.get(model, {"input": 0, "output": 0})
return (input_tokens * rates["input"] + output_tokens * rates["output"]) / 1000

def route_with_logging(query: str, context: str = "") -> dict:
complexity = classify_complexity(query)
model = "gpt-4o-mini" if complexity == "simple" else "claude-sonnet-4-6"

if complexity == "simple":
    response = openai_client.chat.completions.create(
        model=model,
        messages=[
            {"role": "system", "content": context},
            {"role": "user", "content": query}
        ]
    )
    content = response.choices[0].message.content
    usage = response.usage
else:
    response = anthropic_client.messages.create(
        model=model,
        max_tokens=1024,
        system=context,
        messages=[{"role": "user", "content": query}]
    )
    content = response.content[0].text
    usage = response.usage

cost = calculate_cost(model, usage.input_tokens, usage.output_tokens)

print(f"[{model}] {complexity} | ${cost:.5f} | {query[:50]}...")

return {"content": content, "cost": cost, "model": model}

Sample output:

[gpt-4o-mini] simple | $0.00008 | What are your business hours?...
[claude-sonnet-4-6] complex | $0.00340 | Compare the refund policies across...
[gpt-4o-mini] simple | $0.00006 | How do I reset my password?...
Results After 30 Days
Metric Before After
Avg cost per query $0.0034 $0.0007
% queries → mini model 0% 73%
Cache hit rate 0% 31%
Monthly bill $340 $67
Answer quality complaints 2 3
The quality delta was negligible. Three users in a month said an answer felt shallow — all three were simple factual queries that I probably should have cached anyway.

When This Doesn't Work
Be honest about the limits:

Creative writing / long-form content — mini models struggle here, don't route these down
Multi-document synthesis — always route to the capable model
Anything with high stakes (medical, legal, financial) — don't optimize cost here, use the best model
The classify_complexity function above is naive on purpose. You know your query patterns better than I do. Tune the keywords list to your domain.

Next Step
Before you do any of this, model your current costs to know where the money is actually going. I used APICalculators LLM cost calculator — free, no signup, shows cost per model at your actual token volumes. Knowing the delta between models makes it obvious which optimization to prioritize.

Questions or a different routing approach that worked for you? Drop it in the comments.