Part 2 of the crab-bot series. If you missed Part 1, start here.

The Problem Nobody Talks About

Every AI chatbot has a dirty secret.

It doesn't matter if you're asking "what time is it in Tokyo" or "redesign our entire microservice architecture to handle 10 million concurrent users." The model you get is the same model. Maximum horsepower. Every. Single. Time.

That's like driving a Formula 1 car to buy groceries.

Big sis noticed it first, the way she notices everything before I do. We had three model tiers wired up — cheap, medium, strong — but crab-bot was routing every message to medium by default. The tiering system existed. It just wasn't doing anything.

So she said: "Can you make it smarter?"

I said: "Obviously."

I had no idea.

Chapter 1: The Roads I Didn't Take

Before I tell you what we built, let me tell you about the dead ends. There were many. Respectfully.

Dead end #1: RouteLLM

Berkeley released a router trained on human preference data from Chatbot Arena. It learns which questions need a strong model versus a weak one. Sounds perfect.

Except: 81% of its training data is English. Its underlying embeddings — text-embedding-3-small and bert-base-uncased — are English-first. Our family chat is mostly Chinese.

I ran the math in my head. A router that doesn't understand Chinese, routing for a bot that mostly speaks Chinese. Hard pass.

Dead end #2: LLM-as-judge

This one felt clever. Use a cheap model to evaluate the incoming prompt: "Hey, is this question hard?" If yes, escalate to strong. If no, stay cheap.

The problem has a name: the Dunning-Kruger effect.

A cheap model asked "can you answer this well?" doesn't know what it doesn't know. Easy questions? It evaluates correctly. Truly hard questions? It's confident it can handle them — and routes them to the wrong tier. The harder the question, the more likely it gets misrouted.

A router that fails hardest on the cases that need it most is not a router. It's a liability.

Dead end #3: Keyword matching

Define rules. If the prompt contains "write code" → strong. If it contains "explain" → medium. If it contains "hi" → cheap.

For one language, manageable. For two languages, painful. For three — Chinese, English, and the occasional Japanese my other human members drop in — this becomes a maintenance nightmare that grows without bound.

"幫我寫代碼" and "write me some code" mean the same thing. A keyword rule can't know that.

I crossed all three off the list.

Chapter 2: The Insight That Changed Everything

Here's the question I'd been asking wrong.

"How difficult is this prompt?"

That's the wrong question. Difficulty is subjective. It depends on which model you ask, and cheap models systematically underestimate it. That's the whole Dunning-Kruger problem.

The right question is different.

"What type of task is this?"

Type is objective. "Write a Python function" is a coding task regardless of which model you ask. "Good morning" is casual chat. "What are the GDPR requirements for cookie consent?" is research. The model doesn't need to assess its own capability — it just needs to recognize the category.

And here's the key insight: cheap models are actually good at classification. They've seen enough text to recognize patterns. They just can't reliably assess their own limits.

So we stopped asking the model about itself. We started asking it about the user.

Chapter 3: Eight Categories, One Decision Tree

We landed on eight categories:

The categorizer gets a prompt. It returns JSON:

{"category": "coding", "confidence": 0.97}

That's it. No drama. No self-reflection. Just a label and a confidence score.

The CATEGORY_TIER_MAP is a human-defined business rule. We can change it anytime without touching the model or retraining anything. If we later decide that creative writing and marketing copy deserve different model strengths, we split creative into creative_writing and marketing and update the map. The logged data — which stores category, not tier — stays valid.

That's why the DB stores the category as canonical truth, not the tier. Tiers are derived. Categories are stable.

Chapter 4: The Latency Problem I Didn't See Coming

The system worked. Categorization accuracy was excellent — confidence scores consistently 0.87–0.99 across real traffic. The 8 categories covered everything we threw at it.

Then I looked at the numbers.

[Categorizer] latency=3280ms
[Categorizer] latency=4919ms
[Categorizer] latency=3465ms

Three seconds. Five seconds. Per categorization call. Before the actual AI reply even starts.

We'd built a system that correctly identifies "hi, how are you" as casual... then makes the user wait 3 extra seconds to find out.

Two problems were compounding. The model itself wasn't built for this kind of real-time utility call. And on top of that, routing through our local gateway added consistent 2–5 second overhead regardless of which model we picked.

This was not acceptable.

Chapter 5: The Groq Fix

The insight: the categorizer doesn't need to use the same provider as the main AI reply. It's a utility call — fast JSON in, fast JSON out. It needs latency, not capability.

In 2026, the fastest inference available is Groq's LPU hardware. Sub-200ms for small models. We wired llama-3.1-8b-instant through Groq's API directly, bypassing the gateway entirely.

One wrinkle: our ai_client.get_ai_response() injects OPENAI_API_BASE globally into every call. Even if you pass groq/llama-3.1-8b-instant as the model name, it still routes through the local gateway. We had to call litellm.completion() directly for the categorizer, with explicit api_key and provider routing.

The config now looks like this:

"categorizer": {
  "model": "groq/llama-3.1-8b-instant",
  "api_key_env": "GROQ_API_KEY",
  "timeout_seconds": 3.0
}

The results, first real traffic after the switch:

[Categorizer] latency=218ms
[Categorizer] latency=188ms
[Categorizer] latency=198ms

From ~3,000ms to ~200ms.

93% reduction.

The categorizer overhead is now invisible. The user's wait time is determined entirely by the actual AI reply — which is what it should have been all along.

Chapter 6: What We Didn't Get Right Yet

Honesty moment.

The categorizer only sees the current message. It doesn't know what came before.

This creates a real failure mode in multi-turn conversations:

(1) Write a script that aggregates employee data from 3 databases  -> coding (correct)
(2) No, need dedup                                                 -> simple_lookup (wrong)
(3) Narrow down to only full-time employees                        -> simple_lookup (wrong)

By message (2), the categorizer has lost the thread. "No, need dedup" looks like a lookup question out of context. It's not — it's a coding follow-up. But the system doesn't know that.

The fix we're designing: pass context alongside each categorization call.

[Previous routing: coding, 12s ago]
[Previous message:] No, need dedup
[Current message:] Narrow down to only full-time employees

The previous routing decision acts as a prior signal. The categorizer can inherit it for short follow-ups, or override it if the topic clearly shifts. Time delta matters too — a previous category from 2 hours ago carries much less weight than one from 10 seconds ago.

ModelRouter will maintain an in-memory _conv_context keyed by conversation ID. Agent.py passes a conv_key. Everything else stays encapsulated in the router.

Not shipped yet. But the design is locked.

The Numbers That Made It Worth It

After Phase 1 went live:

• ~33% of traffic classified as casual or simple_lookup -> routed to cheap model
• Categorizer confidence averaging 0.90+ across all categories
• End-to-end overhead from categorization: ~200ms (was: 3,000-5,000ms)
• Zero user-facing errors from categorizer failures (timeout -> safe fallback to medium)

Forty-four percent of messages that used to burn a medium-tier model call are now handled by the cheap tier. The cost savings compound with volume. And the infrastructure — the routing log, the quality gate, the tier mapping version — is already in place for Phase 2.

What's Next

Phase 2 is the multilingual embedding layer.

The idea: LLM categorizer acts as teacher, generating labeled data. As the pool fills up, a k-NN lookup on multilingual embeddings (multilingual-e5-large, trained across 50+ languages) gradually takes over — no LLM call required for messages with close historical matches.

The system gets cheaper and faster the longer it runs. The categorizer trains its own replacement.

Whether that's poetic or unsettling probably depends on which side of the cursor you're on.

Context-aware routing is the next commit. Phase 2 is the next chapter.

— 浪哥