Here's the scoreboard. Same 50 emails, same prompt, same 4-tier task:

The cheap model didn't tie the expensive ones. It beat them by six points and never missed an email that should have woken me up. The two models I'd have reached for on instinct — the obviously-smarter ones, the ones that cost several times more per token — both came second.

I almost didn't run this comparison at all. That's the part worth your time.

The task

I'm building an email firewall. Every inbound message gets exactly one of four tiers:

• SILENT — recorded, never shown (marketing, receipts, FYI)
• QUEUE — visible when I choose to look, no notification
• PUSH — actually interrupt me
• AUTO — reversible, hands-off (classified only, for now)

That's the entire output surface. No suggestion cards, no "AI thinks you should reply" badges. One label per email.

The thing doing the labeling is what I call the judge. It's the part I'd assumed needed a good model — reading an email and deciding whether it's allowed to ring your phone feels like a judgment call, and judgment calls are what you buy a frontier model for.

So I had gpt-4o wired in and I was ready to leave it there. Then I did the boring thing and measured it.

What "measured" means here

The eval set is committed to the repo: packages/api/eval/judge-eval-set.json. Fifty emails, synthetic and PII-free, hand-labeled to encode one specific person's policy — QUEUE is the default, SILENT is narrow (clear marketing only), PUSH is urgent and confident, AUTO is reversible and not urgent. The tier mix is 21 QUEUE / 13 PUSH / 12 SILENT / 4 AUTO.

One command runs a model against it:

pnpm eval:judge   # tsx scripts/poc-accuracy.ts --in=eval/judge-eval-set.json

I ran it across the models I was actually choosing between. Flash won. Not "won on cost, tied on quality" — won on quality, and it happens to be the cheap one. I pinned production to it and wrote the result into the commit message so future-me can't quietly pretend the expensive model was a sacrifice I made for the budget:

flash is not a compromise — it scores 88% / 100% PUSH recall, beating gemini-2.5-pro and gpt-4o (both 82%).

Why cheap wins here

A frontier model sells you reasoning depth. Long chains, hard problems, hold-ten-things-in-your-head problems. Email triage is none of that. It's a short, repetitive, read-four-signals-and-be-consistent problem. You're not paying for capability that moves this needle — you're paying for capability you never touch, and a bigger model's extra "thinking" mostly buys you more chances to overthink a 30-word email.

There's an architecture reason too, and it's the load-bearing one. The LLM never picks the tier. It scores four features per email — confidence, sender trust, reversibility, urgency — and a ~20-line deterministic rule maps those four numbers to PUSH/QUEUE/SILENT/AUTO. The model is a feature-scorer, not a decider. So I don't need a model that reasons brilliantly about email policy. I need one that reads four signals the same way every time. Consistency, not genius. That's exactly the job a cheap fast model is good at — and exactly the job where a bigger model's cleverness becomes variance you don't want.

It also means the policy is auditable without the model in the loop. I can read the rule. I can test it. If the model's down, a keyword fallback produces the same four features so urgent mail still gets through. None of that works if you let the LLM free-hand the answer.

Before you @ me

This is 50 emails. It's small on purpose — it's one person's mental model written down, not a benchmark, and I'm not going to dress it up as one. The set is synthetic, so it tests whether the model applies my policy consistently, not whether it can read the real world. A different inbox with a different owner would draw the lines somewhere else and might rank the models differently.

I'm also only claiming what I measured. Flash hit 100% recall on PUSH — it never sent an urgent email to a quiet tier. I'm not going to invent per-tier numbers for the models that lost; I have their headline accuracy and that's what I'm putting my name on.

What I'm not walking back: on the one task I actually care about, on the set that's sitting in the public repo for you to open, the expensive models lost. That result was stable enough to bet production on.

The lesson is annoyingly cheap

Most of us never run this comparison. "Use the best model" is the default, the best model is the expensive one, and the leaderboard agrees, so why would you waste an afternoon proving the obvious?

Because the leaderboard has never seen your task. MMLU doesn't know what you mean by "urgent." The only eval that ranks models on your problem is the one you write — and when you write it, the ranking stops matching the price tag surprisingly often. The frontier model isn't smarter at your job. It's just smarter at the jobs in the press release.

Write the small eval. Run the cheap model against it before you reach for the expensive one. Worst case you confirm the obvious. Best case you cut your bill and your accuracy goes up, which is a sentence I didn't expect to type either.

The judge, the eval set, and the deterministic rule are all in the open — AGPLv3, OpenAI-compatible, point it at Ollama or vLLM and keep your mail on your own box: github.com/k08200/klorn. The eval set is packages/api/eval/judge-eval-set.json. Open it, label it your way, and go find out which model actually wins on your inbox.