Would you hire an engineer based on their SAT score?
Of course not. You look at how they solve problems. How they handle ambiguity. Whether they adapt when their first approach fails. You're evaluating behavior, not just knowledge.
Yet somehow, this is exactly what we do with LLMs. We test them like students — multiple choice, fill in the blank, write a function from a spec — and call it "evaluation." We rank models by MMLU scores and HumanEval pass rates as if those numbers tell us everything we need to know.
They don't. Here's why.
What Are We Actually Measuring?
Let's look at three of the most widely-used LLM benchmarks. Not at their scores, but at what they actually measure.
MMLU: The Encyclopedia Test
MMLU gives an LLM 57-choice multiple choice questions across subjects like law, medicine, and philosophy. Pick the right answer from four options. That's it.
What it measures: breadth of knowledge. How much the model has memorized.
What it doesn't measure: whether the model knows when to apply that knowledge. Whether it can tell the difference between a situation that needs legal reasoning and one that just needs common sense. Whether it knows what it doesn't know.
It's a driving written test. Passing it doesn't mean you can drive.
HumanEval: The Coding Interview Problem
HumanEval shows a function signature and a docstring. The model fills in the body. If the code passes the test cases on the first try, it's a pass. This is measured as pass@1 — first-attempt pass rate.
What it measures: can the model translate a spec into working code in one shot?
What it doesn't measure: what happens when the test fails? Does the model debug systematically or flail randomly? If there's an existing codebase with conflicting patterns, does it notice? Does it know when to refactor instead of patching?
One function. One attempt. That's not how software gets built.
SWE-bench: The First-Day Assignment
SWE-bench is the most realistic of the three. It gives the model a real GitHub issue and access to the full repository. The task: produce a patch that resolves the issue. Evaluation is binary — the repo's test suite either passes or it doesn't.
What it measures: can the model navigate a real codebase and fix a real bug?
What it doesn't measure: anything about the approach path. Did the model grep for the right files efficiently, or did it read half the repository first? Did it understand the existing architecture, or did it brute-force a patch that works but violates every design pattern in the project? Did it learn something from this issue that it could apply to the next one?
SWE-bench evaluates the destination, not the journey.
The Pattern: All Three Measure "First Impressions"
| Benchmark | What it measures | What they all miss |
|---|---|---|
| MMLU | Knowledge recall | Application judgment |
| HumanEval | First-pass coding | Debugging, iteration, adaptation |
| SWE-bench | One-shot bug fixing | Approach path, cross-session learning |
These benchmarks share a fundamental assumption: evaluation happens once, in a single session, with a single correct answer.
But real AI coding agents don't work that way. They work across sessions. They learn from yesterday's mistakes. They reuse context from last week's debugging session. The quality of their work depends not just on what they know, but on how they behave over time.
This isn't a knowledge problem. It's a behavior problem. And no amount of harder questions on MMLU-Pro will solve it.
We Hire Humans by Behavior. Why Do We Test LLMs by Knowledge?
Think about how you hire an engineer.
You glance at their GPA. You look at their GitHub. Maybe you give them a take-home assignment. But none of that is the deciding factor.
The deciding factor comes from the interview. And what do you ask?
- "Tell me about the hardest technical decision you made last year."
- "Walk me through a time you disagreed with a teammate and how you resolved it."
- "Here's a problem. Show me how you'd think about it — not the answer, the thinking."
These are behavioral questions. They don't measure what the candidate knows. They measure how the candidate operates. And they work because past behavior predicts future performance.
Now look at LLM evaluation. Where are the behavioral questions?
There aren't any. We're stuck at the "checking GPA" stage, watching every model score in the 90th percentile and pretending that tells us something useful about how they'll perform on real work.
Same Problem, Different Minds
Here's what behavioral evaluation actually looks like.
Take the same bug ticket and give it to three different models. Don't just check who fixes it — watch how they approach it.
Model A reads the ticket and immediately greps for the relevant code. Within 30 seconds, it has a first patch. It's fast, intuitive, pattern-matching. This model would thrive in rapid prototyping — where speed and gut instinct matter more than architectural rigor.
Model B starts by decomposing the ticket into three sub-tasks. It reproduces each one independently before attempting any fix. It's methodical, structured, systematic. This model belongs on complex architecture work — where missing an edge case costs weeks.
Model C searches git log for similar issues first. It studies existing patches to understand the codebase's conventions before writing anything. It's cautious, precedent-driven, learning from history. This model fits maintenance and bug fixing — where consistency with existing patterns matters more than clever solutions.
All three models fix the bug. Their scores are identical. But their behavioral profiles are completely different. And that difference determines which role each model is actually suited for.
This is what behavioral benchmarks should measure. Not "did the model solve the problem?" but "how did the model solve it?" — and what does that tell us about where it belongs?
A Proposal: Behavioral Benchmarks
I should be clear: this is a proposal, not an established framework. I'm not citing a paper because there isn't one. (Though interestingly, an April 2026 preprint by Tang et al. argues for "in-situ behavioral evaluation" for LLM fairness — suggesting the idea is in the air.) If I'm wrong about any of this, I hope you'll correct me in the comments.
Here's the definition I'm working with:
A Behavioral Benchmark is an evaluation framework that profiles how an LLM approaches problems — its cognitive patterns — rather than just scoring the correctness of its answers.
Where existing benchmarks ask "how many did it get right?", behavioral benchmarks ask "what kind of thinker is this?"
I propose four dimensions to observe:
| Dimension | Observation Question | What It Reveals |
|---|---|---|
| Decomposition | Does it jump straight to execution, or break the problem down first? | Top-down architect vs. bottom-up executor |
| Approach | Does it search for similar patterns, or reason from first principles? | Maintenance engineer vs. innovator |
| Recovery | When stuck, does it change strategy or double down on the same path? | Adaptive vs. persistent |
| Consistency | Does it show the same approach pattern across similar problems? | Predictable vs. creative |
Think of it this way:
- MMLU asks: "What does this candidate know?"
- Behavioral benchmarks ask: "How does this candidate work?"
- And that second question determines role fit.
Why Now
In 2026, coding agents aren't demos anymore. They're daily tools on real engineering teams. And teams are starting to ask questions that our benchmarks can't answer:
- "Which model should I use for our legacy codebase maintenance?"
- "Our junior devs need a pair programmer — which model's debugging style fits them?"
- "We need consistency. Which model produces the most predictable behavior week over week?"
These are role-fit questions. Hiring questions. And we're trying to answer them with SAT scores.
The race for smarter models is maturing. The next frontier isn't a higher MMLU score — it's understanding what each model is actually good for. And we can't get there without behavioral evaluation.
Let's Define This Together
I don't think I've nailed this. The four dimensions I proposed are a starting point, not a destination. Maybe there are better axes. Maybe the whole framing is wrong and someone smarter has already solved this.
Here are a few things I'm probably wrong about — please correct me:
- Decomposition style is a stable trait of a model, not just a reflection of the prompt
- Recovery behavior can be measured without also measuring the harness/framework around the model
- Consistency across sessions is more important for team adoption than raw capability
- Role-fit evaluation will eventually matter more than accuracy benchmarks for enterprise adoption
If you're building coding agents, evaluating models, or just frustrated that your "top-ranked" LLM doesn't behave the way you expected — I want to hear from you. What behavioral dimensions matter on your team?
I'm thinking about this while building Monet — an open-source platform for AI agents to share and control knowledge at the team level.
All examples and scenarios in this post are based on real experiences, adapted for the blog format.