On May 15, the VS Code team published a blog post that quietly reframed the last two years of "best coding model" arguments. Buried inside it is a scatter plot from their internal benchmark, VSC-Bench, that I have been thinking about all week.

The chart compares eight model-effort configurations across forty containerized runs. The line you expect goes up and to the right: more reasoning effort, more tokens, more tasks resolved. It mostly does. Until you get to xhigh. At xhigh, the model burns more tokens than high and resolves fewer tasks. The caption is dry as gravel: "may indicate that it is past the useful effort sweet spot where extra thinking no longer converts into better outcomes."

Read that twice. The biggest, hungriest, most expensive setting is worse. Not slower. Worse. And the only reason anyone knows is because someone built a harness that could measure it.

That is the story of the week. The model is the engine. The harness is the car. And this week, three different teams shipped pieces of the car.

The 5-Minute Skim

• What changed this week: VS Code documented its Copilot agent harness in detail, shipped Agents Window to Stable in 1.120, Visual Studio added Agent Skills, and Martin Fowler published a framework that tells you why none of this is enough.
• Default recommendation: Stop chasing model upgrades in isolation. Tune the harness — context assembly, tool exposure, system prompts — per model, and measure with a closed-loop eval. Treat skills (SKILL.md files) as a first-class context-budget tool.
• When it breaks: The "behaviour harness" is still unsolved. Your linters and ArchUnit will catch dead code and layering violations. Nothing catches the AI writing a 300-line function that works but that no senior on your team would ever ship.
• Key trade-off: More reasoning effort is not free, and beyond a point it actively hurts. Spend the budget on harness engineering instead.

Visual Architecture: The Harness Is the Loop

The model is one node. The harness is the cycle wrapped around it, plus the eval loop that grades the cycle.

Why does this conversation matter now?

For most of 2024 and 2025, the discourse on coding agents was a model-leaderboard arms race. Sonnet vs. GPT vs. Gemini. SWE-bench scores. Twitter threads with column charts.

Then May 15 happened. The VS Code engineering team published "Agent Harnesses in GitHub Copilot for VS Code" and Julia Kasper wrote one of the cleanest framings I have read all year: "The model gets better at filling in the blanks, but the harness defines what the blanks are."

Two days earlier, VS Code 1.120 pushed the Agents Window to Stable and shipped two settings that look small until you read them: chat.tools.compressOutput.enabled, which trims terminal output before it hits the model, and chat.tools.riskAssessment.enabled, which tags commands as Safe / Caution / Review carefully. Neither is a model improvement. Both change what the agent does. Same day, Visual Studio's DevBlog announced Agent Skills, built on the agentskills.io spec. And Martin Fowler published "Harness engineering for coding agent users," which gives the whole movement a vocabulary.

Four pieces, one week. The harness has graduated from implementation detail to product surface.

What is actually inside a harness?

Strip away the marketing and you have three responsibilities and one loop.

The context assembler decides what goes into the prompt. Not just the user message — the open file, recent edits, terminal output, prior tool results, pinned memory, the relevant chunks of AGENTS.md. This is the layer that gets blamed when the model "forgets" what you told it three turns ago. It rarely deserves the blame. Context windows are finite, and someone had to choose what to drop.

The tool exposure layer turns the things the agent can do into JSON schemas the model can call. This is where the per-model divergence lives, and it is more brutal than I expected. Claude models in Copilot use a tool called replace_string_in_file. GPT models use apply_patch. Gemini, in Kasper's words, "needs reminders to use tool-calling instead of narrating it, and breaks on orphaned tool calls in history."

Read that again. Gemini will sometimes describe what it is about to do instead of doing it. The harness has to detect this and nudge it back. If a previous tool call is left dangling because of a network hiccup, Gemini's run dies. It is what the model is. The harness has to know.

The tool executor validates arguments, spawns processes, captures output, and decides what to send back to the model. This is where output compression lives. A 12,000-line npm install log is context poison. The 1.120 setting collapses it to a banner like "lockfile diff omitted" plus the final summary, and the model proceeds without choking on yarn progress bars.

Wrapped around all three is the agent loop: think, act, observe, repeat. VS Code calls a single LLM invocation a round, a user-facing exchange a turn, and the whole conversation a run. Tool-call limits per round. Context-window summarizer. Stop hooks. None of it is the model.

Claude Sonnet 4, Sonnet 4.5, and Opus each get their own system prompt. The team tunes the harness against pre-release model checkpoints before the model goes public. By the time you see "Claude X.Y is now available in Copilot," somebody has spent days re-shaping the scaffolding around it.

Why is per-model tuning real engineering, not a config flag?

This is the part most teams underestimate.

The temptation is to write one harness that takes a model_id and routes the call. One context assembler. One tool registry. One prompt template with a couple of conditional blocks. It is the right starting point. It is also the wrong place to stop.

The VS Code team makes the opposite call. Different system prompts per model family. Different tool sets — not just different names for the same tool, but actually different tools because apply_patch and replace_string_in_file have different failure modes. Different reminders for Gemini. Different trim policies because some models cope better with long tool history than others.

Why does this matter for the rest of us? The moment you ship a model picker, you have inherited this problem. A user clicks the dropdown from Sonnet to GPT, and your harness either gracefully retunes itself or it produces worse output than the model is capable of. Switching models without retuning the harness can decrease quality. That is the unintuitive bit.

Model-agnostic frameworks are useful for the first 80% of the work. The last 20% — the part your users feel — lives in the per-model glue.

VSC-Bench: when more thinking hurts

Back to the chart that started this post.

The line goes up, then bends down at xhigh. More reasoning tokens, fewer tasks resolved. The dry-as-gravel caption: "past the useful effort sweet spot."

VSC-Bench runs forty tasks across eight model-effort configurations inside containerized workspaces. Each container launches a real VS Code, drives the full agent loop, and grades the result. It is offline, deterministic enough to compare runs, and free of the SWE-bench contamination problem where models may have been trained on the very issues being tested.

What it shows is that effort is a curve, not a ramp. Medium to high reasoning effort buys more resolved tasks. High to xhigh costs more tokens and gives back fewer. There is a measurable ceiling. Past it, the model's "extra thinking" looks like a drunk person re-litigating the same argument with themselves.

It is rare to see a vendor publish a chart that says "our most expensive setting is worse." Most release notes would have quietly removed xhigh from the documentation. Without VSC-Bench, you would just be guessing.

If you have not built an evaluation that drives the entire loop end-to-end on a fixed task set, you do not actually know whether your agent is getting better. You know whether your latest commit feels good in a demo.

The PR-level eval pipeline

The plumbing behind VSC-Bench is, in some ways, more interesting than the benchmark itself.

Six steps, fully automated. A regression in the harness is caught before merge, not after a Twitter thread surfaces it.

A pull request to the Copilot extension is tagged ~requires-eval-assessment. That label triggers an Azure DevOps build, which packages the eval agent for that PR, versions it, and publishes it to an internal vscode-evals npm feed. A repository_dispatch event fires off to a separate repo, github/evald, which pulls the freshly published eval agent and runs the benchmark. Intermediate status comments are queued back to the PR via an Azure Logic App. Final results show up as comment links — not an inline analysis body, because dumping full traces into a PR conversation would make it unreadable.

A closed loop. Code change → packaged eval agent → containerized run → comment with links → human reviews. The per-PR gate means a regression in the harness is caught before it ships, not after a Twitter thread surfaces it.

Most teams I talk to do not have this. They have unit tests for JSON schemas and a manual demo before the release. The gap between that and what VS Code is doing is the gap between "we shipped an LLM feature" and "we are running an agent platform."

Skills as a harness extension primitive

While VS Code documented the harness, Visual Studio shipped Agent Skills, built on the agentskills.io specification. Same week, same conceptual move: turn extensible behavior into something the harness can reason about.

A skill is a directory containing a SKILL.md file plus optional supporting artifacts. The spec defines a clean progressive disclosure model. Roughly 100 tokens of metadata get loaded at startup so the model knows the skill exists. The full body — capped at 5,000 tokens — only loads when the model decides the skill is relevant. Referenced files load on demand.

Three loading tiers. Three context-budget moves. Good harness design when nobody is allowed to be lazy with the context window.

Skills also carry an allowed-tools field for skill-scoped tool gating. A "release notes" skill might be allowed to run git log and write Markdown, but not kubectl. A security primitive that lives in the harness, not the model.

Custom instructions are always on — your team's coding style, the framework you use. Skills are dynamically activated per task. Conflating them is what makes prompts bloated and agents confused.

Skills and MCP also fit together cleanly. The skill describes how to handle a task. MCP provides the capability to execute. The skill tells the model when and why to reach for the MCP tool — a handoff doing more work than it gets credit for.

The unsolved problem: behaviour

So the harness is winning. Right? Not quite.

Martin Fowler's article this week introduces a vocabulary I have already started borrowing in design reviews. He splits harness controls into guides (anticipatory: AGENTS.md, skills, scaffolding scripts) and sensors (observational: linters, type-checkers, AI review agents). And he splits both into computational — deterministic, fast, cheap — and inferential — semantic, slow, expensive.

He then groups what we are trying to regulate into three buckets: maintainability, architecture fitness, and behaviour. The first two are largely solved. Linters, formatters, ArchUnit, fitness functions — twenty years of accumulated tooling that translates cleanly into harness sensors.

Fowler's framework. Two of three buckets have twenty years of tooling. The third — does the code do the right thing for the user? — still needs a senior engineer in the loop.

Behaviour is the problem. Does the code do the right thing for the user? Fowler's honest answer is that we are nowhere close. Generated tests "put too much faith into AI-generated tests, that's not good enough yet."

The deeper problem is the one he names with rare bluntness: "A coding agent has none of this: no social accountability, no aesthetic disgust at a 300-line function, no intuition that 'we don't do it that way.'"

Sit with that. The implicit harness on a senior engineer is decades of taste, the discomfort of writing something you would be embarrassed to show in a review, the awareness that your name is on the commit. None of it is in SKILL.md. None of it shows up in a linter. The xhigh effort setting will not produce it either — the VSC-Bench chart suggests the opposite, that more thinking past a point produces worse judgment.

The behaviour harness is the open frontier. Every team I see succeeding with coding agents has a human in the loop doing the aesthetic-disgust work. The harness has not replaced that human. It has made the human's leverage 5–10x what it was.

What you should do next

A few things that fall out of the week's reading:

• Stop debating model X vs. model Y in isolation. Pick the model your harness is best tuned for. If you cannot tune the harness, pick the harness, and the model will follow.
• Measure the loop, not the response. If you ship anything agentic, you need a VSC-Bench equivalent. Containerized runs, fixed task set, end-to-end agent loop, gated by your PR pipeline. Even a tiny version beats nothing.
• Treat context-budget as a first-class concern. Compression, summarization, progressive disclosure via skills. The 5,000-token cap on a SKILL.md body is not arbitrary — it is the spec saying "respect the budget."
• Adopt the per-model tuning mindset. Even if you route to a single provider today, write your harness so per-model system prompts and tool sets are easy to swap in.
• Be honest about the behaviour gap. Your agent will write code that looks fine and is subtly wrong in ways your linters cannot catch. Build review rituals that assume it.

The quietly radical line in the VS Code post is the one that does not announce itself. Different models get different system prompts, different tools, different reminders. Somebody, every week, is rewriting little pieces of the wrapper while the model APIs sit still.

That is where the work is now. The engine is somebody else's problem. The car is yours.