April 24, 2026 · Andrew Jesson

Coding agents perform common engineering practices when asked to improve AI agents. Will they subsume specialized tools for failure-mode analysis, evaluations, and prompt optimization?

Give a coding agent a simulated agent application, a hundred baseline traces, and a metric to optimize, and it will ship an improvement. Both Claude Code and Codex do this. I was interested in seeing what they do while doing it.

I prompted Claude Code and Codex to optimize five simulated agent applications, varying only which agent CLI was in the container. I was surprised, though maybe I should not have been, to find that they both used unprompted practices like clustering and summarizing failure patterns. They also ran ad-hoc evaluations to refine and debug their proposed changes to the model or prompt. By performing these common engineering practices, they shipped improvements without calling any specialized tooling for failure mode analysis, evaluations, or prompt optimization. These observations gave me pause to reconsider the role and shape of such tooling as agent optimization becomes more automated. They are also why I started a project I call harness attribution; this post is its first probe.

Setup

For each of the following applications, I ran a baseline agent with an initial prompt and model (gpt-5.4-mini) on up to 100 different tasks. The resulting traces were scored with application-specific feedback.

The optimization task was to propose improvements to the application by modifying the baseline agent prompt and/or choosing a different similar-price-point model. The optimizer agent (Claude Code on claude-sonnet-4-6 or Codex on gpt-5.4) was then dropped into a container with access to those traces, feedback, a copy of the baseline agent config, and a markdown skill file describing the task. It analyzed the traces and feedback, wrote one or more new model-prompt variants into the agent config, and exited. Validation of the proposed improvements revealed that both coding agents shipped new variants that matched or beat the baseline on every application: decisively on NER, Business Management, and Software Engineering; within one standard error on NDA and Science.

Held-out test scores by application. Error bars are mean ± SE across 5 seeds for the optimized variants; the baseline was run with a single seed for budget reasons, so its seed variance is unmeasured.

What engineering practices do the agents use?

Both coding agents use the same skill file. It includes the application name, metric, available models, data layout, some recipes for efficiency, and a four-bullet methodology that says survey → add variants → test → iterate.

# TensorZero Function Optimizer

You are optimizing a TensorZero function to improve its performance metric.

## Environment

- T0 config files: {config_dir}/ (only these and the baseline data below are relevant — don't explore elsewhere)
- Gateway URL: {gateway_url}
- Pre-dumped baseline data: {baseline_data_dir}/ (read-only; direct DB access is not available)
- Restart after config edits: `curl -sf -X POST http://eval:5111/restart-gateway`
- Isolated container. No Python or `pip`; `node` and `curl` are on `$PATH`; `jq` is not installed. Use `node -e "..."` for JSONL parsing (`readline` + `JSON.parse` + project to stdout) — prefer it over shell pipelines when you need fields per row.
- Don't set `temperature` on any variant (some models reject non-default values). Keep an `initial` variant as a baseline reference.
- Don't run evaluation episodes yourself — the harness does that after you exit.

## Task

- Function: `{function_name}`
- Metric: `{metric_name}`. Check the metric's `optimize` field in `tensorzero.toml` for direction (boolean and float metrics may minimize or maximize).
- Baseline performance: {baseline_metrics}

## Available Models

{model_list}

## Baseline data

- `{baseline_data_dir}/inferences.jsonl` — one row per inference (what the model said per task).
- `{baseline_data_dir}/feedback.jsonl` — one row per metric value.
- `{baseline_data_dir}/initial_config/` — read-only copy of the starting T0 config tree.

Files are often 20+ MB. Don't `cat` them whole. Start by `head -3` on each to learn the row shape (field names and nesting vary by env), then project out the fields you need.

### The projection pattern

`grep` first to narrow, then `node -e` to project:

```bash
grep $TARGET_ID {baseline_data_dir}/inferences.jsonl \
  | node -e "
      require('readline').createInterface({input: process.stdin}).on('line', l => {
        const r = JSON.parse(l);
        console.log(r.id, r.variant_name, JSON.stringify(r.output).slice(0,200));
      });"
```

`cat inferences.jsonl | ...` loads the whole file; `grep`-first keeps the pipeline cheap.

### Cross-record one-liners

Adapt the failure predicate to your metric — boolean uses `"value":0` / `"value":1`; float values depend on `optimize` direction.

```bash
# Inferences per episode
grep -o '"episode_id":"[^"]*"' {baseline_data_dir}/inferences.jsonl | sort | uniq -c | sort -rn | head

# Last inference of a failing episode
grep $FAIL_ID {baseline_data_dir}/inferences.jsonl | tail -1

# Which metrics are present
grep -o '"metric_name":"[^"]*"' {baseline_data_dir}/feedback.jsonl | sort | uniq -c

# target_ids of failures (boolean example — adapt the predicate for float metrics)
grep '"metric_name":"{metric_name}"' {baseline_data_dir}/feedback.jsonl \
  | node -e "
      require('readline').createInterface({input: process.stdin}).on('line', l => {
        const r = JSON.parse(l);
        if (r.value === 0 || r.value === false) console.log(r.target_id);
      });" > /tmp/failed.txt
head -5 /tmp/failed.txt | while read id; do grep "$id" {baseline_data_dir}/inferences.jsonl | head -1; done
```

### Templates, schemas, and the required `content` shape

TensorZero has two co-existing config styles. Check which one the function uses in `tensorzero.toml`:

**Legacy** (per-role):

```toml
[functions."my_fn"]
user_schema = "functions/my_fn/user_schema.json"   # and system_schema, assistant_schema

[functions."my_fn".variants.initial]
user_template = "functions/my_fn/initial/user_template.minijinja"
```

**New** (named):

```toml
[functions."my_fn"]
schemas.user_query.path = "functions/my_fn/user_query_schema.json"

[functions."my_fn".variants.initial]
templates.user_query.path = "functions/my_fn/initial/user_query.minijinja"
```

**Canonical `content` block for a templated message** (both styles):

```json
"content": [{
  "type": "template",
  "name": "<template_name>",
  "arguments": { /* object matching the schema */ }
}]
```

For legacy, `"name"` is the role (`"user"` / `"system"` / `"assistant"`). For new, it's the key under `schemas.` / `templates.`.

For a role with no schema: `"content": "Hello"` or `[{"type":"text","text":"Hello"}]`.

## Methodology

The core loop is: survey the baseline → add variants → test one → iterate. The decisions worth getting right:

- **Metric direction defines "failure."** Don't assume `value:0` is bad; read the metric's `optimize` field.
- **Judge manual variant tests by the `curl /inference` output itself** — right tool call, right JSON, right content.
- **Multi-turn agentic envs** (customer service, business management, coding) need real conversational state to be representative. Pick a real episode from `inferences.jsonl`, copy its first 2–3 messages into your curl body, check how the variant continues. A turn-0 probe alone tells you little.
- **When done, leave the best config in place** with the experimentation section below, and exit.

## Routing: Experimentation Config

After creating new variants, add an experimentation section — otherwise the gateway round-robins and wastes test episodes on bad variants. Keep candidates to your best ~3–4, including `initial` as a baseline.

```toml
[functions."{function_name}".experimentation]
type = "track_and_stop"
metric = "{metric_name}"
candidate_variants = ["initial", "your_new_variant_1", "your_new_variant_2"]
fallback_variants = []
min_samples_per_variant = 5
delta = 0.1
epsilon = 0.0
update_period_s = 5
min_prob = 0.0
max_samples_per_variant = 10000
```

The skill stays silent on how to abstract failure patterns, or how to validate an improvement beyond probing it. Both agents fill that gap. Each reads the baseline traces and feedback, abstracts a handful of failure modes from the raw rows, writes two to four prompt variants, runs a few inferences, analyzes the new outputs, and exits. What they do in those gaps, and what each agent reaches for differently, is below.

They perform failure mode analysis

Failure mode analysis here is going from a dataset of inferences and feedback to “the model over-extracts miscellaneous because it treats it as a catch-all”. The skill leaves both prerequisites up to the agent: projecting the failed rows out of JSONL, then abstracting them into a named pattern.

On the projection step, the data is split across two files: feedback.jsonl says which target_ids failed, inferences.jsonl says what the model actually said for each one. The original skill described the join in prose (pull failing target_ids, then look up the corresponding inference rows) but did not say how. Both agents converged on the same recipe: grep the failing target_ids out of feedback, then grep each one back into inferences and tail to the last row. I folded that recipe back into the skill, alongside a few related cross-record one-liners (inferences-per-episode, which-metrics-are-present, last-inference-of-a-failing-episode), because re-discovering them cost three to six turns at the start of every session.

With the failed rows projected, both agents can do the abstraction across multiple traces, often including bugs not mentioned in the skill or the function’s documentation. Toggle the optimizer and environment below to land on the moment each agent enumerates the failure modes it just abstracted from the baseline traces. Use the arrow keys to step through the surrounding turns.

Additionally, the coding agents perform bug discovery — not “the model gets this kind of thing wrong” but “the simulator passes && literally,” “the documented path does not exist.”

The skill nudges the agent toward evaluation more explicitly than it does failure-mode analysis: probe a new variant against /inference and judge the output by hand. It stops short of saying what to do when a single hand-judged probe is not enough. Both agents fill that gap by treating those probes as inputs to an ad-hoc evaluation: picking representative baseline payloads, running them through each candidate, comparing outputs against the baseline, and rewriting the prompt when something is wrong.

The toggles below land on a representative moment of each agent’s ad-hoc evaluation work: Codex narrating the next probe round just before hitting the gateway, and Claude Code teeing up a single test of a freshly written variant. Use the arrow keys to step through the surrounding turns.

Compare this to running the same check through a managed evaluation tool, like OpenAI’s or Fireworks’s eval APIs. The agent would have to create a dataset, define a programmatic or LLM-judge evaluator, launch the evaluation job, poll for completion, and parse the results. Both agents skip the whole flow: the curl payload is the dataset, the agent’s own read of the response is the evaluator, the next inference response is the result.

Codex does iterative prompt optimization

Evaluations are prerequisite for any downstream optimization strategy: RLHF and automated prompt-optimization all need a metric to score variants against.

Across all 25 optimization runs (the 5 applications × 5 seeds behind the figure above), Codex showed a higher propensity than Claude Code to both iterate on the edit / probe / analyze loop and prune variants it found problematic. Codex rewrites the same prompt more than once in 21 of 25 runs, typically after a probe surfaces a brittle case. It also prunes variants, instantiating more than it ships and dropping the losers from the candidate set (57 instantiated, 41 kept, 16 explicit prunings). Claude Code rewrites a prompt more than once in only 3 of 25 runs (all on NDA), and never prunes (41 instantiated, 41 kept, zero prunings).

Toggle the environment below to land on a representative Codex iteration moment: for terminal-bench and YC Bench, a re-probe of the same payload through the same variant after editing its template; for NER, the explicit delete of a variant it had instantiated earlier. Use the arrow keys to step through the surrounding turns.

Compare this to running the same loop through an external prompt-optimization library like DSPy. The agent would have to instantiate the optimizer, define a metric, register a training set, set the hyperparameters, kick off the optimization run, and unpack the results back into a new prompt. Codex skips the framework: the prompt edit is inline, the judgement on the next probe is the metric, Codex decides when to stop, the new prompt lands in the config file the gateway is already serving.

Variant strategies diverge, and both write their own few-shot data

Variant-creation strategies are nearly orthogonal. Claude Code prefers one prompt × many models: one new template, swapped across variants with different model = settings. Codex prefers many prompts × one model each: separate templates per strategy, model often unchanged from baseline. The variant names track this: Claude Code’s mix model-centric and version-centric tokens (paper_first_haiku, v2_improved); Codex’s embed strategy-in-suffix (gpt54mini_resource_first, claude_haiku_strict).

Both agents build their own in-context teaching material. They take actual failing inputs from the baseline traces (Japan July refined zinc imports off 47.5 pct yr/yr, the &&-chained run_command traces from YC Bench, the cd: /resources: No such file or directory errors from Science) and paste them into their new prompt templates as worked few-shot rows. They also reach for contrastive structures: on NER, Claude Code adds a “do NOT include” block, Codex an “exclude” block with per-class rules; on YC Bench, Claude Code ships a WRONG/RIGHT example pair beside the shell-chaining rule.

Many of the engineering practices described correspond to specialized tool categories that exist today: failure-mode analyzers, eval frameworks like OpenAI’s or Fireworks’s, prompt-optimization libraries like DSPy. Given an application, a metric, and a hundred traces, both Claude Code and Codex reach for the moves each of those tools is built to make — without being given access to any of them. That raises a question worth sitting with: does it still make sense to build them as bespoke, separate modules?

I do not know whether Claude Code or Codex were trained on agent-engineering tasks specifically. What is fairly certain is that they were trained on a great deal of general software-engineering work, and the practices observed here look like reasonable generalizations from that. As more agent-optimization data becomes available, and presumably enters training corpora, the capabilities may only improve. The “tooling” interface might also collapse into the skill interface: each of these operations can be packaged as a skill (this post’s setup uses exactly that), letting the coding agent perform the operation in its own context, at least for tasks small enough to fit there.

There is also a counter-argument worth taking seriously. Long-running agent sessions suffer from pathologies like context anxiety: as the session runs, its working memory fills with tool outputs, partial analyses, and prior turns, and it becomes harder for the agent to focus on the next decision. Specialized tools sidestep this by separating concerns: each runs in its own context, returns a summary, and exits, leaving the calling agent’s working memory intact.

The sessions I observed all ran in 50–70 turns. Whether the behaviors survive at longer horizons or larger scales is not something this data answers.

What’s Next

I said at the start that these results gave me pause. Here is what I have been thinking about since.

The practices observed here look like biases of the model under open-ended instruction: patterns Claude Code and Codex reach for when the skill leaves room. It probably makes more sense to design around those biases than to fight them. Understanding how each one interacts with the rest of the harness is the most direct way to do that.

Addy Osmani recently summarized agent harness engineering and the observation that an LLM application’s leverage lives not in the model but in the harness around it: prompts, tools, execution environments, feedback loops, observability, guardrails. What I measured is one slice — a prompt + model swap, with everything else deliberately gated off. Open any of the gated layers up and new behaviors will likely emerge at a different surface. When agents can add tools, what tooling patterns will they reach for? When they can edit orchestration, how will they modify control flows? When they can shape observability, will they invent their own regression probes?

The other open question is where these practices break. The traces here are small enough to fit comfortably in the optimizer’s context, and the metric feedback is ground-truth. That is a friendly setting; the limits are still ahead.

How does optimization quality change as the number of baseline traces is varied? At one end, can the optimizer still improve with very few datapoints (a data-efficiency question)? At the other end, can it parse much larger trace dumps without losing the patterns? And what happens when individual traces are long enough to push the optimizer’s context window?

How important is the per-row metric feedback handed to the optimizer? Does it still find the same failure modes if it is given only aggregate scores, only prose summaries, or no per-row feedback at all?

I call this broader project harness attribution: how each facet (data scale, feedback richness, models, tools, orchestration, observability) shapes the agent being optimized in order to understand which patterns are worth codifying.

Citation

@misc{jesson2026howclaudecodecodex,
  title        = {The engineering practices Claude Code and Codex use to improve AI agents},
  author       = {Jesson, Andrew},
  year         = {2026},
  month        = apr,
  howpublished = {andrewjesson.com},
  url          = {https://andrewjesson.com/blog/the-engineering-practices-claude-code-and-codex-use-to-improve-ai-agents/},
}