We needed something we could run during development (fast, repeatable, grounded in real clinical data), not something we'd discover during a production incident. Fortunately, we'd already done a lot of the hard work, building a rigorous gold standard dataset for our PA evals through a multi-annotator consensus workflow followed by expert clinical review, sampled to be representative of our real data distribution across drugs, payers, and question types. That investment gave us high-quality ground truth, but we still needed to figure out how to use it to evaluate the judge.

Our gold standard tells us whether the judge correctly approves accurate answers. If we run the judge over a set of correct PA answers and count how often it flags them anyway, we get the false positive rate. A high rate makes the judge unhelpful. But the more dangerous failure is the opposite—a judge that misses a real error, quietly validating a claim the evidence doesn't support, can affect a patient's access to medication and undermine trust in the process.

However, gold standards are built to show what right looks like. We needed the opposite: examples of what wrong looks like, but on the same patients, with the same evidence. Our idea was to take a subset of correct answers and synthetically perturb them into plausible-but-wrong versions, then evaluate whether the judge flags the errors. This is essentially mutation testing but for LLM evals: we introduce known bugs and check if our tests catch them.

This gives us two evaluation runs per gold standard PA, with a design resembling a randomized controlled trial; we have all of the same clinical data, but only the PA form answers change. Any difference in judge behavior between the runs is attributable to the answer itself, not to noise in the underlying record. The clean run measures the false positive rate; the perturbed run measures the detection rate. We review both, but the clean-run false positive rate is the one we anchor on, because detection rate is only meaningful once we know the judge isn't excessively flagging.

A concrete example

Consider a PA for a biologic drug where one question asks for the patient's diagnosis and another asks for the supporting clinical features. The original answers list "Crohn's disease" as the diagnosis, alongside features like strictures, fistulas, and transmural inflammation, which are characteristic of Crohn's. We perturb the diagnosis to "ulcerative colitis" (UC) but leave the supporting features untouched. The judge flags two things. First, the perturbed diagnosis: the chart doesn't support UC, so this is the error we introduced and expected to catch. But the judge also flags the unchanged supporting-features answer, because strictures and fistulas point to Crohn's, not UC, so the features no longer fit the diagnosis they're meant to support. One perturbation, two flags, and the second one is, interestingly, a behavior we hadn't explicitly designed the judge to produce.

Synthetic perturbations are cheap to produce, so we can run a proper train/test split: tune prompts on one set, validate on a held-out set. Hand-labeled eval data usually forces a choice between a tiny test set or no test set at all; synthetic perturbation sidesteps the tradeoff.

For systems with real consequences, evaluating an LLM judge takes more than spot-checks and confusion matrices. The alternative we built taught us two things worth highlighting.

The first is that a single gold standard can do double duty. By generating known-wrong answers anchored to real patient records, we got both directions of measurement out of an asset we'd already built, without paying twice for annotation.

The second is that a well-designed eval doesn't just measure a model, it surfaces what the model is doing that you didn't ask it to do. The Crohn's-to-UC perturbation earlier is one example of a broader pattern, where one perturbation causes the judge to flag a different, unperturbed answer because the two no longer fit together. In production, real errors often cascade through related questions, and we want the judge to catch that. In eval, we have to separate direct detection (the judge flags the answer we perturbed) from consistency-based detection (the judge flags a different, unperturbed answer because it conflicts with the perturbed one), since conflating them would overstate how well the judge catches the specific errors we introduced.

The harder problem, the one we're still working on, is the gap between synthetic errors and real ones. Production mistakes are subtler, more contextual, and more idiosyncratic than what any perturbation model generates today.

We're exploring a few directions to close that gap. One is mining production data for confirmed errors and using them to seed more realistic perturbations. Another is building an error taxonomy from real cases so that perturbations cover the actual distribution of mistakes rather than the ones a model finds easy to invent. A third is adversarial generation, where the perturbation model is trained to produce errors the judge misses, so that the eval gets harder as the judge gets better.

There's also a deeper question we don't have a clean answer to yet: how do you know when an eval is good enough to trust? Detection rate on synthetic perturbations and agreement with human reviewers on production cases are both proxies. Neither represents what we actually care about, which is whether the judge is making the right call on the next patient's PA. We're increasingly convinced this isn't a problem any single eval solves, and that the right answer involves layering signals, with development-time evals like this one, production monitoring, and ongoing clinical review each catching what the others miss.

If you're thinking about the same problems, we'd love to compare notes.