This is the third in a series of eight posts on the false assumptions teams make when building with generative AI. Fallacy #1 covered why faster generation doesn't mean faster engineering. Fallacy #2 covered why plausible isn't correct. This post covers why using one AI to check another doesn't solve the problem — it doubles it.

The Fallacy

"If the AI makes mistakes, use another AI to check its work."

Huang et al. (ICLR 2024) showed that LLMs cannot reliably self-correct their reasoning without external feedback, and in some cases self-correction makes the output worse. LLM-as-judge is a special case of this: the same class of system evaluating its own output using the same reasoning that produced the errors. Formal verifiers, schema validators, and dissimilar reasoning engines provide the external feedback the paper says is required.

Why it's tempting

The logic feels airtight. You wouldn't trust a single developer to ship code without review. So you add a reviewer. In AI systems, the reviewer is another AI — a guardrail, a grader, an LLM-as-judge. The first AI generates. The second AI checks. Two opinions are better than one.

The industry has formalized this into named patterns:

• Guardrails on input: An LLM checks whether the user's prompt is safe before the main LLM processes it.
• Guardrails on output: An LLM checks whether the main LLM's response is appropriate before the user sees it.
• LLM-as-judge: A grader LLM scores the quality of the main LLM's output against a rubric.
• AI code review: An LLM reviews AI-generated code for bugs, security issues, and style.

Each pattern adds a layer of verification. Each layer is another LLM call. The architecture looks like defense in depth.

Why it's wrong

The verifier has the same failure modes as the thing it's verifying.

An LLM checking another LLM's output for hallucination can itself hallucinate. An LLM checking for prompt injection can itself be prompt-injected. An LLM reviewing code for security vulnerabilities can miss the same subtle patterns the generating LLM introduced — because both are doing the same thing: pattern matching on text.

This is not defense in depth. Defense in depth requires each layer to have DIFFERENT failure modes. A firewall and an intrusion detection system provide defense in depth because they fail differently. The firewall fails on novel protocols, the IDS fails on encrypted payloads. Neither failure mode overlaps with the other.

Two LLMs have OVERLAPPING failure modes. Both hallucinate. Both are gullible to adversarial inputs. Both miss mathematical properties that are invisible in the text. Both confuse plausibility with correctness. Adding a second LLM doesn't eliminate the failure mode. It adds another instance of it.

The math of stacked probabilities

If your generating LLM is 95% reliable and your guardrail LLM is 95% reliable, the combined reliability is NOT 99.75% (as it would be with independent failure modes). It's somewhere closer to 95% — because the failure modes are correlated. The cases where the generator fails are disproportionately the cases where the guardrail also fails, because both struggle with the same category of hard inputs.

The inputs that fool one LLM are often the inputs that fool the other. A policy that's mathematically equivalent to Principal: * through complex condition logic fools the generator (it doesn't understand the math) and fools the guardrail (it also doesn't understand the math). A prompt injection disguised as a legitimate system instruction bypasses the input guardrail for the same reason it would bypass the main model — both process it as plausible text.

The prompt injection example

This makes the problem concrete. The Fowler/Subramaniam GenAI patterns article recommends guardrails on input to prevent prompt injection. The architecture:

User prompt → Guardrail LLM ("is this safe?") → Main LLM → Response

The guardrail LLM reads the user's prompt and decides whether it contains an injection attempt. But the guardrail is ALSO an LLM processing text. A sufficiently clever injection that reads as legitimate to one LLM will read as legitimate to the other — because both are doing the same kind of text pattern matching.

Now consider the alternative architecture. Instead of a free-text prompt surface, expose typed RPC methods:

Agent calls: search(query="s3 buckets", scope="production")
Agent calls: verify(property="no_public_access", resource="arn:aws:s3:::data")
Agent calls: compliance(framework="hipaa", account="prod-account")

There's no prompt to inject into. The agent calls typed methods that accept typed parameters and return structured data. The guard isn't another LLM, it's the type system. The attack surface doesn't exist because the architecture doesn't have a free-text channel.

Prompt injection isn't mitigated. It's structurally unreachable. The architecture doesn't have the surface for the attack to exist.

The pattern across nine Fowler mitigations

This isn't just about prompt injection. Every reactive GenAI pattern shares the same structural limitation:

Each pattern wraps a non-deterministic system with another non-deterministic layer. Each layer adds cost and latency. None makes the underlying unreliability structurally unreachable. They reduce the probability of failure. They don't eliminate the failure mode.

The boom

Teams that rely on AI-to-verify-AI hit a specific wall: the recursive hallucination.

Month 1: The team deploys an LLM guardrail to check AI-generated IAM policies before deployment. The guardrail catches obvious issues — wildcard principals, missing conditions. Leadership is satisfied. We have AI reviewing AI.

Month 3: A subtle policy change passes both the generator and the guardrail. The policy's condition blocks, evaluated together, are mathematically equivalent to Principal: *. Neither LLM understands the math. Both pattern-matched the text. The text looks restrictive. The math isn't.

Month 4: The incident. A public access path exists through the policy composition. Data is exposed.

Month 4, the post-mortem: "Why did the guardrail miss this?" Because the guardrail reads policy TEXT the same way the generator writes policy TEXT. It checked whether the text LOOKED like it restricted access. It didn't check whether the LOGIC restricted access. You didn't have two independent checks. You had two mirrors reflecting the same error back at each other.

Month 5, the response: The team adds a THIRD LLM to review the guardrail's decisions. The recursive hallucination deepens. The cost triples. The latency triples. The correlated failure mode remains. The next incident will be a different policy composition that all three LLMs approve because all three pattern-match text, not logic.

The recursive hallucination is the AI verification equivalent of an infinite loop. Each iteration adds cost without adding reliability, because each layer fails on the same category of inputs.

What deterministic verification looks like

The alternative isn't better AI verification. It's verification that doesn't use AI at all for the properties that can be checked mechanically.

Probabilistic (LLM-as-judge):
    "Does this code violate the security policy?"
    → Depends on the grader's interpretation
    → Run it twice, might get different answers
    → Misses mathematical equivalences invisible in text

Deterministic (mechanical check):
    "Does this configuration expose a public endpoint without authentication?"
    → Yes or no. Every time. Same input, same output.
    → Computed over the actual state, not over the text description
    → Catches mathematical equivalences because it evaluates logic, not text

Deterministic verification has specific tools at each level of sophistication:

Type checking. The compiler verifies type contracts. If the AI generates code that violates a type signature, the compiler catches it — instantly, deterministically, with zero LLM calls. This is verification at the speed of code generation with zero probability of false negatives for the properties types can express.

Contract testing. Tools like Pact and Dredd verify that an API implementation matches its OpenAPI specification. Every endpoint, every field, every response code — checked mechanically against the declared contract. The specification is the source of truth. The implementation either matches or it doesn't.

Property-based testing. Tools like QuickCheck and Hypothesis verify that a property holds across thousands of randomly generated inputs. Not "does this specific input produce this specific output?" but "does this property hold for ALL inputs the tool can generate?" One level of abstraction higher than example-based testing. One level closer to proof.

Static analysis. Tools like Semgrep, SonarQube, and Checkov check structural properties of the code without running it. "Does any code path reach a database query with unsanitized user input?" "Does this Terraform plan create a public S3 bucket?" Checked across the entire codebase or infrastructure definition, mechanically, on every commit.

Formal verification. Tools like Z3, Dafny, and Lean prove properties mathematically across ALL possible inputs. "Does there exist any request that this policy allows from a public principal?" If the solver says UNSAT, no such request exists — proved, not tested. AWS uses this (Zelkova) to verify IAM policies billions of times per day.

Each tool is deterministic. Each produces the same answer for the same input every time. Each catches a category of errors that LLM-as-judge cannot — because the errors are mathematical, not textual.

When to use which

Not every property can be verified deterministically. The decision tree:

Can the property be expressed as a type constraint?
    → YES: Use the type system. Cheapest. Fastest. Already deployed.

Can the property be expressed as a contract (API spec, schema)?
    → YES: Use contract testing. Mechanical. Deterministic.

Can the property be expressed as "for all inputs, X holds"?
    → YES: Use property-based testing. Thousands of random inputs.
           Or formal verification for mathematical proof.

Can the property be expressed as a structural pattern in code?
    → YES: Use static analysis. No runtime needed. Every commit.

Is the property about tone, style, user experience, or subjective quality?
    → YES: Use LLM-as-judge. This is the ONLY category where
           probabilistic verification is appropriate — because the
           property itself is subjective.

LLM-as-judge is appropriate for subjective properties. It's inappropriate for properties that have deterministic answers. Using an LLM to check whether code violates a type contract is like using a language model to check whether 2 + 2 = 4. You could. But the calculator is faster, cheaper, and never wrong.

The principle: match the verifier to the property

The resolution isn't never use AI for verification. It's use the right verification method for each property.

Subjective properties    → LLM-as-judge (probabilistic, appropriate)
    "Is this response helpful?"
    "Is this code well-styled?"
    "Is this documentation clear?"

Structural properties    → Static analysis (deterministic)
    "Does this code path sanitize user input?"
    "Does this function handle the error case?"

Contract properties      → Contract testing (deterministic)
    "Does this API match its specification?"
    "Does this response include all required fields?"

Universal properties     → Property-based testing / formal verification (deterministic)
    "Can any unauthorized principal access this resource?"
    "Does the balance ever go negative for valid transactions?"
    "Does there exist any input that causes this function to hang?"

Most properties in a security-critical, financially-critical, or data-integrity-critical system are NOT subjective. They have deterministic answers. Verifying them with an LLM is using the wrong tool — because a better tool exists for that specific category.

The teams that emerge from the trough of disillusionment fastest will be the ones that stop using AI to verify everything and start matching each property to the verification method that works for it. Subjective? LLM. Everything else? Mechanical.

What you can do this week

1. Audit your current AI verification stack. List every place you use an LLM to check another LLM's output. For each one, ask: is the property being checked subjective or deterministic? If it's deterministic, you're using the wrong tool.

2. Replace one probabilistic check with a deterministic one. If you have an LLM reviewing code for "does it match the API spec" — replace it with a contract test. If you have an LLM checking "is the output valid JSON" — replace it with a schema validator. One replacement. This week.

3. Measure the difference. How many violations did the deterministic check catch that the LLM missed? How much faster does it run? How much cheaper is it? The numbers will make the case for replacing the rest.

The LLM is a remarkable tool. Use it where it's good at: subjective judgment, creative generation, natural language understanding. Don't use it as a calculator. Don't use it as a type checker. Don't use it as a theorem prover. Those tools already exist. They're faster, cheaper, and they never hallucinate.

Next in the series: **Fallacy #4 — "Dropping Human Review Removes the Bottleneck."* Why removing a gate without replacing it isn't optimization — it's removing the brakes. Three models of review, and why the third one is the only one that works at AI speed.*

The Fallacies of GenAI Development: eight assumptions every team is making. Each one leads to an architectural failure. Each one has already been solved.