"When AI reviews AI's code, you've built an infinite loop. Here's how we stopped it."

While building an AI code review product (Orange Codens), there was one design problem that dominated everything else: when the reviewer is AI and the fixer is AI, you can very easily build an infinite loop.

Walk it through. Orange reviews a PR and finds a problem. It doesn't stop at the comment — it hands the fix off to another Codens (Purple / Red) to produce a fix PR. But that fix PR is also a PR. Orange reviews it. New code, new findings. Hand off again. Fix. Review again.

A human reviewer stops at "good enough." AI doesn't, and every loop costs LLM tokens. So "it terminates" and "cost is bounded" have to be guaranteed by the architecture, not by how smart the model is. Here's what we wired into Orange, with the real code and the actual non-convergence incidents we hit.

The structure that loops

Built naively, here's how it spins:

PR open
  → Orange review → finding
    → handoff → Purple/Red opens a fix PR
      → that fix PR is open
        → Orange review → new finding
          → handoff → … (forever)

There isn't one place to cut it. "When do we hand off", "whose PRs are eligible", "do we dispatch the same finding twice", "when does the review itself converge" — each needs its own mechanism. In order:

Cut 1: handoff fires at merge, not when a finding is created

This is the big one. We don't dispatch a fix task the moment we find something. We dispatch only when the PR is merged.

The webhook handler:

_REVIEWED_ACTIONS = {"opened", "synchronize", "reopened"}

async def execute(self, payload):
    action = payload.get("action", "")
    pr = payload.get("pull_request") or {}

    if action == "closed":
        # the merge-gated handoff trigger point
        merged = bool(pr.get("merged"))
        return WebhookResult(
            outcome="handoff",
            handoff_repository_id=repository.repository_id,
            handoff_pr_number=int(pr.get("number", 0)),
            handoff_merged=merged,
            handoff_pr_author=str((pr.get("user") or {}).get("login", "")),
        )
    ...

Before merge, it only posts GitHub suggestions. It wakes Purple/Red only on the closed event with merged=true. And a PR closed without merging discards its queued handoffs:

if not merged:
    # the code never lands on main, so drop queued handoffs
    discarded = 0
    for f in open_findings:
        if f.handoff_queued_at is not None:
            f.handoff_queued_at = None
            await finding_repo.update(f)
            discarded += 1
    return {"merged": False, "discarded": discarded}

What changes: an unmerged PR has zero downstream cost. Experimental PRs, drafts, rejected PRs — however many findings they collect, not a single fix task fires unless the code reaches main. Reviews can run all they want, but handoff (the operation that spends money and creates new PRs) always passes through merge — a human (or an explicit gate) decision.

Cut 2: PRs authored by a bot are excluded from auto-handoff

This is the direct break in the loop. Fix PRs are opened by purple-codens[bot] or red-codens[bot]. A PR authored by one of those bots is excluded from auto-handoff.

_CODENS_BOT_LOGINS = {"purple-codens[bot]", "red-codens[bot]", "orange-codens[bot]"}

def _should_handoff(self, finding, policy, is_bot_pr):
    # a human-queued finding dispatches regardless of mode/bot
    if finding.handoff_queued_at is not None:
        return True
    if is_bot_pr:
        return False  # auto-dispatching on a bot PR is the loop
    if policy.mode == HandoffMode.FULL_AUTO:
        return True
    if policy.mode == HandoffMode.THRESHOLD_AUTO:
        threshold = policy.auto_threshold_severity or "high"
        meets_severity = finding.severity.rank >= Severity(threshold).rank
        in_category = (
            not policy.auto_categories or finding.category.value in policy.auto_categories
        )
        return meets_severity and in_category
    return False

Orange still reviews a bot's fix PR (we want to confirm quality), but it does not auto-spawn the next fix task from it. That's the fundamental cut.

Note the first branch. A handoff_queued_at finding (one a human manually queued with "fix this") dispatches even on a bot PR, even when mode is off. Explicit human intent overrides the loop guard. Automatic chaining is stopped, but a human saying "this particular finding on this bot PR is worth fixing" gets through. Automatic vs. manual cleanly splits the safe default from the escape hatch.

Cut 3: findings from verify runs aren't handed off

Closing another loop path. Purple runs its own verify cycle (implement → test → fix). Findings from Orange reviewing those runs are excluded from handoff.

# exclude findings from purple_verify runs (loop guard)
purple_verify_run_ids = {
    r.review_run_id for r in runs if r.triggered_by == TriggeredBy.PURPLE_VERIFY
}

eligible = [
    f for f in open_findings
    if f.review_run_id not in purple_verify_run_ids
    and f.handed_off_task_id is None          # no double-dispatch (idempotency)
    and self._should_handoff(f, policy, is_bot_pr)
]

handed_off_task_id is None matters too: a finding handed off once is never dispatched again. It prevents the quiet divergence of two fix tasks spawning for the same problem.

Incident 1: findings in the same file deadlocked each other's fix PRs

Now the part where it was built to spec and still failed to converge.

In an E2E run, app/api/notes/search.py had 3 findings (SQL injection, a hardcoded AWS key, and one more). Dispatched naively as "one task per finding," each opened a separate fix PR.

Here's the trap. When the SQLi fix PR tries to merge, Orange reviews it and raises a carry-over REQUEST_CHANGES: "the AWS-key finding in the same file is still unresolved." The AWS-key fix PR is blocked for the same reason as long as the SQLi remains. All three fix PRs became unmergeable, each blocked by the others' unfixed findings.

The fix: at handoff time, coalesce findings of "same file × same target" into one task.

# coalesce by (target_codens, file_path). Multiple findings in the same file
# going to the same service become ONE task so the fix PR resolves them
# together. Otherwise each finding's fix PR is blocked by orange's carry-over
# review of the other unfixed findings in that file.
groups: dict[tuple, list[Finding]] = {}
singles: list[Finding] = []
for f in eligible:
    target = self._target_for(f, policy)
    if target == TargetCodens.PURPLE and f.file_path:
        groups.setdefault((target, f.file_path), []).append(f)
    else:
        singles.append(f)

Fix one file's problems together, in one PR. Obvious in hindsight, but "parallelize naively per finding" puts the reviewer (Orange itself) in a position to block its own fixes with its own other findings — a self-deadlock. Get the unit of parallelism wrong and review and fix jam against each other.

Incident 2: new findings on new code keep the review from converging

Another one. When Orange reviewed a bot's fix PR, it kept raising new high-severity findings on the newly written code each round (missing tests, style, etc.) — 4 consecutive REQUEST_CHANGES rounds, then escalation at the cap (max_review_iterations=5). Non-convergence.

The fix changed the decisive-review logic for bot fix PRs:

carry-over (unresolved findings from the previous review) at severity ≥ high → block (confirm they were actually fixed)
new findings block only if they're blockers (the fix introduced a new critical problem)
new findings below critical are recorded inline / in the body but the PR is APPROVED

Behavior on human PRs (COMMENT only, never auto-block) is unchanged. Only for bot fix PRs did we make the judgment favor convergence.

The point: a perfectionist reviewer never converges. If you raise a new minor finding at full marks on every round, the PR never closes. Anchor on "was the previous serious finding resolved", record new minor ones but let them through. Designing the review bar so the review can stop.

Takeaway: termination is guaranteed by the architecture, not the model

When AI is both author and reviewer, the architecture's job is to guarantee termination and bound cost. No amount of model intelligence guarantees that. The wiring does.

The five cuts in Orange:

Handoff fires at merge, not at finding-time (unmerged PRs have zero downstream cost)
Bot-authored PRs are excluded from auto-handoff (the direct loop cut)
Findings from verify runs aren't handed off
A handed-off finding is never dispatched twice (idempotency)
A human-queued finding crosses the guard (escape hatch)

And two we added after hitting them:

Findings in the same file coalesce into one task (avoid self-deadlock)
Bot fix PR review blocks only on carry-over high + new blockers (favor convergence)

Run "AI writes code" in production and you always hit "who reviews it." If you let AI review it too, you have to design the stopping mechanism as part of the deal — otherwise it fixes forever, the cost diverges, or it deadlocks itself.

Orange Codens builds all of this into the product.

https://www.codens.ai/en/

推荐订阅源

DEV Community