Large codebase migrations are not typing problems; they are distributed state machine problems.

When you attempt to execute a multi-step, multi-PR refactor using an LLM—such as the workflows I proposed in this migration-orchestration-skill—you are not just writing code. You are instantiating a distributed system. Your orchestrator is the control plane, your LLM subagents are asynchronous worker nodes, and the local repository is your shared database.

Most AI workflow documentation assumes a "happy path" where agents neatly read instructions, edit files, and check off a Markdown list. This is whiteboard architecture at its most naive. In reality, agents hallucinate, Git locks up, test suites hang, and state files corrupt. To build an orchestrator that actually finishes a migration, we must align our architecture with the mechanical realities of the environment.

The Context & The Constraint

The core business requirement is to safely sequence, verify, and commit a massive architectural change (e.g., migrating a monolith to partitioned micro-databases or rewriting React classes to hooks) without blocking concurrent human feature development.

The fundamental distributed systems headache? You are coordinating asynchronous, non-deterministic compute nodes (LLM agents) attempting concurrent mutations against a rigidly sequential, locally-locked state store (the local filesystem and Git). If you don't engineer strict boundaries, the result is corrupted code, blown context windows, and thousands of dollars in wasted API tokens.

The Naive Approach (The MVP)

The standard MVP implementation breaks the migration into discrete steps (S1..Sn), defines a dependency graph, and instructs an orchestrator to spawn concurrent subagents for any steps that touch "disjoint files." The state of the migration is tracked by having agents read and update a human-readable <slug>-progress.md file. When an agent fails a test suite, it enters a worker → reviewer → fix loop until the tests pass.

Under real-world load, this implodes almost immediately:

• State Corruption: Two agents finish their steps simultaneously and blindly overwrite the progress.md file, wiping out each other's status updates.
• The Thundering Herd: Multiple agents attempt to run npm install or cargo build in parallel. They exhaust I/O, thrash memory, and crash the local machine.
• Lock Contention: Concurrent git operations trigger .git/index.lock collisions, causing unhandled fatal errors that halt the orchestrator entirely.

The Architectural Evolution (Iterative Refinement)

To fix this, we must deconstruct the pipeline and apply mechanical sympathy to every failure domain.

1. State Management & Consensus

• The Pitfall: Using a plain Markdown file (<slug>-progress.md) as a concurrent data store. It guarantees race conditions, phantom reads, and lost updates when multiple agents edit the file.
• The Fix: Introduce a mutex. Force agents to acquire a local file lock before reading or modifying the progress file.
• The Hidden Pitfall: File locks on a local OS are notoriously brittle. If an agent process segfaults, OOMs, or the user forcefully terminates the terminal (Ctrl+C), the lock is never released. The migration is permanently deadlocked.
• The Definitive Fix: Shift to an Append-Only JSONL Ledger for system state, paired with Optimistic Concurrency Control (OCC). Agents do not mutate past lines; they append new state events: {"step": "S2", "status": "passed", "sha": "abc1234", "timestamp": 1719154980}. The Markdown file is strictly a materialized, read-only projection generated by the orchestrator for human consumption. If an agent's expected prior state doesn't match the latest ledger sequence, the append is rejected, and the agent must reconcile.

2. Parallelism & The Edge Reality

• The Pitfall: Executing concurrent steps under the assumption that "disjoint files" (e.g., Agent A edits auth.ts, Agent B edits db.ts) prevents conflicts.
• The Fix: Implement a global queue for Git operations, effectively using pessimistic locking. Agents can write files concurrently but must wait in line to run git add and git commit.
• The Hidden Pitfall: This destroys the throughput of the orchestrator and causes catastrophic memory buffering. Agents are forced to hold pending diffs and LLM context in memory while waiting for the Git lock. This bloats the memory footprint, eventually causing the language runtime's garbage collector to thrash and crash the process.
• The Definitive Fix: Use Git Worktrees. The orchestrator must provision a physical, isolated directory (git worktree add ../S2-branch) for every concurrent step. This gives each agent a completely isolated index, working directory, and execution environment. Parallelism is no longer an illusion; it is mechanically enforced by OS-level filesystem boundaries.

3. The Poison Pill & Operational Blindspots

• The Pitfall: The worker → reviewer → fix loop assumes that every code problem is solvable if the LLM just tries hard enough. It acts as an unbound while(true) loop.
• The Fix: Implement a strict retry counter (e.g., max_attempts=3). If the agent fails, it marks the step as blocked.
• The Hidden Pitfall: A crashed, rate-limited, or hallucinating agent might stall out before reporting back. The ledger permanently shows the step as in_progress. Because downstream steps in the DAG depend on this node, the system indefinitely buffers out-of-order sequences waiting for the missing tail. This is a classic zombie state.
• The Definitive Fix: Lease-based Execution and Dead Letter Queues (DLQ). When an agent claims a step, it writes a lease to the ledger with a hard TTL (e.g., expires_at: 1719155980). If the orchestrator detects an expired lease, the step is violently revoked, marked as a Poison Pill, and sent to the DLQ. The orchestrator halts execution of dependent nodes and pages the human operator. A strict dependency chain cannot skip broken links; it must halt safely and predictably.

The Takeaway

Drawing dependency boxes on a whiteboard is easy. Executing them reliably against physical constraints is Staff-level engineering.

An LLM orchestration framework is not exempt from the physical laws of distributed systems. File lock contention, out-of-order execution, unparseable ASTs, and process crashes are guaranteed. If you rely on "happy path" prompts to sequence a codebase migration, your system will fail. But if you design your AI pipeline with immutable ledgers, isolated execution environments, and strict TTL bounds, you graduate from writing fragile prompt chains to operating a resilient, production-grade migration machine. Real engineering is found in how your system handles failure, not how it behaves when everything goes right.