Twenty four years. Ten DB migrations. Zero downtime.
Except the first one, where I lost seven minutes I couldn't accept.
That seven minutes is why this article exists.
The seven minutes
It was in the mid-2000s. I was running a content distribution system on my own, with a small open-source database underneath. The platform was growing, and at some point we decided to move to a much larger commercial database — an appliance-grade one, the kind you specify by line of business and not by hostname.
I planned the switchover for a thirty-minute maintenance window. I did the work. End-to-end, it took seven minutes.
Seven minutes during which the service was down. End users couldn't reach the catalog. Bookstores couldn't sync. Publishers couldn't see their numbers.
It bothered me more than it should have. The migration was a success. The system came back up. Nobody complained.
But it had been down. Seven minutes that, on paper, the agreement said was acceptable. Seven minutes that, in my head, I never wanted to repeat.
That feeling didn't go away. I started designing every database operation from that day as if seven minutes was the wrong answer.
The contract no one made me write
That platform delivered books, comics, and music — content from publishers and labels through the bookstores that resold it. End users, bookstores, publishers, label owners: all of them sat on top of a single piece of plumbing I was responsible for.
There was no formal SLA written anywhere that said "this never goes down." But there was something stronger than an SLA: a sales-side expectation. Inside the company, it was assumed the service was always reachable. Customers were sold on the assumption that downloads worked at any time of day. Bookstores integrated against that assumption. Publishers settled royalties on top of it.
If I took the service down for an hour, none of those agreements would have technically been broken. But the silent contract — you never notice me running maintenance — would have been.
Once, before a particularly large migration, I had to brief one of the major carriers (they were on the bookstore side, technically a B2B customer). We met around a whiteboard. I drew the sequence. After about five minutes, the lead engineer on their side just nodded and said something like, "okay, if you're doing it, we're fine." We didn't need a recovery plan from them. We didn't need a coordinated test window. They didn't even update their monitoring.
That trust didn't come from documentation. It came from the fact that none of the previous migrations had touched their integration.
So the shape of how I do these migrations started with seven minutes I couldn't accept, and was kept alive by twenty-three years of not breaking that quiet contract again.
The shape
Here is the shape, stripped of vendor names. It is not new. It is not clever. It has just survived.
Step zero: separate the data into two kinds.
- Data A: anything the end user reads. This is what the service actually serves. If this is wrong or unavailable, the service is wrong.
- Data B: aggregates, summaries, derived tables, everything else. Batches write to it. End users never read from it directly.
The rule for Data A is: real-time synchronization, both old and new databases, no exceptions. The rule for Data B is: batches can stop for a while, you'll catch up later.
Step one: well before the migration window, set up two batches that incrementally copy Data A and Data B from the old database to the new one. Use the last-modified timestamp on each row. Take physical deletes into account by occasionally diffing the row sets and removing what's no longer there. Run these for days or weeks until the new database is essentially a copy of the old, plus or minus the most recent few minutes.
Step two: at migration time, stop the batches that write Data B. Run the Data B copy one last time. The aggregate tables on the new database are now identical to the old.
Step three: switch the application logic to a maintenance mode where Data A is written to both databases, but read only from the old one. Every user-facing update now produces two writes: one to the old database (the authoritative one), one to the new database (best effort, the eventually-authoritative one). If the write to the new database fails, it's swallowed — step four catches the drift.
Step four: once all instances of the application are in this dual-write mode, run the Data A copy one final time. This catches anything that was written to the old database between the last sync and the dual-write switchover. After this, both databases agree.
Step five: switch the application logic to read from the new database, while still writing to both. This is the moment of truth. If anything is wrong with the new database, this is when the user feels it.
Step six: switch the application logic to read and write to the new database only. The old database is detached.
For a migration (the new database stays), the batches that write Data B can now resume against the new database, and you're done. For a maintenance bypass (the old database is coming back), you do the same sequence in reverse, and the old database returns to service.
For reference, here's how the application behaves at each step:
| Step | App writes | App reads | What happens |
|---|---|---|---|
| 0 | — | — | Classify rows into Data A and Data B |
| 1 | old | old | Incremental copy batches running |
| 2 | old | old | Stop Data B batches, final Data B copy |
| 3 | old + new | old | Dual-write switchover |
| 4 | old + new | old | Final Data A sync |
| 5 | old + new | new | Read switchover |
| 6 | new | new | Old database detached |
The thing nobody talks about: mixed states
This shape works because each step is resilient to mixed states.
I deploy applications manually. I have for twenty-four years. There is no coordinated rolling restart, no atomic feature flag flip across all instances. When I switch the application logic to dual-write mode in step three, some instances are still in single-write mode (against the old database only) while others are already in dual-write mode. That mixed state can last as long as it takes me to walk through each server.
The shape is designed so that mixed states are correct.
When step three is rolling out: some instances write to old only, some write to both. All instances read from the old. The old database stays authoritative. Reads are consistent.
When step five is rolling out: some instances read from old, some read from new. By this point both databases agree (step four just synced them). Either read is correct.
When step six is rolling out: some instances still dual-write, some write to new only. All instances read from new. The new database is authoritative. The few writes that still hit the old database are harmless — it'll be detached momentarily.
I don't have to wait for a deployment to finish. I don't need a feature flag system to coordinate it. I don't need a service mesh to make it safe. I need the property that the shape stays correct while it's transitioning.
This is the part of the design that is older than every operational tool I see today. And it's the part I haven't found a reason to replace.
Rollback was never a special case
If you'd asked me five years ago whether this design has rollback, I would have said yes, of course. The reverse sequence is the rollback.
A maintenance bypass already runs forward, then backward. That backward leg is rollback. It's executed every single time. Rollback isn't an emergency path. It's a normal part of the workflow.
I've done this kind of migration over ten times in twenty-four years. I have never had to use the reverse sequence as an emergency. Not because nothing ever went wrong. But because the preparation phase — the days and weeks of incremental copying, of double-checking the deletes, of staring at row counts — catches the things that would have gone wrong, before they can.
The boring part of the work is what makes the dramatic part of the work disappear.
Twenty four years later
I'm going to write something now that should probably embarrass me but doesn't: I enjoy this work.
A new database to move into — especially a serious appliance-grade one — is one of the most enjoyable things I get to do. The preparation is meditative. The cutover itself is short and quiet. The week after, when the system is running on the new hardware and the end users have noticed nothing, is satisfying in a way I have not gotten from any other kind of engineering.
Twenty four years. Ten migrations. Zero downtime, after the seven minutes I couldn't accept.
That's the entire story. There are other databases out there now, other appliances, other ways of doing this. There are managed services that do most of the dance automatically. There are tools that take a lot of the carefulness off your hands.
I don't have an argument against any of those. I just know what survives in my hands: a separation of data into two kinds, a dual-write window in the middle, a resilience to mixed states, and a reverse sequence I always treat as ordinary.
This is not what you should do. This is what twenty-four years has taught one specific person to do.
Built with Claude (Opus).
Earlier in this series:
- The Accordion Pattern: Why I stopped writing one fat LLM prompt
- Nobody knows when a job will finish. I'd still like to report it accurately.
- What survives when you build alone for 24 years
- Dynamic isn't enough. Operations is the other half.
- Live report: at this speed, you don't theorize. You eliminate.
- The loop I didn't notice closing
- Abstractions are fine. Starting on them isn't.