Published December 15, 2017 · 9 min read

A few years ago,I worked on data deletion path where we deleted connections for a person. On the deletion path, we wiped the data, confirmed the rows were gone, verified the cache invalidation in the logs. But somehow connections were still showing up in their timeline. Naturally, I checked the database, checked the cache invalidation logic and unsuprisingly, it looks all correct. I double checked the logs again; however, the fucking phantoms kept coming back anyway.

After much more logging, I realized that a read slipped into the tiny window between the cache invalidation and the write completing. The stale relationship got cached again the moment we cleared it. Two weeks chasing a bug that lived in a gap measured in milliseconds but when you have enough users, you hit that lucky window.

That is what state does to you, hiding in the gaps between operations and waiting for the worst possible moment to show up. After enough of these shit, you start to see stateless design differently. When nothing needs to be remembered, there is nothing left to betray you.

Keeping information sounds simple but rarely is. That is why stateless design has become so influential. It challenges the idea that systems should retain internal information between operations. When they do not, they are easier to write, debug, and reason about.

Statefull vs Stateless

Stateless vs. Stateful

A stateful system remembers information between operations. It keeps track of what has happened before, using that stored data to determine how it should behave next. This memory can live in variables, sessions, or persistent storage.

A stateless system works differently. It does not remember anything between executions. Each request or operation is handled in complete isolation. The output depends only on the input, and nothing else.

A simple example in JavaScript can show the difference:

// Stateful: the increment function keeps the state and refers to an inner variable
const increment = _ => (this.i = (this.i || 0) + 1);

// Stateless: the increment function does not keep any state and only calculates a new value
const increment = i => i + 1;

In the first example, the result depends on what happened before. Each call changes the internal value and affects the next result. In the second, the function is independent. It produces the same result for the same input, no matter how many times it runs.

Stateless code is easier to understand, easier to test, and easier to scale because every call stands on its own.

Testing and Debugging

When a program has no state, the only thing that affects its output is its input. This makes testing much simpler. By providing different arguments, you can easily observe how the function behaves and verify that it produces the correct results.

If you find a bug, reproducing it becomes straightforward. You can run the same function with the same inputs and see the same error every time. Once the problem is fixed, you can include that input in your test suite to ensure the bug never returns.

Stateless code builds confidence because it behaves consistently. There are no hidden variables or forgotten side effects. Every test tells you something reliable about how the system will behave in production.

Concurrency

Mutable state is one of the hardest problems in concurrent programming. When multiple threads or processes share data, each one can change it at any moment. This creates a constant risk of conflicts, unpredictable behavior, or even system failure.

If a program has no shared state, these problems disappear. There is nothing for threads to compete over. You no longer need to manage locks, barriers, or semaphores, and you do not have to worry about race conditions or deadlocks.

Removing mutable state turns concurrency from a complex problem into a manageable one. Each operation becomes independent, and the code becomes easier to reason about, test, and scale.

Scaling

When an application keeps no state, scaling becomes much easier. Each instance can handle requests independently, so you can add or remove servers whenever needed. The system does not rely on a single node to remember anything, which makes it flexible and resilient.

This principle is also what makes serverless architectures possible. In a serverless model, you provide functions that run in response to events. If one million events occur, the platform simply runs your function one million times. You do not need to manage servers, balance load, or worry about concurrency. You only pay for the time your code is running.

A stateless design allows systems to grow or shrink effortlessly. It lets developers focus on behavior instead of infrastructure and makes applications more adaptable to real-world demands.

Consensus

Consensus is a genuinely hard problem. The moment you have distributed state, you need multiple nodes to agree on what that state is. That means transactions, locks, and a whole lot of things that can go wrong in ways that are very hard to reproduce. You end up in two-phase commit hell, dealing with lock contention, timeouts that may or maynot mean the operation succeeded, and nodes that disagree on reality. It is one of those problems where the more you understand it, the more you respect why smart people have spent careers on it.

Stateless systems sidestep all of that. No shared state means no need to agree on what it is. If a node fails, you replace it. No recovery protocol, no re-election, no figuring out which node had the most recent version of the truth.

Operational Cost

If you have ever run stateful infrastructure yourself, you know how bad it gets. I brought ZooKeeper into one of our systems. Seemed like the right call at the time. What followed was an education in failure scenarios I did not know existed. Sessions expiring at the wrong moment, nodes falling out of quorum, watches not firing when they should. And keeping ZooKeeper happy is a full time job on its own. It is not the kind of thing you set up and forget. You feed it, you watch it, you wake up at 3am because something in the cluster looked at it wrong. The moment you stop paying attention, it finds a new and creative way to remind you it exists. I was the one who pushed for ZooKeeper in the first place, which made every incident a little more personal.

There is a reason dedicated teams exist to run this stuff. Managing stateful infrastructure is a specialization, not a side quest. The failure modes are too many, the edge cases too weird, and the cost of getting it wrong too high for a product team to absorb on top of everything else. Hand it to people who do nothing else. It is not giving up, it is recognizing that this problem deserves full attention that you are probably not in a position to give it.

Stateful world is still out there

None of this means state is evil. You cannot database your way to stateless. Somewhere, something has to remember things, and that is fine. Databases, file systems, queues, they exist for exactly this reason and they are good at it. The state has to live somewhere, the question is just whether that somewhere is a system purpose built for it or your application code held together with hope and a cache invalidation strategy that almost works.

The mistake is having state in places that were never designed to handle it, built by someone who was absolutely convinced they had a better idea than the people who spent decades solving this problem. I call that bullshit, and I have to remind myself of it every time I sit down to design anything, because I have been that someone more than once. I have looked at a perfectly good database and thought, you know what, I can do this smarter in memory. I could not. Nobody ever can. You just end up with a distributed system that is worse than the one you were trying to avoid, and a postmortem that you have to deal with.

Fault-Tolerant Stateful

This is a term I like to use, even if there might be a better one. A system can be called fault-tolerant stateful if restarting it does not affect how it functions. In such a system, the state can be rebuilt automatically from a fully stateful backend service. No data is lost, and a restart does not disrupt normal operation.

A good example is an application that keeps certain data structures in memory but saves them to a database at critical points. When the application starts again, it can restore those structures directly from the database and continue working as if nothing happened.

If an application truly needs to keep state, this kind of design helps it heal itself. It combines the reliability of stateless design with the persistence of stateful systems.

Where it matters

This pattern shows up everywhere once you start looking for it. APIs are stateless, databases are not. Data pipelines do stateless transformations in parallel and use checkpoints to keep just enough state for recovery. Message queues sit in the middle, stateful by design so the rest of the system does not have to be.

It is the same idea everywhere. Not because stateless is fashionable, but because the people who built these systems got burned enough times to know where state belongs and where it does not.

Keep what you must. Forget everything else. And be honest with yourself about which category something falls into, because that is where most bad decisions happen.

In Conclusion

Stateful applications are popular for good reasons. They are often simpler to start with, easier to reason about when everything is going well, and nobody has to have the stateless architecture conversation in the middle of a sprint. The problems show up later, in production, usually at the worst possible time, as you have seen.

Every layer of state you own is a layer of complexity you are signing up to debug at 3am. The phantom friendships, the ZooKeeper incidents, the two-phase commit hell, none of that is hypothetical. It is just a matter of when.

Avoid state where you can. Push it to systems that were built to handle it when you cannot. And when someone on your team proposes managing it themselves because they have a clever idea, show them this post.