Originally published on lavkesh.com

I recall one of the first design decisions for our payments platform, which was to deploy an in-memory cache for low-latency access to customer account balances. The architecture diagrams looked clean, with Go services using sync.Once-initialized maps in memory, bypassing the database for sub-millisecond reads. However, this approach worked as expected for only three months, until users started reporting inconsistent charges on receipts.

The problem surfaced at peak hours when concurrent updates to the same balance would overwrite each other. For instance, the account balance for user ID 12345 went from $1,200 to $850 to $1,200 again within seconds, leaving the cache in a state that defied the database of record. Engineers stared at the logs, baffled by the mismatch between transactions and cached values, because the team had not accounted for the fact that memory maps are not thread-safe by default in Go.

Debugging revealed the fundamental error: we were optimizing for speed without considering write-through guarantees. The cache treated concurrent requests as idempotent, which they were not. During a single user’s purchase flow, multiple goroutines could validate the balance, each reading a stale value from memory before any had a chance to commit updates. We had to shift to Redis with explicit lock keys and time-to-live settings, adding 4 milliseconds of latency but ensuring atomicity.

The cache also invalidated itself only when a change occurred, not when an upstream source updated. We discovered this when the accounting team reconciled overnight and adjusted balances based on fee settlements - the cache never reflected these updates until it expired naturally. To fix this, we had to implement message queues to broadcast invalidation events across all services. What started as a performance optimization became three nights’ worth of rewriting concurrency models.

This experience taught me two concrete lessons about distributed systems: first, that latency and consistency are a trade-off, not a checkbox; second, that in-memory solutions scale poorly in production when data flows through multiple planes. The original diagrams omitted the reality of cross-service communication, assuming all updates would flow through a single request path. They didn’t, and this oversight led to significant issues.

We eventually replaced the local cache with a Redis cluster and added circuit breakers to handle Redis failures gracefully. The change dropped read throughput by 30% but eliminated 90% of our support tickets related to balance discrepancies. Engineers now run load tests simulating out-of-order updates, which they didn’t before the incident, to ensure the system can handle such scenarios.

The real mistake wasn’t using in-memory storage but treating it as a production-ready solution without stress-testing its edge cases. The team had seen this pattern work in POCs but didn’t account for the messy concurrency of real user behavior. That six-hour debug session in the server room, where we traced race conditions line by line, remains the most expensive but valuable lesson in distributed design I’ve learned.

I keep a screenshot of the problematic cache code on my wall as a reminder: theoretical elegance is worthless if it breaks under real-world load. Production systems demand we question every assumption, even the ones that worked in benchmarks. This experience has stuck with me, and I often think about it when designing new systems.

The next morning after the fix, the team sat in the conference room with stale chai, Go code open on every screen, and a shared understanding that the most obvious optimizations often hide the hardest problems. My mother still asks why we can’t just 'make things fast like the old days.' I tell her that making things fast without making them correct is like boiling a pot on the stove and forgetting to check if the rice is cooked.

I recall that moment in the conference room, with the team reflecting on what we had learned. The experience was a hard lesson in the importance of considering all aspects of a system, not just the theoretical benefits of a particular approach. It’s a lesson that has stayed with me, and one that I try to apply to all my work.