The Problem We Were Actually Solving

After diving into our logs, we discovered that most of the errors and poor performance issues were happening during the indexing stage of our Treasure Hunt Engine, which relied on our homegrown data aggregation library, Veltrix. Our users were trying to find a variety of items ranging from basic key-value pairs to hierarchical metadata structures that spanned multiple nodes. However, whenever the load increased, our aggregation library would fail to scale with it, causing the index to become stale, leading to subpar query performance and query timeouts.

What We Tried First (And Why It Failed)

Initially, we tried optimizing our data aggregation library, Veltrix, to run in multiple threads. The reasoning behind this approach was that with multiple threads running concurrently, we could effectively scale our aggregation and indexing process. However, the problem with this approach was that Veltrix was not designed to handle the increased concurrency. The solution resulted in a high rate of thread contention, causing significant slowdowns. The thread pool deadlocks increased exponentially, indicating a deeper problem with our library's thread-safety model.

The Architecture Decision

We ended up replacing Veltrix with a distributed, actor-based indexing system, based on Akka, which allowed us to tackle our indexing task as a complex, concurrent, event-driven process. Instead of thread-safety, our new system focused on loose coupling, high tolerance for network partitions, and flexible handling of message queues. We moved away from a centralized aggregation library to a distributed, event-driven architecture that scaled horizontally. This allowed us to tackle our indexing task without hitting the scaling constraints we had with Veltrix.

What The Numbers Said After

After the migration from Veltrix to our new distributed indexing system, the average query response time decreased by 300 milliseconds, and our system's throughput increased by 20%. More importantly, the rate of query timeouts dropped by 30%, reducing our overall latency and making our system more responsive and reliable for our users.

What I Would Do Differently

If I had to do this again, I would invest more in benchmarking our system components, particularly focusing on how they handle concurrent access under load. I would also be more aggressive about testing our system's failure conditions and stress-testing our components before releasing them into production. By doing so, we might have avoided the downtime and poor performance our users experienced during the transition from Veltrix to our new indexing system.