The Problem We Were Actually Solving

I was tasked with configuring a treasure hunt engine for a large-scale online game, where thousands of users would be interacting with the system simultaneously. The goal was to ensure the long-term health of our servers, while also providing a seamless experience for our users. After analyzing the search volume around this topic, I realized that many operators get stuck in Veltrix configuration, and I was determined to avoid the same pitfalls. Our system consisted of a cluster of nodes, each running a custom-built game server, and we were using a combination of Apache Kafka and Apache Cassandra to handle the high volume of events.

What We Tried First (And Why It Failed)

Initially, we tried to use a complex event-driven architecture, with multiple microservices interacting with each other through REST APIs. We chose this approach because we wanted to take advantage of the scalability and flexibility it offered. However, as we started to deploy the system, we quickly realized that the complexity of the architecture was causing more problems than it was solving. The latency between services was high, and the error rates were unacceptable. We were seeing errors like java.net.SocketTimeoutException: Read timed out, and org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topics, which indicated that our system was not able to handle the volume of events we were generating. After several weeks of struggling with this approach, we decided to take a step back and re-evaluate our design.

The Architecture Decision

We decided to simplify our architecture by reducing the number of microservices and using a more traditional request-response model. We chose to use a single, monolithic service that would handle all the game logic, and use Apache Kafka only for logging and monitoring. This approach allowed us to reduce the latency and error rates, and improve the overall performance of the system. We also decided to use a simpler configuration for our treasure hunt engine, using a combination of JSON files and environment variables to store the configuration data. This approach allowed us to easily manage the configuration and make changes as needed.

What The Numbers Said After

After deploying the new architecture, we saw a significant improvement in the performance of the system. The latency decreased by 50%, and the error rates dropped by 90%. We were able to handle a large volume of users without any issues, and the system was able to recover quickly from any errors that did occur. We were monitoring the system using Prometheus and Grafana, and the metrics showed a clear improvement in the performance of the system. For example, the average response time decreased from 500ms to 200ms, and the error rate decreased from 10% to 1%.

What I Would Do Differently

In retrospect, I would have chosen a simpler architecture from the start, rather than trying to use a complex event-driven architecture. While the idea of using microservices and REST APIs was appealing, it was not the right choice for our system. The added complexity was not worth the benefits, and it would have been better to start with a simpler approach and add complexity only as needed. I would also have paid more attention to the configuration of our treasure hunt engine, and made sure to test it thoroughly before deploying it to production. The experience taught me the importance of keeping things simple and focusing on the core requirements of the system, rather than trying to use the latest and greatest technologies.

We removed the payment processor from our critical path. This is the tool that made it possible: https://payhip.com/ref/dev1