<h1> From HTTP Chaos to Kafka: How We Fixed Inter-Service Communication in a NestJS Microservices Platform </h1> <blockquote> <p>A technical deep-dive into replacing synchronous HTTP calls with Kafka-based async messaging — covering architecture decisions, NestJS implementation, Redis caching, and BullMQ for background processing.</p> </blockquote> <h2> The Problem: Synchronous HTTP in a Distributed System </h2> <p>On a production NestJS microservices platform I was working on, services were communicating entirely over synchronous HTTP. On paper, this looks fine. In production, under load, it fell apart fast.</p> <p>Here's what we were dealing with:</p> <ul> <li> <strong>Cascading failures</strong> — if Service B was slow, Service A timed out, and the user felt it</li> <li> <strong>Retry storms</strong> — failed requests were retried, multiplying the load on already struggling services</li> <li> <strong>Tight coupling</strong> — every service needed to know the exact endpoint, auth, and contract of every other service it called</li> <li> <strong>Blocking threads</strong> — Node.js event loop threads were being held waiting for HTTP responses</li> </ul> <p>A simplified version of what the architecture looked like:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>User Request │ ▼ [API Gateway] │ HTTP POST /order ▼ [Order Service] │ HTTP POST /inventory/check ← blocking ▼ [Inventory Service] │ HTTP POST /notification/send ← blocking ▼ [Notification Service] </code></pre> </div> <p>Every step in this chain was synchronous. If Notification Service had a spike in latency, it bubbled all the way up to the user's response time. If it went down, orders failed.</p> <h2> The Solution: Kafka-Based Async Messaging </h2> <p>Apache Kafka gave us a way to <strong>decouple producers from consumers</strong> completely. Services no longer call each other — they emit events, and other services react to those events independently.</p> <p>The new architecture:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>User Request │ ▼ [API Gateway] │ HTTP POST /order ▼ [Order Service] ──── kafka: order.created ────▶ [Inventory Service] ──── kafka: order.created ────▶ [Notification Service] ──── kafka: order.created ────▶ [Analytics Service] │ ▼ (immediate response — no waiting) User gets 202 Accepted </code></pre> </div> <p>The Order Service emits a single event. Every downstream service consumes it independently, at its own pace, with its own retry logic. The user gets an immediate response.</p> <h2> NestJS Implementation </h2> <h3> Setting Up the Kafka Module </h3> <p>We used <code>@nestjs/microservices</code> with the Kafka transport. Here's how we configured it:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight typescript"><code><span class="c1">// main.ts</span> <span class="k">import</span> <span class="p">{</span> <span class="nx">NestFactory</span> <span class="p">}</span> <span class="k">from</span> <span class="dl">'</span><span class="s1">@nestjs/core</span><span class="dl">'</span><span class="p">;</span> <span class="k">import</span> <span class="p">{</span> <span class="nx">MicroserviceOptions</span><span class="p">,</span> <span class="nx">Transport</span> <span class="p">}</span> <span class="k">from</span> <span class="dl">'</span><span class="s1">@nestjs/microservices</span><span class="dl">'</span><span class="p">;</span> <span class="k">import</span> <span class="p">{</span> <span class="nx">AppModule</span> <span class="p">}</span> <span class="k">from</span> <span class="dl">'</span><span class="s1">./app.module</span><span class="dl">'</span><span class="p">;</span> <span class="k">async</span> <span class="kd">function</span> <span class="nf">bootstrap</span><span class="p">()</span> <span class="p">{</span> <span class="kd">const</span> <span class="nx">app</span> <span class="o">=</span> <span class="k">await</span> <span class="nx">NestFactory</span><span class="p">.</span><span class="nf">create</span><span class="p">(</span><span class="nx">AppModule</span><span class="p">);</span> <span class="nx">app</span><span class="p">.</span><span class="nx">connectMicroservice</span><span class="o">&lt;</span><span class="nx">MicroserviceOptions</span><span class="o">&gt;</span><span class="p">({</span> <span class="na">transport</span><span class="p">:</span> <span class="nx">Transport</span><span class="p">.</span><span class="nx">KAFKA</span><span class="p">,</span> <span class="na">options</span><span class="p">:</span> <span class="p">{</span> <span class="na">client</span><span class="p">:</span> <span class="p">{</span> <span class="na">brokers</span><span class="p">:</span> <span class="p">[</span><span class="nx">process</span><span class="p">.</span><span class="nx">env</span><span class="p">.</span><span class="nx">KAFKA_BROKER</span><span class="p">],</span> <span class="na">clientId</span><span class="p">:</span> <span class="dl">'</span><span class="s1">order-service</span><span class="dl">'</span><span class="p">,</span> <span class="p">},</span> <span class="na">consumer</span><span class="p">:</span> <span class="p">{</span> <span class="na">groupId</span><span class="p">:</span> <span class="dl">'</span><span class="s1">order-service-consumer</span><span class="dl">'</span><span class="p">,</span> <span class="p">},</span> <span class="p">},</span> <span class="p">});</span> <span class="k">await</span> <span class="nx">app</span><span class="p">.</span><span class="nf">startAllMicroservices</span><span class="p">();</span> <span class="k">await</span> <span class="nx">app</span><span class="p">.</span><span class="nf">listen</span><span class="p">(</span><span class="mi">3000</span><span class="p">);</span> <span class="p">}</span> <span class="nf">bootstrap</span><span class="p">();</span> </code></pre> </div> <h3> Producing Events (Order Service) </h3> <div class="highlight js-code-highlight"> <pre class="highlight typescript"><code><span class="c1">// order.service.ts</span> <span class="k">import</span> <span class="p">{</span> <span class="nx">Injectable</span><span class="p">,</span> <span class="nx">Inject</span> <span class="p">}</span> <span class="k">from</span> <span class="dl">'</span><span class="s1">@nestjs/common</span><span class="dl">'</span><span class="p">;</span> <span class="k">import</span> <span class="p">{</span> <span class="nx">ClientKafka</span> <span class="p">}</span> <span class="k">from</span> <span class="dl">'</span><span class="s1">@nestjs/microservices</span><span class="dl">'</span><span class="p">;</span> <span class="p">@</span><span class="nd">Injectable</span><span class="p">()</span> <span class="k">export</span> <span class="kd">class</span> <span class="nc">OrderService</span> <span class="p">{</span> <span class="nf">constructor</span><span class="p">(</span> <span class="p">@</span><span class="nd">Inject</span><span class="p">(</span><span class="dl">'</span><span class="s1">KAFKA_SERVICE</span><span class="dl">'</span><span class="p">)</span> <span class="k">private</span> <span class="k">readonly</span> <span class="nx">kafkaClient</span><span class="p">:</span> <span class="nx">ClientKafka</span><span class="p">,</span> <span class="p">)</span> <span class="p">{}</span> <span class="k">async</span> <span class="nf">createOrder</span><span class="p">(</span><span class="nx">createOrderDto</span><span class="p">:</span> <span class="nx">CreateOrderDto</span><span class="p">)</span> <span class="p">{</span> <span class="kd">const</span> <span class="nx">order</span> <span class="o">=</span> <span class="k">await</span> <span class="k">this</span><span class="p">.</span><span class="nx">orderRepository</span><span class="p">.</span><span class="nf">save</span><span class="p">(</span><span class="nx">createOrderDto</span><span class="p">);</span> <span class="c1">// Fire and forget — no waiting for downstream services</span> <span class="k">this</span><span class="p">.</span><span class="nx">kafkaClient</span><span class="p">.</span><span class="nf">emit</span><span class="p">(</span><span class="dl">'</span><span class="s1">order.created</span><span class="dl">'</span><span class="p">,</span> <span class="p">{</span> <span class="na">orderId</span><span class="p">:</span> <span class="nx">order</span><span class="p">.</span><span class="nx">id</span><span class="p">,</span> <span class="na">userId</span><span class="p">:</span> <span class="nx">order</span><span class="p">.</span><span class="nx">userId</span><span class="p">,</span> <span class="na">items</span><span class="p">:</span> <span class="nx">order</span><span class="p">.</span><span class="nx">items</span><span class="p">,</span> <span class="na">totalAmount</span><span class="p">:</span> <span class="nx">order</span><span class="p">.</span><span class="nx">totalAmount</span><span class="p">,</span> <span class="na">createdAt</span><span class="p">:</span> <span class="k">new</span> <span class="nc">Date</span><span class="p">().</span><span class="nf">toISOString</span><span class="p">(),</span> <span class="p">});</span> <span class="k">return</span> <span class="p">{</span> <span class="na">orderId</span><span class="p">:</span> <span class="nx">order</span><span class="p">.</span><span class="nx">id</span><span class="p">,</span> <span class="na">status</span><span class="p">:</span> <span class="dl">'</span><span class="s1">processing</span><span class="dl">'</span> <span class="p">};</span> <span class="p">}</span> <span class="p">}</span> </code></pre> </div> <h3> Consuming Events (Inventory Service) </h3> <div class="highlight js-code-highlight"> <pre class="highlight typescript"><code><span class="c1">// inventory.controller.ts</span> <span class="k">import</span> <span class="p">{</span> <span class="nx">Controller</span> <span class="p">}</span> <span class="k">from</span> <span class="dl">'</span><span class="s1">@nestjs/common</span><span class="dl">'</span><span class="p">;</span> <span class="k">import</span> <span class="p">{</span> <span class="nx">EventPattern</span><span class="p">,</span> <span class="nx">Payload</span> <span class="p">}</span> <span class="k">from</span> <span class="dl">'</span><span class="s1">@nestjs/microservices</span><span class="dl">'</span><span class="p">;</span> <span class="p">@</span><span class="nd">Controller</span><span class="p">()</span> <span class="k">export</span> <span class="kd">class</span> <span class="nc">InventoryController</span> <span class="p">{</span> <span class="nf">constructor</span><span class="p">(</span><span class="k">private</span> <span class="k">readonly</span> <span class="nx">inventoryService</span><span class="p">:</span> <span class="nx">InventoryService</span><span class="p">)</span> <span class="p">{}</span> <span class="p">@</span><span class="nd">EventPattern</span><span class="p">(</span><span class="dl">'</span><span class="s1">order.created</span><span class="dl">'</span><span class="p">)</span> <span class="k">async</span> <span class="nf">handleOrderCreated</span><span class="p">(@</span><span class="nd">Payload</span><span class="p">()</span> <span class="nx">data</span><span class="p">:</span> <span class="nx">OrderCreatedEvent</span><span class="p">)</span> <span class="p">{</span> <span class="k">await</span> <span class="k">this</span><span class="p">.</span><span class="nx">inventoryService</span><span class="p">.</span><span class="nf">reserveStock</span><span class="p">(</span><span class="nx">data</span><span class="p">.</span><span class="nx">orderId</span><span class="p">,</span> <span class="nx">data</span><span class="p">.</span><span class="nx">items</span><span class="p">);</span> <span class="p">}</span> <span class="p">}</span> </code></pre> </div> <p>Each service has its own consumer group, so every service receives every <code>order.created</code> event independently.</p> <h2> Adding Redis Caching to Reduce Repeat Load </h2> <p>Kafka solved the async problem. But we still had expensive database reads happening repeatedly for the same data — product catalogue, user profiles, configuration.</p> <p>We introduced Redis as a caching layer using <code>@nestjs/cache-manager</code>:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight typescript"><code><span class="c1">// product.service.ts</span> <span class="k">import</span> <span class="p">{</span> <span class="nx">Injectable</span> <span class="p">}</span> <span class="k">from</span> <span class="dl">'</span><span class="s1">@nestjs/common</span><span class="dl">'</span><span class="p">;</span> <span class="k">import</span> <span class="p">{</span> <span class="nx">Cache</span> <span class="p">}</span> <span class="k">from</span> <span class="dl">'</span><span class="s1">cache-manager</span><span class="dl">'</span><span class="p">;</span> <span class="k">import</span> <span class="p">{</span> <span class="nx">InjectRepository</span> <span class="p">}</span> <span class="k">from</span> <span class="dl">'</span><span class="s1">@nestjs/typeorm</span><span class="dl">'</span><span class="p">;</span> <span class="p">@</span><span class="nd">Injectable</span><span class="p">()</span> <span class="k">export</span> <span class="kd">class</span> <span class="nc">ProductService</span> <span class="p">{</span> <span class="nf">constructor</span><span class="p">(</span> <span class="k">private</span> <span class="nx">cacheManager</span><span class="p">:</span> <span class="nx">Cache</span><span class="p">,</span> <span class="p">@</span><span class="nd">InjectRepository</span><span class="p">(</span><span class="nx">Product</span><span class="p">)</span> <span class="k">private</span> <span class="nx">productRepository</span><span class="p">:</span> <span class="nx">Repository</span><span class="o">&lt;</span><span class="nx">Product</span><span class="o">&gt;</span><span class="p">,</span> <span class="p">)</span> <span class="p">{}</span> <span class="k">async</span> <span class="nf">getProduct</span><span class="p">(</span><span class="nx">productId</span><span class="p">:</span> <span class="kr">string</span><span class="p">):</span> <span class="nb">Promise</span><span class="o">&lt;</span><span class="nx">Product</span><span class="o">&gt;</span> <span class="p">{</span> <span class="kd">const</span> <span class="nx">cacheKey</span> <span class="o">=</span> <span class="s2">`product:</span><span class="p">${</span><span class="nx">productId</span><span class="p">}</span><span class="s2">`</span><span class="p">;</span> <span class="c1">// Check cache first</span> <span class="kd">const</span> <span class="nx">cached</span> <span class="o">=</span> <span class="k">await</span> <span class="k">this</span><span class="p">.</span><span class="nx">cacheManager</span><span class="p">.</span><span class="kd">get</span><span class="o">&lt;</span><span class="nx">Product</span><span class="o">&gt;</span><span class="p">(</span><span class="nx">cacheKey</span><span class="p">);</span> <span class="k">if </span><span class="p">(</span><span class="nx">cached</span><span class="p">)</span> <span class="k">return</span> <span class="nx">cached</span><span class="p">;</span> <span class="c1">// Miss — fetch from DB and cache result</span> <span class="kd">const</span> <span class="nx">product</span> <span class="o">=</span> <span class="k">await</span> <span class="k">this</span><span class="p">.</span><span class="nx">productRepository</span><span class="p">.</span><span class="nf">findOne</span><span class="p">({</span> <span class="na">where</span><span class="p">:</span> <span class="p">{</span> <span class="na">id</span><span class="p">:</span> <span class="nx">productId</span> <span class="p">},</span> <span class="p">});</span> <span class="k">await</span> <span class="k">this</span><span class="p">.</span><span class="nx">cacheManager</span><span class="p">.</span><span class="nf">set</span><span class="p">(</span><span class="nx">cacheKey</span><span class="p">,</span> <span class="nx">product</span><span class="p">,</span> <span class="mi">300</span><span class="p">);</span> <span class="c1">// 5 min TTL</span> <span class="k">return</span> <span class="nx">product</span><span class="p">;</span> <span class="p">}</span> <span class="k">async</span> <span class="nf">invalidateProduct</span><span class="p">(</span><span class="nx">productId</span><span class="p">:</span> <span class="kr">string</span><span class="p">):</span> <span class="nb">Promise</span><span class="o">&lt;</span><span class="k">void</span><span class="o">&gt;</span> <span class="p">{</span> <span class="k">await</span> <span class="k">this</span><span class="p">.</span><span class="nx">cacheManager</span><span class="p">.</span><span class="nf">del</span><span class="p">(</span><span class="s2">`product:</span><span class="p">${</span><span class="nx">productId</span><span class="p">}</span><span class="s2">`</span><span class="p">);</span> <span class="p">}</span> <span class="p">}</span> </code></pre> </div> <p><strong>Cache invalidation</strong> was handled by listening to Kafka events. When a product was updated, the product service emitted a <code>product.updated</code> event, and any consumer that cached that product would invalidate its entry.</p> <h2> BullMQ for Heavy Background Tasks </h2> <p>Not everything belongs on Kafka. For heavy, retry-sensitive background tasks — PDF generation, email sending, report processing — we introduced BullMQ backed by Redis.</p> <p>The key distinction we settled on:</p> <div class="table-wrapper-paragraph"><table> <thead> <tr> <th>Use Case</th> <th>Tool</th> </tr> </thead> <tbody> <tr> <td>Inter-service events (order created, user registered)</td> <td>Kafka</td> </tr> <tr> <td>Heavy background jobs (PDF, email, exports)</td> <td>BullMQ</td> </tr> <tr> <td>Repeated read-heavy data</td> <td>Redis Cache</td> </tr> </tbody> </table></div> <div class="highlight js-code-highlight"> <pre class="highlight typescript"><code><span class="c1">// report.processor.ts</span> <span class="k">import</span> <span class="p">{</span> <span class="nx">Processor</span><span class="p">,</span> <span class="nx">WorkerHost</span> <span class="p">}</span> <span class="k">from</span> <span class="dl">'</span><span class="s1">@nestjs/bullmq</span><span class="dl">'</span><span class="p">;</span> <span class="k">import</span> <span class="p">{</span> <span class="nx">Job</span> <span class="p">}</span> <span class="k">from</span> <span class="dl">'</span><span class="s1">bullmq</span><span class="dl">'</span><span class="p">;</span> <span class="p">@</span><span class="nd">Processor</span><span class="p">(</span><span class="dl">'</span><span class="s1">report-generation</span><span class="dl">'</span><span class="p">)</span> <span class="k">export</span> <span class="kd">class</span> <span class="nc">ReportProcessor</span> <span class="kd">extends</span> <span class="nc">WorkerHost</span> <span class="p">{</span> <span class="k">async</span> <span class="nf">process</span><span class="p">(</span><span class="nx">job</span><span class="p">:</span> <span class="nx">Job</span><span class="o">&lt;</span><span class="nx">ReportJobData</span><span class="o">&gt;</span><span class="p">):</span> <span class="nb">Promise</span><span class="o">&lt;</span><span class="k">void</span><span class="o">&gt;</span> <span class="p">{</span> <span class="kd">const</span> <span class="p">{</span> <span class="nx">userId</span><span class="p">,</span> <span class="nx">reportType</span><span class="p">,</span> <span class="nx">dateRange</span> <span class="p">}</span> <span class="o">=</span> <span class="nx">job</span><span class="p">.</span><span class="nx">data</span><span class="p">;</span> <span class="c1">// This can take 10–30 seconds — fine in a background job</span> <span class="kd">const</span> <span class="nx">reportData</span> <span class="o">=</span> <span class="k">await</span> <span class="k">this</span><span class="p">.</span><span class="nx">analyticsService</span><span class="p">.</span><span class="nf">generateReport</span><span class="p">(</span> <span class="nx">userId</span><span class="p">,</span> <span class="nx">reportType</span><span class="p">,</span> <span class="nx">dateRange</span><span class="p">,</span> <span class="p">);</span> <span class="kd">const</span> <span class="nx">pdfBuffer</span> <span class="o">=</span> <span class="k">await</span> <span class="k">this</span><span class="p">.</span><span class="nx">pdfService</span><span class="p">.</span><span class="nf">generate</span><span class="p">(</span><span class="nx">reportData</span><span class="p">);</span> <span class="k">await</span> <span class="k">this</span><span class="p">.</span><span class="nx">storageService</span><span class="p">.</span><span class="nf">upload</span><span class="p">(</span><span class="s2">`reports/</span><span class="p">${</span><span class="nx">userId</span><span class="p">}</span><span class="s2">/</span><span class="p">${</span><span class="nx">job</span><span class="p">.</span><span class="nx">id</span><span class="p">}</span><span class="s2">.pdf`</span><span class="p">,</span> <span class="nx">pdfBuffer</span><span class="p">);</span> <span class="c1">// Notify user via Kafka when done</span> <span class="k">this</span><span class="p">.</span><span class="nx">kafkaClient</span><span class="p">.</span><span class="nf">emit</span><span class="p">(</span><span class="dl">'</span><span class="s1">report.ready</span><span class="dl">'</span><span class="p">,</span> <span class="p">{</span> <span class="nx">userId</span><span class="p">,</span> <span class="na">reportUrl</span><span class="p">:</span> <span class="s2">`...`</span> <span class="p">});</span> <span class="p">}</span> <span class="p">}</span> </code></pre> </div> <p>BullMQ gives you built-in retries, concurrency control, and job prioritisation — things Kafka is not designed for at the job level.</p> <h2> Results </h2> <p>After the migration:</p> <ul> <li> <strong>Response times dropped significantly</strong> — the critical request path no longer waited on downstream services</li> <li> <strong>System stability improved under load</strong> — a slow consumer no longer caused cascading failures upstream</li> <li> <strong>Architecture became cleanly decoupled</strong> — adding a new service that reacted to <code>order.created</code> required zero changes to the Order Service</li> <li> <strong>Background tasks were offloaded</strong> — the main event loop stayed free for user-facing requests</li> </ul> <h2> Key Lessons Learned </h2> <p><strong>1. Not everything needs Kafka.</strong> We initially over-used it. Short-lived, job-oriented tasks belong in BullMQ. Kafka shines for durable, multi-consumer event streams.</p> <p><strong>2. Consumer group naming matters.</strong> Each service must have a unique <code>groupId</code>. Getting this wrong means multiple instances of the same service all consume the same message — or worse, they compete and only one gets it.</p> <p><strong>3. Schema discipline is non-negotiable.</strong> Once you have three services consuming <code>order.created</code>, changing the event payload becomes a coordination challenge. We moved to versioned event schemas early.</p> <p><strong>4. Idempotency is your responsibility.</strong> Kafka can deliver a message more than once. Every consumer must handle duplicate events gracefully — usually by checking if the work has already been done before processing.</p> <p><strong>5. Dead Letter Queues save you.</strong> We configured DLQ topics for every consumer so failed messages didn't just disappear — they landed somewhere we could inspect and replay.</p> <h2> Final Architecture </h2> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code> ┌─────────────────┐ │ API Gateway │ └────────┬────────┘ │ HTTP ┌────────▼────────┐ │ Order Service │──── Redis Cache └────────┬────────┘ │ Kafka: order.created ┌───────────────────┼───────────────────┐ │ │ │ ┌──────────▼──────┐ ┌────────▼────────┐ ┌──────▼──────────┐ │Inventory Service│ │ Notification │ │Analytics Service│ │ + Redis Cache │ │ Service │ │ │ └─────────────────┘ └────────┬────────┘ └─────────────────┘ │ BullMQ ┌────────▼────────┐ │ Email / PDF │ │ Processor │ └─────────────────┘ </code></pre> </div> <h2> Wrapping Up </h2> <p>Migrating from synchronous HTTP to Kafka-based async messaging was one of the highest-impact architectural changes we made. The system became more resilient, more scalable, and easier to extend.</p> <p>If you're running NestJS microservices and hitting the same walls — timeouts, cascading failures, tight coupling — Kafka is worth the investment. Start with one event type, get comfortable with the consumer group model, and expand from there.</p> <p><em>Ritesh Macwan is a Senior Backend Engineer specialising in NestJS, Kafka, Kubernetes, and distributed systems. Connect on <a href="https://linkedin.com/in/ritesh" rel="noopener noreferrer">LinkedIn</a> or explore projects on <a href="https://github.com/ritesh-7299" rel="noopener noreferrer">GitHub</a>.</em></p>