I built a transactional outbox toolkit for Node.js

If your Node.js service writes to Postgres and publishes events to Kafka or Redpanda, you probably have a silent dual-write bug. I built eventferry to fix it: write your event in the same transaction as your data, and a background relay reliably ships it to the broker. MIT-licensed, zero dep core,

## The bug you might not know you have

You write the order to your database. You commit. You send to Kafka.

  await db.query("INSERT INTO orders ...");
  await kafka.send({ topic: "orders.created", ... });

Looks fine. Until a crash or a Kafka outage hits between those two calls. The DB thinks the order happened; downstream services never hear about it. Quietly broken — until production breaks.

This is the dual-write problem, and it's the reason event-driven Node.js services fail in ways that are very hard to debug after the fact.

## The fix is the transactional outbox pattern

Write the event into an outbox table in the same transaction as your business data. A background relay picks rows off that table and reliably publishes them to the broker.

  ┌─────────────┐   one TX     ┌──────────────┐   relay     ┌───────────────┐
  │  your code  │ ───────────▶ │  outbox tbl  │ ──────────▶ │ Kafka/Redpanda│
  │ (order svc) │   (atomic)   │  (Postgres)  │   publish   │     topic     │
  └─────────────┘              └──────────────┘             └───────────────┘

It's a well-known pattern, but most Node.js implementations get the corners wrong: strict per-aggregate ordering under concurrent relays, the crash-recovery reaper, retry/backoff math, DLQ routing, Schema Registry serialization.

## Why a new library — the honest answer

There are three answers when you Google this:

Debezium is the obvious one. Great, but it's a JVM cluster + Kafka Connect to operate, and events are row-level (not domain-level). For a Node.js team that just wants a library, that's heavy.
pg-boss / BullMQ keep getting suggested for this — they're job queues, not outboxes. There's no atomic dual-write with your business transaction.
A DIY outbox table is what most teams roll. It works until it doesn't; the parts that bite you are exactly the ones you haven't written yet — per-aggregate ordering, the reaper, retry/backoff, DLQ.

eventferry is the "I just want a small library" option.

## Quick start

  import { Relay, PostgresStore, KafkaPublisher } from "@eventferry/all";

  const store = new PostgresStore({ pool });
  const publisher = new KafkaPublisher({
    driver: "kafkajs",
    brokers: ["localhost:19092"],
    idempotent: true,
  });

  // Inside your business transaction:
  await store.enqueue(client, {
    topic: "orders.created",
    aggregateType: "order",
    aggregateId: order.id,
    payload: { orderId: order.id, total: order.total },                                                                                                                                                                                                                                                           
  });

  // Background relay:
  const relay = new Relay({ store, publisher, dlq: { topic: "orders.dlq" } });
  await relay.start();
  process.on("SIGTERM", () => relay.stop());

That's the whole pattern.

## What's inside

✅ Strict per-aggregate ordering across N concurrent relays (FOR UPDATE SKIP LOCKED + a NOT EXISTS guard)
🔄 Crash-recovery reaper — visibility timeout reclaims rows stuck in processing
🔁 Retries with backoff + jitter, DLQ routing for terminal failures
⚡ Low-latency delivery: poll, LISTEN/NOTIFY waker, or WAL streaming relay (same mechanism Debezium uses)
🔒 Type-safe event registry with Standard Schema validation
📦 Schema Registry support (Avro / Protobuf / JSON Schema, Confluent wire format)
🧭 W3C trace propagation (OpenTelemetry-ready)
🪶 Zero-dependency core; pluggable store and broker

Integration tests run against real Postgres + Redpanda via Testcontainers.

## Roadmap

PostgreSQL ships today. MySQL/MariaDB, SQL Server, and MongoDB are next — the relay is database-agnostic; each adapter is the OutboxStore contract. CockroachDB, SQLite, Oracle, and DynamoDB are on the horizon. Full plan with architecture diagrams: ROADMAP.md.

## Try it

  npm i @eventferry/all pg kafkajs

📦 npm: https://www.npmjs.com/package/@eventferry/all
💻 GitHub: https://github.com/SametGoktepe/eventferry — feedback, issues, and stars very welcome ⭐

If you've tried Debezium or pg-boss or a DIY outbox for this and either landed somewhere good or got bitten, I'd love to hear about it in the comments.

推荐订阅源

DEV Community