Your payment provider sends a webhook when an invoice is paid. You verify the HMAC signature, update the subscription, send the welcome email, and move on.

Then the same signed request arrives again. Maybe the provider retried after a timeout. Maybe an attacker captured the request from a misconfigured proxy log. Maybe a developer replayed production traffic into staging and accidentally pointed it at production. The signature is still valid. Your handler runs again. The customer gets a second email, the fulfillment job runs twice, or worse, you credit an account twice.

That is the trap: signature verification answers one question only.

Did someone with the shared secret sign these exact bytes?

It does not answer:

• Is this request recent?
• Have we processed this event before?
• Are two app instances racing to process the same event right now?
• Did the body parser change the bytes before verification?

A production webhook endpoint needs all four answers. This post builds the boring version that survives retries, replay attacks, deploy races, and provider quirks.

The broken handler everybody ships first

Most webhook handlers start like this:

import express from 'express';
import crypto from 'node:crypto';

const app = express();
app.use(express.json());

app.post('/webhooks/billing', async (req, res) => {
  const signature = req.header('x-provider-signature');
  const expected = crypto
    .createHmac('sha256', process.env.WEBHOOK_SECRET!)
    .update(JSON.stringify(req.body))
    .digest('hex');

  if (signature !== expected) {
    return res.status(401).send('bad signature');
  }

  await activateSubscription(req.body.customerId);
  await sendWelcomeEmail(req.body.customerId);

  res.sendStatus(204);
});

This code has three real bugs.

First, it signs JSON.stringify(req.body), not the bytes the provider signed. JSON allows insignificant whitespace and object key ordering differences. Your framework parsed the payload, allocated a JavaScript object, then serialized a new string. That new string is not guaranteed to match the original body.

Second, signature !== expected leaks timing information. Is that the top priority on an internal admin webhook? Probably not. Is crypto.timingSafeEqual one line? Yes. Use it.

Third, it accepts the same valid request forever. If someone records the request today and sends it next month, the HMAC still matches.

Let’s fix the endpoint in layers.

Layer 1: verify the raw body, not parsed JSON

Most providers sign the raw request body. In Express, capture it with express.raw() for this route. Do not put express.json() in front of the webhook route.

import express from 'express';
import crypto from 'node:crypto';

const app = express();

app.post(
  '/webhooks/billing',
  express.raw({ type: 'application/json', limit: '1mb' }),
  async (req, res) => {
    const rawBody = req.body as Buffer;
    const signature = req.header('x-provider-signature');

    if (!signature || !verifySignature(rawBody, signature)) {
      return res.status(401).send('bad signature');
    }

    const event = JSON.parse(rawBody.toString('utf8'));
    await handleBillingEvent(event);

    res.sendStatus(204);
  },
);

function verifySignature(rawBody: Buffer, signatureHeader: string): boolean {
  const expected = crypto
    .createHmac('sha256', process.env.WEBHOOK_SECRET!)
    .update(rawBody)
    .digest('hex');

  const actual = Buffer.from(signatureHeader, 'hex');
  const expectedBuffer = Buffer.from(expected, 'hex');

  if (actual.length !== expectedBuffer.length) return false;
  return crypto.timingSafeEqual(actual, expectedBuffer);
}

If your provider uses a header like t=1715374800,v1=abc123, parse it explicitly. Do not compare the whole header string unless the documentation says that is the signed value.

Also notice the route-level body parser. This matters. A common production bug is this ordering:

app.use(express.json());
app.post('/webhooks/billing', express.raw({ type: 'application/json' }), handler);

By the time the request reaches express.raw(), it has already been consumed by express.json(). Your handler sees an object instead of a buffer. Put webhook routes before global JSON middleware, or mount them on a separate router.

Layer 2: reject old requests with a signed timestamp

Replay protection starts with freshness. The provider should include a timestamp in the signed material. Stripe-style headers are common:

x-provider-signature: t=1715374800,v1=9f2c...

The signature is computed over:

<timestamp>.<raw body>

That timestamp must be part of the HMAC input. If the timestamp is just an unsigned header, an attacker can replace it with the current time and replay the old body.

Here is a verifier with a five-minute tolerance window:

const MAX_SKEW_SECONDS = 5 * 60;

type ParsedSignature = {
  timestamp: number;
  signatures: string[];
};

function parseSignatureHeader(header: string): ParsedSignature | null {
  const parts = header.split(',');
  let timestamp: number | undefined;
  const signatures: string[] = [];

  for (const part of parts) {
    const [key, value] = part.split('=');
    if (key === 't') timestamp = Number(value);
    if (key === 'v1' && value) signatures.push(value);
  }

  if (!timestamp || signatures.length === 0) return null;
  return { timestamp, signatures };
}

function verifySignedRequest(rawBody: Buffer, header: string, now = Date.now()): boolean {
  const parsed = parseSignatureHeader(header);
  if (!parsed) return false;

  const ageSeconds = Math.abs(Math.floor(now / 1000) - parsed.timestamp);
  if (ageSeconds > MAX_SKEW_SECONDS) return false;

  const signedPayload = Buffer.concat([
    Buffer.from(`${parsed.timestamp}.`, 'utf8'),
    rawBody,
  ]);

  const expected = crypto
    .createHmac('sha256', process.env.WEBHOOK_SECRET!)
    .update(signedPayload)
    .digest('hex');

  return parsed.signatures.some((candidate) => safeHexEqual(candidate, expected));
}

function safeHexEqual(a: string, b: string): boolean {
  try {
    const aBuffer = Buffer.from(a, 'hex');
    const bBuffer = Buffer.from(b, 'hex');
    return aBuffer.length === bBuffer.length && crypto.timingSafeEqual(aBuffer, bBuffer);
  } catch {
    return false;
  }
}

Five minutes is a sane default. Too short and you reject legitimate retries when clocks drift or queues back up. Too long and you extend the replay window. Pick a value deliberately, monitor rejections, and keep NTP working on your servers.

This stops captured requests from being replayed next week. It does not stop a valid request from being replayed twice inside the five-minute window. Providers retry inside that window all the time. That is where idempotency comes in.

Layer 3: store event IDs, not vibes

A webhook event should have a stable unique ID from the provider: evt_123, invoice.paid:abc, a delivery ID, or a transaction ID. Use it. If the provider does not send one, derive a fingerprint from the provider name, event type, and raw body hash. A provider-issued ID is better because two legitimate events can have identical bodies.

Create a table for processed webhooks:

CREATE TABLE webhook_events (
  provider text NOT NULL,
  event_id text NOT NULL,
  event_type text NOT NULL,
  received_at timestamptz NOT NULL DEFAULT now(),
  processed_at timestamptz,
  status text NOT NULL CHECK (status IN ('processing', 'processed', 'failed')),
  error text,
  PRIMARY KEY (provider, event_id)
);

Then claim the event before doing side effects:

async function claimWebhookEvent(event: BillingEvent): Promise<'claimed' | 'duplicate'> {
  const result = await db.query(
    `INSERT INTO webhook_events (provider, event_id, event_type, status)
     VALUES ($1, $2, $3, 'processing')
     ON CONFLICT (provider, event_id) DO NOTHING`,
    ['billing-provider', event.id, event.type],
  );

  return result.rowCount === 1 ? 'claimed' : 'duplicate';
}

Now the handler can acknowledge duplicates without repeating side effects:

async function handleBillingEvent(event: BillingEvent) {
  const claim = await claimWebhookEvent(event);
  if (claim === 'duplicate') return;

  try {
    await applyBillingEvent(event);

    await db.query(
      `UPDATE webhook_events
       SET status = 'processed', processed_at = now(), error = NULL
       WHERE provider = $1 AND event_id = $2`,
      ['billing-provider', event.id],
    );
  } catch (error) {
    await db.query(
      `UPDATE webhook_events
       SET status = 'failed', error = $3
       WHERE provider = $1 AND event_id = $2`,
      ['billing-provider', event.id, String(error)],
    );
    throw error;
  }
}

This is better, but there is a policy choice hidden in it: what should happen when the first attempt fails halfway through?

If the handler inserted processing, sent an email, then crashed before marking processed, the next retry will look like a duplicate and return. That might be exactly what you want for non-critical side effects. It is not okay for money movement.

For high-value workflows, make the business operation itself idempotent. The webhook event table prevents duplicate handler execution. The domain tables prevent duplicate outcomes.

CREATE TABLE subscription_activations (
  provider text NOT NULL,
  event_id text NOT NULL,
  customer_id text NOT NULL,
  activated_at timestamptz NOT NULL DEFAULT now(),
  PRIMARY KEY (provider, event_id)
);

async function activateSubscriptionOnce(event: BillingEvent) {
  await db.query('BEGIN');
  try {
    const activation = await db.query(
      `INSERT INTO subscription_activations (provider, event_id, customer_id)
       VALUES ($1, $2, $3)
       ON CONFLICT (provider, event_id) DO NOTHING`,
      ['billing-provider', event.id, event.customerId],
    );

    if (activation.rowCount === 1) {
      await db.query(
        `UPDATE subscriptions
         SET status = 'active', activated_at = now()
         WHERE customer_id = $1`,
        [event.customerId],
      );
    }

    await db.query('COMMIT');
  } catch (error) {
    await db.query('ROLLBACK');
    throw error;
  }
}

That unique constraint is not glamorous. It is the line between “we think this is idempotent” and “the database enforces it while three pods race during a deploy.”

Layer 4: handle concurrent delivery across instances

The Postgres INSERT ... ON CONFLICT DO NOTHING claim is already atomic, so for many systems you can stop there. One instance wins, the rest see duplicates.

A Redis lock is useful when you want to reject concurrent duplicates before they reach heavier processing, or when the event ID claim lives in a slower system. Keep it short-lived and treat it as an optimization, not the source of truth.

import { createClient } from 'redis';

const redis = createClient({ url: process.env.REDIS_URL });
await redis.connect();

async function withEventLock<T>(eventId: string, fn: () => Promise<T>): Promise<T | 'locked'> {
  const key = `webhook-lock:billing-provider:${eventId}`;
  const token = crypto.randomUUID();

  const acquired = await redis.set(key, token, {
    NX: true,
    PX: 2 * 60 * 1000,
  });

  if (acquired !== 'OK') return 'locked';

  try {
    return await fn();
  } finally {
    // Delete only our own lock. Never blindly DEL a lock another worker may own.
    await redis.eval(
      `if redis.call("get", KEYS[1]) == ARGV[1]
       then return redis.call("del", KEYS[1])
       else return 0 end`,
      { keys: [key], arguments: [token] },
    );
  }
}

Do not build correctness on a Redis lock alone. Redis can restart. TTLs can expire while work is still running. Network partitions are real. The final idempotency boundary belongs in the database that records the business outcome.

The complete Express route

Here is the endpoint with all the pieces wired together:

app.post(
  '/webhooks/billing',
  express.raw({ type: 'application/json', limit: '1mb' }),
  async (req, res) => {
    const rawBody = req.body as Buffer;
    const signature = req.header('x-provider-signature');

    if (!signature || !verifySignedRequest(rawBody, signature)) {
      return res.status(401).send('bad signature');
    }

    let event: BillingEvent;
    try {
      event = JSON.parse(rawBody.toString('utf8'));
    } catch {
      return res.status(400).send('invalid json');
    }

    if (!event.id || !event.type) {
      return res.status(400).send('missing event metadata');
    }

    const result = await withEventLock(event.id, async () => {
      const claim = await claimWebhookEvent(event);
      if (claim === 'duplicate') return 'duplicate';

      await applyBillingEvent(event);
      await markWebhookProcessed(event);
      return 'processed';
    });

    // A concurrent copy is already processing this event. Return 204 so the
    // provider does not turn a harmless duplicate into a retry storm.
    if (result === 'locked' || result === 'duplicate') {
      return res.sendStatus(204);
    }

    return res.sendStatus(204);
  },
);

One uncomfortable recommendation: return 204 for duplicates. Teams often want to return 409 Conflict because it feels semantically pure. Providers usually treat non-2xx responses as retryable failures. A duplicate that keeps getting retried is not more correct; it is just noisy. Once you know the event is authentic and already handled or currently being handled, acknowledge it.

For handler failures, return a 5xx so the provider retries. But make sure the retry path is safe because it will happen.

Test the replay path, not just the happy path

A webhook test that only sends one valid event is almost useless. Test the failure modes that caused the incident.

import request from 'supertest';

it('rejects an old but correctly signed event', async () => {
  const body = Buffer.from(JSON.stringify({ id: 'evt_old', type: 'invoice.paid' }));
  const timestamp = Math.floor(Date.now() / 1000) - 3600;
  const signature = signForTest(timestamp, body);

  await request(app)
    .post('/webhooks/billing')
    .set('content-type', 'application/json')
    .set('x-provider-signature', `t=${timestamp},v1=${signature}`)
    .send(body)
    .expect(401);
});

it('processes the same event once across concurrent deliveries', async () => {
  const body = Buffer.from(JSON.stringify({
    id: 'evt_concurrent',
    type: 'invoice.paid',
    customerId: 'cus_123',
  }));
  const timestamp = Math.floor(Date.now() / 1000);
  const signature = signForTest(timestamp, body);

  await Promise.all(
    Array.from({ length: 20 }, () =>
      request(app)
        .post('/webhooks/billing')
        .set('content-type', 'application/json')
        .set('x-provider-signature', `t=${timestamp},v1=${signature}`)
        .send(body)
        .expect(204),
    ),
  );

  const rows = await db.query(
    `SELECT count(*)::int AS count
     FROM subscription_activations
     WHERE provider = $1 AND event_id = $2`,
    ['billing-provider', 'evt_concurrent'],
  );

  expect(rows.rows[0].count).toBe(1);
});

If that second test flakes, your production handler is not idempotent. The fix is not another if statement in JavaScript. The fix is an atomic constraint where the state changes.

Operational details that save you later

Log these fields for every webhook: provider, event ID, event type, timestamp age, verification result, duplicate/processed status, and processing duration. Do not log the raw body unless you have reviewed the data classification. Webhooks often contain emails, addresses, invoice lines, and tokens you do not want in searchable logs.

Alert on these signals:

• signature failures above a low baseline
• timestamp-window rejections
• a sudden spike in duplicates
• events stuck in processing or failed
• handler duration approaching the provider timeout

Keep the raw payload somewhere only if you need audit or replay tooling, and encrypt it if it contains sensitive data. Many teams can store just the event ID, type, status, and a hash of the payload.

Finally, rotate webhook secrets like any other credential. Supporting multiple active signatures makes rotation painless: accept signatures produced by the current and previous secret, deploy the new secret, update the provider, then remove the old secret after the retry window drains.

Practical takeaway

A safe webhook endpoint has four properties:

1. It verifies the signature over the exact raw bytes received.
2. It rejects signed requests outside a short timestamp window.
3. It records provider event IDs with an atomic uniqueness constraint.
4. It makes the business side effect idempotent, not just the HTTP handler.

The signature keeps strangers out. The timestamp limits replay. The event table handles provider retries. The unique business constraint protects you when concurrency, deploys, and partial failures stop being theoretical.

That is the difference between a demo webhook and a production one.

A note from Yojji

Webhook reliability sits at the boundary between backend design, security, and operations, exactly where small shortcuts turn into expensive incidents. Yojji helps teams build and harden custom web and cloud systems, including the idempotent APIs and deployment practices needed when third-party events drive real business workflows.