Idempotency Lessons From an Email Agent

A customer emails your support agent at 9:14 a.m. At 9:15 they get a helpful reply. At 9:16 they get the same reply again, word for word. Nothing crashed. No exception was thrown. Your agent just did exactly what it was told — twice.

I think email agents are the best teacher of idempotency I've seen in years, because the failure mode is so visceral. A duplicate database row is invisible. A duplicate email lands in a human's inbox and makes your product look broken. Building a reply loop on Nylas Agent Accounts (currently in beta) forced me to internalize lessons that apply to any event-driven system, not just email.

Lesson 1: at-least-once is the honest contract

The instinct is to blame the platform: "why did I get the same webhook twice?" But at-least-once delivery is the only honest guarantee a webhook system can make. Per the duplicate-reply docs, if your endpoint doesn't return 200 fast enough, or a transient network blip eats the response, the message.created notification gets delivered again. The alternative — exactly-once — would mean the platform silently drops events whenever it's unsure, and a dropped event is worse than a repeated one.

So duplicates aren't a bug to report. They're a contract to design for. The fix is an atomic check-and-set keyed on the message ID:

const alreadyProcessed = await db.processedMessages.setIfAbsent(messageId, {
  receivedAt: Date.now(),
});
if (alreadyProcessed) return;
await handleMessage(event.data.object);

The atomicity matters more than the storage. In Postgres that's INSERT ... ON CONFLICT DO NOTHING; in Redis it's SET messageId 1 NX EX 86400. A read-then-write sequence reintroduces the race you're trying to close. And give the records a TTL — 24 to 48 hours covers redeliveries without growing the table forever. After that window, a webhook for the same message ID is almost certainly a bug in your own system, not a redelivery, and you want it to surface.

There's a quieter corollary: acknowledge before you act. The docs' example handler calls res.status(200).end() as its first line and only then starts processing. Every second your endpoint spends on an LLM call before responding is a second in which the platform may decide the delivery failed and queue a retry. You can't eliminate redeliveries, but you can stop manufacturing them.

Lesson 2: dedup and locking solve different problems

Here's the part most people miss. Deduplication catches the same event delivered twice. It does nothing about the same event processed twice concurrently. If your handler runs on Lambda or multiple worker processes, two instances can blow past the check-and-set within the same millisecond window.

The docs recommend a per-thread lock with a 30-second TTL, so a crashed worker releases automatically. And inside the lock, a double-check against ground truth: fetch the thread, look at latestDraftOrMessage, and bail if the from address is the agent's own. Between the webhook arriving and your lock being acquired, another worker may have finished the whole job — the thread itself is the only record that can't lie about it.

That layered structure — dedup, then lock, then verify state — generalizes. Idempotency isn't one mechanism. It's a stack of cheap checks, each catching what the previous one can't.

Lesson 3: the best coordination is no coordination

The thorniest duplicates don't come from infrastructure at all. They come from two actors watching the same inbox — two agents, or an agent and a human, both deciding the same message needs a reply. You can't dedup your way out of that; it's not a duplicate event, it's a coordination problem.

The cleanest fix is architectural: one agent, one inbox. Agent Accounts make that nearly free, since each agent gets its own address and its own webhook stream — sales-agent@, support-agent@, scheduling@, each filtering on its own grant_id. No overlap means no conflict to resolve. When humans need visibility, they get read-only IMAP access instead of becoming a second writer.

This is the distributed-systems lesson in miniature: partitioning beats locking whenever you can afford it.

Lesson 4: assume your logic is the next bug

Even with all three layers, you can still build a reply storm. Outbound sends fire message.created too. If your handler forgets to skip the agent's own messages, the agent replies to itself, which triggers another webhook, forever. The first guard is two lines at the top of every handler:

// First check in every handler — skip messages from the agent itself.
const sender = msg.from?.[0]?.email;
if (sender === AGENT_EMAIL) return;

The second guard is a per-thread send budget: more than 3 sends within 5 minutes means something's wrong, so stop and escalate to a human instead of sending.

That's idempotency's underrated cousin — a circuit breaker for when your correct code does something incorrect at volume. Dedup protects you from the platform. The rate limit protects you from yourself.

Lesson 5: the cheapest event to handle is the one that never fires

One more layer sits below all of this. Agent Accounts support server-side rules that sort inbound mail before your webhook handler ever sees it — route automated notifications to a folder the agent doesn't reply in, block spam at the SMTP layer, archive what needs no response.

curl --request POST \
  --url "https://api.us.nylas.com/v3/rules" \
  --header "Authorization: Bearer <NYLAS_API_KEY>" \
  --header "Content-Type: application/json" \
  --data '{
    "match": [{ "field": "from.domain", "operator": "equals", "value": "noreply.example.com" }],
    "actions": [{ "action": "assign_to_folder", "value": "notifications" }],
    "description": "Route automated notifications to a separate folder"
  }'

Your handler then checks which folder a message landed in and skips folders the agent shouldn't touch. Every message you filter out declaratively is a message your idempotency stack never has to be correct about. Shrinking the input space is the idempotency strategy nobody writes blog posts about, because it looks like configuration instead of engineering.

The counterargument worth taking seriously

"This is a lot of machinery for sending email." Fair. If your agent handles ten messages a day from a single-threaded process, a dedup table alone will carry you a long way, and the lock may be premature. The docs themselves note that synthetic concurrent load testing is the only way to surface the race — which implies a single-threaded deployment won't hit it.

But the cost asymmetry should drive the decision. The whole stack is maybe forty lines of code. A double reply to a customer is a trust incident you can't un-send. I'd rather carry the forty lines.

One more habit worth stealing: log every skip. When a message is dropped because it's a duplicate or another worker holds the lock, write that down. Silent idempotency is correct but undebuggable.

If you're building a reply loop, read the prevention recipe end to end, then write a load test that fires the same webhook payload at your handler from five concurrent connections. If exactly one reply goes out, you've earned the right to ship. What's the worst duplicate-action bug you've shipped — and which layer would have caught it?

推荐订阅源

DEV Community