Duplicate processing is not a bug.

It is the default behavior of reliable distributed systems.

Every distributed system eventually faces the same uncomfortable truth:

The moment you introduce:

• Retries
• Failovers
• Message brokers
• Network partitions
• Service crashes

duplicates become inevitable.

Yet many engineers still believe that "exactly-once processing" is something a broker can magically provide.

It isn't.

In this article we'll explore:

• Why exactly-once delivery is mostly a myth
• Why at-least-once is the real-world standard
• How idempotency actually works
• How to build Inbox and Outbox patterns in Go
• How Kafka, PostgreSQL, and Redis fit together in production

🧨 Exactly-Once Is Not a Transport Property

A message broker cannot guarantee exactly-once processing across your entire system.

Exactly-once delivery requires a Single Point of Truth (SPoT) capable of enforcing uniqueness.

Why?

Because delivery, processing, and persistence are separate failure domains.

Even if a broker behaves perfectly:

• Your service can crash after processing
• Your database can commit while ACK fails
• Your consumer can retry unknowingly

This creates unavoidable duplication scenarios.

📊 Delivery Semantics in Real Systems

👉 In practice, everything is at-least-once.

💥 Why Duplicates Happen

Consumer Crash After Processing

func handle(msg Message) error {
    if err := process(msg); err != nil {
        return err
    }

    return ack(msg)
}

`

Failure window:

text
Process Message ✅
Commit Database ✅
Crash Service ❌
ACK Never Sent ❌


Result:

text
Broker thinks processing failed
↓
Message redelivered
↓
Duplicate processing


Network Timeout After Success

`go
err := process(msg)

if err != nil {
return err
}

return broker.Ack(msg)
`

What happens if:

text
ACK sent
↓
Network timeout
↓
Broker never receives ACK


The broker retries.

The operation runs again.

Retry Storms

A temporary latency spike can trigger:

text
Client Retry
↓
Gateway Retry
↓
Service Retry
↓
Consumer Retry


Result:

text
1 failure
↓
50 duplicate executions


🧠 The Real Solution: Idempotency

Instead of preventing duplicates:

text
Wrong Question:
How do I prevent duplicates?


Ask:

text
Correct Question:
How do I make duplicates harmless?


That's where idempotency comes in.

An idempotent operation produces the same final state no matter how many times it executes.

🚨 Naive Payment Service

Let's start with a broken implementation.

`go
func Charge(
ctx context.Context,
req PaymentRequest,
) error {

_, err := db.Exec(ctx, `
    INSERT INTO payments (
        id,
        amount
    )
    VALUES ($1, $2)
`,
    req.ID,
    req.Amount,
)

return err

}
`

Now imagine:

text
Request arrives
↓
Payment inserted
↓
Response lost
↓
Client retries
↓
Payment inserted again


Customer charged twice.

✅ Production Solution #1: Database Unique Constraints

Create an Inbox table.

sql
CREATE TABLE idempotency_keys (
event_id TEXT PRIMARY KEY,
created_at TIMESTAMP DEFAULT now()
);


Attempt to claim the key first.

go
_, err := tx.Exec(ctx,
INSERT INTO idempotency_keys(event_id)
VALUES($1)
`, eventID)

if err != nil {
var pgErr *pgconn.PgError

if errors.As(err, &pgErr) &&
   pgErr.Code == "23505" {

    return nil
}

return err

}
`

The database becomes the source of truth.

⚠️ Common pgx Pitfall

This subtle bug catches many Go teams.

Wrong:

go
import "github.com/jackc/pgconn"


Correct:

go
import "github.com/jackc/pgx/v5/pgconn"


Otherwise:

go
errors.As(err, &pgErr)


silently fails.

Your duplicate protection stops working.

🧪 Testing Duplicate Safety

Let's simulate 100 concurrent requests.

`go
func TestIdempotency(
t *testing.T,
) {
var wg sync.WaitGroup

for i := 0; i < 100; i++ {
    wg.Add(1)

    go func() {
        defer wg.Done()

        processOrder(
            context.Background(),
            "same-event-id",
        )
    }()
}

wg.Wait()

}
`

Expected outcome:

text
100 requests
↓
1 insert succeeds
↓
99 rejected safely


⚡ Redis Optimization

Postgres guarantees correctness.

Redis improves performance.

Use Redis only as:

text
Fast Lock Layer


Not as:

text
Source of Truth


Atomic Lua Guard:

`lua
local key = KEYS[1]

if redis.call("GET", key) then
return 1
end

redis.call(
"SET",
key,
"PENDING",
"EX",
60
)

return 0
`

This protects the database from thundering herds.

📦 Kafka Exactly-Once: What It Actually Means

Many engineers believe:

`text

Kafka Exactly Once

Business Logic Executes Once
`

Wrong.

Kafka guarantees:

text
Producer
↓
Kafka
↓
Consumer


Kafka does NOT guarantee:

text
Producer
↓
Kafka
↓
Consumer
↓
Postgres


The moment you touch an external database:

exactly-once disappears.

⚠️ Disable Auto Commit

Never rely on Kafka auto commits.

Bad:

go
enable.auto.commit=true


Good:

go
consumer, _ := kafka.NewConsumer(
&kafka.ConfigMap{
"enable.auto.commit": false,
},
)


Process first.

Commit later.

`go
err := handleOrder(
ctx,
db,
msg,
)

if err == nil {
consumer.CommitMessage(msg)
}
`

📤 Solving the Dual-Write Problem

This architecture is broken:

text
Insert Order
↓
Publish Event


What if Kafka is down?

text
Order exists
↓
Event lost forever


That's the Dual Write Problem.

🚀 Transactional Outbox Pattern

Store event publishing intent in the same transaction.

sql
CREATE TABLE outbox (
id UUID PRIMARY KEY,
payload JSONB,
status TEXT DEFAULT 'PENDING'
);


go
_, err = tx.Exec(ctx,
INSERT INTO orders(...)
VALUES(...)
`)

_, err = tx.Exec(ctx,
INSERT INTO outbox(...)
VALUES(...)
)
`

Commit both together.

🔄 Outbox Worker

`go
type OutboxWorker struct {
db *pgxpool.Pool
broker Broker
}

func (w *OutboxWorker) Run(
ctx context.Context,
) {
ticker := time.NewTicker(
time.Second,
)

defer ticker.Stop()

for {
    select {
    case <-ctx.Done():
        return

    case <-ticker.C:
        w.processBatch(ctx)
    }
}

}
`

⚡ Scaling Workers Safely

Use PostgreSQL native locking.

sql
SELECT *
FROM outbox
WHERE status = 'PENDING'
FOR UPDATE SKIP LOCKED
LIMIT 10;


Benefits:

• No deadlocks
• No duplicate workers
• Infinite horizontal scaling

💳 The Hardest Part: External Side Effects

Database writes are easy.

External systems are not.

Examples:

• Stripe
• Twilio
• SendGrid
• Payment Gateways

Without idempotency:

text
Duplicate Message
↓
Duplicate Payment
↓
Real Money Lost


Always verify external APIs support idempotency keys.

🛡️ Graceful Shutdown

Kubernetes gives you roughly:

text
30 seconds


before SIGKILL.

Handle shutdown properly.

`go
ctx, stop := signal.NotifyContext(
context.Background(),
syscall.SIGTERM,
)

defer stop()
`

Allow in-flight transactions to finish.

📈 Structured Observability

Track duplicates explicitly.

go
slog.Info(
"duplicate detected",
"event_id",
eventID,
)


If you can't measure duplicates:

you can't prove idempotency works.

🏗️ Final Production Architecture

text
Kafka
│
▼
Inbox Pattern
(Idempotency)
│
▼
PostgreSQL
│
▼
Outbox Pattern
│
▼
Kafka / RabbitMQ


This architecture embraces reality:

`text
At-Least-Once Delivery
+
Idempotency
+

Recovery

Production Reliability
`

🚀 Key Takeaways

• Exactly-once delivery does not exist across distributed systems.
• At-least-once delivery is the industry standard.
• Idempotency transforms duplicates into harmless events.
• PostgreSQL UNIQUE constraints should be the source of truth.
• Redis is an optimization layer, not a consistency layer.
• Kafka auto commits can cause data loss.
• Inbox + Outbox patterns form the backbone of reliable event-driven systems.
• Reliability is not about preventing failures.
• Reliability is about remaining correct despite failures.