Event-driven backends in Node: outbox, idempotency and the failures worth designing for

Adding a message broker feels like adding resilience. It adds asynchrony, and asynchrony is resilient only if you handle the four failure modes it introduces. Skip them and you have built something that loses work quietly, which is worse than something that fails loudly.

The dual-write problem, and the only clean fix

Here is the bug nearly every event-driven system starts with:

// ✗ Two systems, no shared transaction.
await db.orders.insert(order);
await broker.publish('order.created', order);   // if this throws, the event is lost forever

The database commit succeeded and the publish did not. There is no retry that fixes this — the process may be gone. And swapping the order just breaks it the other way: an event for an order that does not exist.

await db.transaction(async (tx) => {
  await tx.orders.insert(order);
  await tx.outbox.insert({
    id: crypto.randomUUID(),
    topic: 'order.created',
    payload: JSON.stringify(order),
    createdAt: new Date(),
  });
});

A relay polls the outbox — or tails the write-ahead log, if your database supports it — publishes, and marks rows sent. If it crashes after publishing and before marking, the message is delivered twice. Which is fine, because of the next section.

Every consumer must be idempotent, without exception

At-least-once delivery is what brokers actually provide. Exactly-once is marketing for a combination of at-least-once and idempotent consumers, and you have to build the second half.

async function onOrderCreated(msg: Message) {
  const key = msg.id;
 
  // Claim the message. A duplicate loses the race and returns.
  const claimed = await db.processed.insertIfAbsent({ key, at: new Date() });
  if (!claimed) return;
 
  await doTheWork(msg.payload);
}

1. Key on the message id, not the payload

   Two legitimately identical payloads are two events. Two deliveries of one id are a duplicate.

2. Claim and work in one transaction where you can

   Otherwise a crash between them leaves the message claimed and the work undone — which is silent loss, and the worst of the available failures.

3. Expire the processed table

   It grows forever otherwise. Retention slightly longer than your maximum retry window is enough.

Ordering: guarantee it only where you need it

Global ordering costs throughput, because it means one consumer. Almost no system needs it.

Per-entity ordering is what people actually mean when they say they need ordering, and it is nearly free: events for one order stay in sequence, while events for different orders process concurrently.

Retries: the defaults will hurt you

1. Exponential backoff with jitter

   Without jitter, everything that failed during an outage retries at the same instant when the dependency recovers — and knocks it over again.

2. Separate retryable from terminal, and stop retrying terminal

   A malformed message will never succeed. Retrying it forever blocks the partition behind it, which is how one bad record stops a whole entity's processing.

3. Cap attempts, then dead-letter

   With the original message, the error, the attempt count and enough context to replay it.

4. Alert on the dead letter queue

   A DLQ nobody watches is a folder where work goes to be forgotten. Alert on depth, and give it a replay path.

Event design outlives every implementation choice

Schemas are the part you cannot refactor later, because consumers you do not control depend on them.

type OrderCreated = {
  id: string;            // message id, for idempotency
  type: 'order.created';
  version: 1;            // schema version, present from day one
  occurredAt: string;    // when it happened, not when it was published
  data: { orderId: string; customerId: string; totalMinor: number; currency: string };
};

1. Name events in the past tense, after facts

   order.created, not create.order. An event is a record of something that happened; a command is a request. Conflating them turns a broker into an RPC layer with worse latency.

2. Version from the first event, not from the first breaking change

   Adding a version field later is itself a breaking change.

3. Additive changes only; new fields optional

   A breaking change means a new event type running alongside the old one until consumers migrate.

4. Decide fat or thin events deliberately

   A fat event carries the data, so consumers need no callback but the payload can be stale. A thin event carries an id, so consumers fetch fresh data but every event becomes a request. Both are defensible; drifting between them is not.

Observability, or you are debugging blind

1. Propagate a correlation id through every event

   From the originating request, into the outbox row, onto the message, into the consumer's logs. Without it a multi-hop flow cannot be reconstructed at all.

2. Watch consumer lag, not just throughput

   Lag is the signal that predicts an incident. Throughput looks healthy right up until it does not.

3. Count duplicates that idempotency rejected

   A rising rate means the relay or the broker is misbehaving, and it is invisible otherwise — because everything is working correctly.

The short version

1. Never dual-write; use a transactional outbox

   One commit, so the event and the data cannot disagree.

2. Make every consumer idempotent, keyed on message id

   At-least-once is what you get. Exactly-once is what you build.

3. Partition for per-entity ordering; do not buy global ordering

   It costs all your parallelism and you almost certainly do not need it.

4. Backoff with jitter, terminal errors dead-lettered, DLQ alerted and replayable

   A queue nobody watches is where work goes to be forgotten.

5. Version events from day one, past tense, additive changes only

   The schema is the part you cannot refactor.