Designing Idempotent Background Jobs: Retries, Locks, and the Exactly-Once Illusion

Every queueing system you'll use — SQS, RabbitMQ, BullMQ, Sidekiq, Pub/Sub — promises "at-least-once" delivery. None of them promise exactly-once, because exactly-once delivery in a distributed system is provably impossible without cooperation from the consumer. The network can drop an ack. A worker can crash after doing the work but before confirming it. A retry can fire while the original attempt is still running.

The uncomfortable truth: your queue's delivery guarantee is not your problem. Your job handler's behavior under duplicate execution is. If you design for "this will run more than once, sometimes concurrently," exactly-once behavior becomes achievable even though exactly-once delivery never will be.

Why retries always create duplicates

A typical job lifecycle looks like this: pull message → process → acknowledge. The failure mode that breaks naive handlers is the gap between "process" and "acknowledge."

// ❌ Bad: work and acknowledgment aren't atomic
async function processPayment(job: Job<PaymentPayload>) {
  await chargeCard(job.data.customerId, job.data.amountCents);
  await sendReceiptEmail(job.data.customerId);
  // If the process crashes here, or the ack never reaches the broker,
  // the queue redelivers this job — and the customer gets charged twice.
}

If the worker dies right after chargeCard succeeds but before the message is acknowledged, most brokers will redeliver the message to another worker. That worker has no idea the charge already went through. This isn't a hypothetical edge case — it's the default behavior of every at-least-once queue under load, deploys, and network partitions.

Idempotency keys: the foundation

The fix is to give every job a stable identity and record that identity before doing anything irreversible. This is the same pattern used for idempotent API requests, applied to background processing.

-- ✅ Good: a dedupe table backed by a unique constraint
CREATE TABLE job_executions (
  idempotency_key TEXT PRIMARY KEY,
  job_type TEXT NOT NULL,
  status TEXT NOT NULL DEFAULT 'processing',
  result JSONB,
  created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
  completed_at TIMESTAMPTZ
);

// ✅ Good: claim the key atomically before doing side effects
async function processPayment(job: Job<PaymentPayload>) {
  const key = job.data.idempotencyKey; // generated once, at job creation time
 
  const claimed = await db.query(
    `INSERT INTO job_executions (idempotency_key, job_type)
     VALUES ($1, 'process_payment')
     ON CONFLICT (idempotency_key) DO NOTHING
     RETURNING idempotency_key`,
    [key],
  );
 
  if (claimed.rowCount === 0) {
    // Already claimed by this run or a previous attempt.
    const existing = await db.query(
      `SELECT status, result FROM job_executions WHERE idempotency_key = $1`,
      [key],
    );
    if (existing.rows[0].status === "completed") {
      return existing.rows[0].result; // safe to no-op, work already done
    }
    throw new RetryableError("Job already in progress, back off and retry");
  }
 
  const result = await chargeCard(job.data.customerId, job.data.amountCents);
  await db.query(
    `UPDATE job_executions SET status = 'completed', result = $2, completed_at = now()
     WHERE idempotency_key = $1`,
    [key, result],
  );
  await sendReceiptEmail(job.data.customerId, result);
  return result;
}

The INSERT ... ON CONFLICT DO NOTHING is the load-bearing line. It converts "did this job already run" from a race condition into a single atomic database operation. The unique constraint does the work that application-level checks can't do safely under concurrency.

Locks solve a different problem than idempotency

It's tempting to reach for a distributed lock (Redis SETNX, Postgres advisory locks, ZooKeeper) and assume that's enough. It isn't. A lock prevents concurrent execution; it does nothing about sequential re-execution after a crash.

// ❌ Bad: a lock only stops concurrency, not retries
async function runNightlyReport(jobId: string) {
  const lockKey = `lock:report:${jobId}`;
  const acquired = await redis.set(lockKey, "1", "NX", "EX", 300);
  if (!acquired) return; // another worker is already on it
 
  await generateAndSendReport(jobId);
  await redis.del(lockKey);
  // If the worker crashes after generateAndSendReport but before del,
  // the lock expires after 300s and the NEXT retry runs the whole thing again.
}

Locks and idempotency keys solve complementary problems:

Use locks to reduce wasted work and contention (why let two workers race to do the same job when one can back off immediately?). Use idempotency keys to guarantee correctness even when the lock fails, expires, or is bypassed. Never treat a lock as a correctness guarantee on its own — TTL expiry alone makes that unsafe.

Making side effects themselves idempotent

Recording that a job ran isn't enough if the side effect — an email, a webhook, a third-party API call — isn't idempotent on its own. Wrapping a non-idempotent action in an idempotency check only protects you if the check happens before the action, atomically.

// ✅ Good: push idempotency down to the external call when the API supports it
async function chargeCard(customerId: string, amountCents: number) {
  return stripe.charges.create(
    { customer: customerId, amount: amountCents, currency: "usd" },
    { idempotencyKey: `charge:${customerId}:${amountCents}:${dayBucket()}` },
  );
}

Most payment processors, email providers, and modern webhook receivers support their own idempotency keys — use them. This gives you two independent layers of protection: your job dedupe table, and the downstream system's own deduplication. When you don't control the downstream system (say, a legacy SOAP endpoint with no idempotency support), the transactional outbox pattern becomes necessary: write the intent to a database table in the same transaction as your business logic, then have a separate relay process deliver it exactly once by tracking delivery state.

Handling the "processing" limbo state

A job that crashed mid-execution leaves your dedupe row stuck at status = 'processing' forever unless you account for it. Don't let a stale claim block retries indefinitely.

// ✅ Good: expire stale claims so crashed jobs can be retried safely
async function reclaimStaleJobs() {
  await db.query(`
    UPDATE job_executions
    SET status = 'failed'
    WHERE status = 'processing'
      AND created_at < now() - interval '10 minutes'
  `);
}

Pick the staleness window based on your job's realistic worst-case runtime, not a guess. Too short, and you'll reclaim jobs that are still legitimately running, causing real duplicate side effects. Too long, and a crashed job blocks progress for longer than necessary.

Testing for idempotency, not just correctness

Standard unit tests verify a job produces the right output once. They don't verify what happens when you run it twice, or twice concurrently. Add that explicitly:

test("processing the same job twice produces one charge", async () => {
  const payload = { idempotencyKey: "test-key-1", customerId: "cus_123", amountCents: 500 };
 
  await Promise.all([
    processPayment({ data: payload } as Job<PaymentPayload>),
    processPayment({ data: payload } as Job<PaymentPayload>),
  ]);
 
  const charges = await getChargesForCustomer("cus_123");
  expect(charges).toHaveLength(1);
});

If this test isn't in your suite for every job that touches money, inventory, or irreversible external calls, it should be. It catches exactly the class of bug that only shows up in production, under load, months after the code was written.

Key Takeaways

1. Exactly-once delivery doesn't exist — design job handlers that behave correctly under at-least-once delivery instead.
2. Use a unique constraint on an idempotency key as the atomic gate before any irreversible side effect runs.
3. Locks reduce contention and wasted work; they do not guarantee correctness after a crash. Don't confuse the two.
4. Push idempotency down to external systems (payment processors, webhook receivers) when they support their own dedupe keys.
5. Reclaim stale "processing" claims on a timeout tuned to your job's real worst-case duration, or crashed jobs will never retry.
6. Write explicit tests that run a job twice — once sequentially, once concurrently — and assert the side effect happened exactly once.