Designing Idempotent Background Jobs: Retries, Locks, and the Exactly-Once Illusion

Every queueing system you'll use — SQS, RabbitMQ, BullMQ, Sidekiq, Pub/Sub — promises "at-least-once" delivery. None of them promise exactly-once, because exactly-once delivery in a distributed system is provably impossible without cooperation from the consumer. The network can drop an ack. A worker can crash after doing the work but before confirming it. A retry can fire while the original attempt is still running.

The uncomfortable truth: your queue's delivery guarantee is not your problem. Your job handler's behavior under duplicate execution is. If you design for "this will run more than once, sometimes concurrently," exactly-once behavior becomes achievable even though exactly-once delivery never will be.