Background Job Processing Patterns
Not everything belongs in the request-response cycle — queues, workers, retries, and dead letter patterns for reliable background processing.
Sending confirmation emails, generating PDF reports, processing image uploads, syncing data with third-party APIs — none of these belong in the request-response cycle. They're slow, unreliable, and their failure shouldn't block the user's action. Background job processing moves this work to a separate pipeline where it can be retried, throttled, and monitored independently.
Why Not Just Use setTimeout?
// ❌ "Background" processing that isn't actually reliable
app.post("/api/orders", async (req, res) => {
const order = await createOrder(req.body);
// This runs in the same process — if the server crashes, the email is lost
setTimeout(async () => {
await sendConfirmationEmail(order);
await updateInventory(order);
await notifyWarehouse(order);
}, 0);
res.json(order);
});If the server restarts, those callbacks are gone. No retry, no visibility, no guarantee the work completes. A proper job queue persists the work and survives crashes.
The Job Queue Pattern
A job queue has three components: producers (enqueue work), a persistent store (hold jobs), and consumers/workers (process jobs).
// Producer: enqueue a job after the main operation
import { Queue } from "bullmq";
import { Redis } from "ioredis";
const connection = new Redis(process.env.REDIS_URL);
const emailQueue = new Queue("email", { connection });
app.post("/api/orders", async (req, res) => {
const order = await createOrder(req.body);
// Job is persisted in Redis — survives server restarts
await emailQueue.add("order-confirmation", {
orderId: order.id,
customerEmail: order.customerEmail,
orderTotal: order.total,
});
res.json(order);
});// Consumer: process jobs from the queue
import { Worker } from "bullmq";
const emailWorker = new Worker(
"email",
async (job) => {
switch (job.name) {
case "order-confirmation":
await sendConfirmationEmail(
job.data.customerEmail,
job.data.orderId,
job.data.orderTotal,
);
break;
case "shipping-notification":
await sendShippingEmail(job.data);
break;
}
},
{
connection,
concurrency: 5, // Process 5 jobs simultaneously
},
);
emailWorker.on("completed", (job) => {
console.log(`Job ${job.id} completed`);
});
emailWorker.on("failed", (job, error) => {
console.error(`Job ${job?.id} failed:`, error.message);
});Retry Strategies
Jobs fail. APIs timeout, services go down, rate limits get hit. A good retry strategy handles transient failures without making permanent failures worse.
// Exponential backoff with jitter
await emailQueue.add(
"order-confirmation",
{ orderId: order.id },
{
attempts: 5,
backoff: {
type: "exponential",
delay: 2000, // 2s, 4s, 8s, 16s, 32s
},
},
);// ❌ Retrying non-retryable errors wastes resources
async function processJob(job: Job) {
try {
await callExternalAPI(job.data);
} catch (error) {
// Retrying a 400 Bad Request will never succeed
throw error; // All errors get retried
}
}
// ✅ Distinguish retryable from permanent failures
async function processJob(job: Job) {
try {
await callExternalAPI(job.data);
} catch (error) {
if (error.status === 429 || error.status >= 500) {
throw error; // Retryable — network/server issue
}
// Permanent failure — don't retry, move to dead letter
throw new UnrecoverableError(error.message);
}
}| Error type | Example | Action |
|---|---|---|
| Transient | Timeout, 503, connection refused | Retry with backoff |
| Rate limit | 429 Too Many Requests | Retry with longer delay |
| Permanent | 400 Bad Request, invalid data | Dead letter queue |
| Bug | Uncaught TypeError | Dead letter queue + alert |
Dead Letter Queues
When a job exhausts all retries, it moves to a dead letter queue for investigation. Never silently discard failed jobs.
// Configure dead letter queue behavior
const orderQueue = new Queue("orders", {
connection,
defaultJobOptions: {
attempts: 5,
backoff: { type: "exponential", delay: 2000 },
removeOnComplete: { age: 86400 }, // Keep completed for 24h
removeOnFail: false, // Never auto-delete failed jobs
},
});
// Monitor dead letter jobs
async function getDeadLetterJobs() {
const failed = await orderQueue.getFailed(0, 100);
return failed.map((job) => ({
id: job.id,
name: job.name,
data: job.data,
failedReason: job.failedReason,
attemptsMade: job.attemptsMade,
timestamp: job.timestamp,
}));
}Job Scheduling
Some jobs need to run on a schedule — daily reports, hourly syncs, periodic cleanup.
// Repeatable jobs with cron expressions
await reportQueue.add(
"daily-revenue-report",
{}, // Data can be empty for scheduled jobs
{
repeat: {
pattern: "0 8 * * *", // Every day at 8 AM
tz: "America/New_York",
},
},
);
await cleanupQueue.add(
"expire-old-sessions",
{},
{
repeat: {
pattern: "*/15 * * * *", // Every 15 minutes
},
},
);Idempotency
Jobs may be delivered more than once (at-least-once delivery). Every job handler must produce the same result whether it runs once or ten times.
// ❌ Not idempotent — charging the customer multiple times
async function processPayment(job: Job) {
await stripe.charges.create({
amount: job.data.amount,
customer: job.data.customerId,
});
}
// ✅ Idempotent — uses an idempotency key to prevent duplicates
async function processPayment(job: Job) {
await stripe.charges.create(
{
amount: job.data.amount,
customer: job.data.customerId,
},
{
idempotencyKey: `order-${job.data.orderId}`,
},
);
}Use unique identifiers (order ID, job ID) as idempotency keys. Check if the work was already done before doing it again.
Key Takeaways
- Move slow, unreliable work out of the request cycle into background jobs
- Persist jobs in a durable store (Redis, PostgreSQL) — in-memory processing is unreliable
- Retry with exponential backoff for transient failures, dead-letter for permanent ones
- Every job handler must be idempotent — jobs can be delivered more than once
- Never silently discard failed jobs — dead letter queues preserve them for investigation
- Monitor queue depth and processing latency — growing queues signal capacity problems
