Distributed Locks: Coordinating Work Across Multiple Nodes

Horizontal scaling solves throughput problems but introduces a new class of bugs: two instances of the same service doing the same work simultaneously. Sending a duplicate email, double-charging a customer, or corrupting shared state — these are race conditions at the infrastructure level, and a single database SELECT ... FOR UPDATE won't save you when the competing process lives on a different machine.

Distributed locks are the tool for this. They're also one of the most misused primitives in backend engineering. The failure modes are subtle, the literature is full of controversy (see the Redlock debate), and most tutorials stop before covering the parts that actually matter in production — lock expiration, renewal, and what happens when the lock holder dies.

The Core Problem

Imagine a cron job that runs every minute, processes overdue invoices, and sends reminder emails. You've scaled to three instances. Without coordination, all three instances wake up at the same time, query the same rows, and send three emails per customer.

The fix isn't "just use a queue" — sometimes the work genuinely has to be a singleton. Scheduled jobs, leader election, and one-shot migrations all share this shape. You need exactly-one execution semantics.

Redis: The Common Choice

Redis's SET key value NX PX ttl command is atomic: it sets a key only if it does not exist, with an expiry. That's the primitive a distributed lock is built on.

import { createClient } from "redis";
import { randomUUID } from "crypto";
 
interface Lock {
  key: string;
  token: string;
  release: () => Promise<void>;
}
 
const RELEASE_SCRIPT = `
  if redis.call("get", KEYS[1]) == ARGV[1] then
    return redis.call("del", KEYS[1])
  else
    return 0
  end
`;
 
async function acquireLock(
  redis: ReturnType<typeof createClient>,
  key: string,
  ttlMs: number,
): Promise<Lock | null> {
  const token = randomUUID();
  const acquired = await redis.set(key, token, {
    NX: true,
    PX: ttlMs,
  });
 
  if (!acquired) return null;
 
  return {
    key,
    token,
    release: async () => {
      // ✅ Lua script ensures we only delete the key we own
      await redis.eval(RELEASE_SCRIPT, { keys: [key], arguments: [token] });
    },
  };
}

Two things deserve attention here. First, the token (a UUID) is stored as the lock value, not just a flag. When releasing, the Lua script checks that the value matches before deleting — this prevents a slow process from releasing a lock that expired and was re-acquired by someone else. Second, the release is atomic: the check-and-delete happens in a single round trip.

The Expiry Trap

Setting a TTL is mandatory. Without it, a crashed process holds the lock forever. But TTLs introduce their own hazard: what if the work takes longer than expected?

// ❌ Fixed TTL — expires while work is still in progress
const lock = await acquireLock(redis, "invoice:process", 5_000);
await processAllInvoices(); // takes 12 seconds on a bad day
 
// ✅ Watchdog extends the lock while work is ongoing
const lock = await acquireLock(redis, "invoice:process", 10_000);
if (!lock) return; // another instance is working
 
const watchdog = setInterval(async () => {
  await redis.pExpire(lock.key, 10_000);
}, 4_000); // renew every 4s, TTL is 10s
 
try {
  await processAllInvoices();
} finally {
  clearInterval(watchdog);
  await lock.release();
}

The watchdog pattern keeps a lock alive as long as the process is healthy. If the process crashes, the interval stops firing, the TTL expires naturally, and another instance can acquire the lock. This is how Redlock and most serious distributed lock libraries handle the expiry problem.

PostgreSQL Advisory Locks

Redis isn't always in the stack. PostgreSQL has a native distributed lock mechanism: advisory locks. They're session-scoped or transaction-scoped, held in shared memory, and visible across all connections to the same database.

-- Session-scoped: held until explicitly released or session ends
SELECT pg_try_advisory_lock(hashtext('invoice:process'));
 
-- Transaction-scoped: released automatically at commit/rollback
SELECT pg_try_advisory_xact_lock(hashtext('invoice:process'));

From TypeScript with pg or a query builder:

import { Pool } from "pg";
 
async function withAdvisoryLock<T>(
  pool: Pool,
  lockKey: string,
  fn: () => Promise<T>,
): Promise<T | null> {
  const client = await pool.connect();
  try {
    // pg_try_advisory_xact_lock is transaction-scoped: auto-released on commit/rollback
    await client.query("BEGIN");
    const { rows } = await client.query<{ acquired: boolean }>(
      "SELECT pg_try_advisory_xact_lock(hashtext($1)) AS acquired",
      [lockKey],
    );
 
    if (!rows[0].acquired) {
      await client.query("ROLLBACK");
      return null; // someone else holds it
    }
 
    const result = await fn();
    await client.query("COMMIT");
    return result;
  } catch (err) {
    await client.query("ROLLBACK");
    throw err;
  } finally {
    client.release();
  }
}

The transaction-scoped variant is safer: if the process crashes mid-work, PostgreSQL rolls back the transaction and releases the lock automatically. No TTL to tune, no watchdog needed. The trade-off is that the lock is only held for the duration of the database transaction — if your work involves non-database side effects (HTTP calls, file writes), the transaction-scoped lock offers no protection once it commits.

Choosing the Right Approach

Handling Acquisition Failure

Deciding what to do when acquireLock returns null is as important as the lock itself. Most callers should skip, not retry blindly.

async function runInvoiceJob(redis: ReturnType<typeof createClient>) {
  const lock = await acquireLock(redis, "jobs:invoice-reminders", 30_000);
 
  if (!lock) {
    // ✅ Another instance is handling it — this is normal, not an error
    console.info({ msg: "Lock unavailable, skipping run" });
    return;
  }
 
  const watchdog = setInterval(
    () => redis.pExpire(lock.key, 30_000),
    10_000,
  );
 
  try {
    await sendInvoiceReminders();
  } catch (err) {
    // Log but don't swallow — let the caller decide on retry policy
    console.error({ msg: "Invoice job failed", err });
    throw err;
  } finally {
    clearInterval(watchdog);
    await lock.release();
  }
}

Returning early on null is the right default for cron jobs and background workers. If you need the work to eventually complete, use a queue instead of a lock — queues give you durable retry semantics, locks don't.

What Distributed Locks Cannot Guarantee

Distributed locks provide mutual exclusion under normal conditions. They don't provide it unconditionally. Clock skew, network partitions, and GC pauses can all cause a process to believe it holds a valid lock when it doesn't. Martin Kleppmann's critique of Redlock is worth reading for the full picture.

The practical implication: for operations where double-execution has severe consequences (financial transactions, irreversible deletes), pair distributed locks with an idempotency key or a database-level constraint. The lock reduces the probability of concurrent execution to near zero; the constraint makes double-execution impossible even if the lock fails.

Key Takeaways

1. Always store a unique token as the lock value — it prevents a slow process from releasing a lock it no longer owns after expiry.
2. Use a watchdog for long-running work — fixed TTLs expire while work is in progress; renew proactively while the process is alive.
3. PostgreSQL advisory locks are underused — if your critical section is already inside a database transaction, xact-scoped locks give you automatic cleanup for free.
4. Returning early on failure is correct behavior — a cron job that skips a run because another instance is working is working as designed.
5. Locks reduce probability, not impossibility — for truly critical operations, add an idempotency constraint at the storage layer as a second line of defense.