Idempotency Keys in Practice: Making Retries Safe

Networks fail in the middle of requests all the time. The client sends a payment request, the server processes it, charges the card, and then the response never makes it back — timeout, dropped connection, whatever. The client, seeing no response, retries. Now you've charged the customer twice.

This isn't a hypothetical edge case. It's the default behavior of every retry mechanism you've ever shipped unless you've explicitly designed against it. Idempotency keys are the standard fix, but most implementations I've seen get the details wrong — they store the key without storing the response, they don't handle concurrent retries, or they let the key collide across different request bodies. Let's fix that.

Why "It's a POST, So It's Not Idempotent" Is the Wrong Framing

REST semantics say GET, PUT, and DELETE are idempotent by convention, POST is not. That's a statement about HTTP method semantics, not about what your business actually needs. Creating an order is a POST, but if the client retries with the same intent, you want the same order, not a second one.

The fix is to decouple idempotency from HTTP method and put it under the caller's control: the client generates a unique key per logical operation and sends it along with the request. The server uses that key to detect duplicates, regardless of verb.

POST /api/payments HTTP/1.1
Idempotency-Key: 7c3a9e02-4b1d-4f8e-9c21-6a1e8f0d5b3a
Content-Type: application/json
 
{"amount": 4999, "currency": "usd", "customerId": "cus_123"}

Storing the Key Isn't Enough — Store the Response

A common mistake: check if the key exists, and if it does, just return a generic "already processed" message. That's useless to the client — they need the actual result of the original request, especially if it included a generated resource ID.

// ❌ Tells the client nothing useful on retry
async function createPayment(req: PaymentRequest, key: string) {
  const existing = await db.idempotencyKeys.findOne({ key });
  if (existing) {
    return { status: 409, body: { error: "duplicate request" } };
  }
  const payment = await chargeCard(req);
  await db.idempotencyKeys.insertOne({ key, createdAt: new Date() });
  return { status: 201, body: payment };
}
 
// ✅ Returns the original response, verbatim, on retry
async function createPayment(req: PaymentRequest, key: string) {
  const existing = await db.idempotencyKeys.findOne({ key });
  if (existing) {
    return { status: existing.statusCode, body: existing.responseBody };
  }
 
  const payment = await chargeCard(req);
 
  await db.idempotencyKeys.insertOne({
    key,
    statusCode: 201,
    responseBody: payment,
    requestHash: hashRequest(req),
    createdAt: new Date(),
  });
 
  return { status: 201, body: payment };
}

The client should be able to retry blindly and get back exactly what it would have gotten the first time — same order ID, same amount, same status.

Guard Against Key Reuse With Different Payloads

If a client sends the same Idempotency-Key with a different request body, that's a bug on their end (or a client-generated collision), and you should reject it loudly rather than silently returning the first response for a different request.

function hashRequest(req: PaymentRequest): string {
  return crypto
    .createHash("sha256")
    .update(JSON.stringify(sortKeys(req)))
    .digest("hex");
}
 
async function createPayment(req: PaymentRequest, key: string) {
  const requestHash = hashRequest(req);
  const existing = await db.idempotencyKeys.findOne({ key });
 
  if (existing) {
    if (existing.requestHash !== requestHash) {
      return {
        status: 422,
        body: { error: "idempotency key reused with a different payload" },
      };
    }
    return { status: existing.statusCode, body: existing.responseBody };
  }
 
  // proceed with the actual operation...
}

The Race Condition Everyone Forgets

Two identical requests can arrive nearly simultaneously — a client retrying aggressively, or a proxy duplicating a request. If your "check, then insert" logic isn't atomic, both requests can pass the existence check before either has written the key, and you end up processing the operation twice anyway.

The fix is to make key insertion atomic and use it as a lock, not just a lookup table.

// ❌ Race condition: two concurrent requests both pass the check
const existing = await db.idempotencyKeys.findOne({ key });
if (!existing) {
  const payment = await chargeCard(req); // both requests get here
  await db.idempotencyKeys.insertOne({ key, ... });
}
 
// ✅ Atomic insert acts as a distributed lock
async function createPayment(req: PaymentRequest, key: string) {
  const requestHash = hashRequest(req);
 
  try {
    await db.idempotencyKeys.insertOne({
      key,
      requestHash,
      status: "processing",
      createdAt: new Date(),
    });
  } catch (err) {
    if (isDuplicateKeyError(err)) {
      return waitForResultOrReturnConflict(key, requestHash);
    }
    throw err;
  }
 
  const payment = await chargeCard(req);
 
  await db.idempotencyKeys.updateOne(
    { key },
    { $set: { status: "completed", statusCode: 201, responseBody: payment } },
  );
 
  return { status: 201, body: payment };
}

A unique index on key in your database makes the insert fail fast for the losing request, which then either polls for the winner's result or returns a 409 telling the client to retry after a short delay.

Handling the "Still Processing" Window

Between the atomic insert and the operation completing, a retry might land in a state where the key exists but there's no response yet. Don't leave the second caller hanging — decide explicitly what happens.

That "stale" branch matters more than people expect. If the process handling the original request crashes after acquiring the lock but before completing the operation, every retry will get stuck in 409 forever unless you have a timeout that reclaims the key.

const STALE_THRESHOLD_MS = 30_000;
 
function isStale(record: IdempotencyRecord): boolean {
  return (
    record.status === "processing" &&
    Date.now() - record.createdAt.getTime() > STALE_THRESHOLD_MS
  );
}

Key Expiration and Storage Costs

Idempotency keys can't live forever — you'll accumulate rows indefinitely otherwise. Most systems expire keys after 24 hours, which comfortably covers realistic retry windows (client timeouts, exponential backoff, manual retries by support staff) without leaving stale entries around.

-- Postgres: TTL via a scheduled job, or a partial index + cron cleanup
CREATE TABLE idempotency_keys (
  key TEXT PRIMARY KEY,
  request_hash TEXT NOT NULL,
  status TEXT NOT NULL,
  status_code INT,
  response_body JSONB,
  created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
 
CREATE INDEX idx_idempotency_created_at ON idempotency_keys (created_at);
 
-- Cleanup job, run hourly
DELETE FROM idempotency_keys WHERE created_at < now() - INTERVAL '24 hours';

If you're on Redis instead of a relational store, this is even simpler — set a TTL on the key at write time and skip the cleanup job entirely.

Where Idempotency Keys Should Live

Not every endpoint needs this. Reserve idempotency keys for operations that are non-idempotent by nature and have real consequences on duplication: payments, order creation, sending notifications, provisioning infrastructure. Don't bother wrapping a PUT /users/:id/profile update — it's already naturally idempotent.

// ✅ Middleware applied selectively to mutation-heavy, consequential routes
router.post("/payments", requireIdempotencyKey, createPayment);
router.post("/orders", requireIdempotencyKey, createOrder);
router.post("/notifications/send", requireIdempotencyKey, sendNotification);
 
// Not needed — PUT is naturally idempotent
router.put("/users/:id/profile", updateProfile);

Key Takeaways

1. Store the response, not just the key — retries should return the exact original result, not a generic duplicate error.
2. Hash the request body and compare it on retry — reused keys with different payloads are a bug that needs a loud error, not silent success.
3. Make key insertion atomic and use it as a lock — a "check then insert" pattern has a race condition under concurrent retries.
4. Handle the in-progress window explicitly — decide what happens when a retry lands while the original request is still processing, and reclaim stale locks.
5. Expire keys — 24 hours is a reasonable default that covers realistic retry windows without unbounded storage growth.
6. Apply idempotency selectively — reserve it for consequential, non-idempotent operations like payments and resource creation, not routes that are already safe to retry.