Request Coalescing: Stopping the Thundering Herd

When a cached value expires under heavy traffic, every request that arrives in the next few milliseconds misses the cache simultaneously. Each one fires the same database query, the same external API call, the same expensive computation — all producing the same result. This is the thundering herd problem, and it's deceptively easy to miss until traffic is high enough that it matters.

The fix isn't a distributed lock or a sleep-and-retry loop. It's inflight request coalescing: deduplicate concurrent in-flight requests for the same resource so only one upstream call ever runs, and every waiting caller shares the resolved value.

Why the Obvious Fixes Don't Work

The most common first attempt is a mutex or Redis lock. Acquire the lock, populate the cache, release. Every other request either waits or returns stale data. This works, but it introduces distributed state, lock timeouts, and a separate failure mode: what happens when the lock holder crashes mid-population?

The next attempt is usually "just extend the TTL." But TTL tuning is a guess, and it trades freshness for stampede avoidance rather than solving the root cause. You're still going to hit this when you deploy a cold instance or explicitly flush the cache.

The actual problem is simpler: multiple callers are doing identical work concurrently when one would suffice. Fix that directly.

Inflight Request Coalescing with a Promise Map

Node.js's single-threaded event loop makes this pattern unusually clean. A Map from resource key to the in-flight Promise is all you need. Any caller that arrives while a fetch is already running gets the same promise. They all resolve together when the upstream call completes.

// ❌ Every concurrent caller fires its own query
async function getUser(id: string): Promise<User> {
  const cached = await cache.get(`user:${id}`);
  if (cached) return cached;
 
  const user = await db.query("SELECT * FROM users WHERE id = $1", [id]);
  await cache.set(`user:${id}`, user, 60);
  return user;
}
 
// ✅ Concurrent callers share one in-flight request
const inflight = new Map<string, Promise<User>>();
 
async function getUser(id: string): Promise<User> {
  const cached = await cache.get(`user:${id}`);
  if (cached) return cached;
 
  const key = `user:${id}`;
  if (inflight.has(key)) return inflight.get(key)!;
 
  const promise = db
    .query("SELECT * FROM users WHERE id = $1", [id])
    .then(async (user) => {
      await cache.set(key, user, 60);
      return user;
    })
    .finally(() => inflight.delete(key));
 
  inflight.set(key, promise);
  return promise;
}

The finally block is critical. It removes the in-flight entry whether the upstream call succeeds or fails. If you only delete on success, a transient error permanently blocks all future callers for that key.

Building a Reusable Coalescer

Repeating the Map + finally pattern inline across every data-fetching function is noisy. Extract it into a utility that wraps any async function:

type Fetcher<T> = () => Promise<T>;
 
class RequestCoalescer {
  private readonly inflight = new Map<string, Promise<unknown>>();
 
  async dedupe<T>(key: string, fetcher: Fetcher<T>): Promise<T> {
    if (this.inflight.has(key)) {
      return this.inflight.get(key) as Promise<T>;
    }
 
    const promise = fetcher().finally(() => this.inflight.delete(key));
    this.inflight.set(key, promise);
    return promise;
  }
 
  get size(): number {
    return this.inflight.size;
  }
}
 
// Usage
const coalescer = new RequestCoalescer();
 
async function getProduct(id: string): Promise<Product> {
  const cached = await cache.get(`product:${id}`);
  if (cached) return cached;
 
  return coalescer.dedupe(`product:${id}`, async () => {
    const product = await db.query(
      "SELECT * FROM products WHERE id = $1",
      [id],
    );
    await cache.set(`product:${id}`, product, 120);
    return product;
  });
}

The coalescer is stateful, so inject it rather than instantiating it inside each function. A single instance per service is the typical approach.

Handling Errors Without Poisoning Callers

There's a subtle bug in naive implementations: if the upstream call throws, every coalesced caller receives the same rejection. That's usually fine — they all get an accurate error. But if you cache the rejected promise itself, future callers who arrive after the failure will still get a rejected promise even though no upstream call is in flight.

The finally block in the example above handles this correctly: the in-flight entry is always deleted on completion, regardless of outcome. Future callers start fresh.

What you might want to cache deliberately is a short-lived negative result — to avoid hammering a resource that's consistently failing. That's a separate concern (circuit breaking), and mixing it with coalescing muddies the implementation.

Bounding the Inflight Map

Under normal operation, the map size stays tiny — one entry per distinct resource key currently being fetched. But if a pathological key space is possible (attacker-controlled IDs, unbounded pagination cursors), it's worth adding a size guard:

class RequestCoalescer {
  private readonly inflight = new Map<string, Promise<unknown>>();
  private readonly maxSize: number;
 
  constructor(maxSize = 1000) {
    this.maxSize = maxSize;
  }
 
  async dedupe<T>(key: string, fetcher: Fetcher<T>): Promise<T> {
    if (this.inflight.has(key)) {
      return this.inflight.get(key) as Promise<T>;
    }
 
    if (this.inflight.size >= this.maxSize) {
      // Fall through: let this caller fetch independently
      return fetcher();
    }
 
    const promise = fetcher().finally(() => this.inflight.delete(key));
    this.inflight.set(key, promise);
    return promise;
  }
}

Exceeding the limit causes a fallback to uncoalesced behavior rather than refusing the request. Thundering herd protection degrades gracefully instead of hard-failing.

Where This Pattern Applies

Request coalescing is useful anywhere work is expensive and idempotent. The canonical use case is cache population, but the same pattern applies to:

The constraint is that the operation must be safe to share: same inputs, same output, no side effects that should run per-caller.

Key Takeaways

1. Thundering herd is a concurrency bug, not a caching bug. Tuning TTL delays the problem; coalescing eliminates it.
2. A Map<string, Promise<T>> is the entire implementation. No external dependencies, no distributed state, no lock timeouts.
3. Always clean up with finally. Deleting the in-flight entry on success only leaves a permanent poison pill for future callers after any failure.
4. Keep the coalescer as an injectable singleton. Per-function instantiation means no deduplication across concurrent callers at different call sites.
5. Fall through gracefully when the map is full. Degrade to uncoalesced behavior rather than rejecting requests outright.
6. Instrument the inflight map size. A persistently large map signals either a slow upstream dependency or a key space problem worth investigating.