Graceful Shutdown in Node.js Production Services

Every time you deploy a Node.js service without a proper shutdown handler, you're gambling with in-flight requests. The process receives SIGTERM, Node.js ignores it by default, Kubernetes eventually sends SIGKILL, and any requests mid-flight get a connection reset. This happens on every rolling deployment, every pod eviction, every scale-down event.

Most teams only discover the problem through user reports — intermittent 503s during peak deploys that are hard to reproduce locally. The fix isn't complicated, but it requires understanding exactly what happens during a Kubernetes termination sequence and wiring up a few moving parts in the right order.

What Happens Without a Shutdown Handler

When Kubernetes terminates a pod, it sends SIGTERM to PID 1. Node.js does not register a default handler for SIGTERM — the signal is silently ignored until terminationGracePeriodSeconds (default: 30 seconds) expires and SIGKILL arrives. At that point the OS forcibly kills the process. No cleanup, no drain, no negotiation.

// ❌ Common pattern — process dies abruptly on SIGKILL
const app = express();
const server = app.listen(3000);
 
// No SIGTERM handler. In-flight requests dropped.
// DB connections closed mid-transaction.
// Queue messages unacknowledged.

The broken behavior is invisible in development because you restart manually with Ctrl+C, which sends SIGINT and Node.js does handle that by default. Production is a different story.

A well-behaved service needs to do four things when it receives SIGTERM:

1. Stop accepting new connections
2. Wait for in-flight requests to finish
3. Close downstream connections (database, cache, queues)
4. Exit with code 0

Registering Signal Handlers

Signal handlers must be registered early — before the server starts, before any connections are established. If the process crashes during startup before handlers are registered, that's acceptable. Once the server is live, it needs a clean exit path.

// ✅ Register before server.listen() — handles both Kubernetes and local dev
function registerShutdownHandlers(shutdown: () => Promise<void>): void {
  let isShuttingDown = false;
 
  const handler = async (signal: string) => {
    // Guard against duplicate signals — Kubernetes can send SIGTERM more than once
    if (isShuttingDown) return;
    isShuttingDown = true;
 
    console.log(`[shutdown] Received ${signal}. Starting graceful shutdown...`);
 
    try {
      await shutdown();
      console.log("[shutdown] Complete. Exiting.");
      process.exit(0);
    } catch (err) {
      console.error("[shutdown] Error during shutdown:", err);
      process.exit(1);
    }
  };
 
  process.on("SIGTERM", () => handler("SIGTERM")); // Kubernetes, docker stop
  process.on("SIGINT", () => handler("SIGINT"));   // Ctrl+C in development
}

The isShuttingDown guard matters. Kubernetes occasionally sends SIGTERM more than once before SIGKILL, and developers running locally sometimes mash Ctrl+C. Running two parallel shutdown sequences is worse than running one.

Draining In-Flight HTTP Requests

server.close() stops the server from accepting new TCP connections, but it has a well-known limitation: it does not close existing keep-alive connections. A client with an open keep-alive connection holds a socket open indefinitely — meaning server.close() can stall for the full grace period before Kubernetes sends SIGKILL.

The fix is to track active connections and force-close idle ones after a timeout:

import { Server, IncomingMessage, ServerResponse } from "http";
 
function createDrainableServer(server: Server): {
  closeWithDrain: (timeoutMs?: number) => Promise<void>;
} {
  const connections = new Set<import("net").Socket>();
 
  server.on("connection", (socket) => {
    connections.add(socket);
    socket.on("close", () => connections.delete(socket));
  });
 
  const closeWithDrain = (timeoutMs = 10_000): Promise<void> => {
    return new Promise((resolve, reject) => {
      server.close((err) => {
        if (err) reject(err);
        else resolve();
      });
 
      // Destroy idle keep-alive connections immediately
      for (const socket of connections) {
        socket.destroy();
      }
 
      setTimeout(() => {
        reject(new Error(`Server drain timed out after ${timeoutMs}ms`));
      }, timeoutMs).unref();
    });
  };
 
  return { closeWithDrain };
}

For services with high keep-alive utilization this matters a lot. Force-destroying idle sockets immediately lets the drain window focus on actually in-flight requests rather than waiting for clients to notice the connection is dead.

Closing Downstream Dependencies in the Right Order

Database pools, queue consumers, and cache clients all need explicit teardown. The order is not arbitrary — getting it wrong causes data corruption or duplicate message processing.

interface ServiceDependencies {
  db: import("pg").Pool;
  cache: import("redis").RedisClientType;
  consumer: import("kafkajs").Consumer;
}
 
async function closeDependencies(deps: ServiceDependencies): Promise<void> {
  // 1. Stop consuming new messages before anything else.
  //    An in-flight message may need the DB — don't close it first.
  await deps.consumer.stop();
  await deps.consumer.disconnect();
 
  // 2. Flush and close cache client
  await deps.cache.quit();
 
  // 3. Close database pool last — lets any final writes from the consumer land
  await deps.db.end();
}

The common mistake is closing the database first. If a queue consumer is mid-processing a message that requires a database write, closing the pool out from under it leaves the message unacknowledged. The message broker will redeliver it, and you've introduced a duplicate processing scenario that your idempotency logic may or may not handle.

The Kubernetes Endpoint Propagation Race

There's a subtle race condition in Kubernetes that even teams with proper shutdown handlers miss. When a pod is terminated, two things happen in parallel:

1. SIGTERM is sent to the pod
2. The pod IP is removed from the Endpoints object

The problem: endpoint propagation is eventually consistent. kube-proxy and your ingress controller update their routing tables asynchronously. For 200–500 milliseconds after SIGTERM is sent, the load balancer may still route new requests to the terminating pod. If you stop accepting connections immediately on SIGTERM, those requests fail with a connection refused.

The fix is cheap: sleep briefly before initiating the actual drain.

async function gracefulShutdown(
  server: ReturnType<typeof createDrainableServer>,
  deps: ServiceDependencies,
): Promise<void> {
  // Allow time for load balancer to drain traffic from this pod.
  // Endpoint propagation typically completes within 2 seconds.
  const PROPAGATION_BUFFER_MS = 2_000;
  await new Promise((resolve) => setTimeout(resolve, PROPAGATION_BUFFER_MS));
 
  // Stop accepting new connections and drain existing ones
  await server.closeWithDrain(20_000);
 
  // Close downstream clients after HTTP is drained
  await closeDependencies(deps);
}

This pattern pairs well with a Kubernetes preStop hook for services that need more precision:

lifecycle:
  preStop:
    exec:
      command: ["/bin/sh", "-c", "sleep 5"]
terminationGracePeriodSeconds: 30

The preStop hook fires before SIGTERM is sent, giving endpoint propagation a head start. Your SIGTERM handler then starts draining an already-quiet server rather than racing against stale routing tables.

Verifying It Works

Graceful shutdown is one of those things that's easy to implement incorrectly and hard to notice until it breaks in production. A minimal smoke test during integration testing:

# Start service, send some long-running requests, then SIGTERM
curl -X POST http://localhost:3000/slow-endpoint &
CURL_PID=$!
 
sleep 0.5
kill -TERM $(lsof -ti:3000)  # Send SIGTERM to the server process
 
wait $CURL_PID
echo "Exit: $?"  # Should be 0 — request completed, not dropped

If the request returns 0, the server drained properly. If it returns 52 (connection reset) or 7 (connection refused), your shutdown handler has a gap.

For more thorough coverage, run shutdown scenarios in your integration test suite against a real server process using child_process.spawn and process.kill(pid, "SIGTERM"). Unit tests won't catch the timing issues that actually cause dropped requests.

Key Takeaways

1. Node.js does not handle SIGTERM by default — register explicit handlers before server.listen() or your process ignores the signal until SIGKILL.
2. server.close() alone isn't enough — track active sockets and destroy idle keep-alive connections to prevent the drain from stalling.
3. Close queue consumers before closing the database — a consumer mid-processing a message needs downstream storage to still be available.
4. Sleep 2–5 seconds before initiating drain — Kubernetes endpoint propagation is asynchronous, and requests still route to your pod immediately after SIGTERM.
5. Set your internal timeout 5 seconds below terminationGracePeriodSeconds — leave a buffer so you exit cleanly before SIGKILL arrives.
6. Write an integration test that sends SIGTERM mid-request — it's the only reliable way to verify the shutdown sequence actually works end to end.