Resilient Distributed Systems: Timeouts and Retries
How to implement timeouts, retries, and backoff strategies that prevent cascading failures in distributed architectures.
In a distributed system, every network call can fail, hang, or take orders of magnitude longer than expected. A downstream service that normally responds in 50ms might take 30 seconds during a deployment, or it might never respond at all. Without timeouts and retries, a single slow service can cascade into a full system outage as threads, connections, and memory fill up waiting for responses that never come.
The challenge is not adding timeouts and retries — it is configuring them so they help during real incidents instead of making things worse.
Timeouts: The Non-Negotiable Default
Every outbound network call needs a timeout. No exceptions. A missing timeout turns a temporary downstream issue into a permanent resource leak in your service.
// ❌ No timeout — hangs indefinitely if downstream is slow
const response = await fetch('https://api.payment-provider.com/charge', {
method: 'POST',
body: JSON.stringify(payload),
});
// ✅ Explicit timeout — fails fast when downstream is unresponsive
const controller = new AbortController();
const timeout = setTimeout(() => controller.abort(), 5000);
try {
const response = await fetch('https://api.payment-provider.com/charge', {
method: 'POST',
body: JSON.stringify(payload),
signal: controller.signal,
});
return await response.json();
} catch (error) {
if (error instanceof DOMException && error.name === 'AbortError') {
throw new TimeoutError('Payment provider did not respond within 5s');
}
throw error;
} finally {
clearTimeout(timeout);
}Choosing Timeout Values
interface TimeoutConfig {
// Connect timeout: how long to wait for TCP connection
connectTimeout: number;
// Read timeout: how long to wait for response after connected
readTimeout: number;
// Total timeout: maximum wall-clock time for the entire operation
totalTimeout: number;
}
// ❌ Timeout too high — defeats the purpose
const bad: TimeoutConfig = {
connectTimeout: 60_000,
readTimeout: 120_000,
totalTimeout: 180_000,
};
// ✅ Based on p99 latency + margin
const good: TimeoutConfig = {
connectTimeout: 1_000, // Most connects finish in <100ms
readTimeout: 3_000, // p99 is 800ms, 3x margin
totalTimeout: 5_000, // Hard ceiling for the operation
};Set timeouts based on the observed p99 latency of the downstream service, not the average. A timeout at 3x the p99 catches genuine failures while allowing normal latency variation.
Retry Strategies
Retries handle transient failures — network blips, brief service restarts, temporary overload. But naive retries amplify problems instead of solving them.
// ❌ Immediate retry — hammers a struggling service
async function naiveRetry<T>(fn: () => Promise<T>, maxRetries: number): Promise<T> {
for (let attempt = 0; attempt <= maxRetries; attempt++) {
try {
return await fn();
} catch (error) {
if (attempt === maxRetries) throw error;
// Retries instantly — adds load to an already struggling service
}
}
throw new Error('Unreachable');
}
// ✅ Exponential backoff with jitter — backs off and spreads retry load
async function retryWithBackoff<T>(
fn: () => Promise<T>,
options: {
maxRetries: number;
baseDelayMs: number;
maxDelayMs: number;
}
): Promise<T> {
const { maxRetries, baseDelayMs, maxDelayMs } = options;
for (let attempt = 0; attempt <= maxRetries; attempt++) {
try {
return await fn();
} catch (error) {
if (attempt === maxRetries) throw error;
if (!isRetryable(error)) throw error;
const exponentialDelay = baseDelayMs * Math.pow(2, attempt);
const jitter = Math.random() * exponentialDelay;
const delay = Math.min(exponentialDelay + jitter, maxDelayMs);
await sleep(delay);
}
}
throw new Error('Unreachable');
}Jitter Prevents Thundering Herds
Without jitter, all clients retry at the same time — creating a synchronized burst that overwhelms the recovering service.
// Without jitter: 1000 clients all retry at exactly t+1s, t+2s, t+4s
// With jitter: 1000 clients retry spread over [0-1s], [0-2s], [0-4s]
function calculateBackoff(attempt: number, baseMs: number, maxMs: number): number {
// Full jitter — recommended by AWS
const ceiling = Math.min(maxMs, baseMs * Math.pow(2, attempt));
return Math.random() * ceiling;
}
// Decorrelated jitter — even better spread
function decorrelatedJitter(
previousDelay: number,
baseMs: number,
maxMs: number
): number {
return Math.min(maxMs, baseMs + Math.random() * (previousDelay * 3 - baseMs));
}Retry Budgets
In a microservice chain, retries multiply. If A calls B with 3 retries and B calls C with 3 retries, a failure at C generates up to 9 requests from B and 27 from A. This amplification can turn a small failure into a system-wide overload.
class RetryBudget {
private attempts = 0;
private successes = 0;
// Only retry if retry rate is below threshold
canRetry(): boolean {
if (this.attempts === 0) return true;
const retryRate = 1 - (this.successes / this.attempts);
return retryRate < 0.2; // Max 20% retry rate
}
recordAttempt(): void {
this.attempts++;
}
recordSuccess(): void {
this.successes++;
}
// Reset counters periodically
reset(): void {
this.attempts = 0;
this.successes = 0;
}
}A retry budget caps the total retry rate across all requests, not per-request. When the system is healthy, the budget is generous. When failures spike, the budget tightens — preventing retry storms.
Idempotency for Safe Retries
Retries are only safe if the operation is idempotent — executing it twice produces the same result as executing it once. Without idempotency, retries can duplicate charges, create double entries, or send multiple emails.
// ❌ Not idempotent — retry creates duplicate payment
app.post('/charge', async (req, res) => {
const result = await paymentProvider.charge(req.body.amount);
await db.payments.create({ amount: req.body.amount, txId: result.id });
return res.json({ success: true });
});
// ✅ Idempotent — uses idempotency key to prevent duplicates
app.post('/charge', async (req, res) => {
const idempotencyKey = req.headers['idempotency-key'] as string;
if (!idempotencyKey) {
return res.status(400).json({ error: 'Idempotency-Key header required' });
}
// Check if this request was already processed
const existing = await db.idempotencyKeys.findUnique({
where: { key: idempotencyKey },
});
if (existing) {
return res.json(existing.response);
}
const result = await paymentProvider.charge(req.body.amount);
const response = { success: true, txId: result.id };
await db.idempotencyKeys.create({
data: { key: idempotencyKey, response },
});
return res.json(response);
});Timeout Hierarchy in Service Chains
In a call chain (API Gateway → Service A → Service B → Database), timeouts must decrease at each hop. If Service A has a 5s timeout and Service B has a 10s timeout, B might still be working when A gives up — wasting resources.
// Service chain timeout configuration
const timeouts = {
apiGateway: 10_000, // 10s — highest timeout, user-facing
serviceA: 7_000, // 7s — must finish before gateway timeout
serviceB: 4_000, // 4s — must finish before A's timeout
database: 2_000, // 2s — must finish before B's timeout
};
// Each level leaves headroom for retries and processing
// Gateway (10s) > A (7s) > B (4s) > DB (2s)Deadline Propagation
Instead of each service setting independent timeouts, propagate a deadline from the originating request.
interface RequestContext {
deadline: number; // Unix timestamp when the entire chain must complete
correlationId: string;
}
async function callDownstream(ctx: RequestContext, url: string): Promise<Response> {
const remainingMs = ctx.deadline - Date.now();
if (remainingMs <= 0) {
throw new DeadlineExceeded('No time remaining for downstream call');
}
// Use remaining time as timeout, with safety margin
const timeout = Math.max(remainingMs - 100, 0);
const controller = new AbortController();
const timer = setTimeout(() => controller.abort(), timeout);
try {
return await fetch(url, {
headers: {
'X-Request-Deadline': String(ctx.deadline),
'X-Correlation-Id': ctx.correlationId,
},
signal: controller.signal,
});
} finally {
clearTimeout(timer);
}
}Deadline propagation ensures that downstream services do not start work they cannot finish before the caller gives up.
Key Takeaways
- Every network call needs a timeout — a missing timeout turns transient failure into permanent resource exhaustion
- Base timeouts on p99, not average — set to 2-3x the observed p99 latency
- Always use exponential backoff with jitter — prevents thundering herd on the recovering service
- Retry budgets prevent amplification — cap total retry rate, not just per-request retries
- Retries require idempotency — without it, retries duplicate side effects
- Timeouts decrease down the call chain — and deadline propagation enforces this automatically
