Debugging Production Incidents: A Systematic Approach
A battle-tested framework for diagnosing and resolving production incidents quickly — from first alert to root cause analysis and prevention.
When Production Breaks
Production incidents are high-stakes, time-pressured situations that expose how well-prepared your team and systems are. The difference between a 5-minute recovery and a 3-hour outage is usually preparation and process, not individual brilliance.
This is the framework I use.
The OODA Loop for Incidents
Military strategist John Boyd's OODA loop maps well to incident response:
Observe → Orient → Decide → Act → repeat
Never skip steps. The most common mistake in incidents is jumping to Act without Observing — making changes based on assumptions rather than evidence, which can make things worse.
Phase 1: Observe — Gather Signal
First 5 minutes: understand the scope and impact.
# What is actually broken?
# Check error rates across services
curl "https://metrics.internal/api/query?q=rate(http_errors_total[5m])"
# Which users are affected?
# Check if it's a subset (region, plan, feature flag cohort)
SELECT
COUNT(*) as affected_users,
COUNT(CASE WHEN region = 'us-east' THEN 1 END) as us_east,
COUNT(CASE WHEN plan = 'free' THEN 1 END) as free_tier
FROM error_events
WHERE created_at > NOW() - INTERVAL '10 minutes';
# When did it start?
# Compare current error rate to baseline
SELECT
date_trunc('minute', created_at) as minute,
COUNT(*) as errors
FROM error_events
WHERE created_at > NOW() - INTERVAL '30 minutes'
GROUP BY 1
ORDER BY 1;Key questions:
- What is the error rate? (3% vs 100% changes the response)
- Which service/endpoint is affected?
- When did it start?
- Did anything deploy around that time?
- Is it getting better or worse?
Phase 2: Orient — Form Hypotheses
Correlate your observations with recent changes. The most likely cause of a production incident is a recent change.
# Check recent deployments
git log --oneline --since="2 hours ago" --all
# → b43224f feat: add fetchDataVersion function [30 min ago]
# Check infrastructure changes
aws cloudtrail lookup-events \
--lookup-attributes AttributeKey=EventName,AttributeValue=UpdateFunctionCode \
--start-time "2 hours ago"
# Check database query performance
SELECT
query,
mean_exec_time,
calls,
total_exec_time
FROM pg_stat_statements
WHERE mean_exec_time > 1000 -- Queries slower than 1 second
ORDER BY mean_exec_time DESC
LIMIT 10;Form a ranked list of hypotheses:
- Most likely: the deployment 30 minutes ago
- Possible: database query suddenly slow
- Less likely: third-party API degradation
Phase 3: Decide — Mitigate vs Fix
Choose the right action based on the situation:
Rollback — fastest when the cause is a recent deployment and rollback is safe. Prefer this when users are actively impacted.
Feature flag off — for incidents caused by a specific feature. Instant with no deployment needed.
Scale up — for capacity-related incidents (high load, memory pressure).
Fix forward — only when rollback isn't possible and you understand the root cause.
# Rollback a Kubernetes deployment
kubectl rollout undo deployment/api-server
# Disable a feature flag (example with Vercel Edge Config)
curl -X PATCH https://api.vercel.com/v1/edge-config/{id}/items \
-H "Authorization: Bearer $TOKEN" \
-d '{"items": [{"operation": "update", "key": "features.new-checkout", "value": {"enabled": false}}]}'
# Scale up if the issue is capacity
kubectl scale deployment/api-server --replicas=10The bias for action: during an active incident, an imperfect mitigation now beats a perfect fix in 30 minutes. Stop the bleeding first.
Phase 4: Verify and Monitor
After applying mitigation, confirm recovery before standing down.
// Monitor key metrics for 10-15 minutes after mitigation
const metricsToWatch = [
"http.error_rate", // Should return to baseline
"db.query_duration_p95", // Should drop if DB was the cause
"api.response_time_p99", // Should recover
"active_user_sessions", // Should stop declining
];
// Set up a dashboard view filtered to last 30 minutes
// Watch for the inflection point where the incident started to resolveDon't declare victory too early. Some issues take 5-10 minutes to propagate through the system after a fix.
Post-Incident: The Blameless Post-Mortem
Within 48 hours, write a post-mortem. The goal is not to assign blame — it's to prevent recurrence.
## Incident: API 500 Errors — 2026-03-15 14:30 UTC
### Summary
For 23 minutes, 8% of API requests returned 500 errors, affecting ~1,200 users.
The cause was a missing database index added in deployment v2.4.1, causing
query timeouts under normal load.
### Timeline
- 14:30 — Deployment v2.4.1 rolled out to production
- 14:38 — Alert: error rate exceeded 5% threshold
- 14:41 — On-call acknowledged, began investigation
- 14:47 — Identified slow query in pg_stat_statements
- 14:52 — Added missing index with CREATE INDEX CONCURRENTLY
- 14:53 — Error rate returned to baseline
### Root Cause
The migration added a `user_id` filter to a query that previously had no
user scope. Without an index on `user_id`, this query did a full table scan
on 8M rows under load.
### Why It Reached Production
- The migration was tested on a dev database with ~500 rows
- Query performance wasn't tested under load
- No slow query monitoring was in place for new migrations
### Action Items
- [ ] Add EXPLAIN ANALYZE to migration PR checklist
- [ ] Set up pg_stat_statements alerts for new slow queries
- [ ] Create load test suite that runs against staging with production-scale dataThe best teams treat post-mortems as learning tools, not punishment. "How did our system allow this?" rather than "who caused this?"
Observability: Invest Before You Need It
The worst time to set up logging and monitoring is during an incident.
// Structured logging — make logs queryable
import pino from "pino";
const logger = pino({
level: process.env.LOG_LEVEL ?? "info",
formatters: {
level: (label) => ({ level: label }),
},
});
// Every important operation logs with context
async function processOrder(orderId: string, userId: string) {
const start = Date.now();
logger.info({ orderId, userId, event: "order.process.start" });
try {
const result = await fulfillOrder(orderId);
logger.info({
orderId,
userId,
event: "order.process.success",
durationMs: Date.now() - start,
});
return result;
} catch (error) {
logger.error({
orderId,
userId,
event: "order.process.error",
error: error.message,
durationMs: Date.now() - start,
});
throw error;
}
}If you can't answer "what was the system doing 30 seconds before the incident?" from your logs, you need better observability.
Production will always surprise you. The question is whether you'll be able to understand what happened.
