Debugging von Produktionsvorfällen: Ein systematischer Ansatz

# What is actually broken?
# Check error rates across services
curl "https://metrics.internal/api/query?q=rate(http_errors_total[5m])"
 
# Which users are affected?
# Check if it's a subset (region, plan, feature flag cohort)
SELECT
  COUNT(*) as affected_users,
  COUNT(CASE WHEN region = 'us-east' THEN 1 END) as us_east,
  COUNT(CASE WHEN plan = 'free' THEN 1 END) as free_tier
FROM error_events
WHERE created_at > NOW() - INTERVAL '10 minutes';
 
# When did it start?
# Compare current error rate to baseline
SELECT
  date_trunc('minute', created_at) as minute,
  COUNT(*) as errors
FROM error_events
WHERE created_at > NOW() - INTERVAL '30 minutes'
GROUP BY 1
ORDER BY 1;

# Check recent deployments
git log --oneline --since="2 hours ago" --all
# → b43224f feat: add fetchDataVersion function  [30 min ago]
 
# Check infrastructure changes
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=UpdateFunctionCode \
  --start-time "2 hours ago"
 
# Check database query performance
SELECT
  query,
  mean_exec_time,
  calls,
  total_exec_time
FROM pg_stat_statements
WHERE mean_exec_time > 1000  -- Queries slower than 1 second
ORDER BY mean_exec_time DESC
LIMIT 10;

# Rollback a Kubernetes deployment
kubectl rollout undo deployment/api-server
 
# Disable a feature flag (example with Vercel Edge Config)
curl -X PATCH https://api.vercel.com/v1/edge-config/{id}/items \
  -H "Authorization: Bearer $TOKEN" \
  -d '{"items": [{"operation": "update", "key": "features.new-checkout", "value": {"enabled": false}}]}'
 
# Scale up if the issue is capacity
kubectl scale deployment/api-server --replicas=10

// Monitor key metrics for 10-15 minutes after mitigation
const metricsToWatch = [
  "http.error_rate", // Should return to baseline
  "db.query_duration_p95", // Should drop if DB was the cause
  "api.response_time_p99", // Should recover
  "active_user_sessions", // Should stop declining
];
 
// Set up a dashboard view filtered to last 30 minutes
// Watch for the inflection point where the incident started to resolve

## Incident: API 500 Errors — 2026-03-15 14:30 UTC
 
### Summary
 
For 23 minutes, 8% of API requests returned 500 errors, affecting ~1,200 users.
The cause was a missing database index added in deployment v2.4.1, causing
query timeouts under normal load.
 
### Timeline
 
- 14:30 — Deployment v2.4.1 rolled out to production
- 14:38 — Alert: error rate exceeded 5% threshold
- 14:41 — On-call acknowledged, began investigation
- 14:47 — Identified slow query in pg_stat_statements
- 14:52 — Added missing index with CREATE INDEX CONCURRENTLY
- 14:53 — Error rate returned to baseline
 
### Root Cause
 
The migration added a `user_id` filter to a query that previously had no
user scope. Without an index on `user_id`, this query did a full table scan
on 8M rows under load.
 
### Why It Reached Production
 
- The migration was tested on a dev database with ~500 rows
- Query performance wasn't tested under load
- No slow query monitoring was in place for new migrations
 
### Action Items
 
- [ ] Add EXPLAIN ANALYZE to migration PR checklist
- [ ] Set up pg_stat_statements alerts for new slow queries
- [ ] Create load test suite that runs against staging with production-scale data

// Structured logging — make logs queryable
import pino from "pino";
 
const logger = pino({
  level: process.env.LOG_LEVEL ?? "info",
  formatters: {
    level: (label) => ({ level: label }),
  },
});
 
// Every important operation logs with context
async function processOrder(orderId: string, userId: string) {
  const start = Date.now();
 
  logger.info({ orderId, userId, event: "order.process.start" });
 
  try {
    const result = await fulfillOrder(orderId);
    logger.info({
      orderId,
      userId,
      event: "order.process.success",
      durationMs: Date.now() - start,
    });
    return result;
  } catch (error) {
    logger.error({
      orderId,
      userId,
      event: "order.process.error",
      error: error.message,
      durationMs: Date.now() - start,
    });
    throw error;
  }
}