Zero-Downtime Deployments: Blue-Green and Canary
A practical guide to zero-downtime deployment strategies: blue-green deployments, canary releases and rolling updates with automated rollback.
The Cost of Deployment Downtime
Every second of downtime during deployment costs revenue, user trust, and team confidence. Developers who fear deployments deploy less frequently. Less frequent deployments mean larger changesets. Larger changesets mean higher risk per deployment. This cycle turns deployment into a stressful, infrequent ceremony instead of a routine, boring operation.
Zero-downtime deployment breaks this cycle. When deployments are safe and invisible to users, teams deploy more often, with smaller changes, and with higher confidence. The strategies covered here—blue-green, canary, and rolling updates—are the building blocks of that capability.
Blue-Green Deployment Architecture
Blue-green deployment maintains two identical production environments. At any time, one (blue) serves live traffic while the other (green) idles or receives the new version. Deployment means switching the router from blue to green.
interface Environment {
name: "blue" | "green";
version: string;
status: "active" | "idle" | "deploying" | "draining";
healthCheckUrl: string;
instances: number;
}
interface BlueGreenState {
activeEnvironment: "blue" | "green";
blue: Environment;
green: Environment;
lastSwitch: Date;
}
class BlueGreenDeployer {
constructor(
private state: BlueGreenState,
private readonly loadBalancer: LoadBalancer,
private readonly healthChecker: HealthChecker
) {}
async deploy(newVersion: string): Promise<void> {
const idle =
this.state.activeEnvironment === "blue"
? this.state.green
: this.state.blue;
console.log(`Deploying ${newVersion} to ${idle.name} environment`);
// Step 1: Deploy to idle environment
idle.status = "deploying";
await this.deployToEnvironment(idle, newVersion);
// Step 2: Health check the new deployment
idle.status = "idle";
const healthy = await this.healthChecker.verify(
idle.healthCheckUrl,
{ retries: 5, intervalMs: 3000 }
);
if (!healthy) {
throw new Error(`Health check failed for ${idle.name} with ${newVersion}`);
}
// Step 3: Switch traffic
console.log(`Switching traffic from ${this.state.activeEnvironment} to ${idle.name}`);
await this.loadBalancer.route(idle.name);
// Step 4: Update state
const previous = this.state.activeEnvironment;
this.state.activeEnvironment = idle.name;
idle.status = "active";
this.state[previous].status = "draining";
this.state.lastSwitch = new Date();
// Step 5: Drain old environment
await this.waitForConnectionDrain(this.state[previous]);
this.state[previous].status = "idle";
console.log(`Deployment complete. ${idle.name} is now active.`);
}
async rollback(): Promise<void> {
const previous =
this.state.activeEnvironment === "blue" ? "green" : "blue";
console.log(`Rolling back to ${previous} environment`);
await this.loadBalancer.route(previous);
this.state.activeEnvironment = previous;
}
private async deployToEnvironment(
env: Environment,
version: string
): Promise<void> {
// Actual deployment logic (container pull, app restart, etc.)
env.version = version;
}
private async waitForConnectionDrain(env: Environment): Promise<void> {
// Wait for in-flight requests to complete
await new Promise((resolve) => setTimeout(resolve, 30000));
}
}The key advantage of blue-green is instant rollback. If the new version has problems, switching back to the previous environment takes seconds because it is still running with the last known good version.
Canary Release Implementation
Canary releases send a small percentage of traffic to the new version while the majority continues on the current version. If metrics stay healthy, traffic gradually shifts to the new version.
interface CanaryConfig {
initialPercentage: number;
steps: number[];
stepDurationMs: number;
rollbackThresholds: {
errorRate: number;
p99Latency: number;
successRate: number;
};
}
// ❌ All-or-nothing deployment — no safety net
async function riskyDeploy(version: string): Promise<void> {
await deployToAllInstances(version);
// If it's broken, ALL users are affected immediately
}
// ✅ Gradual canary rollout with automatic rollback
class CanaryDeployer {
constructor(
private readonly config: CanaryConfig,
private readonly router: TrafficRouter,
private readonly metrics: MetricsCollector
) {}
async deploy(newVersion: string): Promise<boolean> {
// Deploy canary instances with new version
await this.deployCanaryInstances(newVersion);
// Route initial traffic percentage
await this.router.setCanaryWeight(this.config.initialPercentage);
console.log(`Canary started at ${this.config.initialPercentage}% traffic`);
// Gradually increase traffic
for (const percentage of this.config.steps) {
await this.wait(this.config.stepDurationMs);
const healthy = await this.checkCanaryHealth();
if (!healthy) {
console.error(`Canary unhealthy at ${percentage}% — rolling back`);
await this.rollback();
return false;
}
await this.router.setCanaryWeight(percentage);
console.log(`Canary promoted to ${percentage}% traffic`);
}
// Full promotion
await this.promoteCanary(newVersion);
console.log("Canary fully promoted");
return true;
}
private async checkCanaryHealth(): Promise<boolean> {
const canaryMetrics = await this.metrics.getCanaryMetrics();
const baselineMetrics = await this.metrics.getBaselineMetrics();
const thresholds = this.config.rollbackThresholds;
if (canaryMetrics.errorRate > thresholds.errorRate) {
console.error(
`Error rate ${canaryMetrics.errorRate}% exceeds threshold ${thresholds.errorRate}%`
);
return false;
}
if (canaryMetrics.p99Latency > thresholds.p99Latency) {
console.error(
`P99 latency ${canaryMetrics.p99Latency}ms exceeds threshold ${thresholds.p99Latency}ms`
);
return false;
}
return true;
}
private async rollback(): Promise<void> {
await this.router.setCanaryWeight(0);
await this.destroyCanaryInstances();
}
private async promoteCanary(version: string): Promise<void> {
await this.deployToAllInstances(version);
await this.router.setCanaryWeight(0);
await this.destroyCanaryInstances();
}
}Canary releases limit blast radius. If the new version has a bug that affects 1% of requests, only the canary percentage of users are affected during the observation period.
Kubernetes Rolling Update Configuration
Kubernetes natively supports rolling updates. The configuration controls how aggressively new pods replace old ones and what health checks must pass before proceeding.
# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: api-server
spec:
replicas: 6
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1 # At most 1 pod down during update
maxSurge: 2 # At most 2 extra pods during update
template:
spec:
containers:
- name: api
image: registry.example.com/api:v2.3.0
ports:
- containerPort: 3000
readinessProbe:
httpGet:
path: /health/ready
port: 3000
initialDelaySeconds: 5
periodSeconds: 10
failureThreshold: 3
livenessProbe:
httpGet:
path: /health/live
port: 3000
initialDelaySeconds: 15
periodSeconds: 20
failureThreshold: 3
lifecycle:
preStop:
exec:
command: ["/bin/sh", "-c", "sleep 15"]// Health check endpoint implementation
import express from "express";
const app = express();
let isReady = false;
let isShuttingDown = false;
// Readiness probe — am I ready to receive traffic?
app.get("/health/ready", (req, res) => {
if (isReady && !isShuttingDown) {
res.status(200).json({ status: "ready" });
} else {
res.status(503).json({ status: "not-ready" });
}
});
// Liveness probe — am I still alive?
app.get("/health/live", (req, res) => {
res.status(200).json({ status: "alive" });
});
// Graceful shutdown
process.on("SIGTERM", () => {
console.log("SIGTERM received — starting graceful shutdown");
isShuttingDown = true;
// Stop accepting new requests
isReady = false;
// Allow in-flight requests to complete
setTimeout(() => {
process.exit(0);
}, 30000);
});
// Startup initialization
async function initialize(): Promise<void> {
await connectToDatabase();
await warmCaches();
isReady = true;
console.log("Application ready to serve traffic");
}The preStop hook introduces a delay before the container receives SIGTERM, giving the load balancer time to stop routing traffic to the pod. Without this, requests can arrive at a pod that is already shutting down.
Database Migration Strategy for Zero-Downtime
Code deployments are straightforward. Database migrations are where zero-downtime deployments get complicated. The rule is: every migration must be backward-compatible with the previous code version.
// Migration strategy: expand-then-contract
// Step 1: EXPAND — Add new column (backward compatible)
// Deploy migration while old code is still running
const expandMigration = `
ALTER TABLE users ADD COLUMN display_name VARCHAR(255);
-- Old code ignores this column, new code can start using it
`;
// Step 2: MIGRATE DATA — Backfill the new column
const dataMigration = `
UPDATE users SET display_name = name WHERE display_name IS NULL;
-- Run in batches for large tables
`;
// Step 3: Deploy new code that reads from display_name
// Both old and new code work during rolling update
// Step 4: CONTRACT — Remove old column (after all instances run new code)
const contractMigration = `
ALTER TABLE users DROP COLUMN name;
-- Only safe after all old code instances are gone
`;
// Implementation pattern for dual-read during transition
interface UserRepository {
getDisplayName(userId: string): Promise<string>;
}
class TransitionalUserRepository implements UserRepository {
async getDisplayName(userId: string): Promise<string> {
const row = await db.query(
"SELECT display_name, name FROM users WHERE id = $1",
[userId]
);
// Read new column first, fall back to old
return row.display_name || row.name;
}
}Automated Rollback Triggers
Manual rollback is slow. Automated rollback based on metrics catches problems faster than human observation.
interface RollbackPolicy {
metric: string;
threshold: number;
comparison: "gt" | "lt";
windowSeconds: number;
consecutive: number;
}
const rollbackPolicies: RollbackPolicy[] = [
{
metric: "http_error_rate_5xx",
threshold: 5,
comparison: "gt",
windowSeconds: 60,
consecutive: 3,
},
{
metric: "http_latency_p99_ms",
threshold: 2000,
comparison: "gt",
windowSeconds: 120,
consecutive: 2,
},
{
metric: "health_check_success_rate",
threshold: 95,
comparison: "lt",
windowSeconds: 30,
consecutive: 3,
},
];
class AutomatedRollbackMonitor {
private violations = new Map<string, number>();
async evaluate(policies: RollbackPolicy[]): Promise<boolean> {
for (const policy of policies) {
const value = await this.getMetricValue(
policy.metric,
policy.windowSeconds
);
const violated =
policy.comparison === "gt"
? value > policy.threshold
: value < policy.threshold;
const key = policy.metric;
const count = violated
? (this.violations.get(key) ?? 0) + 1
: 0;
this.violations.set(key, count);
if (count >= policy.consecutive) {
console.error(
`Rollback triggered: ${policy.metric} = ${value} (threshold: ${policy.threshold}) for ${count} consecutive checks`
);
return true;
}
}
return false;
}
private async getMetricValue(
metric: string,
windowSeconds: number
): Promise<number> {
// Query metrics backend (Prometheus, Datadog, etc.)
return 0;
}
}Key Takeaways
Zero-downtime deployment is achievable with blue-green, canary, and rolling update strategies. Blue-green provides instant rollback through environment switching. Canary releases limit blast radius by gradually shifting traffic. Rolling updates are Kubernetes-native and work well with proper health checks.
Database migrations must be backward-compatible—use the expand-then-contract pattern to decouple schema changes from code deployments. Implement graceful shutdown in your applications so in-flight requests complete before the process exits. Automate rollback triggers based on error rates and latency thresholds so problems are caught in seconds, not minutes.
The goal is making deployment boring. When deploying is safe, fast, and reversible, teams deploy more often with smaller changes—and smaller changes are inherently less risky.
