Observability Stack: Metrics, Logs and Traces in Practice
Set up a full observability stack with Prometheus metrics, structured JSON logging and OpenTelemetry tracing, and see how each signal connects to the others.
Observability is the ability to understand what's happening inside your system by examining its outputs. Three signals form the foundation: metrics tell you something is wrong, logs tell you what went wrong, and traces tell you where it went wrong across service boundaries. Each signal is useful alone, but they become powerful when connected.
Most teams start with one signal and add others over time. Starting with all three configured correctly from the beginning saves months of retrofitting.
Structured Logging: The Foundation
Logs are your first line of debugging. Unstructured text logs (console.log("something happened")) are almost useless at scale—they can't be queried, aggregated, or correlated.
// ❌ Unstructured logging — impossible to query at scale
console.log("User logged in");
console.log("Order created for user 123");
console.log("Payment failed: insufficient funds");
// grep for "payment"? Returns every line with "payment"
// Find all errors for user 123? Good luck// ✅ Structured JSON logging
import pino from "pino";
const logger = pino({
level: process.env.LOG_LEVEL ?? "info",
formatters: {
level(label: string) {
return { level: label };
},
},
timestamp: pino.stdTimeFunctions.isoTime,
// Add base context to every log line
base: {
service: "api-gateway",
version: process.env.APP_VERSION ?? "unknown",
environment: process.env.NODE_ENV ?? "development",
},
});
// Request-scoped child logger
function createRequestLogger(
req: Request
): pino.Logger {
return logger.child({
requestId: req.headers["x-request-id"] as string,
userId: req.user?.id,
method: req.method,
path: req.path,
userAgent: req.headers["user-agent"],
});
}
// Usage with context
app.post("/api/orders", async (req, res) => {
const log = createRequestLogger(req);
log.info(
{ itemCount: req.body.items.length },
"Order creation started"
);
try {
const order = await createOrder(req.body);
log.info(
{
orderId: order.id,
total: order.total,
duration: Date.now() - req.startTime,
},
"Order created successfully"
);
res.status(201).json(order);
} catch (error) {
log.error(
{
error: (error as Error).message,
stack: (error as Error).stack,
body: req.body,
},
"Order creation failed"
);
res.status(500).json({ error: "Order creation failed" });
}
});
// Output (one JSON object per line):
// {"level":"info","time":"2024-05-10T10:30:00.000Z",
// "service":"api-gateway","requestId":"abc-123",
// "userId":"user-456","method":"POST",
// "path":"/api/orders","itemCount":3,
// "msg":"Order creation started"}Prometheus Metrics: Measuring System Behavior
Metrics are numerical measurements collected over time. They answer "how many," "how fast," and "how much" questions about your system.
import {
collectDefaultMetrics,
register,
Histogram,
Counter,
Gauge,
} from "prom-client";
// Collect Node.js runtime metrics
collectDefaultMetrics();
// HTTP request metrics
const httpRequestDuration = new Histogram({
name: "http_request_duration_seconds",
help: "Duration of HTTP requests in seconds",
labelNames: ["method", "route", "status_code"],
buckets: [
0.01, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10,
],
});
const httpRequestsTotal = new Counter({
name: "http_requests_total",
help: "Total number of HTTP requests",
labelNames: ["method", "route", "status_code"],
});
// Business metrics
const ordersCreated = new Counter({
name: "orders_created_total",
help: "Total number of orders created",
labelNames: ["status"],
});
const activeConnections = new Gauge({
name: "active_connections",
help: "Number of active client connections",
});
// Metrics middleware
function metricsMiddleware(
req: Request,
res: Response,
next: NextFunction
) {
const start = process.hrtime.bigint();
res.on("finish", () => {
const durationNs =
Number(process.hrtime.bigint() - start);
const durationSec = durationNs / 1e9;
const route =
req.route?.path ?? req.path ?? "unknown";
const labels = {
method: req.method,
route,
status_code: String(res.statusCode),
};
httpRequestDuration.observe(labels, durationSec);
httpRequestsTotal.inc(labels);
});
next();
}
// Expose /metrics endpoint for Prometheus scraping
app.get("/metrics", async (req, res) => {
res.set("Content-Type", register.contentType);
res.send(await register.metrics());
});
app.use(metricsMiddleware);OpenTelemetry Distributed Tracing
Traces follow a request as it crosses service boundaries. Each span represents a unit of work within the trace.
import {
NodeTracerProvider,
} from "@opentelemetry/sdk-trace-node";
import {
SimpleSpanProcessor,
} from "@opentelemetry/sdk-trace-base";
import {
OTLPTraceExporter,
} from "@opentelemetry/exporter-trace-otlp-http";
import {
HttpInstrumentation,
} from "@opentelemetry/instrumentation-http";
import {
ExpressInstrumentation,
} from "@opentelemetry/instrumentation-express";
import {
PgInstrumentation,
} from "@opentelemetry/instrumentation-pg";
import {
registerInstrumentations,
} from "@opentelemetry/instrumentation";
import { Resource } from "@opentelemetry/resources";
import {
ATTR_SERVICE_NAME,
ATTR_SERVICE_VERSION,
} from "@opentelemetry/semantic-conventions";
// Initialize tracing — must run before app code
const provider = new NodeTracerProvider({
resource: new Resource({
[ATTR_SERVICE_NAME]: "order-service",
[ATTR_SERVICE_VERSION]:
process.env.APP_VERSION ?? "1.0.0",
}),
});
provider.addSpanProcessor(
new SimpleSpanProcessor(
new OTLPTraceExporter({
url:
process.env.OTEL_EXPORTER_OTLP_ENDPOINT ??
"http://localhost:4318/v1/traces",
})
)
);
provider.register();
// Auto-instrument libraries
registerInstrumentations({
instrumentations: [
new HttpInstrumentation(),
new ExpressInstrumentation(),
new PgInstrumentation(),
],
});Connecting the Three Signals
The real power comes from correlating signals. A trace ID in your logs lets you jump from a log line to the full trace. Tags on your metrics let you drill down from an anomaly to the specific requests.
import { trace, context } from "@opentelemetry/api";
// Enrich logs with trace context
function createCorrelatedLogger(
req: Request
): pino.Logger {
const span = trace.getActiveSpan();
const spanContext = span?.spanContext();
return logger.child({
requestId: req.headers["x-request-id"],
traceId: spanContext?.traceId,
spanId: spanContext?.spanId,
userId: req.user?.id,
method: req.method,
path: req.path,
});
}
// Now logs contain trace IDs:
// {"level":"error","traceId":"abc123def456...",
// "spanId":"789xyz...","msg":"Payment failed"}
//
// Click the traceId in your log viewer →
// see the full distributed trace in Jaeger/Tempo
// Create custom spans for business operations
const tracer = trace.getTracer("order-service");
async function processPayment(
orderId: string,
amount: number
) {
return tracer.startActiveSpan(
"process-payment",
async (span) => {
span.setAttribute("order.id", orderId);
span.setAttribute("payment.amount", amount);
try {
const result =
await paymentGateway.charge(amount);
span.setAttribute(
"payment.transaction_id",
result.transactionId
);
span.setStatus({ code: 0 }); // OK
return result;
} catch (error) {
span.setStatus({
code: 2,
message: (error as Error).message,
});
span.recordException(error as Error);
throw error;
} finally {
span.end();
}
}
);
}Alert Rules: Making Metrics Actionable
Metrics without alerts are just graphs nobody looks at. Define SLO-based alerts that notify on-call when user experience degrades.
# prometheus-alerts.yml
groups:
- name: api-slos
rules:
- alert: HighErrorRate
expr: |
sum(rate(http_requests_total{status_code=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m]))
> 0.01
for: 5m
labels:
severity: critical
annotations:
summary: "Error rate exceeds 1% SLO"
description: >-
Error rate is {{ $value | humanizePercentage }}
over the last 5 minutes
- alert: HighLatency
expr: |
histogram_quantile(0.95,
sum(rate(http_request_duration_seconds_bucket[5m]))
by (le)
) > 0.5
for: 5m
labels:
severity: warning
annotations:
summary: "P95 latency exceeds 500ms SLO"
- alert: OrderCreationSpike
expr: |
rate(orders_created_total{status="failed"}[5m])
> 0.1
for: 3m
labels:
severity: critical
annotations:
summary: "Order creation failures spiking"Key Takeaways
Structured JSON logging with request-scoped context (request ID, user ID, trace ID) transforms logs from unsearchable text into queryable data—every log line should be a structured event that can be filtered, aggregated, and correlated across services. Prometheus metrics with labeled histograms, counters, and gauges quantify system behavior over time—instrument HTTP requests, database queries, and business operations with meaningful labels, and expose a /metrics endpoint for scraping. OpenTelemetry distributed tracing follows requests across service boundaries as a series of spans—auto-instrument HTTP clients, Express, and database drivers at startup, then add custom spans for business-critical operations that need visibility. Connect all three signals by including trace IDs in log lines and matching metric labels to trace attributes, enabling a debugging workflow where metrics detect anomalies, logs provide context, and traces pinpoint the exact service and operation that failed.
