Observability Beyond Monitoring: Traces, Logs, and Metrics

How to build a complete observability stack, connecting distributed traces, structured logs and metrics to debug production faster than monitoring alone.

Published on July 26, 2021Updated on September 9, 20264 min read

observability monitoring distributed-tracing logging metrics devops

Monitoring tells you something is wrong. Observability tells you why. The difference matters when debugging a production issue at 2am. If your dashboards show elevated error rates but you cannot trace a single failed request through your system, you have monitoring without observability.

The three pillars — traces, logs, and metrics — are not useful in isolation. Their power comes from correlation: linking a metric spike to the traces that caused it, and those traces to the log lines that explain the root cause.

Metrics: The Starting Point

Metrics are aggregated numbers over time. They tell you what is happening at a system level but not why.

tstypescript

// ❌ Only tracking high-level metrics
const requestCount = new Counter({
  name: 'http_requests_total',
  help: 'Total HTTP requests',
});
 
app.use((req, res, next) => {
  requestCount.inc();
  next();
});
// You know requests increased, but not which endpoints or status codes

tstypescript

// ✅ Metrics with dimensions for drill-down
import { Counter, Histogram } from 'prom-client';
 
const requestDuration = new Histogram({
  name: 'http_request_duration_seconds',
  help: 'HTTP request duration in seconds',
  labelNames: ['method', 'route', 'status_code'],
  buckets: [0.01, 0.05, 0.1, 0.5, 1, 2, 5, 10],
});
 
const requestErrors = new Counter({
  name: 'http_request_errors_total',
  help: 'HTTP request errors',
  labelNames: ['method', 'route', 'error_type'],
});
 
app.use((req, res, next) => {
  const start = process.hrtime.bigint();
 
  res.on('finish', () => {
    const duration = Number(process.hrtime.bigint() - start) / 1e9;
    const route = req.route?.path ?? 'unknown';
 
    requestDuration
      .labels(req.method, route, String(res.statusCode))
      .observe(duration);
 
    if (res.statusCode >= 400) {
      requestErrors
        .labels(req.method, route, String(res.statusCode))
        .inc();
    }
  });
 
  next();
});

With labeled metrics, you can answer "which endpoint is slow?" and "which error codes are increasing?" directly from the metric data. Use histograms for latency (they give you percentiles), counters for totals, and gauges for current values like queue depth.

Structured Logging

Unstructured logs are unqueryable at scale. Structured logs are searchable, filterable, and can be correlated with traces.

tstypescript

// ❌ Unstructured log lines
console.log(`User ${userId} created order ${orderId} - total: $${total}`);
console.log(`ERROR: Payment failed for order ${orderId}`);
// Good luck searching for "all payment failures over $100 in the last hour"

tstypescript

// ✅ Structured JSON logs with consistent fields
import pino from 'pino';
 
const logger = pino({
  level: process.env.LOG_LEVEL ?? 'info',
  formatters: {
    level: (label) => ({ level: label }),
  },
  timestamp: pino.stdTimeFunctions.isoTime,
});
 
// Application code
logger.info({
  event: 'order_created',
  userId,
  orderId,
  total,
  currency: 'USD',
  itemCount: items.length,
});
 
logger.error({
  event: 'payment_failed',
  orderId,
  userId,
  amount: total,
  errorCode: 'card_declined',
  provider: 'stripe',
  traceId: req.headers['x-trace-id'],
});

Structured logs have two advantages: you can query them (event=payment_failed AND amount>100) and you can include trace IDs that link log lines to distributed traces.

Distributed Tracing

A trace follows a single request through every service it touches. Each segment is a span. Spans form a tree that shows where time was spent.

tstypescript

import { trace, SpanStatusCode } from '@opentelemetry/api';
 
const tracer = trace.getTracer('order-service');
 
async function createOrder(userId: string, items: CartItem[]) {
  return tracer.startActiveSpan('createOrder', async (span) => {
    try {
      span.setAttribute('user.id', userId);
      span.setAttribute('order.item_count', items.length);
 
      // Child span: validate inventory
      const inventory = await tracer.startActiveSpan(
        'validateInventory',
        async (childSpan) => {
          try {
            const result = await inventoryService.check(items);
            childSpan.setAttribute('inventory.available', result.allAvailable);
            return result;
          } finally {
            childSpan.end();
          }
        }
      );
 
      if (!inventory.allAvailable) {
        span.setStatus({
          code: SpanStatusCode.ERROR,
          message: 'Items out of stock',
        });
        throw new Error('Items out of stock');
      }
 
      // Child span: process payment
      const payment = await tracer.startActiveSpan(
        'processPayment',
        async (childSpan) => {
          try {
            const result = await paymentService.charge(userId, total);
            childSpan.setAttribute('payment.provider', 'stripe');
            childSpan.setAttribute('payment.amount', total);
            return result;
          } finally {
            childSpan.end();
          }
        }
      );
 
      span.setAttribute('order.id', payment.orderId);
      return { orderId: payment.orderId, status: 'confirmed' };
    } catch (err) {
      span.recordException(err as Error);
      span.setStatus({ code: SpanStatusCode.ERROR });
      throw err;
    } finally {
      span.end();
    }
  });
}

The trace from this code shows: createOrder (350ms) → validateInventory (50ms) + processPayment (280ms). If payment is slow, you see exactly where the bottleneck is and can drill into the payment service's spans.

Correlating the Three Pillars

The real value comes from linking metrics, logs, and traces together. A metric alert leads to traces, which lead to specific log lines.

tstypescript

// Middleware that connects all three pillars
import { trace, context } from '@opentelemetry/api';
 
function observabilityMiddleware(req: Request, res: Response, next: NextFunction) {
  const span = trace.getActiveSpan();
  const traceId = span?.spanContext().traceId ?? 'no-trace';
 
  // Attach trace ID to logger — all log lines include it
  req.log = logger.child({ traceId, requestId: req.id });
 
  // Attach trace ID to response headers — clients can report it
  res.setHeader('X-Trace-Id', traceId);
 
  const start = process.hrtime.bigint();
 
  res.on('finish', () => {
    const duration = Number(process.hrtime.bigint() - start) / 1e9;
 
    // Metric with trace exemplar
    requestDuration
      .labels(req.method, req.route?.path ?? 'unknown', String(res.statusCode))
      .observe(duration);
 
    // Structured log with trace correlation
    req.log.info({
      event: 'request_completed',
      method: req.method,
      path: req.path,
      statusCode: res.statusCode,
      duration,
    });
  });
 
  next();
}

The correlation flow:

Metric alert: P99 latency for /api/orders exceeds 2s
Trace search: Find traces to /api/orders with duration > 2s in the last 10 minutes
Span analysis: The processPayment span is consistently 1.8s (normally 200ms)
Log drill-down: Filter logs by traceId — see payment_timeout errors from Stripe's EU endpoint

Alerting on the Right Signals

Alert on symptoms (user-facing impact), not causes (CPU usage). Use the RED method for services and USE method for resources.

ymlyaml

# ❌ Alerting on causes — noisy and often irrelevant
groups:
  - name: infrastructure
    rules:
      - alert: HighCPU
        expr: cpu_usage > 80
        # CPU at 81% with no user impact → false alarm at 3am
 
# ✅ Alerting on symptoms — user-facing impact
groups:
  - name: service-health
    rules:
      - alert: HighErrorRate
        expr: |
          sum(rate(http_request_errors_total{status_code=~"5.."}[5m]))
          /
          sum(rate(http_requests_total[5m])) > 0.01
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Error rate exceeds 1% for 5 minutes"
 
      - alert: HighLatency
        expr: |
          histogram_quantile(0.99,
            rate(http_request_duration_seconds_bucket[5m])
          ) > 2
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "P99 latency exceeds 2s for 5 minutes"

The for: 5m clause prevents alerting on transient spikes. A single slow request is noise. Five minutes of elevated latency is a real problem.

Key Takeaways

Metrics show what is happening — use labeled histograms and counters with dimensions for drill-down
Structured logs explain why — use JSON logging with consistent field names and trace IDs
Traces show where time is spent — instrument service boundaries and expensive operations as spans
Correlation is the multiplier — link metrics → traces → logs through trace IDs propagated across services
Alert on symptoms, not causes — error rates and latency percentiles matter more than CPU usage
Use the for clause — require sustained violations before alerting to avoid noise from transient spikes

Observability Beyond Monitoring: Traces, Logs, and Metrics

Metrics: The Starting Point

Structured Logging

Distributed Tracing

Correlating the Three Pillars

Alerting on the Right Signals

Key Takeaways

Related Posts

Observability Stack: Metrics, Logs and Traces in Practice

Monitoring and Alerting Strategies That Reduce Alert Fatigue

OpenTelemetry Instrumentation for Node.js