📜 Centralized Logging Implementation

Why Centralize Logs?

In the previous lesson, metrics gave you quantitative signals — numbers over time. Logs are qualitative: a timestamped record of discrete events, each carrying the detail metrics leave out. Together they complete the picture.

🔧 A car analogy: Metrics are your dashboard gauges — they warn that the engine is overheating. Logs are the mechanic's detailed service notes — they tell you which hose burst, when, and in what order things failed. You need both to actually fix the car.

The trouble is that a modern app is not one program. It is a fleet of containers — microservices, a frontend, databases, a load balancer — each writing logs to its own ephemeral filesystem. When a container restarts, those logs vanish. Centralized logging ships every log to one durable, searchable system so you can investigate the whole system at once.

The Logging Pipeline

Every centralized logging system, regardless of the tools, is built from the same four stages. Data flows left to right:

• Collect — lightweight agents (Fluent Bit, Vector, Promtail, Filebeat) read logs from files or container stdout and forward them. In Kubernetes this is usually a DaemonSet or a sidecar.
• Process — parse unstructured text into fields, drop noise, enrich with context (add the pod name, region, or a GeoIP lookup), and mask sensitive data.
• Store — a system built for high write volume and fast queries, with hot/warm/cold tiers so recent logs are fast and old logs are cheap.
• Analyze — search, dashboards, and alerting on top of the stored logs.

Structured Logging

The single most impactful habit in logging is to emit structured logs — JSON objects with named fields — instead of free-form text. Compare these two lines describing the same event:

# Unstructured — a human can read it, a machine struggles
2026-08-01 15:23:45 ERROR Payment failed for user johndoe amount 99.99 code DECLINED

{
  "timestamp": "2026-08-01T15:23:45.678Z",
  "level": "error",
  "service": "payment-service",
  "message": "Payment failed",
  "userId": "u_12345",
  "amount": 99.99,
  "currency": "USD",
  "errorCode": "CARD_DECLINED"
}

The JSON version can be filtered by errorCode, aggregated by currency, and searched by userId — no fragile text parsing required. Here is how to produce it.

Node.js with Pino

Pino is the modern, fast structured logger for Node. It emits JSON by default:

import pino from 'pino';

const logger = pino({
  level: process.env.LOG_LEVEL ?? 'info',
  base: { service: 'payment-service' }, // fields added to every log line
});

logger.info({ userId: 'u_12345', loginTime: new Date().toISOString() },
  'User logged in');

logger.error(
  { userId: 'u_12345', amount: 99.99, currency: 'USD', errorCode: 'CARD_DECLINED' },
  'Payment failed'
);

Python with structlog

import structlog

structlog.configure(
    processors=[
        structlog.processors.add_log_level,
        structlog.processors.TimeStamper(fmt="iso"),
        structlog.processors.JSONRenderer(),
    ],
)

log = structlog.get_logger(service="payment-service")

log.info("payment_processed", user_id="u_67890",
         payment_id="PMT123456", amount=99.99, currency="USD")

log.error("payment_failed", user_id="u_67890",
          payment_id="PMT123457", amount=149.99,
          error_code="CARD_EXPIRED")

Log Levels Done Right

Log levels let you dial verbosity up in development and down in production. Using them consistently is what makes a log store searchable rather than a swamp.

ELK vs. PLG Stacks

Two open-source stacks dominate. They fill the same pipeline boxes differently.

The ELK / Elastic Stack

Elasticsearch (store & search) + Logstash (process) + Kibana (analyze), with Beats or Fluent Bit collecting. It indexes the full content of every log, making it extremely powerful to search — at the cost of heavy storage and memory.

The PLG Stack (Grafana Loki)

Promtail (collect) + Loki (store) + Grafana (analyze). Loki's trick: it indexes only a small set of labels, not the full log content — like Prometheus, but for logs. That makes it far cheaper to run, and it shares Grafana with your metrics.

Here is a minimal Loki + Promtail + Grafana stack in Docker Compose — lightweight enough to run on a laptop:

# docker-compose.yml — PLG logging stack
services:
  loki:
    image: grafana/loki:2.9.0
    command: -config.file=/etc/loki/local-config.yaml
    ports:
      - "3100:3100"

  promtail:
    image: grafana/promtail:2.9.0
    volumes:
      - /var/log:/var/log:ro
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
      - ./promtail-config.yml:/etc/promtail/config.yml
    command: -config.file=/etc/promtail/config.yml
    depends_on:
      - loki

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    environment:
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
    depends_on:
      - loki

Once running, add Loki as a Grafana data source (http://loki:3100) and query logs with LogQL, Loki's PromQL-inspired language:

# All error logs from the payment service
{service="payment-service"} | json | level="error"

# Count errors per minute
sum(count_over_time({service="payment-service"} | json | level="error" [1m]))

Correlating with Trace IDs

Centralized logs are useful; correlated logs are transformative. When a single user request fans out across five services, you want to see all five services' log lines for that one request, in order. The mechanism is a trace ID: a unique identifier generated at the edge and propagated (via an HTTP header) to every downstream service, which stamps it on every log line it writes.

You can do this manually with a header and a per-request logger, or adopt OpenTelemetry, which standardizes trace-ID generation and propagation across languages and auto-instruments popular frameworks. Here is the lightweight manual version in Express using AsyncLocalStorage so every log inside a request automatically includes the ID:

import { AsyncLocalStorage } from 'node:async_hooks';
import { randomUUID } from 'node:crypto';
import pino from 'pino';

const als = new AsyncLocalStorage();
const base = pino({ base: { service: 'api-gateway' } });

// Wrap the base logger so it always merges in the current trace ID
export const logger = new Proxy(base, {
  get(target, prop) {
    const traceId = als.getStore()?.traceId;
    if (['info', 'warn', 'error', 'debug'].includes(prop)) {
      return (obj, msg) => target[prop]({ ...obj, traceId }, msg);
    }
    return target[prop];
  },
});

// Middleware: reuse an incoming trace ID or mint a new one
export function traceMiddleware(req, res, next) {
  const traceId = req.headers['x-trace-id'] ?? randomUUID();
  res.setHeader('x-trace-id', traceId);
  als.run({ traceId }, () => next());
}

Now a search for traceId="abc123" in Loki or Kibana returns the complete story of one request across every service that honored the header.

Hands-on Exercise

import express from 'express';
import { logger, traceMiddleware } from './logging.js';

const app = express();
app.use(traceMiddleware);

app.get('/checkout', async (req, res) => {
  logger.info({ route: '/checkout' }, 'Checkout started');
  try {
    // ...business logic...
    res.json({ ok: true });
  } catch (err) {
    logger.error({ route: '/checkout', err: err.message }, 'Checkout failed');
    res.status(500).json({ error: 'internal' });
  }
});

app.listen(3000);

Best Practices & Security

Retention & cost

Logs grow fast. Manage them with tiers: hot storage for the last few days (fast queries), warm for a few weeks (ongoing investigations), and cold/archived for long-term compliance (cheap object storage, rarely queried). Compress and rotate aggressively.

Summary & Quiz

📚 Further Reading

• Grafana Loki Documentation
• Elastic Stack Documentation
• Pino — fast Node.js logger
• The Twelve-Factor App — Logs

🚀 What's Next?

You now have metrics and logs. In the final lesson we bring them together into a complete monitoring stack — deploying Prometheus and Grafana, writing PromQL, building dashboards, and wiring up Alertmanager.