🔥 Prometheus and Grafana Monitoring Stack

The Complementary Pair

Prometheus and Grafana do different jobs, and that is exactly why they pair so well.

• Prometheus is the meter — it continuously measures, stores time-series data, evaluates rules, and fires alerts. Originally built at SoundCloud, it is now a graduated CNCF project.
• Grafana is the display — it queries data (from Prometheus and many other sources) and turns it into dashboards, and can alert too.

🏠 A smart-home analogy: Prometheus is the electricity meter on the wall — always measuring, recording, and tripping the breaker at a threshold. Grafana is the smart-home app on your phone — pulling readings from the meter and every other sensor into one clear screen.

How Prometheus Works

Prometheus is pull-based: instead of your apps pushing data to it, Prometheus reaches out and scrapes a /metrics HTTP endpoint on each target at a fixed interval. This makes targets simple (they just expose a page) and lets Prometheus automatically detect a target that has gone down.

The core loop:

1. Discover targets — statically listed, or dynamically via Kubernetes, Consul, or cloud APIs.
2. Scrape each target's /metrics endpoint on the scrape_interval.
3. Store samples in a local time-series database optimized for this shape of data.
4. Query with PromQL, and evaluate alert rules on the same data.
5. Fire triggered alerts to Alertmanager, which dedupes, groups, and routes them.

http_requests_total{method="GET", status="200", route="/api/users"}

The Four Metric Types

Picking the right metric type is the most common early mistake. Here is when to use each:

Declaring one of each with prom-client:

import client from 'prom-client';
const register = new client.Registry();

// Counter — cumulative request count
const requests = new client.Counter({
  name: 'http_requests_total',
  help: 'Total HTTP requests',
  labelNames: ['method', 'route', 'status'],
  registers: [register],
});

// Gauge — a value that moves both ways
const activeSessions = new client.Gauge({
  name: 'active_sessions',
  help: 'Current active user sessions',
  registers: [register],
});

// Histogram — bucketed durations, quantiles computed at query time
const duration = new client.Histogram({
  name: 'http_request_duration_seconds',
  help: 'Request duration in seconds',
  labelNames: ['method', 'route'],
  buckets: [0.1, 0.3, 0.5, 1, 3, 5],
  registers: [register],
});

requests.inc({ method: 'GET', route: '/api', status: '200' });
activeSessions.set(42);
const stop = duration.startTimer({ method: 'GET', route: '/api' });
// ... handle request ...
stop();

PromQL Essentials

PromQL is a functional query language for selecting and aggregating time series. A few patterns cover most day-to-day needs.

Selecting series

# All series for this metric
http_requests_total

# Filter by labels (= exact, != negate, =~ regex)
http_requests_total{status="200", method="GET"}
http_requests_total{status=~"5.."}

# A range vector — the last 5 minutes of samples
http_requests_total[5m]

The functions you will use constantly

# rate() — per-second average over a window; the go-to for counters
rate(http_requests_total[5m])

# Aggregate across series with sum/avg/max ... by (label)
sum(rate(http_requests_total[5m])) by (route)

# Error rate as a percentage
sum(rate(http_requests_total{status=~"5.."}[5m]))
  / sum(rate(http_requests_total[5m])) * 100

# 95th percentile latency from a histogram
histogram_quantile(0.95,
  sum(rate(http_request_duration_seconds_bucket[5m])) by (le))

# CPU utilization per instance (from node_exporter)
100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

Standing Up the Stack

Here is a complete, runnable Docker Compose file bringing up Prometheus, Grafana, Alertmanager, and node_exporter together.

# docker-compose.yml
services:
  prometheus:
    image: prom/prometheus:latest
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
      - ./prometheus/rules:/etc/prometheus/rules
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.retention.time=15d'
    ports: ["9090:9090"]
    restart: unless-stopped

  alertmanager:
    image: prom/alertmanager:latest
    volumes:
      - ./alertmanager/alertmanager.yml:/etc/alertmanager/alertmanager.yml
    ports: ["9093:9093"]
    restart: unless-stopped

  node-exporter:
    image: prom/node-exporter:latest
    ports: ["9100:9100"]
    restart: unless-stopped

  grafana:
    image: grafana/grafana:latest
    depends_on: [prometheus]
    ports: ["3000:3000"]
    volumes:
      - grafana-storage:/var/lib/grafana
      - ./grafana/provisioning:/etc/grafana/provisioning
    environment:
      - GF_SECURITY_ADMIN_PASSWORD__FILE=/run/secrets/grafana_admin
    restart: unless-stopped

volumes:
  grafana-storage:

The Prometheus config tells it what to scrape and where Alertmanager lives:

# prometheus/prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

alerting:
  alertmanagers:
    - static_configs:
        - targets: ['alertmanager:9093']

rule_files:
  - 'rules/*.yml'

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node'
    static_configs:
      - targets: ['node-exporter:9100']

  - job_name: 'web-app'
    static_configs:
      - targets: ['web-app:3000']

Rather than clicking through Grafana's UI, provision the data source as code so the stack is reproducible:

# grafana/provisioning/datasources/prometheus.yml
apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true

Building Dashboards

A Grafana dashboard is a grid of panels, each backed by a query. Panels come in types — time series, stat, gauge, bar, table, heatmap, logs. The art is choosing panels that answer a question at a glance.

A great first dashboard is a RED dashboard for your service — one row, three panels, the whole health of the service in one look:

# Panel 1 — Rate (requests/sec by route), a "time series" panel
sum(rate(http_requests_total[5m])) by (route)

# Panel 2 — Errors (% of requests failing), a "stat" panel with red threshold
sum(rate(http_requests_total{status=~"5.."}[5m]))
  / sum(rate(http_requests_total[5m])) * 100

# Panel 3 — Duration (p50/p95/p99), a "time series" panel with three queries
histogram_quantile(0.50, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))
histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))
histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))

Do not build everything from scratch — Grafana's dashboard library has thousands of community dashboards (the Node Exporter Full dashboard, ID 1860, is a classic) you can import by ID and adapt.

Alerting with Alertmanager

Prometheus evaluates alert rules and, when one fires, hands it to Alertmanager, whose job is to make alerts humane: deduplicate identical alerts, group related ones, silence during maintenance, and route by severity to the right channel.

Recall the alert rules from the monitoring lesson — they live in the rules/ directory Prometheus loads. Alertmanager then decides who gets paged:

# alertmanager/alertmanager.yml
global:
  resolve_timeout: 5m

route:
  group_by: ['alertname', 'job']
  group_wait: 30s        # wait to batch related alerts
  group_interval: 5m
  repeat_interval: 4h    # don't re-page every minute
  receiver: 'team-slack' # default destination
  routes:
    - matchers: [severity="critical"]
      receiver: 'pagerduty'   # wake someone up
    - matchers: [severity="warning"]
      receiver: 'team-slack'  # just notify

receivers:
  - name: 'team-slack'
    slack_configs:
      - api_url: ''
        channel: '#alerts'
        title: '{{ .CommonAnnotations.summary }}'

  - name: 'pagerduty'
    pagerduty_configs:
      - routing_key: ''

Hands-on Exercise

# Should show a non-zero, moving line per route
sum(rate(http_requests_total{job="web-app"}[1m])) by (route)

Production Best Practices

Scaling beyond one Prometheus

A single Prometheus is not horizontally scalable and is bounded by one machine's disk and RAM. When you outgrow it, reach for:

• Functional sharding — several Prometheus servers, each owning part of the fleet.
• Federation — a higher-level Prometheus scraping aggregated series from lower ones for a global view.
• Thanos, Cortex, or Grafana Mimir — systems that add long-term object-storage, a global query view, and high availability on top of Prometheus.

Summary & Quiz

📚 Further Reading

• Prometheus Documentation
• Grafana Documentation
• PromQL Querying Basics
• kube-prometheus — the production Kubernetes stack

🚀 What's Next?

You have now covered the full observability triad — metrics, logs, and dashboards with alerting. Next comes the weekend project, where you will pull everything from this DevOps & Deployment module together into one deployed, monitored application.