📊 Application Monitoring Strategies

Why Monitoring Matters

In a world of distributed systems, microservices, and cloud-native deployments, you cannot fix what you cannot see. Monitoring is the practice of collecting and inspecting signals from a running system so you can answer two questions fast: Is it working right now? and Is it about to stop working?

💡 A useful analogy: Running a service without monitoring is like being a chef who never tastes the food, never checks whether customers are happy, and never times how long orders take. You might be fine — or the kitchen might be on fire and you would be the last to know.

Good monitoring is not about collecting every possible number. It is about collecting the right numbers so that when a pager goes off at 3 a.m., the person on call can understand and fix the problem quickly. Everything in this lesson serves that goal.

Monitoring vs. Observability

These two words get used interchangeably, but they mean different things:

• Monitoring answers known questions — "Is CPU above 90%?" You decide in advance what to watch and set alerts on it.
• Observability is the property of being able to answer new, unanticipated questions about your system from the data it already emits — "Why are only Android users in Brazil seeing slow checkouts?"

Observability rests on three complementary pillars. You need all three; each answers a different kind of question.

The Monitoring Pyramid

Not all metrics are equal. A useful way to organize them is as a pyramid: infrastructure at the base, user experience at the top. Lower layers are easy to measure but far from the user; higher layers are closer to what actually matters but harder to instrument.

RED, USE & the Four Golden Signals

Rather than inventing metrics from scratch, lean on three well-known frameworks. Each answers "which handful of numbers should I watch?" for a different kind of thing.

The RED method — for request-driven services

Coined by Tom Wilkie, RED is perfect for APIs and microservices:

• Rate — requests per second the service is handling
• Errors — how many of those requests are failing
• Duration — how long requests take (as a distribution, not an average)

These map neatly to PromQL. Here is RED for an HTTP service:

# Rate — requests per second, averaged over 5 minutes
sum(rate(http_requests_total[5m]))

# Errors — rate of 5xx responses
sum(rate(http_requests_total{status=~"5.."}[5m]))

# Duration — 95th percentile latency in seconds
histogram_quantile(0.95,
  sum(rate(http_request_duration_seconds_bucket[5m])) by (le))

The USE method — for resources

Brendan Gregg's USE method is the counterpart for infrastructure resources (CPUs, disks, network links):

• Utilization — the percentage of time the resource is busy
• Saturation — how much extra work is queued and waiting
• Errors — the count of error events

🛣️ Highway analogy: Utilization is how many cars are on the road, saturation is the queue backing up at the on-ramp, and errors are the crashes blocking a lane. A road can be 100% utilized and still flowing smoothly — it is saturation that signals real trouble.

The Four Golden Signals — Google SRE's default

The Google SRE book distills monitoring to four signals that combine ideas from both methods:

If you remember nothing else, start every new service with these four. They catch the overwhelming majority of user-facing problems.

Instrumenting a Service

Instrumentation is the code that emits metrics. Prometheus uses a pull model: your app exposes a plain-text /metrics endpoint, and Prometheus scrapes it on an interval. Here is a modern Express service instrumented with prom-client, capturing the Rate, Errors, and Duration of RED in one middleware.

import express from 'express';
import client from 'prom-client';

const app = express();

// A registry holds all metrics; default metrics add Node.js process stats
const register = new client.Registry();
client.collectDefaultMetrics({ register });

// Rate + Errors: a counter labelled by method, route, and status
const httpRequestsTotal = new client.Counter({
  name: 'http_requests_total',
  help: 'Total number of HTTP requests',
  labelNames: ['method', 'route', 'status'],
  registers: [register],
});

// Duration: a histogram bucketed by response time
const httpRequestDuration = new client.Histogram({
  name: 'http_request_duration_seconds',
  help: 'Duration of HTTP requests in seconds',
  labelNames: ['method', 'route', 'status'],
  buckets: [0.05, 0.1, 0.3, 0.5, 1, 3, 5],
  registers: [register],
});

// One middleware records both metrics when the response finishes
app.use((req, res, next) => {
  const stopTimer = httpRequestDuration.startTimer();
  res.on('finish', () => {
    const route = req.route?.path ?? req.path;
    const labels = { method: req.method, route, status: res.statusCode };
    httpRequestsTotal.inc(labels);
    stopTimer(labels);
  });
  next();
});

app.get('/hello', (req, res) => res.json({ message: 'Hello World!' }));

// The endpoint Prometheus scrapes
app.get('/metrics', async (req, res) => {
  res.set('Content-Type', register.contentType);
  res.end(await register.metrics());
});

app.listen(3000, () => console.log('Listening on http://localhost:3000'));

http_requests_total{method="GET",route="/hello",status="200"} 42
http_request_duration_seconds_bucket{le="0.1",method="GET",route="/hello",status="200"} 40
http_request_duration_seconds_sum{method="GET",route="/hello",status="200"} 1.83
http_request_duration_seconds_count{method="GET",route="/hello",status="200"} 42

Python and Java have equivalent libraries — prometheus_client and Micrometer — and the OpenTelemetry SDK offers a vendor-neutral option that works across all of them.

SLOs, SLIs & Alerting

Monitoring without alerting is just passive watching. But alerting badly is worse than not alerting at all — nothing burns out an on-call team faster than a pager that cries wolf. The discipline that fixes this is defining alerts against service level objectives.

The gap between "100%" and your SLO is your error budget. A 99.9% availability SLO means you are allowed to fail 0.1% of requests — roughly 43 minutes a month. Alert not on every error, but when you are burning that budget fast enough to run out. Here is a good and a bad alert:

# prometheus/rules/alerts.yml
groups:
  - name: api-slo
    rules:
      # GOOD: symptom-based, tied to user-facing latency, waits 5m to avoid flapping
      - alert: HighRequestLatency
        expr: |
          histogram_quantile(0.95,
            sum(rate(http_request_duration_seconds_bucket{job="api"}[5m])) by (le)
          ) > 0.5
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "p95 latency above 500ms on the API"
          description: "95th percentile latency has exceeded 500ms for 5 minutes."
          runbook: "https://runbooks.example.com/high-latency"

      # GOOD: fast error-budget burn — page immediately
      - alert: ErrorBudgetBurn
        expr: |
          sum(rate(http_requests_total{status=~"5..",job="api"}[5m]))
            / sum(rate(http_requests_total{job="api"}[5m])) > 0.01
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Error rate above 1% — burning error budget"
          description: "More than 1% of requests are failing over the last 5 minutes."

The Tool Ecosystem

The landscape is large, but tools cluster into a few clear categories. You rarely pick just one — a typical stack combines a metrics tool, a dashboarding tool, a log store, and an alert router.

Hands-on Exercise

sum(rate(http_requests_total{route="/flaky",status=~"5.."}[5m]))
  / sum(rate(http_requests_total{route="/flaky"}[5m])) * 100

- alert: HighRequestLatency
  expr: |
    histogram_quantile(0.95,
      sum(rate(http_request_duration_seconds_bucket[5m])) by (le)) > 0.5
  for: 5m
  labels: { severity: warning }
  annotations:
    summary: "p95 latency above 500ms"

Best Practices

💡 The monitoring test: for every alert you create, ask "if this fires at 3 a.m., is there something a human must do right now?" If the answer is no, it should be a dashboard or a ticket — not a page.

Summary & Quiz

📚 Further Reading

• Google SRE Books — Monitoring & the Four Golden Signals
• Prometheus Documentation
• Grafana & Loki Documentation
• OpenTelemetry Documentation

🚀 What's Next?

Metrics tell you that something broke. To learn what exactly happened, you need logs. Next we build a centralized logging pipeline — structured logs, shippers, and a searchable store — that complements everything you just learned.