Observability basics: logs, metrics, traces and alerts

Production is a black box until it tells you something. It can tell you three ways, and each answers a different question:

Dashboards and alerts are usually built on metrics, because they're cheap to keep for months. When one fires, you drill into traces to find where, and logs to find what exactly.

Structured logs: fields, not sentences

// Hard to search: the useful values are buried in prose
console.log(`order o_123 failed for user 42 after 2310ms: payment timeout`);

// Structured: one JSON object per line, values in named fields
console.log(JSON.stringify({
  level: "error", msg: "order failed", order_id: "o_123",
  duration_ms: 2310, error: "payment timeout",
  trace_id: "c50ed0daa0f6fd6a6087220a3d34fffa",
}));

The second line can be filtered (level = "error" and duration_ms > 2000), counted and joined — no regex over prose. Two habits make it pay off:

• Same field names everywhere — order_id in every service, not orderId here and order there. OpenTelemetry's docs make the point that JSON alone isn't "structured" in a useful sense; a stable schema is.
• Carry the trace ID. With trace_id on every line, one click goes from a slow trace to every log line that request wrote, across services. OpenTelemetry SDKs can add it for you when you route your logging library through them.

What not to log: passwords, tokens, full card numbers, and personal data you don't need — logs are copied, shipped and kept far more widely than your database.

What a Prometheus scrape of a small API looks like, in its text format:

# TYPE http_server_requests_total counter
http_server_requests_total{method="GET",route="/orders/:id",status="200"} 10432
http_server_requests_total{method="GET",route="/orders/:id",status="500"} 17

# TYPE jobs_waiting gauge
jobs_waiting 42

# TYPE http_server_request_duration_seconds histogram
http_server_request_duration_seconds_bucket{le="0.1"} 9210
http_server_request_duration_seconds_bucket{le="0.3"} 10102
http_server_request_duration_seconds_bucket{le="1"} 10398
http_server_request_duration_seconds_bucket{le="+Inf"} 10449
http_server_request_duration_seconds_sum 1342.7
http_server_request_duration_seconds_count 10449

Counter — only goes up (or resets to zero when the process restarts). Requests served, errors, bytes sent. The raw value is meaningless; you query its rate — rate(http_server_requests_total[5m]) — and Prometheus adjusts for resets. Prometheus's naming guide gives accumulating counts a _total suffix. Never use a counter for something that can go down.

Gauge — a value right now that moves both ways: jobs waiting, memory in use, open connections. You read it as is, or take its min/max over a window.

Histogram — the shape of a distribution, for latency and sizes. It's a set of counters, one per bucket, and the buckets are cumulative: le="0.3" (less than or equal) counts every request that took 0.3 s or less, including the 9,210 already counted under le="0.1". le="+Inf" counts everything and equals _count. From these you get percentiles and "what share of requests finished within 300 ms" (Complex tier). Note the unit in the name: Prometheus's guide asks for base units — _seconds, _bytes — not milliseconds.

OpenTelemetry has the same three ideas under its own names — Counter, UpDownCounter and Gauge, and Histogram — plus asynchronous variants that are read once per export.

Cardinality: every label combination is its own series

Prometheus's docs: "every unique combination of key-value label pairs represents a new time series" — so don't use labels for high-cardinality dimensions such as user IDs or email addresses. Do the arithmetic on that histogram:

• 40 routes × 5 methods × 6 status codes = 1,200 label combinations;
• with the 14 bucket boundaries OpenTelemetry recommends for request duration, each combination is 17 series (15 buckets including +Inf, _sum, _count) = 20,400 series — fine;
• add user_id with 50,000 users and it's about a billion.

The usual leak isn't a deliberate user_id; it's a raw URL. path="/orders/o_8f3k2" creates a series per order, where route="/orders/:id" creates one. OpenTelemetry's HTTP conventions require http.route to be low-cardinality for exactly this reason. Put per-request detail — user, order, raw path — on logs and spans, which are priced per event, not per combination.

A service doesn't keep every latency to sort later. It increments bucket counters, and every question has to be answered from those counts. Five requests — 50, 150, 250, 900 and 4,000 ms — with buckets at 100, 300 and 1,000 ms:

"What share finished within 300 ms?" — exact, if a bucket ends there

Read it straight off: count(le=300) / count(+Inf) = 3 / 5 = 60%. Prometheus's histogram guide writes the same thing over a window:

sum(rate(http_request_duration_seconds_bucket{le="0.3"}[5m]))
  /
sum(rate(http_request_duration_seconds_count[5m]))

and warns that it strictly requires a bucket boundary at 0.3. Ask "within 250 ms?" and the counts can't say — the requests between 100 and 300 ms are one number. So put bucket boundaries at your thresholds: if the target is "99% under 300 ms", 300 ms must be a bound.

"What's the p50?" — an estimate

histogram_quantile() finds the bucket the rank falls in and assumes the observations inside it are spread evenly (linear interpolation). The p50 rank is 0.5 × 5 = 2.5, which lands in the 100–300 bucket, three-quarters of the way through its two observations: 100 + 0.75 × 200 = 250 ms. Here that happens to be exact; in general the error is bounded by the width of the bucket the quantile lands in — anywhere from 100 to 300 ms.

Two edge rules from the Prometheus docs matter:

• The lowest bucket is assumed to start at 0.
• A quantile that lands in the +Inf bucket returns the highest finite bound. The p90 rank, 4.5, lands there, so the answer is 1,000 ms — while the real slowest request took 4 s. If your top bound is below your tail, the histogram can't show the tail at all.

Never average percentiles

Each instance of a service can compute its own p99, but you can't combine them. Instance A serves 100 requests, 98 at 10 ms and two at 1 s; instance B serves 100 requests, all at 10 ms. Their p99s are 1,000 and 10 ms; the average, 505 ms, is a latency no request had, and the true p99 of all 200 requests is 10 ms. Prometheus's guide calls averaged quantiles "statistically nonsensical". That's why its summary type, which precomputes quantiles in each process, can't be aggregated across instances, while histogram buckets can: sum the bucket counters from every instance (sum by (le) (rate(...))), then take the quantile.