Observability: Logs, Metrics, and Traces

Observability is the ability to understand a system's internal state from the data it emits. Monitoring is the practice of collecting that data and alerting on it. A system is observable enough when an engineer can answer a question about its behavior, including questions nobody anticipated, without deploying new code.

Three kinds of telemetry cover this: logs, metrics, and traces.

The three pillars

flowchart LR
    A[Application] -->|structured logs| L[Log pipeline]
    A -->|metrics| M[Metrics store]
    A -->|spans| T[Trace backend]
    L --> D[Dashboards and search]
    M --> D
    T --> D
    M --> AL[Alerting]
    AL --> P[On-call engineer]
    P --> D

A typical investigation starts with a metric alert (error rate is high), uses traces to find the failing service and operation, and reads that service's logs for the specific error. A shared trace ID in all three signals makes the transitions direct. For trace details see distributed tracing.

Metrics

A metric is a named time series: a sequence of (timestamp, value) pairs, identified by a name and a set of labels (key-value dimensions such as service, method, status).

Metric types

Counters are used through their rate of change: requests per second is the counter's increase over a window divided by the window length. Histograms allow computing percentiles such as p99 (the value below which 99 percent of observations fall) on the server side. The result is an estimate, interpolated within the bucket that contains the percentile, so its accuracy depends on the bucket boundaries. For example, with buckets at 100 ms and 500 ms, a p99 that falls in that bucket is only known to lie between 100 ms and 500 ms. Histogram buckets from many instances can be summed and a percentile computed afterwards. Client-side summary quantiles cannot be meaningfully averaged across instances.

Percentiles, not averages. Latency distributions have long tails. A mean of 100 ms can hide a p99 of several seconds. Alert and set targets on percentiles (p50, p95, p99).

Push and pull

In the pull model, the metrics system (for example Prometheus) scrapes an HTTP endpoint on each target at an interval. In the push model, applications send metrics to a collector or gateway. Pull makes it easy to detect a target that is down (the scrape fails). Push suits short-lived jobs that may exit before being scraped.

Cardinality

Cardinality is the number of distinct time series, which equals the number of distinct label-value combinations for a metric. Each series consumes memory and storage. A label with unbounded values, such as user ID, request ID, or full URL path, creates one series per value and can overload the metrics store.

Rules:

• Use labels with small, bounded value sets (status class, method, region, route template such as /orders/{id}).
• Put high-cardinality identifiers in logs or trace attributes, not metric labels.
• The series count for a metric is roughly the product of the value counts of its labels. Three labels with 10, 20, and 50 values can produce up to 10,000 series.

RED and USE

Two methods choose what to measure so that dashboards are consistent.

RED applies to request-driven services:

• Rate: requests per second.
• Errors: failed requests per second or as a fraction.
• Duration: latency distribution.

USE applies to resources such as CPU, memory, disks, and network interfaces:

• Utilization: percentage of time the resource is busy.
• Saturation: work queued because the resource is fully used, such as CPU run-queue length or disk I/O queue depth.
• Errors: count of error events.

Google's SRE book names four golden signals: latency, traffic, errors, and saturation. RED covers the first three for services. USE covers saturation for resources.

Alerting

An alert is a rule evaluated on telemetry that notifies a human when a condition holds.

• Alert on symptoms, not causes. A symptom is what users experience: high error rate, high latency, failed checkouts. A cause such as high CPU may or may not affect users. Cause-based alerts create pages that need no action.
• Every page should be actionable and urgent. Non-urgent conditions go to tickets or dashboards.
• Use a duration. Require the condition to hold for a period (for example five minutes) to avoid flapping on brief spikes.
• Tie alerts to SLOs. A service level indicator (SLI) is a measured quantity, such as the fraction of requests that succeed. A service level objective (SLO) is a target for it, such as 99.9 percent over 30 days. The error budget is the allowed failure, 0.1 percent in this example, which is about 43 minutes of full outage in 30 days (0.001 x 43,200 min = 43.2 min). Burn-rate alerts fire when the budget is being consumed faster than the SLO allows, for example when the current error rate is 30 times the allowed rate (a burn rate of 30), which would exhaust the 30-day budget in one day. See scalability, availability, and reliability for SLO definitions.

Logs

A structured log is an event written in a machine-parseable format, usually JSON, with fixed fields (timestamp, level, service, message, trace ID) plus event-specific fields. Plain-text logs need regular expressions to parse. Structured logs can be filtered and aggregated by field.

Practices:

• Include a correlation ID (the trace ID) in every log line so all logs for one request can be found together.
• Use levels consistently: ERROR for failures needing attention, WARN for recoverable anomalies, INFO for significant events, DEBUG for diagnostics, usually disabled in production.
• Do not log secrets, tokens, or personal data.
• Write logs to standard output and let the platform collect them, so the application does not manage files or destinations.

Log pipeline

flowchart LR
    APP[Service stdout] --> AG[Node agent]
    AG --> Q[Buffer or queue]
    Q --> PR[Parse and enrich]
    PR --> ST[(Search index or object storage)]
    ST --> UI[Query UI]

• Agent. Runs on each node and forwards container output.
• Buffer. A queue such as Kafka absorbs bursts so that a slow store does not cause log loss or block applications.
• Processing. Parses fields, adds metadata (host, service, version), and drops or samples noisy events.
• Storage. A full-text index (Elasticsearch-style, see inverted index) gives fast search at high cost. Object storage with a lighter index is cheaper for large volumes. Retention policies delete or archive old data.

Logs are the most voluminous signal, so cost control matters: sampling debug logs, short retention for verbose data, and tiered storage.

Traces and sampling

A trace is the set of spans for one request, each span recording an operation, its start time, duration, and parent. Storing every trace at high traffic is costly, so systems sample:

• Head-based sampling decides at the start of the request (for example keep 1 percent). It is cheap but may discard the rare slow or failed traces.
• Tail-based sampling decides after the trace completes, so it can keep all errors and slow traces. It requires buffering all spans of a trace in one place.

Trade-offs

• Metrics are the cheapest signal and the right basis for alerts, but cannot explain a single failing request.
• Logs and traces explain individual requests but cost more per unit of insight and must be sampled or retained selectively.
• More instrumentation gives more visibility and more cost and overhead. Instrument boundaries (incoming and outgoing requests, queue operations, database calls) first.

Interview questions

• What is the difference between monitoring and observability? Monitoring checks known conditions with predefined dashboards and alerts. Observability is the property that lets you investigate unknown problems from the emitted data, using high-detail signals such as traces and structured logs.
• Why are high-cardinality labels a problem for metrics? Every unique label combination is a separate time series stored and indexed in memory. Labels such as user ID create millions of series and can exhaust the metrics store. Such identifiers belong in logs or traces.
• Why alert on symptoms rather than causes? A symptom alert fires only when users are affected. Cause alerts, such as high CPU, often fire with no user impact and train engineers to ignore pages.
• How do you compute p99 latency across many instances? Export histograms with the same bucket boundaries from every instance, sum the bucket counts, and estimate the percentile from the merged histogram. Averaging per-instance percentiles is incorrect.
• What is the difference between head-based and tail-based trace sampling? Head-based sampling chooses at request start and is cheap but blind to outcome. Tail-based sampling waits for the whole trace and can retain all errors and slow requests, at the cost of buffering spans.