Distributed Tracing and Microservices Observability

A single user request in a microservices system may pass through a gateway and many services, queues, and databases. Logs and metrics from each service alone do not show which downstream call made the request slow or failed. Distributed tracing records the path and timing of one request across all the components it touches. It complements logs and metrics (see observability: logs, metrics, and traces).

Traces and spans

• A trace represents one end-to-end request or transaction. It has a globally unique trace ID.
• A span represents one unit of work within a trace: an incoming request handled by a service, an outgoing call, a database query. A span has a span ID, a name, a start time, a duration, a status, and key-value attributes (for example HTTP method, status code, database statement type). It can also carry timestamped events.
• Each span except the first (the root span) has a parent span ID. The parent links form a tree.
• A span has a kind that describes its role: server (handles an incoming request), client (makes an outgoing request), producer and consumer (send and receive messages), or internal (work inside one process).
• A span can also have links to spans in other traces, used where a strict parent-child relationship does not fit, such as a batch consumer that processes messages from many traces.

graph TD
    R["Span A: gateway POST /orders (0-240 ms)"]
    R --> O["Span B: order-service create (10-230 ms)"]
    O --> I["Span C: inventory-service reserve (20-70 ms)"]
    O --> P["Span D: payment-service charge (80-210 ms)"]
    P --> X["Span E: card provider HTTP call (90-200 ms)"]
    O --> DB["Span F: SQL insert order (215-225 ms)"]

Reading the tree shows that most of the 240 ms is the payment call, and that the inventory reservation completed before payment started. Overlapping sibling spans indicate parallel calls. Gaps between a parent and its children show time spent in the parent itself.

Context propagation

For spans in different processes to join one trace, each service must pass the trace context to the services it calls. Context propagation has two parts:

• In process: the current span context is kept in a context object (thread-local or task-local, or passed explicitly), so that new spans become its children.
• Across processes: the context is serialized into a carrier: HTTP headers, gRPC metadata, or message headers. The receiver extracts it and starts its span as a child.

W3C Trace Context

The W3C Trace Context standard defines the traceparent header:

traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01

A second header, tracestate, carries vendor-specific key-value pairs. The standard baggage header (W3C Baggage) carries user-defined key-value pairs, such as a tenant ID, that propagate through the whole call chain. Baggage is copied into every downstream request, so it should stay small and must not contain sensitive data.

import java.util.Optional;

public class Main {
    record TraceParent(String traceId, String parentId, boolean sampled) {}

    static boolean isHex(String s, int length) {
        if (s.length() != length) {
            return false;
        }
        for (char c : s.toCharArray()) {
            boolean digit = c >= '0' && c <= '9';
            boolean lowerHex = c >= 'a' && c <= 'f';
            if (!digit && !lowerHex) {
                return false;
            }
        }
        return true;
    }

    static boolean allZeros(String s) {
        return s.chars().allMatch(c -> c == '0');
    }

    static Optional<TraceParent> parse(String header) {
        var parts = header.split("-", -1);
        if (parts.length != 4 || !isHex(parts[0], 2) || parts[0].equals("ff")) {
            return Optional.empty();
        }
        var traceId = parts[1];
        var parentId = parts[2];
        if (!isHex(traceId, 32) || allZeros(traceId)) {
            return Optional.empty();
        }
        if (!isHex(parentId, 16) || allZeros(parentId)) {
            return Optional.empty();
        }
        if (!isHex(parts[3], 2)) {
            return Optional.empty();
        }
        int flags = Integer.parseInt(parts[3], 16);
        return Optional.of(new TraceParent(traceId, parentId, (flags & 1) == 1));
    }

    public static void main(String[] args) {
        var ok = "00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01";
        System.out.println(parse(ok));
        System.out.println(parse("00-00000000000000000000000000000000-b7ad6b7169203331-01"));
        System.out.println(parse("garbage"));
    }
}

Optional[TraceParent[traceId=0af7651916cd43dd8448eb211c80319c, parentId=b7ad6b7169203331, sampled=true]]
Optional.empty
Optional.empty

A service that receives a valid traceparent starts a new span with the same trace ID, a newly generated span ID, and the incoming parent ID as its parent. When it calls another service, it sends the same trace ID and its own span ID as the new parent ID. A service that receives no valid header starts a new trace.

Asynchronous boundaries need the same treatment. A producer writes the trace context into message headers, and the consumer extracts it and continues the trace, or starts a new trace linked to the producer's span.

OpenTelemetry

OpenTelemetry (OTel) is a vendor-neutral, open-source standard for telemetry, covering traces, metrics, and logs. It defines:

• API and SDKs for many languages, to create spans and metrics.
• Instrumentation libraries that automatically create spans for common frameworks, HTTP and gRPC clients and servers, and database drivers.
• OTLP, a wire protocol for exporting telemetry.
• The Collector, a standalone process that receives telemetry, processes it (batching, filtering, attribute changes, sampling), and exports it to one or more backends.
• Semantic conventions: standard names for attributes, such as http.request.method, so that data from different services is comparable.

graph LR
    S1["Service A + SDK"] -->|OTLP| C["OpenTelemetry Collector"]
    S2["Service B + SDK"] -->|OTLP| C
    S3["Service C + SDK"] -->|OTLP| C
    C --> B1[("Trace backend: Jaeger, Tempo, or vendor")]
    C --> B2[("Metrics backend")]

Using a Collector keeps export configuration and credentials out of application code and allows changing backends without redeploying services. Spans are exported asynchronously in batches, so tracing adds little latency to requests, but it still consumes CPU, memory, and network.

Sampling

Tracing every request in a high-traffic system produces large volumes of data and cost. Sampling keeps only a portion of traces.

Example with assumed numbers: 1,000 requests per second, 10 spans per trace, and 500 bytes per span produce 1,000 x 10 x 500 = 5 MB/s, or about 430 GB per day. Sampling 1% of traces reduces this to about 4.3 GB per day.

Consistent decisions matter: if each service sampled independently, most traces would be incomplete. Head-based sampling propagates the decision. Tail-based sampling routes spans by trace ID to a specific collector so it sees the whole trace.

Metrics should not be derived from sampled traces alone, because sampling changes counts. Record metrics separately from all requests.

Correlation IDs and logs

A correlation ID is an identifier attached to a request and included in every log line it produces, so logs from many services can be searched together. With tracing in place, the trace ID serves as the correlation ID:

• Attach the current trace ID and span ID to each structured log record.
• From a slow trace, find the exact logs of each span. From an error log, open the full trace.
• Return the trace ID to clients in a response header or error body, so a support request can quote it.

Where tracing is not deployed, a gateway can generate a request ID header (for example X-Request-ID) and every service copies it into its logs and outgoing calls.

Service maps and analysis

Backends aggregate spans into a service map: a graph of services with edges for observed calls, annotated with request rate, error rate, and latency. It shows actual dependencies, including undocumented ones. Analysis uses:

• Critical path: the chain of spans that determines total latency. Only shortening spans on the critical path reduces end-to-end latency.
• Latency breakdown per service and comparison of slow traces with typical ones.
• Error localization: the first span in the tree with an error status.
• Fan-out and N+1 detection: a span with many similar child calls.

Pitfalls

• Broken propagation. One service that fails to forward the context splits the trace into two. Common causes: thread pools and async tasks that lose the in-process context, messaging without header propagation, and proxies that strip headers.
• High-cardinality attributes. Attributes are searchable, but excessive distinct values raise storage and index costs in some backends. Do not put personal or sensitive data in attributes.
• Clock skew. Span timestamps come from different machines. Skewed clocks make children appear to start before parents. Backends may adjust for this.
• Cost. Storage and processing of spans grow with traffic. Use sampling and retention limits.

Interview questions

• What is the difference between a trace and a span? A trace is the whole tree of work for one request, identified by a trace ID. A span is one timed operation within it, with a parent link.
• How does a trace continue across a service boundary? The caller injects the trace ID and its span ID into a header such as traceparent. The callee extracts it and creates a child span with the same trace ID.
• Why use tail-based sampling? It decides after seeing the complete trace, so it can keep all slow or failed requests. Head-based sampling decides before the outcome is known.
• How do you find the cause of high latency for a request? Open its trace and follow the critical path to the longest span that is not explained by its children.
• What links logs to traces? Each log record includes the trace ID and span ID of the active span.