Distributed Tracing 101: Following a Request Across a Hundred Services

Logs tell you what happened inside one service. Metrics tell you the aggregate shape of your system. Only tracing tells you the story of a single request’s journey.

Picture a checkout request in a modern e-commerce platform. It hits an API gateway, which calls an authentication service, which calls a user profile service, which calls an inventory service, which calls a pricing service, which calls a promotions engine, which calls a payment gateway, which calls a fraud detection service, which finally writes to an order database and publishes an event that seventeen other services subscribe to. That’s not a hypothetical — that’s a fairly ordinary architecture in 2025.

Now imagine that request takes 4 seconds instead of the usual 300ms. Which of those nine hops is slow? Your logs are scattered across nine different services, each with its own format, its own timestamps (possibly out of sync by a few hundred milliseconds), and no shared identifier connecting them. Your dashboards show aggregate latency per service, but none of them show this specific request’s path through the system. This is the exact problem distributed tracing was built to solve.

The Core Idea: Trace, Span, and Context Propagation

A trace represents the entire journey of a single request through your system, from the moment it enters at the edge to the moment the last downstream effect completes. A trace is composed of spans — each span represents one unit of work, typically corresponding to one service handling one operation (an HTTP call, a database query, a queue publish).

Every span carries:

• A trace ID, shared by every span in the same request’s journey — this is what lets you reconstruct the whole story later.
• A span ID, unique to that specific unit of work.
• A parent span ID, pointing to whichever span caused this one to start — this is what lets you rebuild the tree structure, not just a flat list of events.
• Timing data: start time and duration, which is what turns the tree into a waterfall diagram you can actually read.
• Attributes: arbitrary key-value metadata — HTTP status code, database query text, user ID, cache hit/miss, whatever context matters for debugging later.

The mechanism that makes all of this possible across service boundaries is context propagation: when Service A calls Service B, it must pass the trace ID and its own span ID (as the new parent) along with the request, typically as HTTP headers. If any service in the chain fails to forward these headers, the trace breaks at that point, and you get two disconnected traces instead of one continuous story. This single failure mode — a missing header forward — is responsible for the overwhelming majority of “why doesn’t my tracing show the full picture” support tickets in every tracing system ever built.

Why OpenTelemetry Changed the Calculus

For years, distributed tracing was hobbled by vendor lock-in. Instrumenting your code meant importing a specific vendor’s SDK, and switching tracing backends later meant re-instrumenting your entire codebase. OpenTelemetry (OTel) fixed this by standardizing the instrumentation layer separately from the backend that stores and visualizes the data.

In practice, this means you instrument your code once, using OTel’s API and SDK, and then configure an exporter to send that data to whichever backend you choose — Jaeger, Tempo, Honeycomb, Datadog, or a dozen others — without touching application code. Switching backends becomes a configuration change, not an engineering project. This is the single biggest reason tracing adoption has accelerated industry-wide over the last several years: the fear of lock-in that used to make tracing a hard sell to engineering leadership has largely evaporated.

OpenTelemetry also unified the three pillars of observability — traces, metrics, and logs — under a common data model, with shared context propagation. This means a log line emitted inside a span can automatically carry that span’s trace ID, letting you jump from “I found a suspicious log line” directly to “here is the full trace this log line belongs to” without manual correlation.

Instrumentation: Automatic vs. Manual

Most OpenTelemetry SDKs offer automatic instrumentation for common frameworks and libraries — your HTTP server, your database client, your message queue consumer. This is usually done through language-specific mechanisms (bytecode manipulation in Java, monkey-patching in Python and Node.js, middleware injection in Go) and requires little more than adding a dependency and an initialization call. Automatic instrumentation gets you 70-80% of the value for maybe 5% of the effort, and it is where every team should start.

The remaining 20-30% comes from manual instrumentation — spans you create explicitly around business logic that matters. Automatic instrumentation will happily tell you that a database call took 40ms, but it won’t tell you that the 200ms your code spent before that call was actually a slow, unnecessary JSON deserialization of a bloated response from a previous step. Manual spans around meaningful chunks of business logic — “calculate shipping options,” “apply promotion rules,” “run fraud scoring” — are what turn a trace from “a list of network calls” into “an actual explanation of where the time went.”

A practical rule: instrument any function that takes a non-trivial, variable amount of time and that a human would plausibly ask “why did this take so long?” about during an incident. Don’t instrument every function — a trace with 400 spans for a single request is as unreadable as no trace at all.

Reading a Trace: The Waterfall View

Once you have traces flowing, the primary interface for reading them is the waterfall (or Gantt-chart) view: each span rendered as a horizontal bar, positioned according to its start time and sized according to its duration, nested under its parent span.

A few patterns become immediately visible once you’re looking at real traces:

• Sequential chains that should be parallel. If Service A calls Service B, waits for the full response, then calls Service C, waits for the full response, and neither call depends on the other’s result, the waterfall will show two bars laid end to end. This is one of the most common and most fixable sources of latency in microservice architectures — the fix is often as simple as issuing both calls concurrently and awaiting both.
• The “long tail” child span. Sometimes a parent span’s duration is dominated by one child that takes 900ms out of a total 950ms, while five other children each take under 10ms. The waterfall makes this instantly obvious in a way that aggregate service-level dashboards never could, because those dashboards show you the service’s average latency, not this specific request’s bottleneck.
• Gaps between spans. Time that isn’t accounted for by any child span usually means work is happening in the parent that hasn’t been instrumented yet — serialization, queueing, or waiting on a lock. A trace with large unexplained gaps is a strong hint that manual instrumentation needs to go deeper.

Sampling: You Cannot (and Should Not) Trace Everything

At meaningful scale, capturing a full trace for every single request is prohibitively expensive, both in the compute overhead of the instrumentation and in the storage and query cost of the backend. This is where sampling strategy becomes a real engineering decision, not an afterthought.

Head-based sampling decides whether to trace a request at the very start, usually based on a random percentage (say, 1% of all traffic) or a simple rule (always trace requests from internal test accounts). It’s cheap and simple, but it has a critical weakness: you decide to sample before you know whether the request will be interesting. A rare, slow, error-producing request has the same 1% chance of being captured as a completely uninteresting one.

Tail-based sampling fixes this by buffering spans for a request until it completes, then deciding retroactively whether to keep the trace — keep 100% of traces that resulted in an error or exceeded a latency threshold, and only a small percentage of the boring, fast, successful ones. This gives you far better signal for debugging at a fraction of the storage cost of tracing everything, but it requires a buffering layer (typically an OpenTelemetry Collector configured for tail sampling) that holds spans in memory until the full trace resolves, which adds infrastructure complexity and a small amount of latency to the export pipeline.

Most mature observability setups end up with a hybrid: a low fixed-percentage head sample for general visibility into “normal” traffic patterns, plus a tail-based rule that guarantees capture of anything that errors or breaches a latency SLO. This combination means you never lose the trace that would have explained your worst incident, while keeping storage costs bounded.

Tracing and Incident Response

The real payoff of tracing shows up during an incident, not during a calm afternoon of dashboard-browsing. When a customer reports “checkout is slow for me,” the traditional response is a scavenger hunt across nine services’ logs, hoping someone thought to log the user ID somewhere useful. With tracing, the response becomes: find the trace for that specific request (often searchable by user ID or request ID if you’ve attached it as a span attribute), open the waterfall, and look directly at which of the nine hops ate the four seconds.

This shift — from “search for clues across disconnected systems” to “look at the one artifact that already contains the whole story” — is the single biggest reduction in mean time to diagnosis that most teams experience when they move from logs-only observability to a tracing-enabled stack. It doesn’t replace logs or metrics; it sits between them, giving you the narrative thread that ties isolated log lines and aggregate graphs into the specific story of what happened to one unlucky request, and by extension, what’s likely happening to every request behind it.

Trace-Based Metrics: Getting the Best of Both Worlds

One increasingly common pattern worth knowing about is deriving metrics directly from spans, sometimes called trace-based or span metrics. Instead of choosing between the low cardinality but low fidelity of pre-aggregated metrics and the high fidelity but high cost of storing every trace, this approach computes aggregate statistics such as request counts, error rates, and latency histograms from the stream of spans as they’re collected, typically inside the OpenTelemetry Collector, before deciding which individual traces to retain in full detail.

This means you get accurate, dashboard-friendly aggregate metrics, including per-endpoint, per-customer, or per-region breakdowns that would be prohibitively expensive to maintain as hand-instrumented custom metrics, derived automatically from the same instrumentation you already added for tracing, with no separate metrics instrumentation effort required. It’s a genuinely efficient reuse of a single instrumentation investment across two of the three observability pillars, and it’s one of the more compelling reasons teams increasingly treat tracing as the primary instrumentation layer, with metrics and even structured logs treated as derived views rather than independently maintained signals.

Getting Started Without Boiling the Ocean

If you’re introducing tracing to an existing system, resist the urge to instrument everything on day one. A pragmatic rollout looks like:

1. Deploy an OpenTelemetry Collector as a central ingestion point, even before every service is instrumented — this gives you a stable place to route data regardless of which backend you eventually choose.
2. Add automatic instrumentation to your edge service (API gateway or load balancer entry point) first, so every trace has a clear root span.
3. Add automatic instrumentation to the two or three services most frequently implicated in past incidents — this is where tracing pays for itself fastest.
4. Verify context propagation is working end-to-end before adding more services; a single broken hop undermines the value of every other hop you’ve instrumented.
5. Only then expand outward, service by service, adding manual spans around business logic as you learn where the automatic instrumentation leaves gaps.

Tracing is not a dashboard you glance at once a week. It’s the diagnostic tool you reach for at 2 AM when a request is slow and nobody knows why — and the only observability signal that can answer that question by showing you the actual, specific path a real request took through your real system.