Distributed tracing

How do you follow one user's request through forty services that each know only their own part?

A user says the checkout page took nine seconds. Behind that page sit forty services. Each has excellent logs, and each reports a median response time in the low tens of milliseconds. Every service is, by its own account, healthy and fast.

So where did the nine seconds go? Here is the difficulty in its purest form: no participant in the system has ever seen the request. Each saw a request — its own inbound call and its own outbound calls — and none of them has any way of knowing that those calls belonged to the same user's journey. The information is not missing from the system. It is scattered across forty places with nothing to join it on.

Suppose you set out to reconstruct the journey by hand. You would open the checkout service's logs and find an entry near the right time. It called three services. You would open each of those, find entries near their right times, and continue. Two problems appear at once. Timestamps alone are ambiguous — at a thousand requests per second, "near the right time" matches hundreds of candidates. And the fan-out grows, so by four levels deep you are guessing among thousands.

What single change would collapse that ambiguity to nothing? Give the request a name at the front door, and make every service that touches it record that name.

That is the mechanism, and everything else is engineering around it. On arrival, the system mints an identifier — a trace ID — and each unit of work records its own entry stamped with that ID. The join is now exact rather than probabilistic: fetch every record carrying that trace ID and you have the whole journey, no matter how many services contributed.

But a flat bag of forty records is not yet a picture. What else must each record carry to reconstruct the shape of the call graph?

Each needs its own identity, and a pointer to whoever called it. So each unit of work — a span — carries its own span ID and a parent span ID. Trace ID says which journey; parent ID says which branch of it. With those three fields plus a start time and a duration, the tree assembles itself. The convention in wide use, W3C Trace Context, carries exactly this in a header called traceparent.

Now the hard part, and the reason this is more of a discipline than a feature. How does the trace ID get from one service to the next?

It has to travel in the request, which means every hop must deliberately read the incoming context and attach it to every outgoing call — in a header over HTTP, in message metadata over a queue. And here the practical failure appears: a single service that receives the context and does not forward it breaks the chain. Everything downstream of that point starts a new trace, so the picture ends abruptly and the missing nine seconds may sit entirely inside the region that vanished. Context propagation is the whole game, and it is where partially-instrumented systems fail.

Two more consequences follow from the design.

The first is cost. A span per unit of work per request is an enormous volume of data, so systems sample. Head-based sampling decides at the front door — keep one percent, discard the rest — which is cheap and predictable but throws away the rare slow request you most wanted. Tail-based sampling buffers the whole trace and decides after the fact, keeping everything that errored or ran long; far more useful, and considerably more expensive to operate because something must hold the spans until the trace completes.

The second is what a trace shows you. Because each span carries a start time and a duration, a trace shows both sequence and concurrency — precisely what the individual medians could not. Nine seconds might be one service taking nine seconds, or sixty sequential calls of a hundred and fifty milliseconds that ought to have run in parallel. Both are invisible in per-service averages and obvious in a trace.

One honest limitation. Clock skew between machines is real, so span timings from different hosts are not perfectly comparable, and a child span can occasionally appear to start before its parent. The parent-child structure is reliable because it comes from explicit IDs rather than from timing; the fine-grained overlap of two spans on different machines is less so.

Three refinements.

The first is a boundary worth being firm about. Tracing tells you where time went and which call failed. It rarely tells you why. A span showing eight seconds in one database call is the end of tracing's contribution; the reason lives in that service's own logs, metrics, and query plans. Tracing is a locator, and treating it as a root-cause tool leads to disappointment.

The second corrects a common assumption: that adding tracing is a matter of switching it on. Instrumentation is only as complete as its least-instrumented hop, and the value is deeply non-linear — traces that break in the middle are close to useless, while traces reaching end to end are transformative. Hence a discipline centred on propagation coverage rather than on span detail.

The third connects tracing to its neighbours. A span is a wide structured event with a parent pointer, so a good tracing setup gives you high-cardinality attributes for free — customer, region, build, feature flag. That is exactly the data metrics could not afford to carry. Tracing is where identity belongs, precisely because it never pre-aggregates.

• W3C Trace Context in detail: the traceparent and tracestate header formats.
• Span attributes as the high-cardinality home metrics cannot afford.

Every term the collection defines is gathered in the glossary.