Distributed Tracing

Why Distributed Tracing Matters

Logs are per-process. Metrics are aggregated. Neither tells you the causal story of one specific request as it crosses service boundaries. In a monolith you can scroll a single log file and follow a request from top to bottom. In microservices that file lives on twelve hosts, and the only thing that ties the entries together is whatever request-id you remembered to thread through every call. A trace is that thread, made first-class.

What logs and metrics can’t answer

The pattern is clear: anything that asks “why was this specific request slow” or “what path did this specific request take” can only be answered by a trace.

The Trace Model

A trace is a tree of spans. Each span has a start time, an end time, and a parent. The root span is the one with no parent — usually the one created at the edge (API gateway, ingress). Every later span carries the same trace_id, so the backend can reassemble the tree from spans that were emitted by twelve different processes.

Notice the “220 ms” on the HTTP POST inventory.reserve span versus “180 ms” on the inner db.update stock span. The 40 ms gap is network and serialization. That’s a real signal you can act on — only visible because you have spans on both sides of the call.

Context Propagation

A trace without context propagation is just a sad list of logs. The tracing SDK on the calling service serializes the active span context into the outbound request; the SDK on the receiving service extracts it and uses it as the parent of its new server-side span. If any link in the chain forgets to propagate, you get two disconnected traces and the flame graph stops at that boundary.

HTTP — W3C Trace Context

# A traceparent header looks like this:
traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
#             ^^ version
#                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ trace_id (16 bytes hex)
#                                                   ^^^^^^^^^^^^^^^^ parent span_id (8 bytes hex)
#                                                                    ^^ trace flags (01 = sampled)

# tracestate carries vendor-specific extras (Datadog, Honeycomb, etc.):
tracestate: dd=p:00f067aa0ba902b7;s:1,honeycomb=...

W3C Trace Context replaced a decade of incompatible header formats (B3 single-header, B3 multi-header, X-Cloud-Trace-Context, etc.). B3 is still common in the wild — if you have an Istio mesh or older Zipkin instrumentation, you’ll see x-b3-traceid, x-b3-spanid, x-b3-sampled. Modern OpenTelemetry SDKs default to W3C and can be configured to also accept B3.

gRPC and async transports

Manual injection into a Kafka producer

from opentelemetry import trace, propagate
from confluent_kafka import Producer

tracer = trace.get_tracer("orders.publisher")
producer = Producer({"bootstrap.servers": "kafka:9092"})

def publish_order(order: dict) -> None:
    with tracer.start_as_current_span("kafka.publish orders") as span:
        span.set_attribute("messaging.system", "kafka")
        span.set_attribute("messaging.destination", "orders")
        span.set_attribute("order.id", order["id"])

        # Inject the active span context into a plain dict, then convert
        # to Kafka header format (list of (key, value-bytes) tuples).
        carrier: dict = {}
        propagate.inject(carrier)
        headers = [(k, v.encode("utf-8")) for k, v in carrier.items()]

        producer.produce(
            topic="orders",
            key=order["id"].encode(),
            value=json.dumps(order).encode(),
            headers=headers,   # traceparent rides here
        )
        producer.flush()

The consumer side does the mirror: read headers into a carrier dict, call propagate.extract(carrier), and use the result as the parent context when starting the consumer span. Done correctly, the producer span and the consumer span (running in a different process, possibly hours later) appear in the same trace as parent and child.

OpenTelemetry

OpenTelemetry has three pieces you need to understand:

• SDK — runs in your app. Creates spans, adds attributes, manages sampling, batches export.
• Auto-instrumentation — libraries that monkey-patch popular frameworks (FastAPI, Express, Spring, gRPC, requests, redis, psycopg, kafka-python) so you get spans without writing tracer code for every HTTP client.
• Collector — a separate process that receives spans over OTLP, runs processors (batching, sampling, attribute scrubbing), and exports to one or more backends. Lets you change backends without touching apps.

Python — FastAPI auto-instrumentation

# pip install opentelemetry-distro opentelemetry-exporter-otlp \
#             opentelemetry-instrumentation-fastapi \
#             opentelemetry-instrumentation-requests

from fastapi import FastAPI
from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.instrumentation.requests import RequestsInstrumentor
from opentelemetry.sdk.trace.sampling import ParentBased, TraceIdRatioBased

# 1. Identify the service in every span (resource attributes).
resource = Resource.create({
    "service.name": "orders",
    "service.version": "2.4.1",
    "deployment.environment": "production",
})

# 2. 10% head sampler — defer to parent if there is one.
sampler = ParentBased(root=TraceIdRatioBased(0.10))

# 3. Provider + OTLP exporter pointing at the local collector.
provider = TracerProvider(resource=resource, sampler=sampler)
provider.add_span_processor(BatchSpanProcessor(
    OTLPSpanExporter(endpoint="http://otel-collector:4317", insecure=True)
))
trace.set_tracer_provider(provider)

# 4. Auto-instrument the framework and the HTTP client.
app = FastAPI()
FastAPIInstrumentor.instrument_app(app)
RequestsInstrumentor().instrument()

tracer = trace.get_tracer("orders")

@app.post("/checkout")
async def checkout(req: CheckoutReq):
    with tracer.start_as_current_span("checkout.handle") as span:
        span.set_attribute("order.id", req.order_id)
        span.set_attribute("cart.size", len(req.items))
        result = await process(req)
        span.set_attribute("order.total", result.total)
        return result

Node.js — Express auto-instrumentation

// npm i @opentelemetry/sdk-node @opentelemetry/auto-instrumentations-node \
//        @opentelemetry/exporter-trace-otlp-grpc @opentelemetry/resources
// File: tracing.js — require this BEFORE express, http, pg, etc.

const { NodeSDK } = require('@opentelemetry/sdk-node');
const { Resource } = require('@opentelemetry/resources');
const { SemanticResourceAttributes } =
    require('@opentelemetry/semantic-conventions');
const { OTLPTraceExporter } =
    require('@opentelemetry/exporter-trace-otlp-grpc');
const { getNodeAutoInstrumentations } =
    require('@opentelemetry/auto-instrumentations-node');
const { ParentBasedSampler, TraceIdRatioBasedSampler } =
    require('@opentelemetry/sdk-trace-base');

const sdk = new NodeSDK({
    resource: new Resource({
        [SemanticResourceAttributes.SERVICE_NAME]: 'payments',
        [SemanticResourceAttributes.SERVICE_VERSION]: '1.8.0',
        [SemanticResourceAttributes.DEPLOYMENT_ENVIRONMENT]: 'production',
    }),
    traceExporter: new OTLPTraceExporter({
        url: 'http://otel-collector:4317',
    }),
    sampler: new ParentBasedSampler({
        root: new TraceIdRatioBasedSampler(0.10),
    }),
    instrumentations: [getNodeAutoInstrumentations({
        // Skip noisy file-system spans in production
        '@opentelemetry/instrumentation-fs': { enabled: false },
    })],
});

sdk.start();
process.on('SIGTERM', () => sdk.shutdown());

Once tracing.js is loaded first (e.g. node -r ./tracing.js server.js), the auto-instrumentations wrap Express, the http client, pg, redis, kafkajs, ioredis, and dozens more. You get a populated trace tree with attributes on day one, before you write a single manual span.

Where the Collector fits

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 5s
    send_batch_size: 1000

  # Strip PII from spans before they leave your network.
  attributes:
    actions:
      - key: user.email
        action: delete
      - key: http.request.header.authorization
        action: delete

  # Tail-based sampling: see ALL spans of a trace, then decide.
  tail_sampling:
    decision_wait: 10s
    num_traces: 100000
    expected_new_traces_per_sec: 2000
    policies:
      - name: errors-keep-100pct
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: slow-keep-100pct
        type: latency
        latency: { threshold_ms: 2000 }
      - name: baseline-1pct
        type: probabilistic
        probabilistic: { sampling_percentage: 1.0 }

exporters:
  otlphttp/tempo:
    endpoint: https://tempo:4318
  otlphttp/honeycomb:
    endpoint: https://api.honeycomb.io
    headers:
      x-honeycomb-team: ${HONEYCOMB_API_KEY}

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [attributes, tail_sampling, batch]
      exporters: [otlphttp/tempo, otlphttp/honeycomb]

Run the collector as a sidecar (per-pod), a DaemonSet (per-node), or a central gateway. The DaemonSet pattern is the production default — one collector per node, low latency for app exporters, and tail sampling that sees a meaningful slice of traffic.

Sampling

Head-based vs tail-based

Common sampling strategies

• Probabilistic (head) — keep N% of traces by hashing trace_id. Simple, predictable. Default starting point: 1–10%.
• Rate-limited (head) — keep at most N traces/sec per service. Caps cost; useful at the edge.
• Parent-based (head) — honor the upstream sampling decision (the 01 in traceparent). Without this, half your trace is sampled and half isn’t — useless.
• Error-aware (tail) — keep 100% of traces with status=ERROR.
• Latency-aware (tail) — keep 100% of traces over P95 latency.
• Attribute-based (tail) — keep 100% of traces from VIP customers, new releases, debug headers, paid tiers.

Reading Traces in Practice

The shapes that signal trouble

Pay attention to the difference between span duration and self time (duration minus time spent in children). A parent span that is 5 s wide but spends 4.9 s in one child has 0.1 s of self time — the child is the problem, not the parent.

Storage Backends and Tools

The OTLP wire protocol means your apps don’t care which backend you pick. The backends differ in storage cost, query model, and how cleanly they integrate with metrics and logs.

Real-World Examples

Google — Dapper (2010). The paper that started the field. Dapper described how Google instruments every RPC, propagates a trace ID through every layer of its stack, and samples aggressively (often well below 1%) because the absolute volume is enough to keep traces statistically useful. The key insight in the paper: tracing has to be always-on, low-overhead, and application-transparent, or engineers will turn it off.

Twitter — Zipkin (2012). Twitter open-sourced its Dapper-inspired tracer. Zipkin defined the B3 propagation headers that dominated the next decade, built the basic UI vocabulary (trace search, span detail, dependency graph) that every later tool copied, and proved that distributed tracing was a generally useful tool, not a Google-only luxury.

Uber — Jaeger (2015). Uber needed something that scaled to thousands of services and donated the result to the CNCF. Jaeger added adaptive sampling (per-service, per-operation rates that the backend computes and pushes down to clients), and remains the reference open-source tracer.

Honeycomb — observability 2.0. Honeycomb’s framing is that the fundamental unit shouldn’t be the metric or the log line; it should be the wide, structured event — which is, basically, a span with a lot of attributes. Their pitch: keep raw events with high cardinality and you can ask questions you didn’t know you’d need to ask. This is why “adding more attributes to your spans” is a core debugging technique now, not a quaint suggestion.

OpenTelemetry — the CNCF merger. OpenTracing (2016) and OpenCensus (2017) competed for years. In 2019 they merged into OpenTelemetry, which is now the second-most-active CNCF project after Kubernetes. Every major vendor — Datadog, New Relic, Honeycomb, Splunk, Dynatrace, AWS, Google, Azure — ships OTel-native ingestion. The format war is over.

Best Practices