Skip to content

Grokem · https://www.grokem.com/programming/software-architecture/microservices/distributed-tracing-with-opentelemetry-in-python-trace-one-request-across-two-services

MicroservicesTutorialIntermediate level

Distributed tracing with OpenTelemetry in Python: trace one request across two services

Build two Flask services that pass W3C Trace Context headers, so one request produces spans that share a single trace. You check the result by comparing trace data in two terminals.

Krishna Sharma

Technical Writer

Published
  • 25 min to complete
  • 0 sec

Tested with
  • Python3.10
  • opentelemetry-api
  • opentelemetry-sdk
  • Flask
  • requests
Your progress
  1. Distributed tracing with OpenTelemetry in Python: trace one request across two services7 min(finished)
    0%

Before you start

5
  • Python 3.10 or higher, the minimum the OpenTelemetry Python documentation lists
  • pip and the ability to create a virtual environment
  • Three terminal windows on one machine
  • Free local ports 5001 and 5002
  • Working knowledge of Flask and HTTP headers

A distributed trace records the path of a single request as it moves through several services. It is made of spans, and each span is one unit of work. For the spans from different services to join one trace, the caller must pass its trace context to the callee. That mechanism is called propagation.

In this tutorial you build the smallest system that shows it: two Flask services, api1 and api2. api1 starts a span, injects the current context into HTTP headers using the W3C Trace Context format, and calls api2. api2 extracts that context and starts its own span inside the same trace. Both services print their spans with ConsoleSpanExporter, so you can check the result without running a backend.

The OpenTelemetry Python documentation recommends instrumentation libraries or auto-instrumentation for real systems, because they propagate context for you. You inject and extract by hand here so you see exactly what crosses the service boundary. The code follows the manual propagation example in the official Python propagation page.

Versions: the instructions are written for Python 3.10, the minimum version the OpenTelemetry Python page states. That page lists the Python tracing API and SDK as stable. It does not pin package versions for the example, so install the current releases of opentelemetry-api and opentelemetry-sdk from PyPI and note them for your records.

Step 1 of 9: Create a virtual environment and install the packages

Create an isolated environment, activate it with the command for your shell, and install the OpenTelemetry API and SDK plus Flask and requests. The API and SDK are the two packages the OpenTelemetry Python page installs with pip. Flask and requests are the web framework and HTTP client the example imports.

A virtual environment keeps the tutorial's dependencies away from your system Python. The commands use bash syntax, so adjust the activation line if you use another shell.

The flow you are about to build looks like this:

one trace, same trace IDcurlGETapi1port 5002api1_spantraceparent andbaggage headersapi2port 5001api2_spanConsoleSpanExporterConsoleSpanExporter
One request, two services, one trace.
bash
python -m venv .venv
source .venv/bin/activate
pip install opentelemetry-api opentelemetry-sdk flask requests
python --version

Step 2 of 9: Write the receiving service api2.py

Create api2.py. It listens on port 5001, reads the traceparent and baggage headers, extracts them into a context, and starts api2_span with that context as its parent.

Passing context=ctx2 to start_span is what attaches this span to the caller's trace. Without it, api2 would begin a new, unrelated trace.

The service also sets up its own TracerProvider with a BatchSpanProcessor and ConsoleSpanExporter. Each service in a real system configures its own SDK, so this mirrors production structure.

python
from flask import Flask, request
from opentelemetry import trace, baggage
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import ConsoleSpanExporter, BatchSpanProcessor
from opentelemetry.baggage.propagation import W3CBaggagePropagator

app = Flask(__name__)

trace.set_tracer_provider(TracerProvider())
trace.get_tracer_provider().add_span_processor(BatchSpanProcessor(ConsoleSpanExporter()))
tracer = trace.get_tracer(__name__)

@app.route('/')
def hello():
    # Log the headers received from API 1
    headers = dict(request.headers)
    print(f"Received headers: {headers}")

    carrier = {'traceparent': headers['Traceparent']}
    ctx = TraceContextTextMapPropagator().extract(carrier=carrier)
    print(f"Received context: {ctx}")

    b2 = {'baggage': headers['Baggage']}
    ctx2 = W3CBaggagePropagator().extract(b2, context=ctx)
    print(f"Received context2: {ctx2}")

    # Start a new span inside the propagated trace
    with tracer.start_span("api2_span", context=ctx2):
        print(baggage.get_baggage('hello', ctx2))
        return "Hello from API 2!"

if __name__ == '__main__':
    app.run(port=5001)

Step 3 of 9: Write the sending service api1.py

Create api1.py. It listens on port 5002. Inside api1_span it sets one baggage item, injects baggage and trace context into an empty headers dict, prints the dict, and calls api2 with those headers.

Injection must happen while the span is current. The injected traceparent then identifies api1_span as the parent for whatever the receiver does next.

python
from flask import Flask
import requests
from opentelemetry import trace, baggage
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator
from opentelemetry.baggage.propagation import W3CBaggagePropagator
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import ConsoleSpanExporter, BatchSpanProcessor

app = Flask(__name__)

trace.set_tracer_provider(TracerProvider())
trace.get_tracer_provider().add_span_processor(BatchSpanProcessor(ConsoleSpanExporter()))
tracer = trace.get_tracer(__name__)

@app.route('/')
def hello():
    with tracer.start_as_current_span("api1_span") as span:
        ctx = baggage.set_baggage("hello", "world")
        headers = {}
        W3CBaggagePropagator().inject(headers, ctx)
        TraceContextTextMapPropagator().inject(headers, ctx)
        print(headers)
        response = requests.get('http://127.0.0.1:5001/', headers=headers)
        return f"Hello from API 1! Response from API 2: {response.text}"

if __name__ == '__main__':
    app.run(port=5002)

Step 4 of 9: Start the receiving service in terminal 1

Run api2.py first and leave it running. The sender calls it on every request, so it must already be listening on port 5001.

bash
source .venv/bin/activate
python api2.py

Step 5 of 9: Start the sending service in terminal 2

Run api1.py in a second terminal with the same virtual environment active. This is the entry point of the request, so it creates the root span of the trace.

bash
source .venv/bin/activate
python api1.py

Step 6 of 9: Send one request to api1 from terminal 3

Call api1 once. A single request gives you one trace to inspect, which makes the comparison in the next step unambiguous.

bash
curl http://127.0.0.1:5002/

Step 7 of 9: Compare the trace identifier in both terminals

Look at terminal 2 (api1). It printed the headers dict, which contains a traceparent entry. In the W3C format that value carries the trace ID and the ID of the span that made the call.

Then look at the span output in each terminal. The console exporter prints finished spans after a short batching delay, so wait a few seconds. If nothing appears, stop the service with Control-C, which flushes the batch.

You are checking that api1_span in terminal 2 and api2_span in terminal 1 show the same trace ID, and that api2_span records api1_span as its parent. The exact field layout of the console output depends on your SDK release, so read the trace and parent fields it prints rather than relying on a fixed format.

The same trace ID on both sides is the proof that the request was followed across the service boundary.

Step 8 of 9: Confirm that baggage crossed the boundary

In terminal 1, find the line printed by print(baggage.get_baggage('hello', ctx2)). It should read world.

This shows that baggage, which travels in its own baggage header next to traceparent, was injected by api1 and extracted by api2. Trace context and baggage are separate propagators, and the example uses both.

Step 9 of 9: Replace the manual code with instrumentation libraries in your own services

For your real services, switch from hand-written inject and extract calls to the instrumentation libraries. The OpenTelemetry Python page states that libraries for frameworks such as Flask and Django propagate context across services for you, and that they are better tested and easier to use than manual propagation.

The propagation page does not list the install commands for each library. Search the OpenTelemetry Registry, linked from the Python documentation, for the package that matches your framework. The opentelemetry-instrumentation-{instrumentation} naming pattern is the one the Python page documents.

Keep the manual version only for places no library covers, such as a custom transport.


Expected result

curl prints: Hello from API 1! Response from API 2: Hello from API 2! Terminal 2 (api1) prints the injected headers dict, including traceparent and baggage entries. Terminal 1 (api2) prints the received headers, the extracted contexts, and the line world. After a short batching delay, each terminal prints one span from ConsoleSpanExporter: api1_span in terminal 2 and api2_span in terminal 1. Both spans carry the same trace ID, and api2_span names api1_span as its parent.

Troubleshooting

ProblemLikely causeFix
KeyError: 'Traceparent' in the api2 terminalapi2 was called directly, for example from a browser or curl on port 5001, so the trace headers were never injected.Send requests to api1 on port 5002. api1 adds the headers and forwards the call.
requests raises a connection error and curl returns a 500 from api1api2 is not running or is not listening on 127.0.0.1:5001.Start api2.py first and keep terminal 1 open, then repeat the request.
Address already in use when starting a serviceAnother process holds port 5001 or 5002.Stop the other process, or change the port in app.run and in the requests.get URL in api1.py so both agree.
No span output appears in either terminalBatchSpanProcessor exports spans in batches, so output is delayed until the batch is flushed.Wait a few seconds, or stop the service with Control-C to flush pending spans.
The two spans show different trace IDsapi2 started its span without the extracted context, or extraction used the wrong header, so it began a new trace.Check that api2.py passes context=ctx2 to start_span and that the traceparent printed by api1 matches the Received headers line in api2.
ModuleNotFoundError for opentelemetry, flask or requestsThe virtual environment is not active or the packages were installed elsewhere.Activate .venv in that terminal and rerun the pip install command from step 1.

Frequently asked questions

Do I have to inject and extract context by hand?

No. The OpenTelemetry Python documentation recommends instrumentation libraries, which propagate context across services for you. Libraries exist for frameworks such as Flask, Django and Celery. Manual propagation is shown here to make the mechanism visible.

Which header format carries the trace between services?

OpenTelemetry propagates context with the W3C Trace Context HTTP headers. This tutorial also sends baggage in its own header.

Do I need a Collector or backend to follow a request?

Not for this check. ConsoleSpanExporter prints spans to each service's terminal. For a shared view of all spans, send telemetry to a Collector or backend, as described in the Collector quick start.

Which Python version does OpenTelemetry Python support?

The Python documentation states that OpenTelemetry-Python supports Python 3.10 and higher.

Are Python traces stable?

Yes. The language status table lists traces as stable for Python. Logs are listed as in development.

Can propagation reach a database?

Some Python instrumentations support sqlcommenter, which appends key-value pairs such as traceparent to query statements. This lets context reach database logs when they are enabled.

Next steps

  • Send spans to the OpenTelemetry Collector. The Collector quick start runs the otel/opentelemetry-collector:0.165.0 image with OTLP ports 4317 (gRPC) and 4318 (HTTP) open. Find the OTLP exporter packag
  • Add more services to the chain and confirm the trace ID stays the same at every hop.
  • Replace manual propagation with the Flask instrumentation library, and read the Python manual instrumentation page for custom spans and attributes.
  • Read the observability primer to connect traces with logs and metrics.

References

From Grokem: https://www.grokem.com/programming/software-architecture/microservices/distributed-tracing-with-opentelemetry-in-python-trace-one-request-across-two-services

Finished this article?

How did this strike you?

Be the first to react – it is anonymous.

Community

Questions about this?

Ask in Questions and help and get help from other readers.

Ask a questionBrowse Questions and help

Comments

No comments yet. Share what worked for you, or ask a question.

Keep learning

Picked for this article

Related reading, discussions and learning paths on the same subject.

All in Microservices

From the community

Learning paths

Learning pathBeginner

Infrastructure basics for data work

The ground a data project stands on: the command line, how a name finds a server, packaging an app in a container, where files and databases live, keeping secrets out of code, backups you have tested and a cloud bill you control.

7 lessonsabout 15 minStart

Choose what we may use. You can change this at any time with “Privacy choices” at the foot of every page. Details are in the Cookie policy and the Privacy policy.

Strictly necessary

Remembers your theme, the lessons you marked as done and this choice. It stays on your device, tracks nothing and cannot be switched off.

Always on
What exactly is collected, and why

Google Analytics

Collects
The pages you view, your device and browser type, your language, how you reached the site and your approximate location (country and city, worked out from your IP address).
Why
To understand which pages and lessons are used, so we can improve them.
Stored
Two cookies, _ga and _ga_<id>, for up to 2 years. Google acts for us; the data is processed by Google, including outside your country.

Grokem’s own counts

Collects
The pages you open and how long you stay, how far you scroll, where you came from, your device type, browser, screen size, language and time zone. Your IP address is turned into a code that changes every day; it is never stored as it is.
Why
To see how long lessons are read and where readers leave. Not shared with anyone.
Stored
A random visitor ID, a visit ID and the time of your last page view in your browser’s storage (gr_vid, gr_sid, gr_last). No name or email is attached.