How to use auto-instrumentation for Uvicorn with multiple worker processes

See original GitHub issue

Describe your environment

Using the following Dockerfile to simplify testing:

FROM tiangolo/uvicorn-gunicorn-fastapi:python3.8-slim

RUN pip install --no-cache-dir \
      opentelemetry-distro==0.19b0 \
      opentelemetry-instrumentation-fastapi==0.19b0

Save the file as Dockerfile, and build the Docker image it using the following command in the same folder:

docker build -t test-uvicorn .

Steps to reproduce

The Uvicorn server only has one registered endpoint at the root path. Running the server, and then accessing the endpoint (in this example, at http://localhost:5000/) will display any collected spans in the terminal.

Running the Uvicorn server with multiple workers disabled, and OpenTelemetry auto-instrumentation:

docker run --rm -p 5000:5000 test-uvicorn opentelemetry-instrument --trace-exporter console_span uvicorn main:app --host 0.0.0.0 --port 5000

Running the Uvicorn server with multiple workers enabled, and OpenTelemetry auto-instrumentation:

docker run --rm -p 5000:5000 test-uvicorn opentelemetry-instrument --trace-exporter console_span uvicorn main:app --host 0.0.0.0 --port 5000 --workers 2

What is the expected behavior?

Both modes (single-worker, and multi-worker) must correctly auto-instrument the application, and display collected spans.

What is the actual behavior?

Only the single-worker command displays collected spans. Multi-worker Uvicorn does not display any spans.

Additional context

Uvicorn uses multiprocessing to spawn workers. (https://github.com/encode/uvicorn/blob/bf1c64e2c1/uvicorn/main.py#L381-L384, and https://github.com/encode/uvicorn/blob/bf1c64e2c1/uvicorn/subprocess.py#L38). Could be related to the same issue that is blocking Gunicorn from working (#171).

Submitting the issue here, as apparently the auto-instrumentation code will be moved to this repository, based on https://github.com/open-telemetry/opentelemetry-python/issues/1532.

Issue Analytics

  • State:open
  • Created 2 years ago
  • Comments:13 (8 by maintainers)

github_iconTop GitHub Comments

1reaction
srikanthccvcommented, Jul 30, 2022

Please try running with latest rc2 version and let us know if it changes anything.

0reactions
bilbofcommented, Jul 31, 2022

The problem is caused by Uvicorn using multiprocessing.spawn rather than multiprocessing.fork (which uses os.fork), so the OTel post fork hook isn’t running.

It’d be great if we could fix this here rather than at uvicorn, since others are experiencing the same problem with multiprocessing (e.g. #2185). Would be interested in your thoughts on how to resolve this.

Also, is the fork process model doc still accurate? The workaround in this case is to use gunicorn to run uvicorn apps (which is recommended by Uvicorn) which I’ve confirmed has worked (see my demo). So issue #2038 could be closed.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Deployment - Uvicorn
When running locally, use --reload to turn on auto-reloading. The --reload and --workers arguments are mutually exclusive.
Read more >
Working With Fork Process Models - OpenTelemetry Python
The BatchSpanProcessor spawns a thread to run in the background to export spans to the telemetry backend. During the fork, the child process...
Read more >
Server Workers - Gunicorn with Uvicorn - FastAPI
When deploying applications you will probably want to have some replication of processes to take advantage of multiple cores and to be able...
Read more >
Setting up OpenTelemetry in Django gUnicorn ASGI
I was working on integrating Signoz APM in one of the k8s clusters ... Sample Worker processes workers = 4 worker_class = "uvicorn.workers....
Read more >
Automatic Instrumentation - Python - OpenTelemetry
Overview This example demonstrates how to use auto-instrumentation in ... This reduces the amount of work required to integrate OpenTelemetry into your ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found