How to use auto-instrumentation for Uvicorn with multiple worker processes
See original GitHub issueDescribe your environment
Using the following Dockerfile to simplify testing:
FROM tiangolo/uvicorn-gunicorn-fastapi:python3.8-slim
RUN pip install --no-cache-dir \
opentelemetry-distro==0.19b0 \
opentelemetry-instrumentation-fastapi==0.19b0
Save the file as Dockerfile, and build the Docker image it using the following command in the same folder:
docker build -t test-uvicorn .
Steps to reproduce
The Uvicorn server only has one registered endpoint at the root path. Running the server, and then accessing the endpoint (in this example, at http://localhost:5000/) will display any collected spans in the terminal.
Running the Uvicorn server with multiple workers disabled, and OpenTelemetry auto-instrumentation:
docker run --rm -p 5000:5000 test-uvicorn opentelemetry-instrument --trace-exporter console_span uvicorn main:app --host 0.0.0.0 --port 5000
Running the Uvicorn server with multiple workers enabled, and OpenTelemetry auto-instrumentation:
docker run --rm -p 5000:5000 test-uvicorn opentelemetry-instrument --trace-exporter console_span uvicorn main:app --host 0.0.0.0 --port 5000 --workers 2
What is the expected behavior?
Both modes (single-worker, and multi-worker) must correctly auto-instrument the application, and display collected spans.
What is the actual behavior?
Only the single-worker command displays collected spans. Multi-worker Uvicorn does not display any spans.
Additional context
Uvicorn uses multiprocessing to spawn workers. (https://github.com/encode/uvicorn/blob/bf1c64e2c1/uvicorn/main.py#L381-L384, and https://github.com/encode/uvicorn/blob/bf1c64e2c1/uvicorn/subprocess.py#L38).
Could be related to the same issue that is blocking Gunicorn from working (#171).
Submitting the issue here, as apparently the auto-instrumentation code will be moved to this repository, based on https://github.com/open-telemetry/opentelemetry-python/issues/1532.
Issue Analytics
- State:
- Created 2 years ago
- Comments:13 (8 by maintainers)
Top Related StackOverflow Question
Please try running with latest rc2 version and let us know if it changes anything.
The problem is caused by Uvicorn using multiprocessing.spawn rather than multiprocessing.fork (which uses os.fork), so the OTel post fork hook isn’t running.
It’d be great if we could fix this here rather than at uvicorn, since others are experiencing the same problem with multiprocessing (e.g. #2185). Would be interested in your thoughts on how to resolve this.
Also, is the fork process model doc still accurate? The workaround in this case is to use gunicorn to run uvicorn apps (which is recommended by Uvicorn) which I’ve confirmed has worked (see my demo). So issue #2038 could be closed.