Running uvicorn with watchfiles caused CPU usage >= 99% in docker container

See original GitHub issue

Description

We have a fairly simple ASGI app served using uvicorn. To facilitate debug, we mount git repository in the host to docker. However, when running uvicorn with command

uvicorn main:app --port 8100 --host 0.0.0.0 --reload

will cause cpu to run at 100%, even if the app crashed with ImportError or other error. In such case, Ctrl+C will not be able to terminate uvicorn.

I tried to use command such as

watchfiles "sleep 100" provider_api

With or without WATCHFILES_FORCE_POLLING=1, no such error occurred.

docker file python:3.8 watchfiles version 0.16.1 uvicorn version 0.18.2

The host runs on Oracle ampere arm64 instance. It might having similar issue with M1 Mac?

Example Code

root@4af5866a294c:/# uvicorn main:app --port 8100 --host 0.0.0.0 --reload
INFO:     Will watch for changes in these directories: ['/']
INFO:     Uvicorn running on http://0.0.0.0:8100 (Press CTRL+C to quit)
INFO:     Started reloader process [170] using WatchFiles
Process SpawnProcess-1:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.8/site-packages/uvicorn/_subprocess.py", line 76, in subprocess_started
    target(sockets=sockets)
  File "/usr/local/lib/python3.8/site-packages/uvicorn/server.py", line 60, in run
    return asyncio.run(self.serve(sockets=sockets))
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/uvicorn/server.py", line 67, in serve
    config.load()
  File "/usr/local/lib/python3.8/site-packages/uvicorn/config.py", line 479, in load
    self.loaded_app = import_from_string(self.app)
  File "/usr/local/lib/python3.8/site-packages/uvicorn/importer.py", line 24, in import_from_string
    raise exc from None
  File "/usr/local/lib/python3.8/site-packages/uvicorn/importer.py", line 21, in import_from_string
    module = importlib.import_module(module_str)
  File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/provider_api/main.py", line 25, in <module>
    from .models.slot_profile import SlotBulkUpdateRequest
ImportError: attempted relative import with no known parent package

Watchfiles Output

$ watchfiles 'echo reloaded' . --verbose
[02:26:04] watchfiles v0.16.1 👀  path="/" target="echo reloaded" (command) filter=DefaultFilter...
[02:26:04] running "echo reloaded" as command
reloaded
watcher: INotifyWatcher { channel: Sender { .. }, waker: Waker { inner: Waker { fd: File { fd: 5, path: "anon_inode:[eventfd]", read: true, write: true } } } }
[02:26:10] rust notify timeout, continuing
[02:26:15] rust notify timeout, continuing
raw-event: Event { kind: Modify(Metadata(Any)), paths: ["/run.py"], attr:tracker: None, attr:flag: None, attr:info: None, attr:source: None }
raw-event: Event { kind: Access(Close(Write)), paths: ["/run.py"], attr:tracker: None, attr:flag: None, attr:info: None, attr:source: None }
[02:26:17] 1 change detected: {(<Change.modified: 2>, '/run.py')}
[02:26:17] process already dead, exit code: 0
reloaded
[02:26:22] rust notify timeout, continuing

Operating System & Architecture

Linux-5.15.0-1016-oracle-aarch64-with-glibc2.35 #20-Ubuntu SMP Mon Aug 8 07:08:08 UTC 2022

Environment

Docker

Python & Watchfiles Version

python: 3.9.7 | packaged by conda-forge | (default, Sep 2 2021, 18:58:59) [GCC 9.4.0], watchfiles: 0.16.1

Rust & Cargo Version

No response

Issue Analytics

  • State:closed
  • Created a year ago
  • Comments:8 (5 by maintainers)

github_iconTop GitHub Comments

1reaction
baishicommented, Sep 7, 2022

I verified the behavior and it expressed on Oracle Linux on aarch64 and my desktop amd64 Debian.

It looks like there is circular loop in /proc folder and seemingly the aarch64’s way of managing /proc make it easier to encounter circular loop.

Also WATCHFILES_FORCE_POLLING doesn’t solve my problem.

If you think there is nothing can be done on watchfiles side, you can close this ticket.

0reactions
samuelcolvincommented, Sep 11, 2022

Thanks for your understanding.

Read more comments on GitHub >

github_iconTop Results From Across the Web

High CPU usage when using --reload #338 - encode/uvicorn
On my mac, I find this keeps the docker containers running with very low CPU when inactive - 1-3% using ctop with a...
Read more >
tiangolo/uvicorn-gunicorn-fastapi - Docker Image
You can use the config file from the base image as a starting point for yours. WORKERS_PER_CORE. This image will check how many...
Read more >
Settings - Uvicorn
Use the following options to configure Uvicorn, when running from the command line. ... CLI options and the arguments for uvicorn.run() take precedence...
Read more >
Diagnosing high CPU usage on Docker for Mac - Stack Overflow
I'm running a 2018 MBP 2.8 GHz Core i7 with 4 cores. I tried tweaking the number of CPU cores for the Docker...
Read more >
high CPU usage (#345091) · Issues - GitLab.org
0 and runs in a docker. I couldn't find any crontab scrips in this docker. But indeed there was a /tmp/.git directory and...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found