PoolTimeout when num tasks in asyncio.gather() exceeds client max_connections
See original GitHub issueChecklist
- Reproducible on 0.13.3
- This issue seems similar but it’s closed and was supposedly fixed
Describe the bug
If the number of tasks executed via asyncio.gather(...) is greater than max_connections, i get a PoolTimeout. It seems like maybe this is happening because the tasks that have completed aren’t releasing their connections upon completion.
I’m new to asyncio so it’s possible I’m doing something wrong, but haven’t been able to find any documentation or issues that cover this case definitively.
To reproduce
import asyncio
import httpx
async def main() -> None:
url = "https://www.example.com"
max_connections = 2
timeout = httpx.Timeout(5.0, pool=2.0)
limits = httpx.Limits(max_connections=2)
client = httpx.AsyncClient(timeout=timeout, pool_limits=limits)
async with client:
tasks = []
for _ in range(max_connections + 1):
tasks.append(client.get(url))
await asyncio.gather(*tasks)
if __name__ == "__main__":
loop = asyncio.get_event_loop()
try:
loop.run_until_complete(main())
finally:
loop.close()
Expected behavior
I would expect all tasks to complete, rather than getting a PoolTimeout on the nth task, where n = max_connections + 1.
Actual behavior
Getting a PoolTimeout on the nth task, where n = max_connections + 1.
Debugging material
Traceback (most recent call last):
File "test_async.py", line 21, in <module>
loop.run_until_complete(main())
File "/Users/redacted/.pyenv/versions/3.6.9/lib/python3.6/asyncio/base_events.py", line 484, in run_until_complete
return future.result()
File "test_async.py", line 16, in main
await asyncio.gather(*tasks)
File "/Users/redacted/.pyenv/versions/3.6.9/lib/python3.6/site-packages/httpx/_client.py", line 1416, in get
timeout=timeout,
File "/Users/redacted/.pyenv/versions/3.6.9/lib/python3.6/site-packages/httpx/_client.py", line 1242, in request
request, auth=auth, allow_redirects=allow_redirects, timeout=timeout,
File "/Users/redacted/.pyenv/versions/3.6.9/lib/python3.6/site-packages/httpx/_client.py", line 1273, in send
request, auth=auth, timeout=timeout, allow_redirects=allow_redirects,
File "/Users/redacted/.pyenv/versions/3.6.9/lib/python3.6/site-packages/httpx/_client.py", line 1302, in _send_handling_redirects
request, auth=auth, timeout=timeout, history=history
File "/Users/redacted/.pyenv/versions/3.6.9/lib/python3.6/site-packages/httpx/_client.py", line 1338, in _send_handling_auth
response = await self._send_single_request(request, timeout)
File "/Users/redacted/.pyenv/versions/3.6.9/lib/python3.6/site-packages/httpx/_client.py", line 1374, in _send_single_request
timeout=timeout.as_dict(),
File "/Users/redacted/.pyenv/versions/3.6.9/lib/python3.6/contextlib.py", line 99, in __exit__
self.gen.throw(type, value, traceback)
File "/Users/redacted/.pyenv/versions/3.6.9/lib/python3.6/site-packages/httpx/_exceptions.py", line 359, in map_exceptions
raise mapped_exc(message, **kwargs) from None # type: ignore
httpx._exceptions.PoolTimeout
Environment
- OS: macOS 10.14.6
- Python version: 3.6.9
- HTTPX version: 0.13.3
- Async environment: asyncio
- HTTP proxy: no
- Custom certificates: no
Additional context
I commented on this issue, but it’s closed so figured it would be better to create a new one.
Issue Analytics
- State:
- Created 3 years ago
- Reactions:8
- Comments:13 (7 by maintainers)
Top Results From Across the Web
Coroutines and Tasks — Python 3.11.1 documentation
This section outlines high-level asyncio APIs to work with coroutines and Tasks. Coroutines, Awaitables, Creating Tasks, Task Cancellation, Task Groups, ...
Read more >How to get httpx.gather() with return_exceptions=True to ...
AsyncClient for the first time and trying to figure out how to complete my list of tasks when some number of them may...
Read more >API Reference — aioredis 1.3.0 documentation - Read the Docs
get_event_loop() if not specified). timeout (float greater than 0 or None) – Max time to open a connection, otherwise raise asyncio.TimeoutError exception ...
Read more >Limit concurrency with semaphore in Python asyncio
AsyncClient() async def make_one_request(url: str, num: int) -> httpx. ... num)) tasks.append(task) results = await asyncio.gather(*tasks) ...
Read more >Python Asyncio: The Complete Guide
What is Asyncio gather(); How to use Asyncio gather() ... a very large number of seemingly simultaneous functions in your Python programs.
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Have confirmed that the given example now works in
httpx0.21 (Fixed due to the substantial reworking in the latest httpcore.)I’m planning at getting stuck into this one pretty soon yup. It’s a bit of an involved one, but I know what we need to do to resolve it.