PoolTimeout when num tasks in asyncio.gather() exceeds client max_connections

See original GitHub issue

Checklist

  • Reproducible on 0.13.3
  • This issue seems similar but it’s closed and was supposedly fixed

Describe the bug

If the number of tasks executed via asyncio.gather(...) is greater than max_connections, i get a PoolTimeout. It seems like maybe this is happening because the tasks that have completed aren’t releasing their connections upon completion.

I’m new to asyncio so it’s possible I’m doing something wrong, but haven’t been able to find any documentation or issues that cover this case definitively.

To reproduce

import asyncio
import httpx

async def main() -> None:
    url = "https://www.example.com"
    max_connections = 2
    timeout = httpx.Timeout(5.0, pool=2.0)
    limits = httpx.Limits(max_connections=2)
    client = httpx.AsyncClient(timeout=timeout, pool_limits=limits)

    async with client:
        tasks = []
        for _ in range(max_connections + 1):
            tasks.append(client.get(url))
        await asyncio.gather(*tasks)

if __name__ == "__main__":
    loop = asyncio.get_event_loop()
    try:
        loop.run_until_complete(main())
    finally:
        loop.close()

Expected behavior

I would expect all tasks to complete, rather than getting a PoolTimeout on the nth task, where n = max_connections + 1.

Actual behavior

Getting a PoolTimeout on the nth task, where n = max_connections + 1.

Debugging material

Traceback (most recent call last):
  File "test_async.py", line 21, in <module>
    loop.run_until_complete(main())
  File "/Users/redacted/.pyenv/versions/3.6.9/lib/python3.6/asyncio/base_events.py", line 484, in run_until_complete
    return future.result()
  File "test_async.py", line 16, in main
    await asyncio.gather(*tasks)
  File "/Users/redacted/.pyenv/versions/3.6.9/lib/python3.6/site-packages/httpx/_client.py", line 1416, in get
    timeout=timeout,
  File "/Users/redacted/.pyenv/versions/3.6.9/lib/python3.6/site-packages/httpx/_client.py", line 1242, in request
    request, auth=auth, allow_redirects=allow_redirects, timeout=timeout,
  File "/Users/redacted/.pyenv/versions/3.6.9/lib/python3.6/site-packages/httpx/_client.py", line 1273, in send
    request, auth=auth, timeout=timeout, allow_redirects=allow_redirects,
  File "/Users/redacted/.pyenv/versions/3.6.9/lib/python3.6/site-packages/httpx/_client.py", line 1302, in _send_handling_redirects
    request, auth=auth, timeout=timeout, history=history
  File "/Users/redacted/.pyenv/versions/3.6.9/lib/python3.6/site-packages/httpx/_client.py", line 1338, in _send_handling_auth
    response = await self._send_single_request(request, timeout)
  File "/Users/redacted/.pyenv/versions/3.6.9/lib/python3.6/site-packages/httpx/_client.py", line 1374, in _send_single_request
    timeout=timeout.as_dict(),
  File "/Users/redacted/.pyenv/versions/3.6.9/lib/python3.6/contextlib.py", line 99, in __exit__
    self.gen.throw(type, value, traceback)
  File "/Users/redacted/.pyenv/versions/3.6.9/lib/python3.6/site-packages/httpx/_exceptions.py", line 359, in map_exceptions
    raise mapped_exc(message, **kwargs) from None  # type: ignore
httpx._exceptions.PoolTimeout

Environment

  • OS: macOS 10.14.6
  • Python version: 3.6.9
  • HTTPX version: 0.13.3
  • Async environment: asyncio
  • HTTP proxy: no
  • Custom certificates: no

Additional context

I commented on this issue, but it’s closed so figured it would be better to create a new one.

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Reactions:8
  • Comments:13 (7 by maintainers)

github_iconTop GitHub Comments

8reactions
tomchristiecommented, Dec 16, 2021

Have confirmed that the given example now works in httpx 0.21 (Fixed due to the substantial reworking in the latest httpcore.)

7reactions
tomchristiecommented, Sep 25, 2020

I’m planning at getting stuck into this one pretty soon yup. It’s a bit of an involved one, but I know what we need to do to resolve it.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Coroutines and Tasks — Python 3.11.1 documentation
This section outlines high-level asyncio APIs to work with coroutines and Tasks. Coroutines, Awaitables, Creating Tasks, Task Cancellation, Task Groups, ...
Read more >
How to get httpx.gather() with return_exceptions=True to ...
AsyncClient for the first time and trying to figure out how to complete my list of tasks when some number of them may...
Read more >
API Reference — aioredis 1.3.0 documentation - Read the Docs
get_event_loop() if not specified). timeout (float greater than 0 or None) – Max time to open a connection, otherwise raise asyncio.TimeoutError exception ...
Read more >
Limit concurrency with semaphore in Python asyncio
AsyncClient() async def make_one_request(url: str, num: int) -> httpx. ... num)) tasks.append(task) results = await asyncio.gather(*tasks) ...
Read more >
Python Asyncio: The Complete Guide
What is Asyncio gather(); How to use Asyncio gather() ... a very large number of seemingly simultaneous functions in your Python programs.
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found