ValueError: need at most 63 handles, got a sequence of length 65

See original GitHub issue

Black runs just fine with a single file or a folder without subfolders. However, when specifying a folder with subfolders I’m getting the errors below.

Since I don’t have another server with the same Windows version, I can’t exclude a Windows installation issue, even if other python scripts and tools are working fine.

Python 3.7

Exception in thread QueueManagerThread:
Traceback (most recent call last):
  File "c:\python37\lib\threading.py", line 917, in _bootstrap_inner
    self.run()
  File "c:\python37\lib\threading.py", line 865, in run
    self._target(*self._args, **self._kwargs)
  File "c:\python37\lib\concurrent\futures\process.py", line 354, in _queue_management_worker
    ready = wait(readers + worker_sentinels)
  File "c:\python37\lib\multiprocessing\connection.py", line 868, in wait
    ready_handles = _exhaustive_wait(waithandle_to_obj.keys(), timeout)
  File "c:\python37\lib\multiprocessing\connection.py", line 800, in _exhaustive_wait
    res = _winapi.WaitForMultipleObjects(L, False, timeout)
ValueError: need at most 63 handles, got a sequence of length 65

Python 3.6

Exception in thread Thread-1:
Traceback (most recent call last):
  File "c:\python36\lib\threading.py", line 916, in _bootstrap_inner
    self.run()
  File "c:\python36\lib\threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "c:\python36\lib\concurrent\futures\process.py", line 270, in _queue_management_worker
    ready = wait([reader] + sentinels)
  File "c:\python36\lib\multiprocessing\connection.py", line 859, in wait
    ready_handles = _exhaustive_wait(waithandle_to_obj.keys(), timeout)
  File "c:\python36\lib\multiprocessing\connection.py", line 791, in _exhaustive_wait
    res = _winapi.WaitForMultipleObjects(L, False, timeout)
ValueError: need at most 63 handles, got a sequence of length 64

Operating system: Windows Server 2012 R2 (x64) Python version: 3.6.6, 3.7.0, 3.7.1rc2 (x64) Black version: 18.6b4, 18.9b0 Does also happen on master: Yes

Issue Analytics

State:
Created 5 years ago
Comments:7

Top GitHub Comments

1reaction

rbtcollinscommented, Apr 17, 2019

Steve Dower has offered to review patches to Python itself to fix this; I’d be happy to argue for backporting them existing releases, if someone would like to fix this at source, rather than band-aiding it.

(edited, it was late is my only excuse). In short: multiprocessing is using a thread per process to track the processes, which is quite wasteful on both unix and windows where nonblocking primitives exist, but nevertheless, thats what it is doing. black would need to limit itself to 60 processes to avoid the limit in question as a result; which is probably fine - one would need a hecka lot of code to run out of patience at that concurrency, at least for now.

0reactions

kevinkleinQCcommented, Jul 28, 2020

I’m encountering this issue with black 19.10b0 and python 3.8.3 on Windows Server 2012.

$ black dbcheck
Exception in thread QueueManagerThread:
Traceback (most recent call last):
  File "D:\Personal\Kevin_Klein\envs\dbcheck\lib\threading.py", line 932, in _bootstrap_inner
    self.run()
  File "D:\Personal\Kevin_Klein\envs\dbcheck\lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "D:\Personal\Kevin_Klein\envs\dbcheck\lib\concurrent\futures\process.py", line 362, in _queue_management_worker
    ready = mp.connection.wait(readers + worker_sentinels)
  File "D:\Personal\Kevin_Klein\envs\dbcheck\lib\multiprocessing\connection.py", line 879, in wait
    ready_handles = _exhaustive_wait(waithandle_to_obj.keys(), timeout)
  File "D:\Personal\Kevin_Klein\envs\dbcheck\lib\multiprocessing\connection.py", line 811, in _exhaustive_wait
    res = _winapi.WaitForMultipleObjects(L, False, timeout)
ValueError: need at most 63 handles, got a sequence of length 63