Multiprocessing issue 'Can't pickle <function ....'

See original GitHub issue

Hey, the following code is not working in ptpython on python version 3.

from multiprocessing.pool import Pool

def do(x):
    return x 

pool = Pool(processes=16,  maxtasksperchild=100)
done_x = []
for x in pool.imap_unordered(do, [1, 2, 3, 4, 5]):
    done_x.append(x)

It produces the following error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/multiprocessing/pool.py", line 695, in next
    raise value
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/multiprocessing/pool.py", line 385, in _handle_tasks
    put(task)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/multiprocessing/connection.py", line 206, in send
    self._send_bytes(ForkingPickler.dumps(obj))
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/multiprocessing/reduction.py", line 50, in dumps
    cls(buf, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <function do at 0x10388c268>: attribute lookup do on __main__ failed
Can't pickle <function do at 0x10388c268>: attribute lookup do on __main__ failed

I have tested the code on Python 3.5.2 on ubuntu and mac os. It is working correctly in default python’s REPL, in jupyter, and by running it directly.

Issue Analytics

  • State:open
  • Created 6 years ago
  • Reactions:3
  • Comments:5 (1 by maintainers)

github_iconTop GitHub Comments

3reactions
ttarter3commented, Apr 9, 2018

@danai-antoniou I updated Anaconda to the latest version and that fixed the problem for me. Not sure which package was the issue. I read online it could be multiprocessing, Ipython, or pickle.

1reaction
danai-antonioucommented, Apr 9, 2018

@jonathanslenders Any update on this? Also experiencing this bug

Read more comments on GitHub >

github_iconTop Results From Across the Web

Python multiprocessing PicklingError: Can't pickle <type ...
To get a PicklingError something must be put on the Queue which is not picklable. It could be the function or its arguments....
Read more >
Python “multiprocessing” “Can't pickle…” - TedChen - Medium
The 'pickle' issue in 'multiprocessing' is because of passing objects as arguments between process. There is 3 methods to start process in 'multiprocessing'....
Read more >
Multiprocessing and Pickle, How to Easily fix that?
However, the multiprocess tasks can't be pickled; it would raise an error failing to pickle. That's because when dividing a single task over ......
Read more >
Ways to Solve Can't Pickle local object Error - Python Pool
In this program, we are going to see how to rectify the attribute error while multiprocessing. Import multiprocessing. Create a function.
Read more >
Multiprocessing Can't pickle local object - Python Forum
Incase anyone else faces the same error with the same code structure, adding global keyword word to the child function before you define...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found