Pickling error from joblib.Parallel()

See original GitHub issue

When running a block of code from within Parallel(), it returns the error - _pickle.PicklingError: Could not pickle the task to send it to the workers.

Is anyone facing similar issues? Any help appreciated!

Description - Inside the (to be parallelized) function, I am using Spacy’s english language model to extract Named Entities from text. I encounter this error when running the script from command line. If I use interactive program like a jupyter notebook, the call to Parallel() sometimes works fine and sometimes throws the described error, basically not consistent.

The error trace looks like this -

$ python namedAliases_associatedEntities_v6.py
[Parallel(n_jobs=-1)]: Using backend LokyBackend with 4 concurrent workers.
[Parallel(n_jobs=-1)]: Batch computation too fast (0.0770s.) Setting batch_size=4.
joblib.externals.loky.process_executor._RemoteTraceback:
"""
Traceback (most recent call last):
  File "/home/ubuntu/.local/lib/python3.5/site-packages/joblib/externals/loky/backend/queues.py", line 151, in _feed
    obj, reducers=reducers)
  File "/home/ubuntu/.local/lib/python3.5/site-packages/joblib/externals/loky/backend/reduction.py", line 145, in dumps
    p.dump(obj)
  File "/home/ubuntu/.local/lib/python3.5/site-packages/joblib/parallel.py", line 290, in __getstate__
    for func, args, kwargs in self.items]
  File "/home/ubuntu/.local/lib/python3.5/site-packages/joblib/parallel.py", line 290, in <listcomp>
    for func, args, kwargs in self.items]
  File "/home/ubuntu/.local/lib/python3.5/site-packages/joblib/parallel.py", line 278, in _wrap_non_picklable_objects
    wrapped_obj = CloudpickledObjectWrapper(obj)
  File "/home/ubuntu/.local/lib/python3.5/site-packages/joblib/parallel.py", line 208, in __init__
    self.pickled_obj = dumps(obj)
  File "/home/ubuntu/.local/lib/python3.5/site-packages/joblib/externals/cloudpickle/cloudpickle.py", line 917, in dumps
    cp.dump(obj)
  File "/home/ubuntu/.local/lib/python3.5/site-packages/joblib/externals/cloudpickle/cloudpickle.py", line 268, in dump
    return Pickler.dump(self, obj)
  File "/usr/lib/python3.5/pickle.py", line 408, in dump
    self.save(obj)
  File "/usr/lib/python3.5/pickle.py", line 475, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/ubuntu/.local/lib/python3.5/site-packages/joblib/externals/cloudpickle/cloudpickle.py", line 408, in save_function
    self.save_function_tuple(obj)
  File "/home/ubuntu/.local/lib/python3.5/site-packages/joblib/externals/cloudpickle/cloudpickle.py", line 573, in save_function_tuple
    save(state)
  File "/usr/lib/python3.5/pickle.py", line 475, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/ubuntu/.local/lib/python3.5/site-packages/dill/_dill.py", line 893, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/usr/lib/python3.5/pickle.py", line 810, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/lib/python3.5/pickle.py", line 836, in _batch_setitems
    save(v)
  File "/usr/lib/python3.5/pickle.py", line 475, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/ubuntu/.local/lib/python3.5/site-packages/dill/_dill.py", line 893, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/usr/lib/python3.5/pickle.py", line 810, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/lib/python3.5/pickle.py", line 841, in _batch_setitems
    save(v)
  File "/usr/lib/python3.5/pickle.py", line 475, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/ubuntu/.local/lib/python3.5/site-packages/joblib/externals/cloudpickle/cloudpickle.py", line 408, in save_function
    self.save_function_tuple(obj)
  File "/home/ubuntu/.local/lib/python3.5/site-packages/joblib/externals/cloudpickle/cloudpickle.py", line 573, in save_function_tuple
    save(state)
  File "/usr/lib/python3.5/pickle.py", line 475, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/ubuntu/.local/lib/python3.5/site-packages/dill/_dill.py", line 893, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/usr/lib/python3.5/pickle.py", line 810, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/lib/python3.5/pickle.py", line 836, in _batch_setitems
    save(v)
  File "/usr/lib/python3.5/pickle.py", line 475, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/ubuntu/.local/lib/python3.5/site-packages/dill/_dill.py", line 893, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/usr/lib/python3.5/pickle.py", line 810, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/lib/python3.5/pickle.py", line 836, in _batch_setitems
    save(v)
  File "/usr/lib/python3.5/pickle.py", line 475, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/ubuntu/.local/lib/python3.5/site-packages/joblib/externals/cloudpickle/cloudpickle.py", line 408, in save_function
    self.save_function_tuple(obj)
  File "/home/ubuntu/.local/lib/python3.5/site-packages/joblib/externals/cloudpickle/cloudpickle.py", line 573, in save_function_tuple
    save(state)
  File "/usr/lib/python3.5/pickle.py", line 475, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/ubuntu/.local/lib/python3.5/site-packages/dill/_dill.py", line 893, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/usr/lib/python3.5/pickle.py", line 810, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/lib/python3.5/pickle.py", line 836, in _batch_setitems
    save(v)
  File "/usr/lib/python3.5/pickle.py", line 475, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/ubuntu/.local/lib/python3.5/site-packages/dill/_dill.py", line 893, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/usr/lib/python3.5/pickle.py", line 810, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/lib/python3.5/pickle.py", line 841, in _batch_setitems
    save(v)
  File "/usr/lib/python3.5/pickle.py", line 520, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/lib/python3.5/pickle.py", line 592, in save_reduce
    "args[0] from __newobj__ args has the wrong class")
_pickle.PicklingError: args[0] from __newobj__ args has the wrong class
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "namedAliases_associatedEntities_v6.py", line 283, in <module>
    list_of_named_entities = Parallel(n_jobs=-1, verbose=10, backend = 'loky')(map(delayed(processNER), all_articles))
  File "/home/ubuntu/.local/lib/python3.5/site-packages/joblib/parallel.py", line 996, in __call__
    self.retrieve()
  File "/home/ubuntu/.local/lib/python3.5/site-packages/joblib/parallel.py", line 899, in retrieve
    self._output.extend(job.get(timeout=self.timeout))
  File "/home/ubuntu/.local/lib/python3.5/site-packages/joblib/_parallel_backends.py", line 517, in wrap_future_result
    return future.result(timeout=timeout)
  File "/home/ubuntu/.local/lib/python3.5/site-packages/joblib/externals/loky/_base.py", line 431, in result
    return self.__get_result()
  File "/home/ubuntu/.local/lib/python3.5/site-packages/joblib/externals/loky/_base.py", line 382, in __get_result
    raise self._exception
_pickle.PicklingError: Could not pickle the task to send it to the workers.

Issue Analytics

  • State:open
  • Created 5 years ago
  • Comments:7 (2 by maintainers)

github_iconTop GitHub Comments

1reaction
alexander-soarecommented, Nov 8, 2020

I’m getting this issue too. Worth noting that I’m in a notebook environment and using autoreload:

%reload_ext autoreload
%autoreload 2

The error seems to be thrown up when I update one of my modules in between cell runs.

0reactions
JohnCHarringtoncommented, Apr 21, 2021

I have a very similar issue trying to run PyMC3 model training with joblib (and I also had dill in the trace). As dmparker said above, I tried using an old version of joblib (0.12.1) and it works fine.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Could not pickle the task to send it to the workers - Stack ...
Same issue. I solved by changing the backend from loky to threading in Parallel .
Read more >
Serialization of un-picklable objects - Joblib
First, define functions which cannot be pickled with the standard pickle protocol. They cannot be serialized with pickle because they are defined in...
Read more >
When using multiple workers, I get a pickling error. Can I still ...
sbi uses joblib to parallelize simulations, which in turn uses pickle or cloudpickle to serialize the simulator. Almost all simulators will be picklable...
Read more >
PicklingError: Could not pickle the task to send it to the workers.
I am using sklearn in a databricks notebook to fit an estimator in parallel. Sklearn uses joblib with loky backend to do this....
Read more >
Multiprocessing and Pickle, How to Easily fix that?
However, the multiprocess tasks can't be pickled; it would raise an error failing to pickle. That's because when dividing a single task over ......
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found