`RuntimeError: Kernel didn't respond in 60 seconds`, when trying to run papermill with python multiprocessing

Hi, I might just be overlooking something, but I think I’m still experiencing this issue even after upgrading nbconvert. It seems to be an upstream issue with nbconvert, because I get the same issues when calling the execute API directly. Let me know if I should migrate this question to that repository.

To replicate:

import multiprocessing as mp
import nbconvert
assert "5.6." in nbconvert.__version__
from nbconvert.preprocessors import ExecutePreprocessor
import nbformat
import os
import papermill as pm

def run_pm(fn):
    pm.execute_notebook(fn, fn, request_save_on_cell_execute = False)
    
def run(fn):
    with open(fn) as f:
        nb = nbformat.read(f, as_version = 4)        
    ep = ExecutePreprocessor(timeout = None, kernel_name = "python3")
    ep.startup_timeout = 300    
    ep.preprocess(nb, {"metadata": {"path": os.getcwd() + "/"}})    
    with open(fn, "w", encoding = "utf-8") as f:
        nbformat.write(nb, f)
        
fn = "test.ipynb"

test.ipynb has a single cell that prints the word “testing”. The following works fine:

run_pm(fn)
run(fn)

But the following two code snippets each break

pool = mp.Pool(1)
pool.map(run_pm, [fn])
pool.close()
pool.join()

pool = mp.Pool(1)
pool.map(run, [fn])
pool.close()
pool.join()

with error code RuntimeError: Kernel didn't respond in 60 seconds in the first case and RuntimeError: Kernel didn't respond in 300 seconds in the second.

I’m using Python 3.7. I’ve been able to replicate this with both nbconvert 5.6.0 and 5.6.1.

Thanks!

Wondering: what is the status for this issue? I can confirm that this problem is still present in Python3 when using Papermill. Will multiprocessing become doable with Papermill?