python luigi died unexpectedly with exit code -11

问题

I have a data pipeline with luigi that works perfectly fine if I put 1 worker to the task. However, if I put > 1 workers, then it dies (unexpectedly with exit code -11) in a stage with 2 dependencies. The code is rather complex, so a minimum example would be difficult to give. The gist of the matter is that I am doing the following things with gensim:

1. Building a dictionary from some texts.
2. Building a corpus from said texts and the dictionary (requires (1)).
3. Training an LDA model from the corpus and dictionary (requires (1) and (2)).

For some reason, step (3) crashes every time I put more than one worker, even if (1) and (2) are already completed...

Any help would be greatly appreciated!

EDIT: Here is an example of the logging info. TrainLDA is task (3). There are still two tasks after that that require TrainLDA. All earlier tasks finished correctly. I substituted TrainLDA's arguments for ... so that the output would be more readable. The additional info are just print statements we put to help us know what is happening.

DEB

UG: Pending tasks: 3
DEBUG: Asking scheduler for work...
INFO: [pid 28851] Worker Worker(salt=514562349, workers=4, host=felipe.local, username=Felipe, pid=28825) running   TrainLDA(...)
INFO: Done
INFO: There are no more tasks to run at this time
INFO: TrainLDA(...) is currently run by worker Worker(salt=514562349, workers=4, host=felipe.local, username=Felipe, pid=28825)
==============================
Corriendo LDA de spanish con nivel de limpieza stopwords
==============================
Número de tópicos: 40
DEBUG: Asking scheduler for work...
INFO: Done
INFO: There are no more tasks to run at this time
INFO: TrainLDA(...) is currently run by worker Worker(salt=514562349, workers=4, host=felipe.local, username=Felipe, pid=28825)
DEBUG: Asking scheduler for work...
INFO: Done
INFO: There are no more tasks to run at this time
INFO: TrainLDA(...) is currently run by worker Worker(salt=514562349, workers=4, host=felipe.local, username=Felipe, pid=28825)
INFO: Worker task TrainLDA(...) died unexpectedly with exit code -11
DEBUG: Asking scheduler for work...
INFO: Done
INFO: There are no more tasks to run at this time
INFO: There are 2 pending tasks possibly being run by other workers
INFO: There are 2 pending tasks unique to this worker
INFO: Worker Worker(salt=514562349, workers=4, host=felipe.local, username=Felipe, pid=28825) was stopped. Shutting down Keep-Alive thread

来源：https://stackoverflow.com/questions/32978429/python-luigi-died-unexpectedly-with-exit-code-11