keras + scikit-learn wrapper, appears to hang when GridSearchCV with n_jobs >1

Question

UPDATE: I have to re-write this question as after some investigation I realise that this is a different problem.

Context: running keras in a gridsearch

Answer 1

TLDR Answer: You can't because your Keras model can't be serialized, and serialization is needed for parallelizing in Python with joblib.

This problem is much detailed here: https://www.neuraxle.org/stable/scikit-learn_problems_solutions.html#problem-you-can-t-parallelize-nor-save-pipelines-using-steps-that-can-t-be-serialized-as-is-by-joblib

The solution to parallelize your code is to make your Keras estimator serializable. This can be done using savers as described at the link above.

If you're lucky enough to be using TensorFlow v2's prebuilt Keras module, the following practical code sample will reveal to be useful to you as you'd practically just need to take the code and modify it with yours:

• https://github.com/guillaume-chevalier/seq2seq-signal-prediction

In this example, all the saving and loading code is all pre-written for you using Neuraxle-TensorFlow, and this makes it parallelizeable if you use Neuraxle's AutoML methods (e.g.: Neuraxle's grid search and Neuraxle's own parallelism things).