keras + scikit-learn wrapper, appears to hang when GridSearchCV with n_jobs >1

Question

UPDATE: I have to re-write this question as after some investigation I realise that this is a different problem.

Context: running keras in a gridsearch

Answer 1

Are you using a GPU? If so, you can't have multiple threads running each variation of the params because they won't be able to share the GPU.

Here's a full example on how to use keras, sklearn wrappers in a Pipeline with GridsearchCV: Pipeline with a Keras Model

If you really want to have multiple jobs in the GridSearchCV, you can try to limit the GPU fraction used by each job (e.g. if each job only allocates 0.5 of the available GPU memory, you can run 2 jobs simultaneously)

See these issues:

• Limit the resource usage for tensorflow backend

• GPU memory fraction does not work in keras 2.0.9 but it works in 2.0.8