Can I send callbacks to a KerasClassifier?

Question

I want the classifier to run faster and stop early if the patience reaches the number I set. In the following code it does 10 iterations of fitting the model.

Answer 1

Reading from here, which is the source code of KerasClassifier, you can pass it the arguments of fit and they should be used. I don't have your dataset so I cannot test it, but you can tell me if this works and if not I will try and adapt the solution. Change this line :

estimators.append(('mlp', KerasClassifier(build_fn=create_baseline, nb_epoch=300, batch_size=16, verbose=0, callbacks=[...your_callbacks...])))

A small explaination of what's happening : KerasClassifier is taking all the possibles arguments for fit, predict, score and uses them accordingly when each method is called. They made a function that filters the arguments that should go to each of the above functions that can be called in the pipeline. I guess there are several fit and predict calls inside the StratifiedKFold step to train on different splits everytime.

The reason why it takes forever to fit and it fits 10 times is because one fit is doing 300 epochs, as you asked. So the KFold is repeating this step over the different folds :

• calls fit with all the parameters given to KerasClassifier (300 epochs and batch size = 16). It's training on 9/10 of your data and using 1/10 as validation.

EDIT :

Ok, so I took the time to download the dataset and try your code... First of all you need to correct a "few" things in your network :

• your input have a 60 features. You clearly show it in your data prep :

  X = dataset[:,:60].astype(float)
  

  so why would you have this :

  model.add(Dropout(0.2, input_shape=(33,)))
  

  please change to :

  model.add(Dropout(0.2, input_shape=(60,)))
  

• About your targets/labels. You changed the objective from the original code (binary_crossentropy) to categorical_crossentropy. But you didn't change your Y array. So either do this in your data preparation :

  from keras.utils.np_utils import to_categorical
  encoded_Y = to_categorical(encoder.transform(Y))
  

  or change your objective back to binary_crossentropy.

• Now the network's output size : 122 on the last dense layer? your dataset obviously has 2 categories so why are you trying to output 122 classes? it won't match the target. Please change back your last layer to :

  model.add(Dense(2, init='normal', activation='softmax'))
  

  if you choose to use categorical_crossentropy, or

  model.add(Dense(1, init='normal', activation='sigmoid'))
  

  if you go back to binary_crossentropy.

So now that your network compiles, I could start to troubleshout.

here is your solution

So now I could get the real error message. It turns out that when you feed fit_params=whatever in the cross_val_score() function, you are feeding those parameters to a pipeline. In order to know to which part of the pipeline you want to send those parameters you have to specify it like this :

fit_params={'mlp__callbacks':calls}

Your error was saying that the process couldn't unpack 'callbacks'.split('__', 1) into 2 values. It was actually looking for the name of the pipeline's step to apply this to.

It should be working now :)

results = cross_val_score(pipeline, X, encoded_Y, cv=kfold, fit_params={'mlp__callbacks':calls})

BUT, you should be aware of what's happening here... the cross validation actually calls the create_baseline() function to recreate the model from scratch 10 times an trains it 10 times on different parts of the dataset. So it's not doing epochs as you were saying, it's doing 300 epochs 10 times. What is also happening as a consequence of using this tool : since the models are always differents, it means the fit() method is applied 10 times on different models, therefore, the callbacks are also applied 10 different times and the files saved by ModelCheckpoint() get overriden and you find yourself only with the best model of the last run.

This is intrinsec to the tools you use, I don't see any way around this. This comes as consequence to using different general tools that weren't especially thought to be used together with all the possible configurations.