Early Stopping and Callbacks with Keras when using SageMaker

问题

I am using sagemaker to train a keras model. I need to implement early stoping approach when training the model.

Is there a way to pass callbacks such as EarlyStopping, Histories..etc.

In traditional way, we used to pass this as a parameter to keras's fit function:

results = model.fit(train_x_trim, train_y_trim, 
                    validation_data=(test_x, test_y), 
                    epochs=FLAGS.epoch,  
                    verbose=0, 
                    callbacks=[tboard, checkpointer, early_stopping, history])

However, if using SageMaker, we need to call SageMaker's fit function instead which doesn't support callbacks.

from sagemaker.tensorflow import TensorFlow 
iris_estimator = TensorFlow(entry_point='training_code.py', 
                            role=role, output_path=model_location, 
                            code_location=custom_code_upload_location, 
                            train_instance_count=1, 
                            train_instance_type='ml.c4.xlarge', 
                            training_steps=1000, 
                            evaluation_steps=100)

Any idea how to implement callbacks in SageMaker ?

回答1:

I apologize for the late response.

It looks like the Keras code you specified above is essentially your algorithm code. This would be defined in your user script, which would be "training_code.py" in the SageMaker Python SDK example you provided.

Starting with TensorFlow 1.11, the SageMaker predefined TensorFlow containers have support for "script mode". You should be able to specify your Keras callbacks within your user script.

For more information: https://github.com/aws/sagemaker-python-sdk/blob/master/src/sagemaker/tensorflow/README.rst#tensorflow-sagemaker-estimators-and-models

来源：https://stackoverflow.com/questions/53486118/early-stopping-and-callbacks-with-keras-when-using-sagemaker