Keras: How to use fit_generator with multiple outputs of different type

Question

In a Keras model with the Functional API I need to call fit_generator to train on augmented images data using an ImageDataGenerator. The problem is my model has two outputs:

Answer 1

The example below might be self-explanatory! The 'dummy' model takes 1 input (image) and it outputs 2 values. The model computes the MSE for each output.

x = Convolution2D(8, 5, 5, subsample=(1, 1))(image_input)
x = Activation('relu')(x)
x = Flatten()(x)
x = Dense(50, W_regularizer=l2(0.0001))(x)
x = Activation('relu')(x)

output1 = Dense(1, activation='linear', name='output1')(x)
output2 = Dense(1, activation='linear', name='output2')(x)

model = Model(input=image_input, output=[output1, output2])
model.compile(optimizer='adam', loss={'output1': 'mean_squared_error', 'output2': 'mean_squared_error'})

The function below generates batches to feed the model during training. It takes the training data x and the label y where y=[y1, y2]

batch_generator(x, y, batch_size, is_train):
    sample_idx = 0
    while True:
       X = np.zeros((batch_size, input_height, input_width, n_channels), dtype='float32')
       y1 = np.zeros((batch_size, mask_height, mask_width), dtype='float32')
       y2 = np.zeros((batch_size, 1), dtype='float32')

       # fill up the batch
       for row in range(batch_sz):
           image = x[sample_idx]
           mask = y[0][sample_idx]
           binary_value = y[1][sample_idx]
           # transform/preprocess image
           image = cv2.resize(image, (input_width, input_height))
           if is_train:
               image, mask = my_data_augmentation_function(image, mask)
           X_batch[row, ;, :, :] = image
           y1_batch[row, :, :] = mask
           y2_batch[row, 0] = binary_value
           sample_idx += 1

       # Normalize inputs
       X_batch = X_batch/255.
       yield(X_batch, {'output1': y1_batch, 'output2': y2_batch} ))

Finally, we call the fit_generator()

    model.fit_generator(batch_generator(X_train, y_train, batch_size, is_train=1))

Answer 2

If you have separated both mask and binary value you can try something like this:

generator = ImageDataGenerator(rotation_range=5.,
                                width_shift_range=0.1, 
                                height_shift_range=0.1, 
                                horizontal_flip=True,  
                                vertical_flip=True)

def generate_data_generator(generator, X, Y1, Y2):
    genX = generator.flow(X, seed=7)
    genY1 = generator.flow(Y1, seed=7)
    while True:
            Xi = genX.next()
            Yi1 = genY1.next()
            Yi2 = function(Y2)
            yield Xi, [Yi1, Yi2]

So, you use the same generator for both input and mask with the same seed to define the same operation. You may change the binary value or not depending on your needs (Y2). Then, you call the fit_generator():

model.fit_generator(generate_data_generator(generator, X, Y1, Y2),
                epochs=epochs)

Answer 3

The best way to achieve this seems to be to create a new generator class expanding the one provided by Keras that parses the data augmenting only the images and yielding all the outputs.