gcloud ml-engine returns error on large files

Question

I have a trained model that takes in a somewhat large input. I generally do this as a numpy array of the shape (1,473,473,3). When I put that to JSON I end up getting about

Answer 1

What I typically do is to have the json refer to a file in Google Cloud Storage. See the serving input function here for example:

https://github.com/GoogleCloudPlatform/training-data-analyst/blob/61ab2e175a629a968024a5d09e9f4666126f4894/courses/machine_learning/deepdive/08_image/flowersmodel/trainer/model.py#L119

Users would first upload their file to gcs and then invoke prediction. But this approach has other advantages, since the storage utilities allow for parallel and multithreaded uploads.

Answer 2

I encountered the same error when trying to run predictions on AI Platform with large images. I solved the payload limit problem by first encoding images to PNG format before sending them to the AI Platform.

My Keras model doesn't take PNG encoded images as an input, so I needed to convert the Keras model to a Tensorflow Estimator and define its serving input function containing the code to decode the PNG encoded images back to the format my model expects.

Example code when the model expects two different grayscale images as an input:

import tensorflow as tf
from tensorflow.keras.estimator import model_to_estimator
from tensorflow.estimator.export import ServingInputReceiver

IMG_PNG_1 = "encoded_png_image_1"
IMG_PNG_2 = "encoded_png_image_2"


def create_serving_fn(image_height, image_width):
    def serving_input_fn():
        def preprocess_png(png_encoded_img):
            img = tf.reshape(png_encoded_img, shape=())
            img = tf.io.decode_png(img, channels=1)
            img = img / 255
            img = tf.expand_dims(img, axis=0)
            return img

        # receiver_tensors worked only when the shape parameter wasn't defined
        receiver_tensors = {
            IMG_PNG_1: tf.compat.v1.placeholder(tf.string),
            IMG_PNG_2: tf.compat.v1.placeholder(tf.string)
        }

        img_1 = preprocess_png(png_encoded_img=receiver_tensors[IMG_PNG_1])
        img_2 = preprocess_png(png_encoded_img=receiver_tensors[IMG_PNG_2])

        input_img_1 = tf.compat.v1.placeholder_with_default(img_1, shape=[None, image_height, image_width, 1])
        input_img_2 = tf.compat.v1.placeholder_with_default(img_2, shape=[None, image_height, image_width, 1])

        features = {
            "model_input_1": input_img_1,
            "model_input_2": input_img_2,
        }

        return ServingInputReceiver(features=features, receiver_tensors=receiver_tensors)

    return serving_input_fn

# Convert trained Keras model to Estimator
estimator = model_to_estimator(keras_model=model)
save_path = "location_of_the_SavedModel"
export_path = estimator.export_saved_model(
    export_dir_base=save_path,
    serving_input_receiver_fn=create_serving_fn(1000, 1000)
)