tf object detection api - extract feature vector for each detection bbox

Question

I\'m using Tensorflow object detection API and working on pretrainedd ssd-mobilenet model. is there a way to extact the last global pooling of the mobilenet for each bbox as a f

Answer 1

Support for feature extraction was added in a recent PR: (https://github.com/tensorflow/models/pull/7208). To use this functionality, you can re-export the pretrained models using the exporter tool.

For reference, this was the script I used:

#!/bin/bash
# NOTE: run this from tf/models/research directory

# Ensure that the necessary modules are on the PYTHONPATH
PYTHONPATH=".:./slim:$PYTHONPATH"

# Modify this to ensure that Tensorflow is accessible to your environment
conda activate tf37

# pick a model from the model zoo
ORIG_MODEL="faster_rcnn_inception_resnet_v2_atrous_oid_v4_2018_12_12"

# point at wherever you have downloaded the pretrained model
ORIG_MODEL_DIR="object_detection/pretrained/${ORIG_MODEL}"

# choose a destination where the updated model will be stored
DEST_DIR="${ORIG_MODEL_DIR}_with_feats"
echo "Re-exporting model from $ORIG_MODEL_DIR"

python3 object_detection/export_inference_graph.py \
     --input_type image_tensor \
     --pipeline_config_path "${ORIG_MODEL_DIR}/pipeline.config" \
     --trained_checkpoint_prefix "${ORIG_MODEL_DIR}/model.ckpt" \
     --output_directory "${DEST_DIR}"

To use the re-exported model, you can update the run_inference_for_single_image in the example notebook to include detection_features as an output:

def run_inference_for_single_image(image, graph):
    with graph.as_default():
        with tf.Session() as sess:
            # Get handles to input and output tensors
            ops = tf.get_default_graph().get_operations()
            all_tensor_names = {output.name for op in ops for output in op.outputs}
            tensor_dict = {}
            for key in ['num_detections', 'detection_boxes', 'detection_scores', 'detection_classes',
                        'detection_masks', 'detection_features']:
                tensor_name = key + ':0'
                if tensor_name in all_tensor_names:
                    tensor_dict[key] = tf.get_default_graph().get_tensor_by_name( tensor_name)
            if 'detection_masks' in tensor_dict:
                # The following processing is only for single image
                detection_boxes = tf.squeeze(tensor_dict['detection_boxes'], [0])
                detection_masks = tf.squeeze(tensor_dict['detection_masks'], [0])
                # Reframe is required to translate mask from box coordinates to image coordinates and fit the image size.
                real_num_detection = tf.cast(tensor_dict['num_detections'][0], tf.int32)
                detection_boxes = tf.slice(detection_boxes, [0, 0], [real_num_detection, -1])
                detection_masks = tf.slice(detection_masks, [0, 0, 0], [real_num_detection, -1, -1])
                detection_masks_reframed = utils_ops.reframe_box_masks_to_image_masks( detection_masks, detection_boxes, image.shape[1], image.shape[2])
                detection_masks_reframed = tf.cast( tf.greater(detection_masks_reframed, 0.5), tf.uint8)
                # Follow the convention by adding back the batch dimension
                tensor_dict['detection_masks'] = tf.expand_dims( detection_masks_reframed, 0)
            image_tensor = tf.get_default_graph().get_tensor_by_name('image_tensor:0')

            # Run inference
            output_dict = sess.run(tensor_dict, feed_dict={image_tensor: image})

            # all outputs are float32 numpy arrays, so convert types as appropriate
            output_dict['num_detections'] = int(output_dict['num_detections'][0])
            output_dict['detection_classes'] = output_dict[ 'detection_classes'][0].astype(np.int64)
            output_dict['detection_boxes'] = output_dict['detection_boxes'][0]
            output_dict['detection_scores'] = output_dict['detection_scores'][0]
            output_dict['detection_features'] = output_dict['detection_features'][0]
            if 'detection_masks' in output_dict:
                output_dict['detection_masks'] = output_dict['detection_masks'][0]
    return output_dict