How to evaluate a pretrained model in Tensorflow object detection api

Question

Trying work with the recently released Tensorflow Object Detection API, and was wondering how I could evaluate one of the pretrained models they provided in their model zoo? ex.

Answer 1

Try:

python eval.py --logtostderr --checkpoint_dir=training --eval_dir=path/to/eval_dir --pipeline_config_path=path/to/pretrained_model.config

For example:

python eval.py --logtostderr --checkpoint_dir=training --eval_dir=images/val \
  --pipelineline_config_path=training/faster_rcnn_inception_v2.config

Note:

The training dir contains all your training checkpoints. During training Tensorflow generates a checkpoint file inside this directory with all your checkpoint metadata in it so you do not need to create another one. If you wish to evaluate your trained custom model after generating your inference graph then ensure your change your original pretrained_model/model.chpt to your new_trained_model/model.ckpt in the .config you used for training. You should get a similar output:

Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.457
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.729
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.502
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.122
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.297
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.659
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.398
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.559
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.590
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.236
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.486
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.746
INFO:tensorflow:Writing metrics to tf summary.
INFO:tensorflow:DetectionBoxes_Precision/mAP: 0.456758
INFO:tensorflow:DetectionBoxes_Precision/mAP (large): 0.659280
INFO:tensorflow:DetectionBoxes_Precision/mAP (medium): 0.296693
INFO:tensorflow:DetectionBoxes_Precision/mAP (small): 0.122108
INFO:tensorflow:DetectionBoxes_Precision/mAP@.50IOU: 0.728587
INFO:tensorflow:DetectionBoxes_Precision/mAP@.75IOU: 0.502194
INFO:tensorflow:DetectionBoxes_Recall/AR@1: 0.397509
INFO:tensorflow:DetectionBoxes_Recall/AR@10: 0.558966
INFO:tensorflow:DetectionBoxes_Recall/AR@100: 0.590182
INFO:tensorflow:DetectionBoxes_Recall/AR@100 (large): 0.745691
INFO:tensorflow:DetectionBoxes_Recall/AR@100 (medium): 0.485964
INFO:tensorflow:DetectionBoxes_Recall/AR@100 (small): 0.236275
INFO:tensorflow:Losses/Loss/BoxClassifierLoss/classification_loss: 0.234645
INFO:tensorflow:Losses/Loss/BoxClassifierLoss/localization_loss: 0.139109
INFO:tensorflow:Losses/Loss/RPNLoss/localization_loss: 0.603733
INFO:tensorflow:Losses/Loss/RPNLoss/objectness_loss: 0.206419