How to make an annotation tensor with inconsistent number of objects per image?

Question

I was following this tutorial from tf/models/research/object_detection/colab_tutorials where the author uses TF object detection API to fine-tune a RetinaNet model trained on CO