Running object detection using Tensorflow.js

Question

I am working on object detection using Tensorflow.js. I am trying to run custom object detection tensorflow.js model in a browser. I could able to convert tensorflow model to te

Answer 1

Though the python model is not added to the question, but the extracted nodes of the model, the size of the tensor and their type provide an insight to identify the detection_classes, detection_scores, and detection_boxes.

The first tensor has the size 400 and corresponds to detection_boxes. The dataSync operator returns a completely flatten array. The size 400 most likely corresponds to the shape [100, 4]. This is corroborated by the shape of the other tensors that is discussed below. [100, 4] means that there are 100 bounding boxes in the input - most likely an image. Concretely, it means that the first four elements corresponds to the first bounding box, and so on...

The second tensor corresponds to the detections_scores. There are 100 detections scores for the 100 bounding boxes. The first element of this array correspond to the first four elements of the first array (detection_boxes array)

The third array corresponds to the detection_classes. It is an array of 100 integers where each value is the index of the matched label.

The forth array correspond to the num_detections. It contains how many detections there are: 100

I looked at the above output but I couldn't get class labels

To get the label (string), the index taken from the detection_classes should be used against the json (dictionary in python) or array that contains all the labels and their indexes.

It is noteworthy to indicate that for the js model to return the same output as the python model, all the processing done to the image in python prior to feeding the model should be replicated in js.

Answer 2

Finally, I could figure out the problem and it was related to the size of an input frame.

SSD model needs shape of [1,300,300,3] image/frame as input. I added this in my code and got the solution. Using the following line (in inference.html), we can feed (300,300,3) shape of image as an input to the model:

 <img id="img" src="257.jpg" width="300" height="300"/>

Using the following lines in index.js:

 tf_img = tf_img.expandDims(0);
 console.log(tf_img.shape)  // Image dimension is  [1, 300, 300, 3]

We obtain image shape of [1,300,300,3] which is needed by SSD.