How to make my multi-object tracker more efficient？

Question

Since the MultiTracker in OpenCV cannot handle trackers individually, I decided to build my own.

It would be something like this with more a