How to limit the performance overhead caused by Tensor.to() in Torch?

Question

I\'m working on a real-time object tracking network based on Tracktor and the mmdetection framework. I\'m simulating a constant stream of input data by running it with a batch s