How do I improve performance of a pixel mapping Mat?

Question

I have a custom image projection that I create a pixel mapping with. I store the mapping in a cv::Mat so that the location of source pixel is stored at the dest