I have implemented the Dense optical flow algorithm in python and got the flow output as a 2D array having displacement of that pixel from the prev frame. So the flow matrix wou