Unable to reduce the loss in the Optical Flow Model called as FlowNet

Question

I have been trying to emulate the DL architecture called as FlowNet that is show in the figure below.

It is the model know as FlowNetSimple. The refinement st