Neural network doesn't converge for a single input

Question

I am trying to reproduce a neural network for shape and appearance disentangling. The network was written in Tensorflow and I want to write it in PyTorch. The Model looks as fol