Why my Faster R-CNN can't convergence when batch size > 1?

Question

I build faster rcnn model by keras.And I train it well when batch size = 1.The losses convergence normally. When I set batch size = 2 or bigger, the training processing is e