Why is my Network not converging with SGD but with Adam?

Question

I am currently training a network based on pretrained EfficientNet B0. The network is supposed to classify skin lesion images as part of the ISIC challenge. I managed to get