The loss did not converge when writing a training loop from scratch with TensorFlow 2.2 and Tensorflow 2.3

Question

If I use "fit" in training my network. The loss function converges, and the metric (accuracy) is significantly improved.

However, I write the training proce