Is my implementation of stochastic gradient descent correct?

Question

I am trying to develop stochastic gradient descent, but I don\'t know if it is 100% correct.

• The cost generated by my stochastic gradient descent algorithm is

Answer 1

There is a reason for small value of the learning rate. Briefly, when the learning rates decrease with an appropriate rate, and subject to relatively mild assumptions, stochastic gradient descent converges almost surely to a global minimum when the objective function is convex or pseudoconvex, and otherwise converges almost surely to a local minimum. This is in fact a consequence of the Robbins-Siegmund theorem.

Robbins, Herbert; Siegmund, David O. (1971). "A convergence theorem for non negative almost supermartingales and some applications". In Rustagi, Jagdish S. Optimizing Methods in Statistics. Academic Press