Parallelizing ML models which rely on stochastic gradient descent?

Question

I have been reading about different NLP models like word2vec and GloVe, and how these can be parallelized because they are mostly just dot products. However, I am a bit confused