gradient-descent

I started to learn the machine learning last week. when I want to make a gradient descent script to estimate the model parameters, I came across a problem: How to choose a appropriate learning rate and variance。I found that，different (learning rate，variance) pairs may lead to different results, some times you even can't convergence. Also, if change to another training data set, a well-chose （learning rate，variance）pair probably will not work. For example(script below)，when I set the learning rate to 0.001 and variance to 0.00001, for 'data1', I can get the suitable theta0_guess and theta1

I am doing a neural network regression with 4 features. How do I determine the size of mini-batch for my problem? I see people use 100 ~ 1000 batch size for computer vision with 32*32*3 features for each image, does that mean I should use batch size of 1 million? I have billions of data and tens of GB of memory so there is no hard requirement for me not to do that. I also observed using a mini-batch with size ~ 1000 makes the convergence much faster than batch size of 1 million. I thought it should be the other way around, since the gradient calculated with a larger batch size is most

This question already has an answer here: Why do we need to explicitly call zero_grad()? 4 answers The method zero_grad() needs to be called during training. But the documentation is not very helpful | zero_grad(self) | Sets gradients of all model parameters to zero. Why do we need to call this method? In PyTorch , we need to set the gradients to zero before starting to do backpropragation because PyTorch accumulates the gradients on subsequent backward passes. This is convenient while training RNNs. So, the default action is to accumulate (i.e. sum) the gradients on every loss.backward() call

what is the benefit of using Gradient Descent in the linear regression space? looks like the we can solve the problem (finding theta0-n that minimum the cost func) with analytical method so why we still want to use gradient descent to do the same thing? thanks When you use the normal equations for solving the cost function analytically you have to compute: Where X is your matrix of input observations and y your output vector. The problem with this operation is the time complexity of calculating the inverse of a nxn matrix which is O(n^3) and as n increases it can take a very long time to

I'm trying to implement a simple gradient descent for linear regression. It works normally if I compute the gradient manually (by using the analytical expression), but now i was trying to implement it with autograd from the mxnet module. This is the code from mxnet import autograd, np, npx npx.set_np() def main(): # learning algorithm parameters nr_epochs = 1000 alpha = 0.01 # read data, insert column of ones (to include bias with other parameters) data = pd.read_csv("dataset.txt", header=0, index_col=None, sep="\s+") data.insert(0, "x_0", 1, True) # insert column of "1"s as x_0 m = data.shape