Can someone explain to me the difference between a cost function and the gradient descent equation in logistic regression?

Question

I\'m going through the ML Class on Coursera on Logistic Regression and also the Manning Book Machine Learning in Action. I\'m trying to learn by implementing everything in Pytho

Answer 1

It's strange to think about it, but there is more than one measure for how "accurately" a line fits to data points.

To access how accurately a line fits the data, we have a "cost" function which which can compare predicted vs. actual values and provide a "penalty" for how wrong it is.

penalty = cost_funciton(predicted, actual)

A naive cost function might just take the difference between the predicted and actual.

More sophisticated functions will square the value, since we'd rather have many small errors than one large error.

Additionally, each point has a different "sensitivity" to moving the line. Some points react very strongly to movement. Others react less strongly.

Often, you can make a tradeoff, and move TOWARD a point that is sensitive, and AWAY from a point that is NOT sensitive. In that scenario , you get more than you give up.

The "gradient" is a way of measuring how sensitive each point is to moving the line.

This article does a good job of describing WHY there is more than one measure, and WHY some points are more sensitive than others:

https://towardsdatascience.com/wrapping-your-head-around-gradient-descent-with-pictures-3fbd810235f5?source=friends_link&sk=7117e5de8c66bd4a4c2bb2a87a928773