Linear vs nonlinear neural network?

Question

I\'m new to machine learning and neural networks. I know how to build a nonlinear classification model, but my current problem has a continuous output. I\'ve been searching for

Answer 1

I had the same struggle, most online courses use ANNs for classification, but you never actually solve a regression problem with them in the courses.

What does make an ANN non-linear? The activation function.

Even if you have an ANN with thousands of perceptrons and hidden units, if all the activations are linear (or not activated at all) you are just training a plain linear regression.

But be careful, some activations functions (like sigmoid), have a range of values that act as a linear function and you may get stuck with a linear model even with non-linear activations.

How to predict continuous output with an ANN? The same way as when you classify.

It is the same problem, you just backpropagate the error (label - prediction) and update the weights. But don't forget to CHANGE THE ACTIVATION FUNCTION of the output layer to a continuous function (maybe ReLu if all labels are positive or don't activate the output at all), the intermediate hidden layers can be activated however you wish.

For small regression problems with ANNs you may need to start with a veeeeeery small learning rate since there will be lots of variance since the error will be "unbounded" at first.

Hope this helps :)