Why does the Logistic Regression cost go negative and not correct?

问题

I am implementing logistic regression in Matlab. The data is normalized (mean and std). I understand that depending on your learning rate you may overshoot the optimal point. But doesn't that mean your cost starts going up? In my case the cost goes into negative territory, I don't understand why.

Here is the standard (I think?) cost and weight update rule

function J = crossEntropyError(w, x, y)
  h = sigmoid(x*w);
  J = (-y'*log(h) - (1-y')*log(1-h));
end

Weight update:

function w = updateWeights(alpha, w, x, y)      
  h = sigmoid(x*w);
  gradient = x'*(h-y);
  w = w - alpha*gradient;
end

This is what happens with my cost, x-axis is the iteration:

This makes no sense. When hitting 0, isn't it supposed to self-correct and go in the other direction? That is, since the derivative points to the minimum. I've played with the learning rate, here it's set to a tiny 0.0001. But it makes no difference, same pattern. What's the issue? There must be something really wrong here, I can't find it though.

回答1:

So I realized my mistake, it's quite silly. I was using a dataset where the labels are not boolean (0 or 1) which resulted in the cross entropy error above. The code was correct, but not suitable for labels with non-boolean data.

I would delete the question but I wouldn't want my account to be blocked. Maybe it can help someone?

回答2:

Please try this cost function

J = -1/(m)*sum(y.*log(sigmoid(x*w))+(1-y).*log(1-sigmoid(x*w);

as

m = sieze(x)

and

gradient = zeros(size(theta)); 

 for i = 1:size(theta),
     gradient(i)= (1/m)*sum((sigmoid(X*theta)-y).*X(:,i));

 end;

来源：https://stackoverflow.com/questions/40224707/why-does-the-logistic-regression-cost-go-negative-and-not-correct