问题
For a very simple classification problem where I have a target vector [0,0,0,....0] and a prediction vector [0,0.1,0.2,....1] would cross-entropy loss converge better/faster or would MSE loss? When I plot them it seems to me that MSE loss has a lower error margin. Why would that be?
Or for example when I have the target as [1,1,1,1....1] I get the following:
回答1:
You sound a little confused...
- Comparing the values of MSE & cross-entropy loss and saying that one is lower than the other is like comparing apples to oranges
- MSE is for regression problems, while cross-entropy loss is for classification ones; these contexts are mutually exclusive, hence comparing the numerical values of their corresponding loss measures makes no sense
- When your prediction vector is like
[0,0.1,0.2,....1](i.e. with non-integer components), as you say, the problem is a regression (and not a classification) one; in classification settings, we usually use one-hot encoded target vectors, where only one component is 1 and the rest are 0 - A target vector of
[1,1,1,1....1]could be the case either in a regression setting, or in a multi-label multi-class classification, i.e. where the output may belong to more than one class simultaneously
On top of these, your plot choice, with the percentage (?) of predictions in the horizontal axis, is puzzling - I have never seen such plots in ML diagnostics, and I am not quite sure what exactly they represent or why they can be useful...
If you like a detailed discussion of the cross-entropy loss & accuracy in classification settings, you may have a look at this answer of mine.
回答2:
As complement to the accepted answer, I will answer the following questions
- What is the interpretation of MSE loss and cross entropy loss from probability perspective.
- Why cross entropy is used for classification and MSE is used for linear regression?
TL;DR Use MSE loss if (random) target variable is from Gaussian distribution and categorical cross entropy loss if (random) target variable is from Multinomial distribution.
MSE(Mean squared error)
One of the assumptions of the linear regression is multi-variant normality. From this it follows that the target variable is normally distributed(more on the assumptions of linear regression can be found here and here).
Gaussian distribution(Normal distribution) with mean
and variance 
is given by=%5Cfrac%7B1%7D%7B%5Csqrt%7B2%5Cpi%5Csigma%5E%7B2%7D%7D%7De%5E%7B-%5Cfrac%7B(x-%5Cmu)%5E2%7D%7B2%5Csigma%5E2%7D%7D)
Often in machine learning we deal with distribution with mean 0 and variance 1(Or we transform our data to have mean 0 and variance 1). In this case the normal distribution will be,
=%5Cfrac%7B1%7D%7B%5Csqrt%7B2%5Cpi%7D%7De%5E%7B-%5Cfrac%7Bx%5E2%7D%7B2%7D%7D)
This is called standard normal distribution.For normal distribution model with weight parameter
and precision(inverse variance) parameter 
, the probability of observing a single target t given input x is expressed by the following equation
=%5Cmathcal%7BN%7D(t|y(x,%5Cmathbf%7Bw%7D),%5Cbeta%5E%7B-1%7D))
, where )
is mean of the distribution and is calculated by model as=%5Csum_%7Bi=1%7D%5E%7Bm%7Dw_ix%5Ei)
Now the probability of target vector
given input 
can be expressed by
=%5Cprod_%7Bn=1%7D%5E%7BN%7D%5Cmathcal%7BN%7D(t_n|y(x_n,%5Cmathbf%7Bw%7D),%5Cbeta%5E%7B-1%7D)=)
)%5E2%7D%7B2%7D%7D)
Taking natural logarithm of left and right terms yields
=%5Cln%20%5Cprod_%7Bn=1%7D%5E%7BN%7D%5Cfrac%7B%5Cbeta%7D%7B%5Csqrt%7B2%5Cpi%7D%7De%5E%7B-%5Cbeta%5Cfrac%7B(t_n-y(x_n,w))%5E2%7D%7B2%7D%7D)
-t_n%5Cright%7D%5E2%2b%5Cfrac%7BN%7D%7B2%7D%5Cln%5Cbeta-%5Cfrac%7BN%7D%7B2%7D%5Cln(2%5Cpi)=)
)
Where
is log likelihood of normal function. Often training a model involves optimizing the likelihood function with respect to . Now maximum likelihood function for parameter is given by (constant terms with respect to can be omitted),
=-%5Cfrac%7B%5Cbeta%7D%7B2%7D%5Csum_%7Bn=1%7D%5EN%5Cleft%7By(x_n,w)-t_n%5Cright%7D%5E2)
For training the model omitting the constant
doesn't affect the convergence.
=%5Csum_%7Bn=1%7D%5EN%5Cleft%7By(x_n,w)-t_n%5Cright%7D%5E2)
This is called squared error and taking the mean yields mean squared error.=%5Cfrac%7B1%7D%7BN%7D%5Csum_%7Bn=1%7D%5EN%5Cleft%7By(x_n,w)-t_n%5Cright%7D%5E2)
,
Cross entropy
Before going into more general cross entropy function, I will explain specific type of cross entropy - binary cross entropy.
Binary Cross entropy
The assumption of binary cross entropy is probability distribution of target variable is drawn from Bernoulli distribution. According to Wikipedia
Bernoulli distribution is the discrete probability distribution of a random variable which takes the value 1 with probability p and the value 0 with probability q=1-p
Probability of Bernoulli distribution random variable is given by
=p%5Ek(1-p)%5E%7B1-k%7D)
, where 
and p is probability of success.
This can be simply written as
=p%5Ey(1-p)%5E%7B1-y%7D)
Taking negative natural logarithm of both sides yields
=-y%5Cln(p)%2d(1-y)%5Cln(1-p))
, this is called binary cross entropy.
Categorical cross entropy
Generalization of the cross entropy follows the general case when the random variable is multi-variant(is from Multinomial distribution ) with the following probability distribution
=%5Cprod_%7Bn=1%7D%5E%7BN%7Dp_n%5E%7By_n%7D(1-p_n)%5E%7B1-y_n%7D=%7Bp_n%7D%5E%7B%5Csum_%7Bn=1%7D%5E%7BN%7Dy_n%7D(1-p_n)%5E%7Bn-%5Csum_%7Bn=1%7D%5E%7BN%7Dy_n%7D%7D)
Taking negative natural logarithm of both sides yields categorical cross entropy loss.
=-(%5Csum_%7Bn=1%7D%5E%7BN%7Dy_n%5Cln(p_n)%2b(1-y_n)%5Cln(1-p_n)))
,
Conclusion
Cross entropy is used when the target variable is from Bernoulli distribution and MSE is used when the target variable is from Normal distribution.
来源：https://stackoverflow.com/questions/49322197/comparing-mse-loss-and-cross-entropy-loss-in-terms-of-convergence