问题
[SOLVED]
My mistake was that I did not realise that entropy is 0 if all are of one type. Thus if all are positive, entropy is 0 and if all are negative it is zero as well. Entropy will be 1 if equal amount are positive and negative.
It does not make sense that one would get negative information gain.
However based on this example I am getting a negative information gain.
here is the data:
And if I calculate the information gain on the Humidity attribute I get this:
Obviously I am missing something here.
EDIT: To clarify how I understand it.
Entropy of the whole system is defined as:
Which in this case then is:
And the information gain per atribute is defined as:
Which for humidity I calculate to:
Entropy of system - (1/4)Entropy of Humidity Normal - (3/4)Entropy of Humidity High
As per this Libre Office Calc:
Or is my understanding of the formula for information gain for an attribute incorrect?
回答1:
To begin with, I'm assuming your S variable is EnjoySport. (I think you could phrase the text more clearly, BTW.)
So the entropy of S is 0.8113, but that's the last part with which I agree.
The entropy of S given Normal is 0, as it is deterministic.
The entropy of S given High is 0.91829583405448945, but you need to multiply that by 0.75, because that is the probability of Normal. So that gives you 0.68872187554086706.
The difference is non-negative, as expected.
Note that the Information gain is the expected difference in Entropy, and the expectation needs to take into account the probability of the conditioned event.
来源：https://stackoverflow.com/questions/31362320/why-am-i-getting-a-negative-information-gain