SPARK ML, Naive Bayes classifier: high probability prediction for one class

Question

I am using Spark ML to optimise a Naive Bayes multi-class classifier.

I have about 300 categories and I am classifying text documents. The training set is balanced enoug

Answer 1

Lets start with a very informal description of Naive Bayes classifier. If C is a set of all classes and d is a document and x_i are the features, Naive Bayes returns:

Since P(d) is the same for all classes we can simplify this to

where

Since we assume that features are conditionally independent (that is why it is naive) we can further simplify this (with Laplace correction to avoid zeros) to:

Problem with this expression is that in any non-trivial case it is numerically equal to zero. To avoid we use following property:

and replace initial condition with:

These are the values you get as the raw probabilities. Since each element is negative (logarithm of the value in (0, 1]) a whole expression has negative value as well. As you discovered by yourself these values are further normalized so the maximum value is equal to 1 and divided by the sum of the normalized values

It is important to note that while values you get are not strictly P(c|d) they preserve all important properties. The order and ratios are exactly (ignoring possible numerical issues) the same. If none other class gets prediction close to one it means that, given the evidence, it is a very strong prediction. So it is actually something you want to see.