I am using Mallet api to extract topic from twitter data and I have already extracted topics which are seems good topic. But I am facing problem to estimating K.
For example I fixed K value from 10 to 100. So, I have taken different number of topics from the data. But, now I would like to estimate which K is best. There are some algorithm I know as
- Perplexity
- Empirical likelihood
- Marginal likelihood (Harmonic mean method)
- Silhouette
I found a method model.estimate() which may be used to estimate with different value of K. But I am not getting any idea to show the value of K is best for the model. Does anyone give some idea about it with some sample code? Thanks.
I think the best algorithm is human judgement. Create topic models with different numbers of topics and look at them and take what you like. Sometimes you want to fine tune the number of topics (Say, you don't want a certain topic to be split into two, or you want a certain topic to be there and not merged into another one).
来源：https://stackoverflow.com/questions/31729227/how-to-evaluate-the-best-k-for-lda-using-mallet