Using gensim I was able to extract topics from a set of documents in LSA but how do I access the topics generated from the LDA models?
When printing the lda.print_topics(10) the code gave the following error because print_topics() return a NoneType:
Traceback (most recent call last):
File "/home/alvas/workspace/XLINGTOP/xlingtop.py", line 93, in <module>
for top in lda.print_topics(2):
TypeError: 'NoneType' object is not iterable
The code:
from gensim import corpora, models, similarities
from gensim.models import hdpmodel, ldamodel
from itertools import izip
documents = ["Human machine interface for lab abc computer applications",
"A survey of user opinion of computer system response time",
"The EPS user interface management system",
"System and human system engineering testing of EPS",
"Relation of user perceived response time to error measurement",
"The generation of random binary unordered trees",
"The intersection graph of paths in trees",
"Graph minors IV Widths of trees and well quasi ordering",
"Graph minors A survey"]
# remove common words and tokenize
stoplist = set('for a of the and to in'.split())
texts = [[word for word in document.lower().split() if word not in stoplist]
for document in documents]
# remove words that appear only once
all_tokens = sum(texts, [])
tokens_once = set(word for word in set(all_tokens) if all_tokens.count(word) == 1)
texts = [[word for word in text if word not in tokens_once]
for text in texts]
dictionary = corpora.Dictionary(texts)
corpus = [dictionary.doc2bow(text) for text in texts]
# I can print out the topics for LSA
lsi = models.LsiModel(corpus_tfidf, id2word=dictionary, num_topics=2)
corpus_lsi = lsi[corpus]
for l,t in izip(corpus_lsi,corpus):
print l,"#",t
print
for top in lsi.print_topics(2):
print top
# I can print out the documents and which is the most probable topics for each doc.
lda = ldamodel.LdaModel(corpus, id2word=dictionary, num_topics=50)
corpus_lda = lda[corpus]
for l,t in izip(corpus_lda,corpus):
print l,"#",t
print
# But I am unable to print out the topics, how should i do it?
for top in lda.print_topics(10):
print top
After some messing around, it seems like print_topics(numoftopics) for the ldamodel has some bug. So my workaround is to use print_topic(topicid):
>>> print lda.print_topics()
None
>>> for i in range(0, lda.num_topics-1):
>>> print lda.print_topic(i)
0.083*response + 0.083*interface + 0.083*time + 0.083*human + 0.083*user + 0.083*survey + 0.083*computer + 0.083*eps + 0.083*trees + 0.083*system
...
I think syntax of show_topics has changed over time:
show_topics(num_topics=10, num_words=10, log=False, formatted=True)
For num_topics number of topics, return num_words most significant words (10 words per topic, by default).
The topics are returned as a list – a list of strings if formatted is True, or a list of (probability, word) 2-tuples if False.
If log is True, also output this result to log.
Unlike LSA, there is no natural ordering between the topics in LDA. The returned num_topics <= self.num_topics subset of all topics is therefore arbitrary and may change between two LDA training runs.
Are you using any logging? print_topics prints to the logfile as stated in the docs.
As @mac389 says, lda.show_topics() is the way to go to print to screen.
you can use:
for i in lda_model.show_topics():
print i[0], i[1]
Here is sample code to print topics:
def ExtractTopics(filename, numTopics=5):
# filename is a pickle file where I have lists of lists containing bag of words
texts = pickle.load(open(filename, "rb"))
# generate dictionary
dict = corpora.Dictionary(texts)
# remove words with low freq. 3 is an arbitrary number I have picked here
low_occerance_ids = [tokenid for tokenid, docfreq in dict.dfs.iteritems() if docfreq == 3]
dict.filter_tokens(low_occerance_ids)
dict.compactify()
corpus = [dict.doc2bow(t) for t in texts]
# Generate LDA Model
lda = models.ldamodel.LdaModel(corpus, num_topics=numTopics)
i = 0
# We print the topics
for topic in lda.show_topics(num_topics=numTopics, formatted=False, topn=20):
i = i + 1
print "Topic #" + str(i) + ":",
for p, id in topic:
print dict[int(id)],
print ""
I think it is alway more helpful to see the topics as a list of words. The following code snippet helps acchieve that goal. I assume you already have an lda model called lda_model.
for index, topic in lda_model.show_topics(formatted=False, num_words= 30):
print('Topic: {} \nWords: {}'.format(idx, [w[0] for w in topic]))
In the above code, I have decided to show the first 30 words belonging to each topic. For simplicity, I have shown the first topic I get.
Topic: 0
Words: ['associate', 'incident', 'time', 'task', 'pain', 'amcare', 'work', 'ppe', 'train', 'proper', 'report', 'standard', 'pmv', 'level', 'perform', 'wear', 'date', 'factor', 'overtime', 'location', 'area', 'yes', 'new', 'treatment', 'start', 'stretch', 'assign', 'condition', 'participate', 'environmental']
Topic: 1
Words: ['work', 'associate', 'cage', 'aid', 'shift', 'leave', 'area', 'eye', 'incident', 'aider', 'hit', 'pit', 'manager', 'return', 'start', 'continue', 'pick', 'call', 'come', 'right', 'take', 'report', 'lead', 'break', 'paramedic', 'receive', 'get', 'inform', 'room', 'head']
I don't really like the way the above topics look so I usually modify my code to as shown:
for idx, topic in lda_model.show_topics(formatted=False, num_words= 30):
print('Topic: {} \nWords: {}'.format(idx, '|'.join([w[0] for w in topic])))
... and the output (first 2 topics shown) will look like.
Topic: 0
Words: associate|incident|time|task|pain|amcare|work|ppe|train|proper|report|standard|pmv|level|perform|wear|date|factor|overtime|location|area|yes|new|treatment|start|stretch|assign|condition|participate|environmental
Topic: 1
Words: work|associate|cage|aid|shift|leave|area|eye|incident|aider|hit|pit|manager|return|start|continue|pick|call|come|right|take|report|lead|break|paramedic|receive|get|inform|room|head
Recently, came across a similar issue while working with Python 3 and Gensim 2.3.0. print_topics() and show_topics() weren't giving any error but also not printing anything. Turns out that show_topics() returns a list. So one can simply do:
topic_list = show_topics()
print(topic_list)
You can also export the top words from each topic to a csv file. topn controls how many words under each topic to export.
import pandas as pd
top_words_per_topic = []
for t in range(lda_model.num_topics):
top_words_per_topic.extend([(t, ) + x for x in lda_model.show_topic(t, topn = 5)])
pd.DataFrame(top_words_per_topic, columns=['Topic', 'Word', 'P']).to_csv("top_words.csv")
The CSV file has the following format
Topic Word P
0 w1 0.004437
0 w2 0.003553
0 w3 0.002953
0 w4 0.002866
0 w5 0.008813
1 w6 0.003393
1 w7 0.003289
1 w8 0.003197
...
****This code works fine but I want to know the topic name instead of Topic: 0 and Topic:1, How do i know which topic this word comes in**?**
for index, topic in lda_model.show_topics(formatted=False, num_words= 30):
print('Topic: {} \nWords: {}'.format(idx, [w[0] for w in topic]))
Topic: 0
Words: ['associate', 'incident', 'time', 'task', 'pain', 'amcare', 'work', 'ppe', 'train', 'proper', 'report', 'standard', 'pmv', 'level', 'perform', 'wear', 'date', 'factor', 'overtime', 'location', 'area', 'yes', 'new', 'treatment', 'start', 'stretch', 'assign', 'condition', 'participate', 'environmental']
Topic: 1
Words: ['work', 'associate', 'cage', 'aid', 'shift', 'leave', 'area', 'eye', 'incident', 'aider', 'hit', 'pit', 'manager', 'return', 'start', 'continue', 'pick', 'call', 'come', 'right', 'take', 'report', 'lead', 'break', 'paramedic', 'receive', 'get', 'inform', 'room', 'head']
来源：https://stackoverflow.com/questions/15016025/how-to-print-the-lda-topics-models-from-gensim-python