lda

I'm new to Sagemaker and am running some tests to measure the performance of NTM and LDA on AWS compared with LDA mallet and native Gensim LDA model. I'm wanting to inspect the trained models on Sagemaker and look at stuff like what words have the highest contribution for each topic. And also to get a measure of model coherence. I have been able to successfully get what words have the highest contribution for each topic for NTM on Sagemaker by downloading the output file untarring it and unzipping to expose 3 files params, symbol.json and meta.json. However, when I try to do the same process

from nltk.tokenize import RegexpTokenizer from stop_words import get_stop_words from nltk.stem.porter import PorterStemmer from gensim import corpora, models import gensim import csv import jieba import codecs from mpl_toolkits.mplot3d import axes3d import matplotlib.pyplot as plt import pyLDAvis.gensim from gensim import corpora from gensim.models import LdaModel def is_number(s): try: float(s) return True except ValueError: pass try: import unicodedata unicodedata.numeric(s) return True except (TypeError, ValueError): pass return False info = [] def data_g(filename): csv_reader = csv.reader

说明： 原文： http://chrisstrelioff.ws/sandbox/2014/11/13/getting_started_with_latent_dirichlet_allocation_in_python.html 本文包含了上文的主要内容。 关于LDA： LDA漫游指南 使用的python库lda来自： https://github.com/ariddell/lda 。 gensim 库也含有lda相关函数。 安装 $ pip install lda --user 示例 from __future__ import division, print_function import numpy as np import lda import lda.datasets # document-term matrix X = lda.datasets.load_reuters() print("type(X): {}".format(type(X))) print("shape: {}\n".format(X.shape)) print(X[:5, :5]) '''输出： type(X): <type 'numpy.ndarray'> shape: (395L, 4258L) [[ 1 0 1 0 0] [ 7 0 2 0 0] [ 0 0 0 1 10] [ 6 0 1