gensim.LDAMulticore throwing exception:

问题

I am running LDAMulticore from the python gensim library, and the script cannot seem to create more than one thread. Here is the error:

  Traceback (most recent call last):
  File "/usr/lib64/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib64/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib64/python2.7/multiprocessing/pool.py", line 97, in worker
    initializer(*initargs)
  File "/usr/lib64/python2.7/site-packages/gensim/models/ldamulticore.py", line 333, in worker_e_step
    worker_lda.do_estep(chunk)  # TODO: auto-tune alpha?
  File "/usr/lib64/python2.7/site-packages/gensim/models/ldamodel.py", line 725, in do_estep
    gamma, sstats = self.inference(chunk, collect_sstats=True)
  File "/usr/lib64/python2.7/site-packages/gensim/models/ldamodel.py", line 655, in inference
    ids = [int(idx) for idx, _ in doc]
TypeError: 'int' object is not iterable
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.7/threading.py", line 765, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib64/python2.7/multiprocessing/pool.py", line 325, in _handle_workers
    pool._maintain_pool()
  File "/usr/lib64/python2.7/multiprocessing/pool.py", line 229, in _maintain_pool
    self._repopulate_pool()
  File "/usr/lib64/python2.7/multiprocessing/pool.py", line 222, in _repopulate_pool
    w.start()
  File "/usr/lib64/python2.7/multiprocessing/process.py", line 130, in start
    self._popen = Popen(self)
  File "/usr/lib64/python2.7/multiprocessing/forking.py", line 121, in __init__
    self.pid = os.fork()
OSError: [Errno 12] Cannot allocate memory

I'm creating my LDA model like this:

ldamodel = LdaMulticore(corpus, num_topics=50, id2word = dictionary, workers=3)

I have actually asked another question about this script, so the full script can be found here:

Gensim LDA Multicore Python script runs much too slow

If it's relevant, I'm running this on a CentOS server. Let me know if I should include any other information.

Any help is appreciated!

回答1:

OSError: [Errno 12] Cannot allocate memory sounds like you are running out of RAM.

Check your available free memory and swap.

You can try to to reduce the number of threads with the workers parameter or the number of documents to be used in each training chunk with the chunksize parameter.

来源：https://stackoverflow.com/questions/54432791/gensim-ldamulticore-throwing-exception