multiprocessing

问题 I'm trying to create a code to download N files at the same type using pysftp and multiprocessing libs. I made a basic python training, got pieces of codes and combined them into one, but I can't get it work out. I'd appreciate if somebody helps me with that. The error occurs after the vFtp.close() command. In the part that suppose to start simultaneous downloads. from multiprocessing import Pool import pysftp import os vHost='10.11.12.13' vLogin='admin' vPwd='pass1234' vFtpPath='/export/home

问题 All the examples that I see for using spacy just read in a single text file (that is small in size). How does one load a corpus of text files into spacy? I can do this with textacy by pickling all the text in the corpus: docs = textacy.io.spacy.read_spacy_docs('E:/spacy/DICKENS/dick.pkl', lang='en') for doc in docs: print(doc) But I am not clear as to how to use this generator object (docs) for further analysis. Also, I would rather use spacy, not textacy. spacy also fails to read in a single

问题 All the examples that I see for using spacy just read in a single text file (that is small in size). How does one load a corpus of text files into spacy? I can do this with textacy by pickling all the text in the corpus: docs = textacy.io.spacy.read_spacy_docs('E:/spacy/DICKENS/dick.pkl', lang='en') for doc in docs: print(doc) But I am not clear as to how to use this generator object (docs) for further analysis. Also, I would rather use spacy, not textacy. spacy also fails to read in a single

问题 The documentation for multiprocessing states the following about Pool.join() : Wait for the worker processes to exit. One must call close() or terminate() before using join() . I know that Pool.close() prevents any other task from being submitted to the pool; and that Pool.join() waits for the pool to finish before proceeding with the parent process. So, why can I not call Pool.join() before Pool.close() in the case when I want to reuse my pool for performing multiple tasks and then finally

问题 The documentation for multiprocessing states the following about Pool.join() : Wait for the worker processes to exit. One must call close() or terminate() before using join() . I know that Pool.close() prevents any other task from being submitted to the pool; and that Pool.join() waits for the pool to finish before proceeding with the parent process. So, why can I not call Pool.join() before Pool.close() in the case when I want to reuse my pool for performing multiple tasks and then finally

问题 The Multiprocessing module is quite confusing for python beginners specially for those who have just migrated from MATLAB and are made lazy with its parallel computing toolbox. I have the following function which takes ~80 Secs to run and I want to shorten this time by using Multiprocessing module of Python. from time import time xmax = 100000000 start = time() for x in range(xmax): y = ((x+5)**2+x-40) if y <= 0xf+1: print('Condition met at: ', y, x) end = time() tt = end-start #total time

问题 I am dealing with a huge table where I have to do query. I decided to do so by chunking my data based on user_id and every time read and write into the sql. from sqlalchemy import create_engine engine = create_engine('mysql+pymysql://') q1 = "SELECT max(id) FROM users" max_users = pd.read_sql(q1, engine) max_users = max_users.iloc[0][0] # since user_ids start from 1 to ... I make the split based on that data = range(max_users) chunks = [list(data[x:x+1000]) for x in range(0, len(data), 1000)]

问题 I am confused as to what _number_left is supposed to return. I assumed it was the number tasks remaining in the pool, but it does not appear to provide the correct value in my code. For example, if I have a pool of 10 workers counting to the number 1000, I would expect result._number_left to countdown from 1000. However, it only tells me I have 40 left until the code is about finished. Am I missing something here? Code: import multiprocessing import time def do_something(x): print x time

问题 I am confused as to what _number_left is supposed to return. I assumed it was the number tasks remaining in the pool, but it does not appear to provide the correct value in my code. For example, if I have a pool of 10 workers counting to the number 1000, I would expect result._number_left to countdown from 1000. However, it only tells me I have 40 left until the code is about finished. Am I missing something here? Code: import multiprocessing import time def do_something(x): print x time

问题 In a file called test.py I have print 'i am cow' import multi4 print 'i am cowboy' and in multi4.py I have import multiprocessing as mp manager = mp.Manager() print manager I am confused by the way this code operates. At the command line, if I type python and then in the python environment if I type import test.py I get the expected behavior: Python 2.7.3 (default, Apr 10 2012, 23:31:26) [MSC v.1500 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information