multiprocessing

问题 I have to process a huge pandas.DataFrame (several tens of GB) on a row by row bases, where each row operation is quite lengthy (a couple of tens of milliseconds). So I had the idea to split up the frame into chunks and process each chunk in parallel using multiprocessing . This does speed-up the task, but the memory consumption is a nightmare. Although each child process should in principle only consume a tiny chunk of the data, it needs (almost) as much memory as the original parent process

问题 I am using concurrent.futures module to do multiprocessing and multithreading. I am running it on a 8 core machine with 16GB RAM, intel i7 8th Gen processor. I tried this on Python 3.7.2 and even on Python 3.8.2 import concurrent.futures import time takes list and multiply each elem by 2 def double_value(x): y = [] for elem in x: y.append(2 *elem) return y multiply an elem by 2 def double_single_value(x): return 2* x define a import numpy as np a = np.arange(100000000).reshape(100, 1000000)

问题 I have two large Pandas Dataframes (1GB+) with data that needs to be processed by multiple workers. I'm able to perform the operations without issues in a toy example with much smaller Dataframes (DFs). Below is my reproducible example. I've tried several routes: I am unable to take advantage of chunk . The DFs need to be sliced into specific pieces on an index before each piece is fed to the workers. And chunk can only slice them to an arbitrary length. Using starmap : This is what you see

问题 I assumed that the multiprocessing package used pickle to send things between processes. However, pickle pays attention to the __getstate__ and __setstate__ methods of an object. Multiprocessing seems to ignore them. Is this correct? Am I confused? To replicate, install docker, and type into command line $ docker run python:3.4 python -c "import pickle import multiprocessing import os class Tricky: def __init__(self,x): self.data=x def __setstate__(self,d): self.data=10 def __getstate__(self)

问题 This question already has answers here : Compulsory usage of if __name__==“__main__” in windows while using multiprocessing [duplicate] (2 answers) Closed 2 years ago . I'm fairly new to python programming and need some help understanding the python interpreter flow, especially in the case of multiprocessing. Please note that I'm running python 3.7.1 on Windows 10. Here is my simple experimental code and the output. import multiprocessing import time def calc_square(numbers, q): for n in

问题 Have called an external function which returns multiple values. def get_name(full_name): # you code return first_name, last_name In simple function call, I can get the results. from names import get_name first, last= get_name(full_name) But I need to use threading for the call to get the result values for the first and last variables. I failed in using a simple threading call. first, last= Threading.thread(get_name, args= (full_name,) Please help me to get the return values of the function

问题 Have called an external function which returns multiple values. def get_name(full_name): # you code return first_name, last_name In simple function call, I can get the results. from names import get_name first, last= get_name(full_name) But I need to use threading for the call to get the result values for the first and last variables. I failed in using a simple threading call. first, last= Threading.thread(get_name, args= (full_name,) Please help me to get the return values of the function

问题 I try to run this example code on Pycharm2018.3.3, it didn't work out. But the same code can run on the IDLE without any error. My environment is Python3.7 + windows10. from multiprocessing import Process, Queue def f(q): q.put([42, None, 'hello']) if __name__ == '__main__': q = Queue() p = Process(target=f, args=(q,)) p.start() print(q.get()) # prints "[42, None, 'hello']" p.join() Process Process-1: Traceback (most recent call last): File "C:\Users\WYM\AppData\Local\Programs\Python\Python37

问题 Using Python 2.7 on a Windows machine, I have a large pandas DataFrame (about 7 million rows and 20+ columns) from a SQL query that I'd like to filter by looping through IDs then run calculations on the resulting filtered data. I'd also like to do this in parallel. I know that if I try to do this with standard methods from the multiprocessing package in Windows, each process will generate a new instance of that large DataFrame for its own use and my memory will be eaten up. So I'm trying to

问题 Now I am in one of the threads of process A , I need to create new process B in current thread, and run in process B function MyFunc() . How can I do it ? I found how to create a child process from current process: click . But how can I run MyFunc() in this new process ? This 2 processes should run async, and not wait each other like in this example: // Wait until child process exits. WaitForSingleObject( pi.hProcess, INFINITE ); ETA: I work on windows 回答1: I assume you are running under