Unable to send multiple arguments to concurrrent.futures.Executor.map()

前端 未结 1 681
耶瑟儿~
耶瑟儿~ 2021-01-29 02:39

I am trying to combine the solutions provided in both of these SO answers - Using threading to slice an array into chunks and perform calculation on each chunk and reassemble th

相关标签:
1条回答
  • 2021-01-29 03:20

    Using the solution provided by user mkorvas as shown here - How to pass a function with more than one argument to python concurrent.futures.ProcessPoolExecutor.map()? I was able to solve my problem as shown in the solution here -

    import psutil
    import numpy as np
    import sys
    from concurrent.futures import ThreadPoolExecutor
    from functools import partial
    
    def main():
       testThread()
    
    def testThread():
    
       minLat = -65.76892
       maxLat =  66.23587
       minLon =  -178.81404
       maxLon =  176.2949
       latGrid = np.arange(minLat,maxLat,0.05)
       lonGrid = np.arange(minLon,maxLon,0.05)
       print(latGrid.shape,lonGrid.shape)
       gridLon,gridLat = np.meshgrid(latGrid,lonGrid)
       grid_points = np.c_[gridLon.ravel(),gridLat.ravel()]
       print(grid_points.shape)
       n_jobs = psutil.cpu_count(logical=False)
       chunk = np.array_split(grid_points,n_jobs,axis=0)
       x = ThreadPoolExecutor(max_workers=n_jobs) 
    
    
      maxDistance = 4.3
      func = partial(performCalc,maxDistance)
    
      results = x.map(func,chunk)
    
    
     def performCalc(maxDistance,chunk):
    
         print(maxDistance)
         return chunk
    
    main()
    

    What apparently one needs to do(and I do not know why and maybe somebody can clarify in another answer) is you need to switch the order of input to the function performCalc()

    as shown here -

          def performCalc(maxDistance,chunk):
    
              print(maxDistance)
              return chunk
    
    0 讨论(0)
提交回复
热议问题