Julia parallel programming - Making existing function available to all workers

Question

I am faced with the following problem:

I have a function called TrainModel that runs for a very long time on a single thread. When it finishes computing

Answer 1

There is a Unix-only solution based on the PTools.jl package (https://github.com/amitmurthy/PTools.jl).

It relies on parallelism via forking instead of the Julia in-built mechanism. Forked processes are spawned with the same workspace as the main process, so all functions and variables are directly available to the workers.

This is a similar to the Fork clusters in R parallel package, so it can be used as the mclapply function.

The function of interest is pfork(n::Integer, f::Function, args...) and one noticeable difference with mclapply in R is that the function f must take as first argument the index of the worker.

An example:

Pkg.add("PTools")
Pkg.checkout("PTools") #to get the last version, else the package does not build at the time of writing

using PTools
f(workid,x) = x[workid] + 1
pfork(3, f, [1,2,3,4,5]) #Only the three first elements of the array will be computed

3-element Array{Any,1}:  
 2  
 3  
 4  

I expect that an interface to pfork will be built so that the first argument of the function will not need to be the index of the worker, but for the time being it can be used to solve the problem