Julia parallel programming - Making existing function available to all workers

Question

I am faced with the following problem:

I have a function called TrainModel that runs for a very long time on a single thread. When it finishes computing

Answer 1

There is a Unix-only solution based on the PTools.jl package (https://github.com/amitmurthy/PTools.jl).

It relies on parallelism via forking instead of the Julia in-built mechanism. Forked processes are spawned with the same workspace as the main process, so all functions and variables are directly available to the workers.

This is a similar to the Fork clusters in R parallel package, so it can be used as the mclapply function.

The function of interest is pfork(n::Integer, f::Function, args...) and one noticeable difference with mclapply in R is that the function f must take as first argument the index of the worker.

An example:

Pkg.add("PTools")
Pkg.checkout("PTools") #to get the last version, else the package does not build at the time of writing

using PTools
f(workid,x) = x[workid] + 1
pfork(3, f, [1,2,3,4,5]) #Only the three first elements of the array will be computed

3-element Array{Any,1}:  
 2  
 3  
 4  

I expect that an interface to pfork will be built so that the first argument of the function will not need to be the index of the worker, but for the time being it can be used to solve the problem

Answer 2

The approach to return the function seems elegant but unfortunately, unlike JavaScript, Julia does not resolve all the variables when creating the functions. Technically, your training function could produce the source code of the function with literal values for all the trained parameters. Then pass it to each of the worker processes, which can parse it in their environment to a callable function.

I suggest to return a data structure that contains all the information to produce the trained function: weights of an ANN, support vectors, decision rules ... Define a the "trained" function on the worker processes, such that it will utilized the trained parameters. You might want to have the ability of saving the results of the training to disk anyway, so that you can easily re-produce your computations.