Julia parallel programming - Making existing function available to all workers

Question

I am faced with the following problem:

I have a function called TrainModel that runs for a very long time on a single thread. When it finishes computing

Answer 1

The approach to return the function seems elegant but unfortunately, unlike JavaScript, Julia does not resolve all the variables when creating the functions. Technically, your training function could produce the source code of the function with literal values for all the trained parameters. Then pass it to each of the worker processes, which can parse it in their environment to a callable function.

I suggest to return a data structure that contains all the information to produce the trained function: weights of an ANN, support vectors, decision rules ... Define a the "trained" function on the worker processes, such that it will utilized the trained parameters. You might want to have the ability of saving the results of the training to disk anyway, so that you can easily re-produce your computations.