How to set up cluster slave nodes (on Windows)

Question

I need to run thousands* of models on 15 machines (each of 4 cores), all Windows. I started to learn parallel, snow and snowfall packa

Answer 1

It's a shame how all these APIs (like parallel/snow/snowfall) are complex to work with, a lots of docs but not what you need... I have found an API which is very simple and goes straight to the ideas I sketched!! It is redis and doRedis R package (as recommended here). Finally a very simple tutorial is present! Just modified a bit and got this:

The workers need only R, doRedis package and this script:

require(doRedis)    
redisWorker('jobs', '10.0.0.7') # IP of the server

The master needs redis server running (installed the experimental windows binaries for Windows), and this R code:

require(doRedis)
registerDoRedis('jobs')
foreach(j=1:10,.combine=sum,.multicombine=TRUE) %dopar%
    ... # whatever you need to run
removeQueue('jobs')

Adding/removing workers is fully dynamic, no need to specify IPs at master, automatic "load balanancing", simple and no need for tons of docs! This solution fulfills all the requirements and even more - as stated in ?registerDoRedis:

The doRedis parallel back end tolerates faults among the worker processes and automatically resubmits failed tasks.

I don't know how complex this would be using the parallel/snow/snowfall with SOCKS/MPI/PVM/NWS, if it would be possible at all, but I guess very complex...

The only disadvantages of using redis I found:

• It is a database server. I wonder if this API exist somewhere without the need to install the database server which I don't need at all. I guess it must exist!
• There is a bug in the current doRedis package ("object '.doRedisGlobals' not found") with no solution yet and I am not able to install the old working doRedis 1.0.5 package into R 3.0.1.