R - problem with foreach %dopar% inside function called by optim

Question

Calling a function that includes foreach %dopar% construct from optim causes an error:

> workers <- startWorkers(6) # 6 cores
> 
> registerDoSMP(         


        

Answer 1

[[Edited]]

Your pf function and your "static table" x must be distributed to all worker nodes. You must read the documentation for your parallel library on how that works.

It seems to be that when run through optim, the pf function it finds is another one (probably stats::pf, which does not have an isPrebuilt argument).

Can you try renaming your pf function (for example to mypf)?

mypf <- pf # renaming the function

maxProb2 <- function(wp) {
  r <- foreach (i=s0:s1, .combine=c) %dopar% { mypf(i,x[i,5],wp,isPrebuilt=TRUE) }
  cat("w=",wp,"max=",sum(r),"\n")
  sum(r)
}

Or, if your pf function is part of a package with a namespace (say, mypackage), you could reference it like this: mypackage::pf

maxProb2 <- function(wp) {
  r <- foreach (i=s0:s1, .combine=c) %dopar% { mypackage::pf(i,x[i,5],wp,isPrebuilt=TRUE) }
  cat("w=",wp,"max=",sum(r),"\n")
  sum(r)
}

Answer 2

Quick fix for problem with foreach %dopar% is to reinstall these packages:

install.packages("doSNOW")

install.packages("doParallel") 

install.packages("doMPI")

As mentioned in various threads at StackOverflow, these are responsible for parallelism in R. Bug which existed in old versions of these packages is now removed. It worked in my case. I should mention that it will most likely help even though you are not using these packages in your project/package.

Answer 3

I ran into the same problem an the issue is with the environment not being included in the sub-threads. Your error

Error in { : task 1 failed - "could not find function "simple_fn""

can be reproduced by this very simple example:

simple_fn <- function(x)
    x+1

test_par <- function(){
    library("parallel")
    no_cores <- detectCores()
    library("foreach")
    cl<-makeCluster(no_cores)
    library("doSNOW")
    registerDoSNOW(cl)
    out <- foreach(i=1:10) %dopar% {
        simple_fn(i)
    }

    stopCluster(cl)
    return(out)
}

test_par()

Now all you need to to is to change the foreach(i=1:10) into foreach(i=1:10, .export=c("simple_fn")). If you want to export your complete global environment then just write .export=ls(envir=globalenv()) and you will have it for better or worse.