doParallel “foreach” inconsistently inherits objects from parent environment: “Error in { : task 1 failed - ”could not find function…"

Question

I have a problem with foreach that I just can\'t figure out. The following code fails on two Windows computers I\'ve tried, but succeeds on three Linux computers, all runnin

Answer 1

@Tensibai is right. When trying to use doParallel on Windows, you have to "export" the functions that you want to use that are not in the current scope. In my experience, the way I've made this work is with the following (redacted) example.

format_number <- function(data) {
  # do stuff that requires stringr
}

format_date_time <- function(data) {
  # do stuff that requires stringr
}

add_direction_data <- function(data) {
  # do stuff that requires dplyr
}

parse_data <- function(data) {
  voice_start <- # vector of values
  voice_end <- # vector of values
  target_phone_numbers <- # vector of values
  parse_voice_block <- function(block_start, block_end, number) {
    # do stuff
  }

  number_of_cores <- parallel::detectCores() - 1
  clusters <- parallel::makeCluster(number_of_cores)
  doParallel::registerDoParallel(clusters)
  data_list <- foreach(i = 1:length(voice_start), .combine=list,
                       .multicombine=TRUE, 
                       .export = c("format_number", "format_date_time", "add_direction_data"), 
                       .packages = c("dplyr", "stringr")) %dopar% 
                       parse_voice_block(voice_start[i], voice_end[i], target_phone_numbers[i])
  doParallel::stopCluster(clusters)
  output <- plyr::rbind.fill(data_list)
}

Since the first three functions aren't included in my current environment, doParallel would ignore them when firing up the new instances of R, but it would know where to find parse_voice_block since it's within the current scope. In addition, you need to specify what packages should be loaded in each new instance of R. As Tensibai stated, this is because you're not running forking the process, but instead firing up multiple instances of R and running commands simultaneously.

Answer 2

It's rather unfortunate that when you register doParallel using:

registerDoParallel(2)

then doParallel uses mclapply on Linux and Mac OS X, but clusterApplyLB with an implicitly created cluster object on Windows. This often causes code to work on Linux but fail on Windows because the workers are clones of the master when using mclapply due to fork. For that reason, I usually test my code using:

cl <- makePSOCKcluster(2)
registerDoParallel(cl)

to make sure I'm loading all necessary packages and exporting all necessary functions and variables, and then switch back to registerDoParallel(2) to get the benefit of mclapply on platforms that support it.

Note that the .packages and .export options are ignored when doParallel uses mclapply, but I recommend always using them for portability.

---

The auto-export feature of foreach doesn't work quite as smoothly when using it inside a function because foreach is rather conservative about what to auto-export. It seems pretty safe to auto-export variables and functions that are defined in the current environment, but outside of that seems risky to me because of the complexity of R's scoping rules.

I tend to agree with your comment that your two work-arounds aren't very stable for an actively developed package, but if f and g are defined in package foo, then you should use the foreach .package option to load the package foo on the workers:

g <- function(){
    r = foreach(x = 1:4, .packages='foo') %dopar% {
        return(x + f())
    }
    return(r)
}

Then f will be in the scope of g even though it is neither implicitly or explicitly exported by foreach. However, this does require that f is an exported function of foo (as opposed to an internal function), since the code executed by the workers isn't defined in foo, so it can only access exported functions. (Sorry for using the term "export" in two different ways, but it's hard to avoid.)

I'm always interested to hear comments such as yours because I'm always wondering if the auto-export rules should be tweaked. In this case, I'm thinking that if a foreach loop is executed by a function that is defined in a package, the cluster workers should auto-load that package without the need for the .packages option. I'll try to look into that and perhaps add this to the next release of doParallel and doSNOW.