mclapply

I'm using the doSNOW package and more specifically the parLapply function to perform reclassification (and subsequently other operations) on a list of big raster datasets (OS: Windows x64). The code looks a little like this minimalistic example: library(raster) library(doSNOW) #create list containing test rasters x <- raster(ncol=10980,nrow=10980) x <- setValues(x,1:ncell(x)) list.x <- replicate( 9 , x ) #setting up cluster NumberOfCluster <- 8 cl <- makeCluster(NumberOfCluster) registerDoSNOW(cl) junk <- clusterEvalQ(cl,library(raster)) #perform calculations on each raster list.x <- parLapply

Based on this post here , I tried to write a script, seen here: library(parallel) library(doParallel) cl<-makeCluster(2,outfile='') registerDoParallel(cl) foreach(i=1:5, .packages='parallel') %dopar% { system.time(mclapply(1:10, function(x){rnorm(1e5)},mc.cores=2)) } stopCluster(cl) It worked intially but is now throwing up error codes: Error in unserialize(node$con) : error reading from connection Calls: <Anonymous> ... doTryCatch -> recvData -> recvData.SOCKnode -> unserialize Execution halted Error in unserialize(socklist[[n]]) : error reading from connection Error in unserialize(node$con)

I am using mclapply from within RStudio and would like to have an output to the console from each process but this seems to be suppressed somehow (as mentioned for example here: Is mclapply guaranteed to return its results in order? ). How could I get R Studio to print something like x <- mclapply(1:20, function(i) cat(i, "\n")) to the console? I've tried print(), cat(), write() but they all seem not to work. I also tried to set mc.silent = FALSE explicitly without an effect. Parallel processing with GUI's is problematic. I write a lot of parallel code and it's constantly crashing my colleague

I am implementing a parallel processing system which will eventually be deployed on a cluster, but I'm having trouble working out how the various methods of parallel processing interact. I need to use a for loop to run a big block of code, which contains several large list of matrices operations. To speed this up, I want to parallelise the for loop with a foreach(), and parallelise the list operations with mclapply. example pseudocode: cl<-makeCluster(2) registerDoParallel(cl) outputs <- foreach(k = 1:2, .packages = "various packages") { l_output1 <- mclapply(l_input1, function, mc.cores = 2)

I love the setting .progress = 'text' in plyr's llply . However, it causes my much anxiety to not know how far along an mclapply (from package multicore ) is since list items are sent to various cores and then collated at the end. I've been outputting messages like *currently in sim_id # ....* but that's not very helpful because it doesn't give me an indicator of what percentage of list items are complete (although it is helpful to know that that my script isn't stuck and moving along). Can someone suggest other ideas that would allow me to look at my .Rout file and get a sense of progress? I

I am using R 3.0.1 on Platform: x86_64-apple-darwin10.8.0 (64-bit) I am trying to use tm_map from the tm library. But when I execute the this code library(tm) data('crude') tm_map(crude, stemDocument) I get this error: Warning message: In parallel::mclapply(x, FUN, ...) : all scheduled cores encountered errors in user code Does anyone know a solution for this? I suspect you don't have the SnowballC package installed, which seems to be required. tm_map is supposed to run stemDocument on all the documents using mclapply . Try just running the stemDocument function on one document, so you can