snowfall | 易学教程

parallel regression in R (maybe with snowfall)

阅读更多关于 parallel regression in R (maybe with snowfall)

问题 I'm trying to run R in parallel to run a regression. I'm trying to use the snowfall library (but am open to any approach). Currently, I'm running the following regression which is taking an extremely long time to run. Can someone show me how to do this? sales_day_region_ctgry_lm <- lm(log(sales_out+1)~factor(region_out) + date_vector_out + factor(date_vector_out) + factor(category_out) + mean_temp_out) I've started down the following path: library(snowfall) sfInit(parallel = TRUE, cpus=4,

R: making cluster in doParallel / snowfall hangs

阅读更多关于 R: making cluster in doParallel / snowfall hangs

问题 I've got two servers on a LAN with fresh installs of Centos 6.4 minimal and R 3.0.1. Both computers have doParallel, snow, and snowfall packages installed. The servers can ssh to each other fine. When I attempt to make clusters in either direction, I get a prompt for a password, but after entering the password, it just hangs there indefinately. makePSOCKcluster("192.168.1.1",user="username") How can I troubleshoot this? edit: I also tried calling makePSOCKcluster on the above-mentioned

R: making cluster in doParallel / snowfall hangs

阅读更多关于 R: making cluster in doParallel / snowfall hangs

Using snow (and snowfall) with AWS for parallel processing in R

阅读更多关于 Using snow (and snowfall) with AWS for parallel processing in R

问题 In relation to my earlier similar SO question , I tried using snow/snowfall on AWS for parallel computing. What I did was: In the sfInit() function, I provided the public DNS to socketHosts parameter like so sfInit(parallel=TRUE,socketHosts =list("ec2-00-00-00-000.compute-1.amazonaws.com")) The error returned was Permission denied (publickey) I then followed the instructions (I presume correctly!) on http://www.imbi.uni-freiburg.de/parallel/ in the 'Passwordless Secure Shell (SSH) login'

How to manage parallel processing with animated ggplot2-plot?

阅读更多关于 How to manage parallel processing with animated ggplot2-plot?

问题 I'm trying to build an animated barplot with ggplot2 and magick that's growing on a "day per day" base. Unfortunately, I've got tenthousands of entries in my dataset (dates for each day for several years and different categories), which makes processing very slow. Thus, I'm using the snow package to speed up processing time. However, I ran into trouble when splitting my data and calling ggplot() in a cluster. magick requires to split the data per date for animation and snow requires splitting

How to setup workers for parallel processing in R using snowfall and multiple Windows nodes?

阅读更多关于 How to setup workers for parallel processing in R using snowfall and multiple Windows nodes?

问题 I’ve successfully used snowfall to setup a cluster on a single server with 16 processors. require(snowfall) if (sfIsRunning() == TRUE) sfStop() number.of.cpus <- 15 sfInit(parallel = TRUE, cpus = number.of.cpus) stopifnot( sfCpus() == number.of.cpus ) stopifnot( sfParallel() == TRUE ) # Print the hostname for each cluster member sayhello <- function() { info <- Sys.info()[c("nodename", "machine")] paste("Hello from", info[1], "with CPU type", info[2]) } names <- sfClusterCall(sayhello) print

How to log using futile logger from within a parallel method in R?

阅读更多关于 How to log using futile logger from within a parallel method in R?

问题 I am using futile logger in R for logging. I have a parallel algorithm implemented using snowfall in R. Each core of the parallel process logs an intermediate output in the logger. But this output is not showing up in the logger? Can we log using futile logger from within a parallel job using snowfall? adding how it was done: My specific case was a bit different. I am calling a C function from R using a shared object that I created. The function is an iterative algorithm and I need output to

R: making cluster in doParallel / snowfall hangs

阅读更多关于 R: making cluster in doParallel / snowfall hangs

I've got two servers on a LAN with fresh installs of Centos 6.4 minimal and R 3.0.1. Both computers have doParallel, snow, and snowfall packages installed. The servers can ssh to each other fine. When I attempt to make clusters in either direction, I get a prompt for a password, but after entering the password, it just hangs there indefinately. makePSOCKcluster("192.168.1.1",user="username") How can I troubleshoot this? edit: I also tried calling makePSOCKcluster on the above-mentioned computer with a host that IS capable of being used as a slave (from other computers), but it still hangs. So,

How to output a message in snowfall?

阅读更多关于 How to output a message in snowfall?

问题 I am conducting a simulation study using snowfall package on Windows 7. I like to print out a message for every 10 runs to main R console to monitor the progress, but it fails to do so. ie. nothing is printed Any help will be much appreciated. runsim = function(nsim,n,mean,var){ cov = 0 for(i in 1:nsim){ if ( i %% 10==0) cat("\n Running simulation",i) dat = function1(n,mean,var) cov = ... } cov / nsim } sfExport("function1","runsim") sfLibrary(library1) wrapper = function(n){ runsim(100,n,0.5

Communication of parallel processes: what are my options?

阅读更多关于 Communication of parallel processes: what are my options?

问题 I'm trying to dig a bit deeper into parallelziation of R routines. What are my options with respect to the communication of a bunch of "worker" processes regarding the communication between the respective workers ? the communication of the workers with the " master " process? AFAIU, there's no such thing as a " shared environment/shared memory " that both the master as well as all worker processes have access to, right? The best idea I came up with so far is to base the communication on