Parallel processing in R with H2O

Question

I am setting up a piece of code to parallel processes some computations for N groups in my data using foreach.

I have a computation that involves a call to

Answer 1

Following Erin LeDell's answer, I just wanted to add that in many cases a decent practical solution can be something in between #1 and #3. To increase CPU utilization and still save RAM you can use multiple H2O instances in parallel, but they each can use multiple cores without much performance loss relative to running more instances with only one core.

I ran an experiment using a relatively small 40MB dataset (240K rows, 22 columns) on a 36 core server.

• Case 1: Use all 36 cores (nthreads=36) to estimate 120 GBM models (with default hyper-parameters) sequentially.

• Case 2: Use foreach to run 4 H2O instances on this machine, each using 9 cores to estimate 30 GBM default models sequentially (total = 120 estimations).

• Case 3: Use foreach to run 12 H2O instances on this machine, each using 3 cores to estimate 10 GBM default models sequentially (total = 120 estimations).

Using 36 cores estimating a single GBM model on this dataset is very inefficient. CPU utilization in Case 1 is jumping around a lot, but is on average below 50%. So there is definitely something to gain using more than one H2O instance at a time.

• Runtime Case 1: 264 seconds
• Runtime Case 2: 132 seconds
• Runtime Case 3: 130 seconds

Given the small improvement from 4 to 12 H2O instances, I did not even run 36 H2O instances each using one core in parallel.