“foreach” loop : Using all cores in R (especially if we are sending sql queries inside foreach loop)

Question

I intend to use \"foreach\" to uitlize all the cores in my CPU. The catch is i need to send a sql query inside the loop. The script is working fine with normal \'for\' loop,

Answer 1

My suggestion is this: Move the database queries outside the loop, and lock access so you dont do parallel database queries. I think that will speed things up too, as you won't have parallel disk access, while still being able to do parallel processing.

Meaning (pseudo code) db = connect to database threadlock = lock();

parfor { threadlock.lock result = db query (pull all data here, as you cant process while you load without keeping the database locked) thread.unlock process resulting data (which is now just data, and not a sql object). }