Running parallel R on multiple hosts [closed]

问题

Can you please provide a script to run parallel cluster on 2 hosts ( amazon ec2) from scratch on Ubuntu Linux Machine ?

Specifications

The hosts are connected from local machine via identity key named amazon_key.pem
hostnames are supposed to be connected via internal IP address provided by Amazon cloud
keep the hostnames as rserver1 and rserver2 while setting up the cloud

回答1:

All these commands are run from local system. This has been written in such a way so that one can automate this code based on their need.

   HOST1=ip_of_server1
   HOST2=ip_of_server2

People who do not have pem file can avoid this. Else put your exact location of the pem(key) file present in your local system.

pem_file_loc="~"

for finding out the private IP , my region is eu-west-1 , please replace with your region. people who already knows private IP can ignore this and simply fill PIP1 and PIP2

PIP1=$(aws ec2 describe-instances --region eu-west-1 --filter "Name=ip-address,Values=${HOST1}" --query 'Reservations[].Instances[].[PrivateIpAddress]' --output text) #for finding out the private IP
PIP2=$(aws ec2 describe-instances --region eu-west-1 --filter "Name=ip-address,Values=${HOST2}" --query 'Reservations[].Instances[].[PrivateIpAddress]' --output text)

for ease of usage only. some people may not need this based on how you spin up your ec2 instance.

SSH_ARGS="-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -i ${pem_file_loc}/amazon_key.pem"

copy the pem file to the amazon machine , if pem file is not present then ignore this

rsync -e "ssh ${SSH_ARGS}" ${pem_file_loc}/amazon_key.pem ubuntu@${HOST1}:~/
rsync -e "ssh ${SSH_ARGS}" ${pem_file_loc}/amazon_key.pem ubuntu@${HOST2}:~/

Run this command to set up the ssh authentication on HOST1 , people without pem file , can directly login and run the steps between the two EOF . Also note that i am setting up rserver1 and rserver2 for the ease of coding. People who already have many host machines can avoid this and instead use the private IP for all below commands.

ssh -T $SSH_ARGS ubuntu@${HOST1} <<EOF
sudo sh -c 'echo ${PIP1} rserver1 >> /etc/hosts'
sudo sh -c 'echo ${PIP2} rserver2 >> /etc/hosts'
rm -rf ~/.ssh/id_rsa.pub ~/.ssh/id_rsa
ssh-keygen -t rsa -N '' -f ~/.ssh/id_rsa
cat ~/.ssh/id_rsa.pub | ssh -i amazon_key.pem -o StrictHostKeyChecking=no ubuntu@rserver2 'cat >> ~/.ssh/authorized_keys'
cat ~/.ssh/id_rsa.pub | ssh -i amazon_key.pem  -o StrictHostKeyChecking=no ubuntu@rserver1 'cat >> ~/.ssh/authorized_keys' #required for clustering
EOF

Run the same for HOST2

ssh -T $SSH_ARGS ubuntu@${HOST2} <<EOF
sudo sh -c 'echo ${PIP1} rserver1 >> /etc/hosts'
sudo sh -c 'echo ${PIP2} rserver2 >> /etc/hosts'
rm -rf ~/.ssh/id_rsa.pub ~/.ssh/id_rsa
ssh-keygen -t rsa -N '' -f ~/.ssh/id_rsa
cat ~/.ssh/id_rsa.pub | ssh -i amazon_key.pem  -o StrictHostKeyChecking=no ubuntu@rserver1 'cat >> ~/.ssh/authorized_keys'
EOF

These are the commands i took from the blog http://www.win-vector.com/blog/2016/01/running-r-jobs-quickly-on-many-machines/ .

Run these commands on R server of host1. Cross verify the number of cores you want to keep. My case i used as 11 . empirically it is good to keep as detectCores() - 1

machineAddresses <- list(
  list(host='rserver1',user='ubuntu',
       ncore=11),
  list(host='rserver2',user='ubuntu',
       ncore=11)
)

spec <- lapply(machineAddresses,
               function(machine) {
                 rep(list(list(host=machine$host,
                               user=machine$user)),
                     machine$ncore)
               })

spec <- unlist(spec,recursive=FALSE)

library("doParallel")
cl <- makeCluster(type='PSOCK',master=primary,spec=spec)
registerDoParallel(cl)
#this is purely based on your need , there are many articles on how to run parallel loops , the focus is mainly on multiple hosts
clusterExport(cl, varlist=ls(.GlobalEnv)) 

print(cl)
##run your commands
stopCluster(cl)

Just in case the command hangs , confirm the setup by running system("ssh ubuntu@rserver1") and system("ssh ubuntu@rserver2") . These commands should work if ssh is properly setup.

来源：https://stackoverflow.com/questions/44912893/running-parallel-r-on-multiple-hosts

标签

remote-server

ssh-keys

doparallel

rparallel