calling spark-ec2 from within an EC2 instance: ssh connection to host refused

Question

In order to run Amplab\'s training exercises, I\'ve create a keypair on us-east-1 , have installed the training scripts (git clone git://github.com/amplab

Answer 1

This is most likely caused by SSH taking a long time to start up on the instances, causing the 120 second timeout to expire before the machines could be logged into. You should be able to run

./spark-ec2 -i ~/.ssh/myspark.pem -r us-east-1  -k myspark --copy launch --resume try1

(with the --resume flag) to continue from where things left off without re-launching new instances. This issue will be fixed in Spark 1.2.0, where we have a new mechanism that intelligently checks the SSH status rather than relying on a fixed timeout. We're also addressing the root causes behind the long SSH startup delay by building new AMIs.