Running Spark EC2 scripts with IAM role

问题

I am trying to run Spark EC2 scripts to launch a cluster under an IAM role which my user under my root account can assume.

According to this JIRA ticket, we can now specify --profile when running Spark EC2 scripts, and the comments on the pull request say that the --profile option refers to what I believe as the AWSCLI profile.

When I run the scripts as

ec2/spark-ec2 -k key-name -i key-name.pem -s 1 --profile myprofile --instance-type=t2.medium launch test-cluster

I get

Profile "myprofile" not found!

However, running

aws s3 ls s3://mybucket --profile myprofile

works as intended, leading my to think the IAM role was specified correctly in the ~/.aws/config (I don't think you specify IAM roles in the ~/.aws/credentials).

However, when I add a test profile to the ~/.aws/credentials as

[foobar]
aws_secret_access_key=xxxxxxx
aws_access_key_id=xxxxxxx

Spark finds the foobar profile. However, after adding

[foobar]
role_arn = arn:aws:iam::12345:role/MY_ROLE
aws_secret_access_key=xxxxxxx
aws_access_key_id=xxxxxxx

Spark finds the foobar profile, but it does not correctly log into the IAM role. I get

boto.exception.EC2ResponseError: EC2ResponseError: 400 Bad Request
<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>InvalidKeyPair.NotFound</Code><Message>The key pair 'key-name' does not exist</Message></Error></Errors><RequestID>fcebd475-a895-4a5b-9a29-9783fd6b7f3d</RequestID></Response>

This is because the key pair key-name does not exist under my user, but it does exist under the IAM role I need to assume. This tells me Spark is not properly logging into the IAM role.

My ~/.aws/config:

[default]
region = us-east-1
aws_secret_access_key = xxxxx
aws_access_key_id = xxxxx

[profile myprofile]
role_arn = arn:aws:iam::12345:role/MY_ROLE
source_profile = default

My ~/.aws/credentials:

[default]
aws_secret_access_key = xxxxx
aws_access_key_id = xxxxx

Side note- also tried:

Assuming the role manually with

aws sts assume-role --role-arn arn:aws:iam::12345:role/MY_ROLE --role-session-name temp-session

then exporting the AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN, and AWS_ACCESS_KEY_ID to the environment variables. I then ran the EC2 scripts without any profile specified and got

boto.exception.EC2ResponseError: EC2ResponseError: 401 Unauthorized
<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>AuthFailure</Code><Message>AWS was not able to validate the provided access credentials</Message></Error></Errors><RequestID>11402f6e-074c-478c-84c1-11fb92ad0bff</RequestID></Response>

Side note- also tried:

According to this JIRA on Spark scripts with IAM roles, we can specify --instance-profile-name (is an instance profile the only way of using an IAM role this way? ie.. would I have to ask our admin for IAM list/create permissions to launch a cluster with an IAM role?). I have tried using arn:aws:iam::12345:role/MY_ROLE and MY_ROLE but get

boto.exception.EC2ResponseError: EC2ResponseError: 400 Bad Request
<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>InvalidParameterValue</Code><Message>Value (arn:aws:iam::12345:role/MY_ROLE) for parameter iamInstanceProfile.name is invalid. Invalid IAM Instance Profile name</Message></Error></Errors><RequestID>ffeffef9-acad-4a34-a925-31f6b5bbbb3e</RequestID></Response>

回答1:

I managed assigning a role to an ec2 instance by providing the '--instance-profile-name' parameter with the spark-ec2 script which you can pass a profile name.

Inside the instance make sure to run