Spark is inventing his own AWS secretKey

孤街浪徒 提交于 2019-11-29 11:30:20

I ran into a similar issue. Requests that were using valid AWS credentials returned a 403 Forbidden, but only on certain machines. Eventually I found out that the system time on those particular machines were 10 minutes behind. Synchronizing the system clock solved the problem.

Hope this helps!

(updated as my original one was downvoted as clearly considered unacceptable)

The AWS auth protocol doesn't send your secret over the wire. It signs the message. That's why what you see isn't what you passed in.

For further information, please reread.

It is very intriguing this random passkey. Maybe AWS SDK is getting the password from OS environment.

In hadoop 2.8, the default AWS provider chain shows the following list of providers:

BasicAWSCredentialsProvider EnvironmentVariableCredentialsProvider SharedInstanceProfileCredentialsProvider

Order, of course, matters! the AWSCredentialProviderChain, get the first keys from the first provider that provides that information.

            if (credentials.getAWSAccessKeyId() != null &&
                credentials.getAWSSecretKey() != null) {
                log.debug("Loading credentials from " + provider.toString());
                lastUsedProvider = provider;
                return credentials;
            } 

See the code in "GrepCode for AWSCredentialProviderChain".

I face similar problem using profile credentials. SDK was ignoring the credentials inside ~/.aws/credentials (as good practice, I encourage you to not store credentials inside the program in any way).

My solution...

Set the credentials provider to use ProfileCredentialsProvider

sc._jsc.hadoopConfiguration().set("fs.s3a.endpoint", "s3.eu-central-1.amazonaws.com") # yes, I am using central eu server.
sc._jsc.hadoopConfiguration().set('fs.s3a.aws.credentials.provider', 'com.amazonaws.auth.profile.ProfileCredentialsProvider')
易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!