PySpark using IAM roles to access S3

Question

I\'m wondering if PySpark supports S3 access using IAM roles. Specifically, I have a business constraint where I have to assume an AWS role in order to access a given bucket. Th

Answer 1

You could try the approach in Locally reading S3 files through Spark (or better: pyspark).

However I've had better luck with setting environment variables (AWS_ACCESS_KEY_ID etc) in Bash ... pyspark will automatically pick these up for your session.