Airflow s3 connection using UI

Question

I\'ve been trying to use Airflow to schedule a DAG. One of the DAG includes a task which loads data from s3 bucket.

For the purpose above I need to setup s3 connecti

Answer 1

Assuming airflow is hosted on an EC2 server.

just create the connection as per other answers but leave everything blank in the configuration apart from connection type which should stay as S3

The S3hook will default to boto and this will default to the role of the EC2 server you are running airflow on. assuming this role has rights to S3 your task will be able to access the bucket.

this is a much safer way than using and storing credentials.

Answer 2

Another option that worked for me was to put the access key as the "login" and the secret key as the "password":

Conn Id: <arbitrary_conn_id>
Conn Type: S3
Login: <aws_access_key>
Password: <aws_secret_key>

Leave all other fields blank.