Can you set up a transfer from an S3 bucket subfolder to a GCS bucket with only the credentials to access the S3 subfolder, not the root folder?

Question

I\'m looking to set up a transfer job to take files stored within an S3 bucket and load them to a GCS bucket. The credentials that I have give me access to the folder that conta

Answer 1

Paul,

Most likely your IAM role is missing s3:ListBucket permission. Can you update your IAM role to have s3:ListBucket , s3:GetBucketLocation and try again?

Answer 2

According to the documentation link:

The Storage Transfer Service uses the project-[$PROJECT_NUMBER]@storage-transfer-service.iam.gserviceaccount.com service account to move data from a Cloud Storage source bucket.

The service account must have the following permissions for the source bucket:

storage.buckets.get Allows the service account to get the location of the bucket. Always required.

storage.objects.list Allows the service account to list objects in the bucket. Always required.

storage.objects.get Allows the service account to read objects in the bucket. Always required.

storage.objects.delete Allows the service account to delete objects in the bucket. Required if you set deleteObjectsFromSourceAfterTransfer to true.

The roles/storage.objectViewer and roles/storage.legacyBucketReader roles together contain the permissions that are always required. The roles/storage.legacyBucketWriter role contains the storage.objects.delete permissions. The service account used to perform the transfer must be assigned the desired roles.

You have to set this permissions on your AWS bucket.