gsutil cannot copy to s3 due to authentication

Question

I need to copy many (1000+) files to s3 from GCS to leverage an AWS lambda function. I have edited ~/.boto.cfg and commented out the 2 aws authentication parameters

Answer 1

I had a similar problem. Here is what I ended up doing on a GCE machine:

Step 1: Using gsutil, I copied files from GCS to my GCE hard drive Step 2: Using aws cli (aws s3 cp ...), I copied files from GCE hard drive to s3 bucket

The above methodology has worked reliably for me. I tried using gsutil rsync but it fail unexpectedly.

Hope this helps