How to download data from Amazon's requester pay buckets?

Question

I have been struggling for about a week to download arXiv articles as mentioned here: http://arxiv.org/help/bulk_data_s3#src.

I have tried lots of things: s3Browse

Answer 1

At the bottom of this page arXiv explains that s3cmd gets denied because it does not support access to requester pays bucket as a non-owner and you have to apply a patch to the source code of s3cmd. However, the version of s3cmd they used is outdated and the patch does not apply to the latest version of s3cmd.

Basically you need to allow s3cmd to add "x-amz-request-payer" header to its HTTP request to buckets. Here is how to fix it:

1. Download the source code of s3cmd.
2. Open S3/S3.py with a text editor.

3. Add this two lines of code at the bottom of __init__ function:

   if self.s3.config.extra_headers:
       self.headers.update(self.s3.config.extra_headers)
   

4. Install s3cmd as instructed.