How to download data from Amazon's requester pay buckets?

Question

I have been struggling for about a week to download arXiv articles as mentioned here: http://arxiv.org/help/bulk_data_s3#src.

I have tried lots of things: s3Browse

Answer 1

Requester Pays is a feature on Amazon S3 buckets that requires the user of the bucket to pay Data Transfer costs associated with accessing data.

Normally, the owner of an S3 bucket pays Data Transfer costs, but this can be expensive for free / Open Source projects. Thus, the bucket owner can activated Requester Pays to reduce the portion of costs they will be charged.

Therefore, when accessing a Requester Pays bucket, you will need to authenticate yourself so that S3 knows whom to charge.

I recommend using the official AWS Command-Line Interface (CLI) to access AWS services. You can provide your credentials via:

aws configure

and then view the bucket via:

aws s3 ls s3://arxiv/pdf/

and download via:

aws s3 cp s3://arxiv/pdf/arXiv_pdf_1001_001.tar .

UPDATE: I just tried the above myself, and received Access Denied error messages (both on the bucket listing and the download command). When using s3cmd, it says ERROR: S3 error: Access Denied. It would appear that the permissions on the bucket no longer permit access. You should contact the owners of the bucket to request access.