Amazon Web Services S3 Request Limit

Question

I\'m using AWS to run some data processing. I have 400 spot instances in EC2 with 4 processes each, all of them writing to a single bucket in S3. I\'ve started to get a (app

Answer 1

To add to what James said, there are some internals about S3 partitioning that have been discussed and can be used to mitigate this in the future because exponential backoff is required.

See here: http://aws.typepad.com/aws/2012/03/amazon-s3-performance-tips-tricks-seattle-hiring-event.html

Briefly, don't store everything with the same prefix or there is a higher likelihood you will have these errors.Find some way to make the very first character in the prefix be as random as possible to avoid hotspots in S3's internal partitioning.