Throttling S3 commands with aws cli

Question

I\'m running a backup script using AWS CLI to perform an S3 sync command every night on my MediaTemple server. This has run without fail for months, but I updated my Plesk insta

Answer 1

The AWS CLI S3 transfer commands (which includes sync) have the following relevant configuration options:

• max_concurrent_requests -
  • Default: 10
  • The maximum number of concurrent requests.
• multipart_threshold -
  • Default: 8MB
  • The size threshold the CLI uses for multipart transfers of individual files.
• multipart_chunksize -
  • Default: 8MB
  • When using multipart transfers, this is the chunk size that the CLI uses for multipart transfers of individual files.

This isn't so granular as throttling packets per second, but it seems like setting a lower concurrent request value and lowering both multipart threshold and chunksize will help. If the values you pasted are close to average, I would start with these values and tweak until you're reliably not exceeding the limits anymore:

$ aws configure set default.s3.max_concurrent_requests 8
$ aws configure set default.s3.multipart_threshold 6MB
$ aws configure set default.s3.multipart_chunksize 6MB