问题
I have problem with aws-cli, I did a yum update, it updated awscli (among other things) and now awscli fails on large files (e.g. 5.1 Gb) with SignatureDoesNotMatch. The exact same command (to same bucket) with smaller files works. The big file still works if I use boto from python.
It copies all parts but two it looks like (i.e. it was counted up to 743 of 745 parts), and then the error message comes.
Looks like a bug in awscli? I could not find anything about it when I google around though.
This is all executed on an ec2 instance.
$ /usr/bin/aws --version
aws-cli/1.5.4 Python/2.6.9 Linux/3.14.20-20.44.amzn1.x86_64
$ ls -hl
-rw-rw-r-- 1 user user 4 Nov 4 21:14 small
-rw-rw-r-- 1 user user 5.1G Nov 4 20:02 big
$ /usr/bin/aws s3 cp small s3://mybucket/uploaded.ext
upload: ./small to s3://mybucket/uploaded.ext
$ /usr/bin/aws s3 cp big s3://mybucket/uploaded.ext
upload failed: ./big to s3://mybucket/uploaded.ext
A client error (SignatureDoesNotMatch) occurred when calling the UploadPart operation: The request signature we calculated does not match the signature you provided. Check your key and signing method.
Edit: thought I found it, but still have this problem. I would prefer to stick with awscli since it is what we use all over the code, and it is considerable faster than boto (at least when it works)
Edit2: it actually gone from bad to worse, I can repeat the same copy command in the console, sometimes it goes through, sometimes it does not
回答1:
The aws-cli can certainly handle uploads of 5GB and higher.
I launched an EC2 instance and was able to complete it successfully:
$ /usr/bin/aws --version
aws-cli/1.3.6 Python/2.6.9 Linux/3.10.35-43.137.amzn1.x86_64
$ dd if=/dev/zero of=small bs=1K count=1
1+0 records in
1+0 records out
1024 bytes (1.0 kB) copied, 0.000388469 s, 2.6 MB/s
$ dd if=/dev/zero of=big bs=5M count=1000
1000+0 records in
1000+0 records out
5242880000 bytes (5.2 GB) copied, 73.733 s, 71.1 MB/s
$ aws s3 mb s3://target77
make_bucket: s3://target77/
$ aws s3 cp small s3://target77/small
upload: ./small to s3://target77/small
$ aws s3 cp big s3://target77/big
upload: ./big to s3://target77/big
$ aws s3 ls s3://target77/
2014-11-17 09:44:42 5242880000 big
2014-11-17 09:44:29 1024 small
So far, so good!
I then updated to a newer version of the CLI and tried again:
$ sudo yum update
...
$ /usr/bin/aws --version
aws-cli/1.5.5 Python/2.6.9 Linux/3.10.35-43.137.amzn1.x86_64
$ aws s3 cp small s3://target77/small
upload: ./small to s3://target77/small
$ aws s3 cp big s3://target77/big
upload failed: ./big to s3://target77/bigs) remaining
A client error (SignatureDoesNotMatch) occurred when calling the UploadPart operation: The request signature we calculated does not match the signature you provided. Check your key and signing method.
I repeated the experiment with the EC2 instance and S3 bucket in the same region, and in different regions. Both situations eventually gave the error, but also sometimes worked.
So, it does seem to be an issue with the updated aws-cli. Plus, the error occurred after a random number of 'parts' when I tried re-executing the copy command.
The SignatureDoesNotMatch error is related to security credentials rather than the Amazon S3 service. It sounds similar to issues that people have reported on GitHub: Recurring SignatureDoesNotMatch client error #995
So, I'd recommend using an earlier version of aws-cli until the issue is resolved.
回答2:
It wouldn't surprise me if it's a bug in aws-cli. I'd recommend using s3cmd instead for copying files to/from S3 and doing all your other S3-based interactions.
来源：https://stackoverflow.com/questions/26746218/cannot-copy-large-5-gb-files-with-awscli-1-5-4