I've a script to download file from AWS S3 and it works from a separate server outside of AWS. But when I put this script on an EC2 instance and tried it then it returns error "SignatureDoesNotMatch - The request signature we calculated does not match the signature you provided. Check your key and signing method."
But it works on the other server. The ec2 is on the same region as the s3 host. I'm guessing it has something to do with the host. I tried these host/url but it returns the same error.
Anyone able to download s3 files from ec2 with curl? I need to use only curl. Please answer if you know how to do it through curl. Thanks.
https://s3-ap-southeast-1.amazonaws.com/$bucket/$file
https://s3.amazonaws.com/$bucket/$file
https://$bucket.s3-ap-southeast-1.amazonaws.com/$file
https://$bucket.s3.amazonaws.com/$file
#!/bin/sh file="file-name" bucket="bucket-name" resource="/${bucket}/${file}" contentType="application/x-compressed-tar" dateValue="`date +'%a, %d %b %Y %H:%M:%S %z'`" stringToSign="GET\n\n${contentType}\n${dateValue}\n${resource}" s3Key="xxxxxxxxxxxxxxxxxx" s3Secret="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" signature=$(echo -en "${stringToSign}" | openssl sha1 -hmac "${s3Secret}" -binary | base64) curl -H "Host: s3-ap-southeast-1.amazonaws.com" \ -H "Date: $dateValue" \ -H "Content-Type: $contentType" \ -H "Authorization: AWS ${s3Key}:${signature}" \ https://s3-ap-southeast-1.amazonaws.com/$bucket/$file -o $file
Rather than writing a CURL command, consider using the AWS Command-Line Interface (CLI).
It has a aws s3 cp command that can copy content to/from Amazon S3 buckets (and even between buckets).
Try this...
#!/bin/sh S3_KEY='/file.txt' S3_SECRET_KEY='xxx' S3_ACCESS_KEY='AKIAxxx' S3_HOST=yourbucket.s3.amazonaws.com AMZ_DATE=$(date -u "+%Y%m%dT%H%M%SZ") DATE=$(/bin/echo ${AMZ_DATE} | cut -b 1-8) REGION=ap-southeast-1 CANONICAL_REQUEST="GET\n${S3_KEY}\n\nhost:${S3_HOST}\nx-amz-content-sha256:UNSIGNED-PAYLOAD\nx-amz-date:${AMZ_DATE}\n\nhost;x-amz-content-sha256;x-amz-date\nUNSIGNED-PAYLOAD" STRING_TO_SIGN="AWS4-HMAC-SHA256\n${AMZ_DATE}\n${DATE}/${REGION}/s3/aws4_request\n$(/bin/echo -en ${CANONICAL_REQUEST} | sha256sum | cut -f 1 -d' ')" AWS_SIG_V4_AND_S3_SECRET_KEY=AWS4${S3_SECRET_KEY} DATE_HMAC_HEX=$(/bin/echo -n "${DATE}" | openssl sha256 -hmac "${AWS_SIG_V4_AND_S3_SECRET_KEY}" | cut -f 2 -d' ') DATE_HMAC_BIN=$(/bin/echo -n "${DATE_HMAC_HEX}" | xxd -r -p) REGION_HMAC_HEX=$(/bin/echo -n "${REGION}" | openssl sha256 -hmac "${DATE_HMAC_BIN}" | cut -f 2 -d' ') REGION_HMAC_BIN=$(/bin/echo -n "${REGION_HMAC_HEX}" | xxd -r -p) SERVICE_HMAC_HEX=$(/bin/echo -n "s3" | openssl sha256 -hmac "${REGION_HMAC_BIN}" | cut -f 2 -d' ') SERVICE_HMAC_BIN=$(/bin/echo -n "${SERVICE_HMAC_HEX}" | xxd -r -p) SIGNING_KEY_HEX=$(/bin/echo -n "aws4_request" | openssl sha256 -hmac "${SERVICE_HMAC_BIN}" | cut -f 2 -d' ') SIGNING_KEY_BIN=$(/bin/echo -n "${SIGNING_KEY_HEX}" | xxd -r -p) SIGNATURE_HEX=$(/bin/echo -ne "${STRING_TO_SIGN}" | openssl sha256 -hmac "${SIGNING_KEY_BIN}" | cut -f 2 -d' ') curl -vv https://${S3_HOST}/file.txt \ -H "Host: ${S3_HOST}" \ -H "x-amz-content-sha256: UNSIGNED-PAYLOAD" \ -H "x-amz-date: ${AMZ_DATE}" \ -H "Authorization: AWS4-HMAC-SHA256 Credential=${S3_ACCESS_KEY}/${DATE}/${REGION}/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=${SIGNATURE_HEX}"
This is using AWS Signature V4. I have tested this and it works on both CentOS 7.3 and Ubuntu LTS 16.04 for my bucket in ap-southeast-1 region. I would highly suggest using John Rotenstein's suggestion.
