Set cache-control for entire S3 bucket automatically (using bucket policies?)

I need to set cache-control headers for an entire s3 bucket, both existing and future files and was hoping to do it in a bucket policy. I know I can edit the existing ones and I know how to specify them on put if I upload them myself but unfortunately the app that uploads them cannot set the headers as it uses s3fs to copy the files there.

Dan Williams

There are now 3 ways to get this done: via the AWS Console, via the command line, or via the s3cmd command line tool.

AWS Console Instructions

This is now the recommended solution. It is straight forward, but it can take some time.

Log in to AWS Management Console
Go into S3 bucket
Select all files by route
Choose "More" from the menu
Select "Change metadata"
In the "Key" field, select "Cache-Control" from the drop down menu max-age=604800Enter (7 days) for Value
Press "Save" button

(thanks to @biplob - please give him some love below)

AWS Command Line Solution

Originally, when I created this bucket policies were a no go, so I figured how to do it using aws-cli, and it is pretty slick. When researching I couldn't find any examples in the wild, so I thought I would post some of my solutions to help those in need.

NOTE: By default, aws-cli only copies a file's current metadata, EVEN IF YOU SPECIFY NEW METADATA.

To use the metadata that is specified on the command line, you need to add the '--metadata-directive REPLACE' flag. Here are a some examples.

For a single file

aws s3 cp s3://mybucket/file.txt s3://mybucket/file.txt --metadata-directive REPLACE \
--expires 2034-01-01T00:00:00Z --acl public-read --cache-control max-age=2592000,public

For an entire bucket (note --recursive flag):

aws s3 cp s3://mybucket/ s3://mybucket/ --recursive --metadata-directive REPLACE \
--expires 2034-01-01T00:00:00Z --acl public-read --cache-control max-age=2592000,public

A little gotcha I found, if you only want to apply it to a specific file type, you need to exclude all the files, then include the ones you want.

Only jpgs and pngs:

aws s3 cp s3://mybucket/ s3://mybucket/ --exclude "*" --include "*.jpg" --include "*.png" \
--recursive --metadata-directive REPLACE --expires 2034-01-01T00:00:00Z --acl public-read \
--cache-control max-age=2592000,public

Here are some links to the manual if you need more info:

Known Issues:

"Unknown options: --metadata-directive, REPLACE"

this can be caused by an out of date awscli - see @eliotRosewater's answer below

S3cmd tool

S3cmd is a "Command line tool for managing Amazon S3 and CloudFront services". While this solution requires a git pull it might be a simpler and more comprehensive solution.

For full instructions, see @ashishyadaveee11's post below

Hope it helps!

Now, it can change easily from AWS console.

Log in to AWS Management Console
Go into S3 bucket
Select all files by route
Choose "More" from the menu
Select "Change metadata"
In the "Key" field, select "Cache-Control" from the drop down menu
max-age=604800Enter (7 days) for Value
Press "Save" button

It takes time to execute depends on your bucket files. Redo from the beginning if you accidentally close the browser.

ashishyadaveee11

steps

git clone https://github.com/s3tools/s3cmd
Run s3cmd --configure (You will be asked for the two keys - copy and paste them from your confirmation email or from your Amazon account page. Be careful when copying them! They are case sensitive and must be entered accurately or you'll keep getting errors about invalid signatures or similar. Remember to add s3:ListAllMyBuckets permissions to the keys or you will get an AccessDenied error while testing access.)
./s3cmd --recursive modify --add-header="Cache-Control:public ,max-age= 31536000" s3://your_bucket_name/

roens

Were it that my reputation score were >50, I'd just comment. But it's not (yet) so here's another full answer.

I've been banging my head on this problem for a while now. Until I found & read the docs. Sharing that here in case it helps anyone else:

Amazon CloudFront Documentation: Specifying How Long Objects Stay in a CloudFront Edge Cache (Expiration)

What ended up reliably working for me was this command. I chose a 1 second expiration time for testing to verify expected results:

aws s3 cp \
  --metadata-directive REPLACE \
  --cache-control max-age=1,s-maxage=1 \
  s3://bucket/path/file \
  s3://bucket/path/file

--metadata-directive REPLACE is required when "cp" modifying metadata on an existing file in S3
max-age sets Browser caching age, in seconds
s-maxage sets CloudFront caching, in seconds

Likewise, if setting these Cache-Control header values on a file while uploading to S3, the command would look like:

aws s3 cp \
  --cache-control max-age=1,s-maxage=1 \
  /local/path/file \
  s3://bucket/path/file

I don't think you can specify this at the bucket level but there are a few workarounds for you.

Copy the object to itself on S3 setting the appropriate cache-control headers for the copy operation.
Specify response headers in the url to the files. You need to use pre-signed urls for this to work but you can specify certain response headers in the querystring including cache-control and expires. For a full list of the available options see: http://docs.amazonwebservices.com/AmazonS3/latest/API/RESTObjectGET.html?r=5225

To those attempting to use Dan's answer and getting the error:

"Unknown options: --metadata-directive, REPLACE"

I ran into the issue, and the problem was that I installed awscli using

sudo apt-get install awscli

This installed an old version of the awscli which is missing the --metadata-directive command. So I used sudo apt-get remove awscli to remove it.

Then reinstalled following the procedure from amazon: http://docs.aws.amazon.com/streams/latest/dev/kinesis-tutorial-cli-installation.html

The only difference is that I had to use sudo -H because of permission issues which others might run into also.

You can always configure a lambda with a trigger on PUTOBJECT on S3, the lambda will simply change the header of this particular object that was just put.

Then you can run the copy command mentioned above one last time, and all the new objects will be fixed by the lambda.

UPDATE:

Here is a good place to start from: https://www.aaronfagan.ca/blog/2017/how-to-configure-aws-lambda-to-automatically-set-cache-control-headers-on-s3-objects/

来源：https://stackoverflow.com/questions/10435334/set-cache-control-for-entire-s3-bucket-automatically-using-bucket-policies

标签

amazon-s3

s3fs