Uploading ZIP file to S3, use EC2 to Unzip

Question

I have a Codeigniter web app that is uploading many tiny files every hour to Amazon S3, which is causing my S3 request charges to shoot up real fast. One way to overcome thi

Answer 1

I have been using this service to unzip files full of thousands of tiny image files, each zip I upload is about 4GB, and costs around $1 to unzip using http://www.cloudzipinc.com/service/s3_unzip, maybe that might help someone.

Having said that, you might find it easier to use Python with the Boto library. That will work far more efficiently than PHP.

Answer 2

S3 is just storage. Whatever file you upload is the file that is stored. You cannot upload a zip file then extract it once its in S3. If you wrote the application the best thing I could say is to try to re-design how you store the files. S3 requests are pretty cheap... you must be making a lot of requests.

Answer 3

You can write code in a lambda to unzip a file of S3 bucket, you just have to use it, AWS Lambda will do this for you.

Referece:

1. https://github.com/carloscarcamo/aws-lambda-unzip-py/blob/master/unzip.py
2. https://github.com/mehmetboraezer/aws-lambda-unzip

Answer 4

If you have an EC2 machine in the same region I would suggest you upload it there zipped and then it drop it to s3 from there unzipped. S3 cannot unzip it on its own as its all static.

Theres no charges between ec2 and s3 so ec2 can handle the unzipping and then write it out into your s3 bucket without additional transfer charges.