How to efficiently import many large JSON files directly from S3 into MongoDB

Question

I have compressed JSON files in S3 and I would like to set up MongoDB in EC2 to server json documents contained in these files. The compressed files are >100M and there are 1000

Answer 1

The alternative of using s3cmd is to use aws s3 which has fewer features but comes installed with AWS CLI.

The command would look like:

aws S3 cp  - | \
mongoimport \
--db  \
--collection  \

The - sends the file to stdout