How to efficiently import many large JSON files directly from S3 into MongoDB

Question

I have compressed JSON files in S3 and I would like to set up MongoDB in EC2 to server json documents contained in these files. The compressed files are >100M and there are 1000

Answer 1

The alternative of using s3cmd is to use aws s3 which has fewer features but comes installed with AWS CLI.

The command would look like:

aws S3 cp <your s3 URI> - | \
mongoimport \
--db <dbName> \
--collection <collectionName> \

The - sends the file to stdout

Answer 2

You don't need to store intermediate files, you can pipe the output of s3 file to stdout and you can get input to mongoimport from stdin.

Your full command would look something like:

s3cmd get s3://<yourFilename> - | mongoimport -d <dbName> -c <collectionName>

note the - which says send the file to stdout rather than to a filename.