How do you import Big Data public data sets into AWS?

元气小坏坏 提交于 2019-12-06 07:55:16

问题


Loading any of Amazon's listed public data sets (http://aws.amazon.com/datasets) would take a lot of resources and bandwidth. What's the best way to import them into AWS so you start working with them quickly?


回答1:


You will need to create a new EBS Instance using the Snapshot-ID for the public dataset. That way you won't need to pay for transfer.

But be careful, some data sets are only available in one region, most likely denoted by a note similar to this. You should register your EC2 instance in the same region then.

These datasets are hosted in the us-east-1 region. If you process these from other regions you will be charged data transfer fees.




回答2:


FYI : SDBExplorer uses Multithreaded BatchPutAttributes to achieve high write throughput while uploading bulk data to Amazon SimpleDB. SDB Explorer allows multiple parallel uploads. If you have the bandwidth, you can take full advantage of that bandwidth by running number of BatchPutAttributes processes at once in parallel queue that will reduce the time spend in processing. SDBExplorer supports Import data from MySql and CSV to Amazon SimpleDB.

http://www.sdbexplorer.com

Disclosure : I am the developer of SDBExplorer.



来源:https://stackoverflow.com/questions/8991426/how-do-you-import-big-data-public-data-sets-into-aws

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!