问题
since amazon web service need to pay, so just wanna ask ppl who had worked on it before i jump into it, and confirm some knowledge about it.
Question one: In Amazon auto scaling service, it says can scale up and down instances. that does this mean?
does it mean changing the type of instance? or can start/stop more/less instance base on the condition define?
Question two: can the auto scaling framework work with map reduce?
for example, if i have a extreme case, i will have endless tasks, and the time to finish the task will be vary, and might be very computation intensive. so can i use auto scaling service to scale the map reduce?
Thanks
回答1:
re. question one: "scale up and scale down" means they can add more instances as needed and the number of instances can then be reduced (i.e. virtual machine instances can be shut down) when they're no longer required.
re. question two: I would argue that auto-scaling does not really make sense for map reduce, at least if you're thinking about Hadoop: One of the main ideas behind Map/Reduce & Hadoop is that the data is split up among a lot of nodes and the nodes can then process the data that is close to them - ideally on their local disk or at least on another machine in close (physical) proximity. Starting a new instance would of course be possible, but that makes sense only as long as you do not have to copy the data you want to process over to the new instance. Hope this explanation makes sense to you... In any case, check out Amazon's Elastic Map Reduce offering, which does what you want, at least if I understood your question correctly.
回答2:
You could take a look at Themis, an EMR autoscaling framework developed at Atlassian. Current features include reactive autoscaling (based on current cluster load) as well as proactive autoscaling (based on predefined schedules), it comes with a Web UI, and the tool is very easy to configure.
(Apologies for posting in an old thread, but the answer may still be interesting for readers discovering this thread.)
来源：https://stackoverflow.com/questions/3955822/can-amazon-auto-scaling-service-work-with-elastic-map-reduce-service