What's a good way to collect logs from Amazon EC2 instances?

Question

My app is hosted on an Amazon EC2 cluster. Each instance writes events to log files. I need to collect (and data mine) over these logs at the end of each day. What\'s a reco

Answer 1

Use syslog-ng, a popular tool that transfers log messages over TCP, optionally encrypted.

http://www.balabit.com/network-security/syslog-ng/

Answer 2

This question is old now (December 2014) but still ranks highly during a Google search on this topic.

Amazon now provides a way to do some of this through CloudWatch. It has the capability to pattern-match the log message and trigger alarms based on things happening in the application. Depending on the nature of the data-mining that needs to be done, it may be possible to use their API to fetch the desired, aggregate events. See http://aws.amazon.com/blogs/aws/cloudwatch-log-service/