What's the best tool to parse log files? [closed]

Answer 1

on linux, you can use just the shell(bash,ksh etc) to parse log files if they are not too big in size. The other tools to go for are usually grep and awk. However, for more programming power, awk is usually used. If you have big files to parse, try awk.

Of course, Perl or Python or practically any other languages with file reading and string manipulation capabilities can be used as well.

Answer 2

I find this list invaluable when dealing with any job that requires one to parse with python.

I wouldn't use perl for parsing large/complex logs - just for the readability (the speed on perl lacks for me (big jobs) - but that's probably my perl code (I must improve)).

However if grep suits your needs perfectly for now - there really is no reason to get bogged down in writing a full blown parser. Simplest solution is usually the best, and grep is a fine tool.

Answer 3

try Nagios Log Monitoring

The reason this tool is the best for your purpose is this:

It requires no installation of foreign packages. Which means, there's no need to install any perl dependencies or any silly packages that may make you nervous.

There is little to no learning curve. You don't need to learn any programming languages to use it. All you need to do is know exactly what you want to do with the logs you have in mind, and read the pdf that comes with the tool.

If the log you want to parse is in a syslog format, you can use a command like this:

./NagiosLogMonitor 10.20.40.50:5444 logrobot autofig /opt/jboss/server.log 60m 'INFO' '.' 1 2 -show

Even if your log is not in a recognized format, it can still be monitored efficiently with the following command:

./NagiosLogMonitor 10.20.40.50:5444 logrobot autonda /opt/jboss/server.log 60m 'INFO' '.' 1 2 jbosslogs -ndshow

To parse a log for specific strings, replace the 'INFO' string with the patterns you want to watch for in the log. If you want to search for multiple patterns, specify them like this 'INFO|ERROR|fatal'.

If efficiency and simplicity (and safe installs) are important to you, this Nagios tool is the way to go.