What's the best tool to parse log files? [closed]

Answer 1

There's a Perl program called Log_Analysis that does a lot of analysis and preprocessing for you.

Answer 2

In the end, it really depends on how much semantics you want to identify, whether your logs fit common patterns, and what you want to do with the parsed data.

If you can use regular expressions to find what you need, you have tons of options. Perl is a popular language and has very convenient native RE facilities. I personally feel a lot more comfortable with Python and find that the little added hassle for doing REs is not significant.

If you want to do something smarter than RE matching, or want to have a lot of logic, you may be more comfortable with Python or even with Java/C++/etc. For instance, it is easy to read line-by-line in Python and then apply various predicate functions and reactions to matches, which is great if you have a ruleset you would like to apply.

Answer 3

Another possible interpretation of your question is "Are there any tools that make log monitoring easier?", and to answer that I would suggest you have a look at Splunk or maybe Log4view.

Answer 4

All scripting languages are good candidates: Perl, Python, Ruby, PHP, and AWK are all fine for this. Using any one of these languages are better than peering at the logs starting from a (small) size.

Wearing Ruby Slippers to Work is an example of doing this in Ruby, written in Why's inimitable style. Here's a basic example in Perl. I suggest you choose one of these languages and start cracking.

Answer 5

A big advantage Perl has over Python is that when parsing text is the ability to use regular expressions directly as part of the language syntax. For example:

if ($line =~ m/^Regex/) {
    ... code goes here
}

Perl also assigns capture groups directly to $1, $2, etc, making it very simple to work with. Depending on the format and structure of the logfiles you're trying to parse, this could prove to be quite useful (or, if it can be parsed as a fixed width file or using simpler techniques, not very useful at all).

It's all just syntactic sugar, really, and other languages also allow you use regular expressions and capture groups (indeed, the linked article shows how to do it in Python). You just have to write a bit more code and pass around objects to do it.

Answer 6

Learning a programming language will let you take you log analysis abilities to another level.

Any dynamic or "scripting" language like Perl, Ruby or Python will do the job. What you should use really depends on external factors. Among the things you should consider:

• does work already use a suitable langauge?
• do you know anyone who can mentor you in a suitable language?
• try each language a little and see which language fits you better.

Personally, for the above task I would use Perl. YMMV.

Several reasons to like Perl:

Powerful one-liners - if you need to do a real quick, one-off job, Perl offers some really great short-cuts. See perlrun -n for one example

Multi-paradigm language - Perl has support for imperative, functional and object-oriented programming methodologies.

Sigils - those leading punctuation characters on variables like $foo or @bar. They are a bit like hungarian notation without being so annoying.

Moose - an incredible new OOP system that provides powerful new OO techniques for code composition and reuse.

Strictures - the use strict pragma catches many errors that other dynamic languages gloss over at compile time. I miss it terribly when I use Python or PHP.

Self-discipline - Perl gives you the freedom to write and do what you want, when you want. This means that you have to learn to write clean code or you will hurt. Fortunately, there are tools to help a beginner. Perl::Critic does lint-like analysis of code for best practices.