unable to filter badly-formatted messages in syslog-ng

问题

I am contacting you regarding an issue I have with syslog-NG. some of our devices (mainly HP switches and SANs) are sending syslog messages that do not respect the syslog RFC 5424 it seems.

to give you a few examples :

if I sniff the network interface of the server, we can see these "wrong" messages like that (as you can see, after the PRI, we can see the PROGRAM, then timezone and fields separated with commas. in short, mixed fields, missing fields etc. not a standard syslog message) :

<190>raslogd: 2017/03/08-16:03:20, [SEC-1203], 53642, WWN 10:00:50:eb:1a:6c:21:38 | FID 128, INFO, cswc-mo8x-SAN01, Login information: Login successful via TELNET/SSH/RSH. IP Addr: 1.2.3.4

therefore, I am currently unable to filter these messages. I tried to define a regular filter and try to print fields such as MSGHDR, MSG, PRI etc.. but couldn't see anything.

the ONLY WAY of having this message filtered is by just defining the external interface, and a file as output, without any template.

e.g if I do a filter like this :

destination d_INCOMING_ALL   { file("/app/syslog-ng/logs/incoming_all.log"); };

log {
  source(s_EXTERNAL);
  destination(d_INCOMING_ALL);
};

I can see messages in the log file, but formatted, somehow (I suppose syslog-NG reformats them) :

[root@xxxxxxxxxxxx logs]# grep -i cswc incoming_all.log
Mar  9 09:44:20 cswc-mo8x-hpsan01 raslogd: 2017/03/09-08:34:50, [SEC-1203], 53647, WWN 10:00:50:eb:1a:6c:21:38 | FID 128, INFO, cswc-mo8x-SAN01, Login information: Login successful via TELNET/SSH/RSH. IP Addr: 1.2.3.4
[root@xm1p1034vmo logs]#

the problem is that I cannot filter these messages like that (we receive logs from more than 1000 devices) there, i need to filter messages coming from these devices ! and the only way I can do it is on the hostname (cswc-) or program name (raslogd)

so I tried to display the fields by adding a template to that file, example :

destination d_test { 
    file ("/app/syslog-ng/logs/test_olivier.log" 
    template("pri=${priority} date=${ISODATE} host=${HOST} program=${PROGRAM} message=${MSG}\n") 
    ); 
};

but nothing works, nothing is displayed. I tried all fields, MSG, MESSAGE, MSGHDR etc.. can't manage do display ANYTHING. the only working thing is the parsing without filters or templates.

naturally, if I tred all kind of filters, like these below, it does not work (as fields are mixed) :

filter f_is_SAN     {
    host("cswc.*" flags(ignore-case));
};

same for :

filter f_is_SAN     {
    match(".*cswc.*" flags(ignore-case));
};

any hints on how I create filters for these messages coming from these devices (on hostname or programname) ?

thanks regards,

回答1:

mmh thanks Robert, i tried with a single junction like this, and it works ! (it works even without the junction) unfortunately, my other filters are not working anymore (I'm just showing 1 here, but i have more than 20, original file is much bigger)

can't we listen on two sources using the same port, but with differents flags ? :(

source s_EXTERNAL {
    udp();
};

source s_EXTERNAL_NOPARSE {
    udp(flags(no-parse) persist-name('noparse'));
};

destination d_INCOMING_ALL { file("/app/syslog-ng/logs/incoming_all.log"); };
destination d_OUTGOING_ISERIES   { file("/app/syslog-ng/logs/outgoing_iseries.log"); };
destination d_olivier2 { file ("/app/syslog-ng/logs/test_olivier2.log" template("host=${HOST} message=${MESSAGE}\n") ); };

filter f_is_iSeries {
  match ("PowerTech.*Interact" value("MESSAGE") flags(ignore-case));
};

filter f_is_Network     {
  host("cswc.*" flags(ignore-case))
  or program("raslogd" flags(ignore-case));
};

# -------- not working anymore
log {
 source(s_EXTERNAL);
 filter(f_is_iSeries);
 destination(d_OUTGOING_ISERIES);
 flags(final);
};

# ------- working now
log {
 source(s_EXTERNAL_NOPARSE);
 junction {
    channel {
            filter(f_is_Network);
    };
 };
 destination(d_olivier2);
};

i can now see this log in the test d_olivier2 dest :

[root@xm3p1034vmo etc]# cat ../logs/test_olivier2.log
host=cswc-mo8x-hpsan01 message=<190>raslogd: 2017/03/23-16:29:44, [SEC-1203], 53656, WWN 10:00:50:eb:1a:6c:21:38 | FID 128, INFO, cswc-mo8x-SAN01, Login information: Login successful via TELNET/SSH/RSH. IP Addr: 1.2.3.4

but nothing in /app/syslog-ng/logs/outgoing_iseries.log :(

来源：https://stackoverflow.com/questions/42692552/unable-to-filter-badly-formatted-messages-in-syslog-ng

标签

switch-statement

syslog

syslog-ng

san