In my /etc/fluent/fluent.conf
<
I have been facing warn: pattern not match in fluentd, and because of this my filter section was not working. And then I took this warning seriously and resolved this by creating a regex. So, my td-agent.conf is as follows:
<source>
@type tail
format **/^([^ ]*) (?<host>[^ ]*) [^ ]* "(?<method>\S+) (?<path>[^ ]* +\S*)? (?<code>[^ ]*) (?<size>[^ ]*) (?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?/**
path /var/log/apache2/access.log
pos_file /var/log/td-agent/httpd.access.pos
tag s3.apache.access
</source>
<filter **>
@type grep
<regexp>
key path
pattern \/aws\/project_launch\/view\/[\w\W]*
</regexp>
</filter>
<match **>
@type s3
aws_key_id xxxxxx
aws_sec_key xxxxxx
s3_bucket bucketalifluentd
s3_region eu-west-1
path logs_viewshare/
buffer_path /var/log/td-agent/buffer/s3
time_slice_format %Y-%m-%d/%H
time_slice_wait 2m
</match>
Please note that, the apache2 logs may be different for you, because you have made a different configuration in apache2.conf. You can use rubular
for creating regex in ruby, because fluentd/td-agent has been written in ruby. After that you can view buffer that has collected your logs in directory /var/log/td-agent/buffer/s3.xxx
It seems that tail plugin does not support the format for apache log format "vhost_combined" but "combined".
How about changing the apache configuration file as follows:
/etc/apache2/conf-available/other-vhosts-access-log.conf
Before:CustomLog ${APACHE_LOG_DIR}/other_vhosts_access.log vhost_combined
(change vhost_combined to combined)
After:CustomLog ${APACHE_LOG_DIR}/other_vhosts_access.log combined