记录filebeat 收集nginx 的json格式直接送入ES问题 Cannot index event publisher.Event

1、问题现象

原来的nginx日志格式为

47.106.166.195 - - [05/May/2020:19:21:55 +0800] "GET /Login/index.html HTTP/2.0" 200 3500 "https://newds.sxms.com" "Chrome/57"
47.103.76.244 - - [05/May/2020:19:22:05 +0800] "GET / HTTP/2.0" 302 20 "-" "Chrome/57"
47.103.76.244 - - [05/May/2020:19:22:05 +0800] "GET /Login/index.html HTTP/2.0" 200 3498 "https://newds.ssms.com" "Chrome/57"

调整nginx配置

添加格式JSON 到http段

log_format main
    '{"@timestamp":"$time_iso8601",'
    '"host":"$hostname",'
    '"server_ip":"$server_addr",'
    '"client_ip":"$remote_addr",'
    '"xff":"$http_x_forwarded_for",'
    '"domain":"$host",'
    '"url":"$uri",'
    '"referer":"$http_referer",'
    '"args":"$args",'
    '"upstreamtime":"$upstream_response_time",'
    '"responsetime":"$request_time",'
    '"request_method":"$request_method",'
    '"status":"$status",'
    '"size":"$body_bytes_sent",'
    '"request_body":"$request_body",'
    '"request_length":"$request_length",'
    '"protocol":"$server_protocol",'
    '"upstreamhost":"$upstream_addr",'
    '"file_dir":"$request_filename",'
    '"http_user_agent":"$http_user_agent"'
  '}';
    access_log  /var/log/nginx/access.log  main;

虚拟主机好像得引用

类似这

        access_log  /www/wwwlogs/wbn4.cn.access.log main;
        error_log  /www/wwwlogs/wbn4.cn.error.log ;

重启nginx 看日志输出

{"@timestamp":"2020-05-19T00:23:22+08:00","host":"newds.111.com","server_ip":"172.26.19.121","client_ip":"47.106.166.195","xff":"-","domain":"newds.111.com","url":"/index.php","referer":"https://newds.111.com","args":"s=Login/index.html","upstreamtime":"0.029","responsetime":"0.036","request_method":"GET","status":"200","size":"3499","request_body":"-","request_length":"92","protocol":"HTTP/2.0","upstreamhost":"127.0.0.1:9000","file_dir":"/www/wwwroot/wbn4.cn/index.php","http_user_agent":"Chrome/57"}
{"@timestamp":"2020-05-19T00:23:27+08:00","host":"newds.s1111.com","server_ip":"172.26.19.121","client_ip":"120.79.161.84","xff":"-","domain":"newds.s1111.com","url":"/zfadmin.php","referer":"-","args":"-","upstreamtime":"0.026","responsetime":"0.025","request_method":"GET","status":"200","size":"689","request_body":"-","request_length":"68","protocol":"HTTP/2.0","upstreamhost":"127.0.0.1:9000","file_dir":"/www/wwwroot/wbn4.cn/zfadmin.php","http_user_agent":"Chrome/57"}

filebeat配置如下

[ root@[ newds.sx11.com ]:/etc/yum.repos.d# ] cat /etc/filebeat/filebeat.yml |grep -v "#\|^$"
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /www/wwwlogs/wbn4.cn.*.log
  json.keys_under_root: true
  json.overwrite_keys: true
  json.add_error_key: true
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 1
setup.kibana:
output.elasticsearch:
  hosts: ["admin.sxdsms.com:9200"]
  username: "elastic"
processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

重启filebear 查看日志

[ root@[ newds.sxdsms.com ]:/etc/yum.repos.d# ] /etc/init.d/filebeat status
● filebeat.service - Filebeat sends log files to Logstash or directly to Elasticsearch.
   Loaded: loaded (/usr/lib/systemd/system/filebeat.service; disabled; vendor preset: disabled)
   Active: active (running) since Tue 2020-05-19 00:16:03 CST; 1min 1s ago
     Docs: https://www.elastic.co/products/beats/filebeat
 Main PID: 3383 (filebeat)
   CGroup: /system.slice/filebeat.service
           └─3383 /usr/share/filebeat/bin/filebeat -environment systemd -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebe...

May 19 00:16:44 newds.sxdsms.com filebeat[3383]: 2020-05-19T00:16:44.925+0800        INFO        [index-management]        idxmgmt/std.go:295        Loaded index template.
May 19 00:16:44 newds.sxdsms.com filebeat[3383]: 2020-05-19T00:16:44.969+0800        INFO        [index-management]        idxmgmt/std.go:306        Write alias successfully generated.
May 19 00:16:45 newds.sxdsms.com filebeat[3383]: 2020-05-19T00:16:45.012+0800        INFO        [publisher_pipeline_output]        pipeline/output.go:111        Connection to...established
May 19 00:16:45 newds.sxdsms.com filebeat[3383]: 2020-05-19T00:16:45.059+0800        WARN        [elasticsearch]        elasticsearch/client.go:384        Cannot index event publisher.Ev...
May 19 00:16:45 newds.sxdsms.com filebeat[3383]: 2020-05-19T00:16:45.059+0800        WARN        [elasticsearch]        elasticsearch/client.go:384        Cannot index event publisher.Ev...
May 19 00:16:45 newds.sxdsms.com filebeat[3383]: 2020-05-19T00:16:45.059+0800        WARN        [elasticsearch]        elasticsearch/client.go:384        Cannot index event publisher.Ev...
May 19 00:16:47 newds.sxdsms.com filebeat[3383]: 2020-05-19T00:16:47.637+0800        WARN        [elasticsearch]        elasticsearch/client.go:384        Cannot index event publisher.Ev...
May 19 00:16:47 newds.sxdsms.com filebeat[3383]: 2020-05-19T00:16:47.637+0800        WARN        [elasticsearch]        elasticsearch/client.go:384        Cannot index event publisher.Ev...
May 19 00:16:50 newds.sxdsms.com filebeat[3383]: 2020-05-19T00:16:50.631+0800        WARN        [elasticsearch]        elasticsearch/client.go:384        Cannot index event publisher.Ev...
May 19 00:17:03 newds.sxdsms.com filebeat[3383]: 2020-05-19T00:17:03.583+0800        INFO        [monitoring]        log/log.go:145        Non-zero metrics in the last 30s        {"monit...
Hint: Some lines were ellipsized, use -l to show in full.

报错

Cannot index event publisher.Event

Google后没有具体的答案

我的解决办法就是用kibana清除了这个索引并删除

后面输出就达到我的要求

{
"_index": "filebeat-7.7.0",
"_type": "_doc",
"_id": "3xyXKHIBcyw5b4ojZ7OZ",
"_version": 1,
"_score": 1,
"_source": {
"@timestamp": "2020-05-18T16:21:46.000Z",
"agent": {
"hostname": "newds.sxdsms.com",
"id": "ec7e241d-79d2-43e9-8d14-fdc78943dca9",
"version": "7.7.0",
"type": "filebeat",
"ephemeral_id": "93610033-9750-4f81-b910-dcaac9a72b60"
},
"size": "20",
"referer": "-",
"status": "302",
"server_ip": "172.26.19.121",
"upstreamtime": "0.022",
"ecs": {
"version": "1.5.0"
},
"responsetime": "0.023",
"domain": "newds.sxdsms.com",
"file_dir": "/www/wwwroot/wbn4.cn/index.php",
"http_user_agent": "Chrome/57",
"request_length": "58",
"host": {
"hostname": "newds.sxdsms.com",
"architecture": "x86_64",
"os": {
"codename": "Core",
"platform": "centos",
"version": "7 (Core)",
"family": "redhat",
"name": "CentOS Linux",
"kernel": "3.10.0-1062.12.1.el7.x86_64"
},
"id": "20191008171109952517774891784520",
"containerized": false,
"ip": [
"172.26.19.121"
],
"name": "newds.sxdsms.com",
"mac": [
"00:16:3e:09:c0:ef"
]
},
"log": {
"offset": 53613254,
"file": {
"path": "/www/wwwlogs/wbn4.cn.access.log"
}
},
"upstreamhost": "127.0.0.1:9000",
"args": "-",
"client_ip": "47.92.27.176",
"request_method": "GET",
"request_body": "-",
"input": {
"type": "log"
},
"protocol": "HTTP/2.0",
"xff": "-",
"url": "/index.php"
}
}

来源：oschina

链接：https://my.oschina.net/kcw/blog/4283379

标签

ElasticSearch

Chrome

php