fluentd

Google container engine logging to Stackdriver Error Reporting

由 和自甴很熟 提交于 2019-12-20 06:12:28
问题 I'm currently trying to log errors to Stackdriver Error Reporting from Google Container Engine. I'm using the built-in fluentd-based Stackdriver Logging agent from GKE which works great. However, when I log an error according to the specification(https://cloud.google.com/error-reporting/docs/formatting-error-messages), I do not see it appear in Stackdriver Error Reporting The payload I see in Stackdriver Logging is { insertId: "xatjb4fltv246" jsonPayload: { stream: "event" message: "path was

kubernetes资源清单之DaemonSet

由 血红的双手。 提交于 2019-12-14 15:59:27
什么是 DaemonSet？ DaemonSet 确保全部（或者某些）节点上运行一个 Pod 的副本。当有节点加入集群时，也会为他们新增一个 Pod 。 当有节点从集群移除时，这些 Pod 也会被回收。删除 DaemonSet 将会删除它创建的所有 Pod。 如何调度 Daemon Pod？ 正常情况下，Pod 运行在哪个机器上是由 Kubernetes 调度器来选择的。然而，由 Daemon Controller 创建的 Pod 已经确定了在哪个机器上（Pod 创建时指定了 .spec.nodeName），因此： 1.DaemonSet Controller 并不关心一个节点的 unschedulable 字段。 2.DaemonSet Controller 可以创建 Pod，即使调度器还没有启动，这对集群启动是非常有帮助的。 与 Daemon Pod 通信 与 DaemonSet 中的 Pod 进行通信，几种可能的模式如下： 1.Push：配置 DaemonSet 中的 Pod 向其它 Service 发送更新，例如统计数据库。它们没有客户端。 2.NodeIP 和已知端口：DaemonSet 中的 Pod 可以使用 hostPort，从而可以通过节点 IP 访问到 Pod。客户端能通过某种方法知道节点 IP 列表，并且基于此也可以知道端口。 3.DNS：创建具有相同 Pod

Configure fluentd to properly parse and ship java stacktrace,which is formatted using docker json-file logging driver,to elastic as single message

由 穿精又带淫゛_ 提交于 2019-12-13 07:11:42
问题 Our service runs as a docker instance. Given limitation is that the docker logging driver cannot be changed to anything different than the default json-file driver. The (scala micro)service outputs a log that looks like this {"log":"10:30:12.375 [application-akka.actor.default-dispatcher-13] [WARN] [rulekeepr-615239361-v5mtn-7]- c.v.r.s.logic.RulekeeprLogicProvider(91) - decision making have failed unexpectedly\n","stream":"stdout","time":"2017-05-08T10:30:12.376485994Z"} {"log":"java.lang

How to add timestamp & key for elasticsearch using fluent

由 时光怂恿深爱的人放手 提交于 2019-12-13 02:14:01
问题 I have configured fluentd and elasticsearch and they both are working fine. I am tailing a file and then reading its data and publishing it to elasticsearch. Below is the json data: {"time": "2018-05-14T11:37:30.339593", "Data count": 78, "Data status": "Sent", "DataId": "332"} and below is the fluentd configuration file: <source> @type tail time_key time path /home/user/file.json format json tag first </source> <match *first*> @type elasticsearch hosts 192.168.196.118:9200 user <username>

How to make fluent-logger-java work with logback appender in springboot

由 风流意气都作罢 提交于 2019-12-13 02:13:25
问题 I am trying to integrate fluent-logger-java with a springboot project with a logback appender. And I am able to send data to a fluentd aggregator with the sample code given in https://github.com/fluent/fluent-logger-java But when added as a logback appender, it throws NullPointerException. I have tried to use logback-more-appenders from https://mvnrepository.com/artifact/com.sndyuk/logback-more-appenders/1.4.3. But it is not sending any data from the application. To reproduce the issue,

How can I send the data from fluentd in kubernetes cluster to the elasticsearch in remote standalone server outside cluster?

由 Deadly 提交于 2019-12-12 10:08:09
问题 I have three kubernetes cluster environments set up in GCP. I have installed Fluentd as daemonset in all these environments to collect the logs from all the pods. I have also installed elasticsearch and kibana in a separate server outside the cluster. I need to feed the logs in fluentd to the elasticsearch in remote server and thereby run a centralised logging platform. How can I send the data from fluentd to the elasticsearch in remote server? The error received is: error_class=Fluent:

Logging to FluentD with dotnet core?

由 北城以北 提交于 2019-12-12 02:19:39
问题 I'm using fluentd as a log aggregator in front of an elasticsearch db. I started working on a dotnet core api and want to add my logs there as well. Is there a good package for this? So far I've found the Nlog and the fluentd target are not compatible with core. 回答1: Use Serilog with the corresponding Fluentd sink 来源： https://stackoverflow.com/questions/40849950/logging-to-fluentd-with-dotnet-core

Google Cloud Logging + google-fluentd Dropping Messages

由 泪湿孤枕 提交于 2019-12-11 11:46:12
问题 I have a rather small (1-2 node) kubernetes cluster running in GKE with ±40 Pods running. The problem at hand is that it's not logging to the GCE Console properly. I see lots of messages from the fluentd container(s) in the following format: $ kubectl logs fluentd-cloud-logging-gke-xxxxxxxx-node-xxxx 2016-02-02 23:30:09 +0000 [warn]: Dropping 10 log message(s) error_class="Google::APIClient::ClientError" error="Project has not enabled the API. Please use Google Developers Console to activate

Fluentd plugin for http with base authenticatoin

由 狂风中的少年 提交于 2019-12-11 05:38:30
问题 Is there any way to handle http request with basic authentication in Fluentd? Or any in_http plugin extension for that? 回答1: One way of doing it is putting nginx as a reverse proxy that does the authentication and then forwards the call to fluentd. 来源： https://stackoverflow.com/questions/38146565/fluentd-plugin-for-http-with-base-authenticatoin

Send Kubernetes cluster logs to AWS Elasticsearch

由 霸气de小男生 提交于 2019-12-10 17:22:11
问题 I have a testing Kubernetes cluster and I created elasticsearch on AWS which include Kibana for the log management. Endpoint: https://search-this-is-my-es-wuktx5la4txs7avvo6ypuuyri.ca-central-1.es.amazonaws.com As far as I googled, I have to send logs from fluentd. Then I tried to implement DaemonSet using this article. No luck. Could you please share any good documentation to me, please 回答1: Kibana provides visualization capabilities on top of the content indexed on an Elasticsearch cluster.
订阅 fluentd