cockroachdb/kubernetes logging lots of errors on stackdriver

天涯浪子 提交于 2019-12-11 12:32:18

问题


I've just noticed from a very nasty gcp bill that cockroachDB has logged 1.5tb of errors on stackdriver, costing me several hundred dollars in just a few days. Sadly I had left it on 100% logging. The errors look like this and are piling up multiple times per second.

E  I180712 11:18:41.963205 106 server/status/runtime.go:223  [n2]
runtime stats: 1.5 GiB RSS, 283 goroutines, 254 MiB/54 MiB/441 MiB GO alloc/idle/total, 
918 MiB/1.1 GiB CGO alloc/total, 
2175.51cgo/sec, 
0.16/0.02 %(u/s)time, 0.00 %gc (1x)

Does anybody know what they mean, and how to stop them?


回答1:


These are all CockroachDB logs, not just errors. This is indicated by the I prefix (meaning Info) in the CockroachDB log line. The listed log lines show basic memory information for the cockroach process. This is logged every 10 seconds.

If you wish to persist logs I would recommend filtering by severity.

This can be done by CockroachDB itself when redirecting logs to stderr by using: --logtostderr=Level where Level is one of Info, Warning, Error, or Fatal.

If you are saving the raw logs, you could do a quick pass to discard anything not starting with a desired prefix. This will however not be as accurate as the --logtostderr method as you would need to handle multi-line log entries.




回答2:


Marc's answer is the "right" one, but as a workaround you could exclude specific logs from Stackdriver: https://cloud.google.com/logging/docs/exclusions

Log exclusions give fine grain control, and wouldn't require re-configuring or restarting the job.



来源:https://stackoverflow.com/questions/51304889/cockroachdb-kubernetes-logging-lots-of-errors-on-stackdriver

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!