问题
I'm using graphite for pretty long time, and first time facing issue with some metrics getting… lost?
Through tcpdump -nA dst port 2003 I can see that metrics are delivered to Graphite node.
Also, some of them are getting created in whisper database, and seen in /var/log/carbon/updates.log
But most of them are not appearing anywhere.
So my question is: how do I debug it? How do I prove that Graphite really receives these metrics from eth0?
I couldn't find any debug logs except for updates.log in carbon.
Log:
sudo tcpdump -An dst port 2003 | grep 172_31_00_01 | grep requests
backend.dev.172_31_00_01.requests.max 60554.34 1453734067 backend.dev.172_31_00_01.requests.mean 16714.87 1453734067 backend.dev.172_31_00_01.requests.min 2.93 1453734067 backend.dev.172_31_00_01.requests.stddev 12185.74 1453734067 backend.dev.172_31_00_01.requests.p50 16415.87 1453734067 backend.dev.172_31_00_01.requests.p75 20314.51 1453734067 backend.dev.172_31_00_01.requests.p95 41526.36 1453734067 backend.dev.172_31_00_01.requests.p98 54370.59 1453734067 backend.dev.172_31_00_01.requests.p99 60368.68 1453734067 backend.dev.172_31_00_01.requests.p999 60553.31 1453734067 backend.dev.172_31_00_01.requests.count 3141 1453734067 backend.dev.172_31_00_01.requests.m1_rate 2.02 1453734067 backend.dev.172_31_00_01.requests.m5_rate 1.95 1453734067 backend.dev.172_31_00_01.requests.m15_rate 1.20 1453734067 backend.dev.172_31_00_01.requests.mean_rate 0.66 1453734067 backend.dev.172_31_00_01.requests.mark_sessionid_active.max 152.59 1453734067 backend.dev.172_31_00_01.requests.mark_sessionid_active.mean 41.86 1453734067 backend.dev.172_31_00_01.requests.mark_sessionid_active.min 0.82 1453734067 backend.dev.172_31_00_01.requests.mark_sessionid_active.stddev 24.84 1453734067 backend.dev.172_31_00_01.requests.mark_sessionid_active.p75 57.51 1453734067 backend.dev.172_31_00_01.requests.mark_sessionid_active.p95 85.78 1453734067
$ pwd
/var/lib/graphite/whisper/backend/dev/172_31_00_01/requests
$ ls -Rl
.:
total 1796
drwxr-xr-x 2 _graphite _graphite 4096 Jan 25 14:25 mark_sessionid_active
-rw-r--r-- 1 _graphite _graphite 1831744 Jan 25 15:05 mean.wsp
./mark_sessionid_active:
total 3584
-rw-r--r-- 1 _graphite _graphite 1831744 Jan 25 15:05 min.wsp
-rw-r--r-- 1 _graphite _graphite 1831744 Jan 25 15:05 stddev.wsp
PS: It's not new installation, it works for several months now, and no metrics was lost until today.
回答1:
There is MAX_CREATES setting in carbon.conf. Setting it to high value (like 1000) or inf solves this.
回答2:
Check the LOG_DIR variable in carbon.conf. In my case it's /var/log/carbon/ and I can see a lot of logs in there like console.log, creates.log, listener.log. I believe creates.log is the one you are after.
If the .wsp file is created but you cannot see it in Graphite directly, try trying to render it anyhow using the URL API and see if it works.
来源：https://stackoverflow.com/questions/34995825/graphite-some-metrics-lost-but-seen-in-tcpdump