Adding custom jmx metrics to google cloud monitoring collectd configuration

问题

I've added the JVM Monitoring plugin as described here

That's all working great and I can, but now I'd like to add more JMX metrics. e.g. MemoryPool specific counters So I've added this config to /opt/stackdriver/collectd/etc/collectd.d/jvm-sun-hotspot.conf

<MBean "jvm_localhost_MemoryPool">
    ObjectName "java.lang:type=MemoryPool,name=*"
    InstanceFrom "name"
    <Value>
        Type "gauge"
        InstancePrefix "memorypool-usage_used"
        Table false
        Attribute "Usage.used"
    </Value>
</MBean>

and

Collect "jvm_localhost_MemoryPool"

in the Connection section

It seems to be valid collectd config but when it's getting sent to Stackdriver/Google Cloud Monitoring it is rejected.

012    {#012      "index": 261,#012      "valueErrors": [#012        {#012          "error": {#012            "code": 3,#012            "message": "Unsupported
 collectd id: plugin: \"jvm\" type: \"gauge\" type_instance: \"memorypool-usagecommitted\""#012          }#012        }#012      ]#012    },#012    {#012
"index": 262,#012      "valueErrors": [#012        {#012          "error": {#012            "code": 3,#012            "message": "Unsupported collectd id: plug
in: \"jvm\" type: \"gauge\" type_instance: \"memorypool-usageinit\""#012          }#012        }#012      ]#012    },#012    {#012      "index": 263,#012
"valueErrors": [#012        {#012          "error": {#012            "code": 3,#012            "message": "Unsupported collectd id: plugin: \"jvm\" type: \"gau
ge\" type_instance: \"memorypool-usagemax\""#012          }#012        }#012      ]#012    },#012    {#012      "index": 264,#012      "valueErrors": [#012
    {#012          "error": {#012            "code": 3,#012            "message": "Unsupported

Now from what I understand it needs to be added as a custom metric, but this document suggests it would be auto created. And indeed when I look at the list of builtin jvm metrics I can't see how they map onto the existing ones in the collectd configuration.

e.g. how is os-open_fd_count mapped to os/open_files ?

It would be helpful to see the actual api request being sent by Google's custom collectd implementation but I can't see a way of increasing logging.

I can see from this post that it's probably custom metrics where I want to be looking but how do I do that in the collectd configuration?

I've tried

InstancePrefix "custom.googleapis.com/memorypool-usage"

but still no joy.

Has anyone done this before or can offer any suggestions on what I'm doing wrong?

回答1:

The troubleshooting documents [1] could be helpful to determine what points need to be transformed, as well as to ensure your transformations behave as expected.

[1] https://cloud.google.com/monitoring/agent/custom-metrics-agent#troubleshooting

回答2:

To get this logging I needed to add stackdriver_metric_type metadata.

The complete chain is now

<Chain "GenericJMX_jvm">
    <Rule "rewrite_custom_jmx">
        <Match regex>
            Plugin "^GenericJMX$"
            PluginInstance "^jvm.*$"
            TypeInstance "^memorypool-usage_used$"
        </Match>
        <Target "set">
            MetaData "stackdriver_metric_type" "custom.googleapis.com/jvm/memorypool/usage_used"
            MetaData "label:pool" "%{plugin_instance}"
        </Target>
        <Target "replace">
            MetaData "label:pool" "jvm" ""
        </Target>
    </Rule>
    <Rule "rewrite_genericjmx_to_jvm">
        <Match regex>
            Plugin "^GenericJMX$"
            PluginInstance "^jvm.*$"
        </Match>
        <Target "replace">
            PluginInstance "jvm" ""
        </Target>
        <Target "set">
            Plugin "jvm"
        </Target>
        Target "return"
    </Rule>
</Chain>

The plugin instance is the pool name (e.g. G1 Eden Space) which is why I copied it into the 'pool' label.

This did auto-create the metric in Stackdriver but I also used the following body to the projects.metricDescriptors.create method to add description and units.

{
  "name": "projects/yourprojecthere/metricDescriptors/custom.googleapis.com/jvm/memorypool/usage_used",
  "labels": [
    {
      "key": "pool",
      "description": "Name of the JVM memory pool."
    }
  ],
  "metricKind": "GAUGE",
  "valueType": "DOUBLE",
  "unit": "By",
  "description": "Current size in bytes of the memory pool.",
  "type": "custom.googleapis.com/jvm/memorypool/usage_used",
  "monitoredResourceTypes": [
    "gce_instance"
  ]
}

The resulting graph in metric explorer looks like this

来源：https://stackoverflow.com/questions/62414258/adding-custom-jmx-metrics-to-google-cloud-monitoring-collectd-configuration

标签

stackdriver

google-cloud-stackdriver

collectd

google-cloud-monitoring