telegraf - exec plugin - aws ec2 ebs volumen info - metric parsing error, reason: [missing fields] or Errors encountered: [ invalid number]

Question

Machine - CentOS 7.2 or Ubuntu 14.04/16.xx

Telegraf version: 1.0.1

Python version: 2.7.5

Answer 1

It seems like the rules are very strict, I should have looked more closely.

Syntax of the output of any program that you can to consume MUST match or follow INFLUX LINE PROTOCOL format shown below and also all the RULES which comes with it.

For ex:

weather,location=us-midwest temperature=82 1465839830100400200
  |    -------------------- --------------  |
  |             |             |             |
  |             |             |             |
+-----------+--------+-+---------+-+---------+
|measurement|,tag_set| |field_set| |timestamp|
+-----------+--------+-+---------+-+---------+

You can read more about what's measurement, tag, field and optional(timestamp) here: https://docs.influxdata.com/influxdb/v1.2/write_protocols/line_protocol_tutorial/

Important rules are:

1) There must be a , and no space between measurement and tag set.

2) There must be a space between tag set and field set.

3) For tag keys, tag values, and field keys always use a backslash character \ to escape if you want to escape any character in measurement name, tag or field set name and their values!

4) You can't escape \ with \

5) Line Protocol handles emojis with no problem :)

6) TAG / TAG set (tags comma separated) in OPTIONAL

7) FIELD / FIELD set (fields, comma separated) - At least ONE is required per line.

8) TIMESTAMP (last value shown in the format) is OPTIONAL.

9) VERY IMPORTANT QUOTING rules are below:

a) Never double or single quote the timestamp. It’s not valid Line Protocol. '123123131312313' or "1231313213131" won't work if that # is valid.

b) Never single quote field values (even if they’re strings!). It’s also not valid Line Protocol. i.e. fieldname='giga' won't work.

c) Do not double or single quote measurement names, tag keys, tag values, and field keys. NOTE: THIS does say !!! tag values !!!! so careful.

d) Do not double quote field values that are ONLY in floats, integers, or booleans format, otherwise InfluxDB will assume that those values are strings.

e) Do double quote field values that are strings.

f) AND the MOST IMPORTANT one (which will save you from getting BALD): If a FIELD value is set without double quote / i.e. you think it's an integer value or float in one line (for ex: anyone will say fields size or iops) and in some other lines (anywhere in the file that telegraf will read/parse using exec plugin) if you have a non-integer value set (i.e. a String), then you'll get the following error message Errors encountered: [ invalid number error.

So to fix it, the RULE is, if any possible FIELD value for a FIELD key is a string, then you MUST make sure to use " to wrap it (in every lines), it doesn't matter whether it has value 1, 200 or 1.5 in some lines (for ex: iops can be 1, 5) and in some other lines that value (iops can be None).

Error message: Errors encountered: [ invalid number

[vagrant@myvagrant ~] $ telegraf --config-directory=/etc/telegraf --test --input-filter=exec
2017/03/10 11:13:18 I! Using config file: /etc/telegraf/telegraf.conf
* Plugin: inputs.exec, Collection 1
2017-03-10T11:13:18Z E! Errors encountered: [ invalid number metric parsing error, reason: [invalid field format], buffer: [awsebsvol,host=myvagrant ], index: [25]]

So, after all this learning, it's clear that first I was missing the Influx Line protocol format and ALSO the RULES!!

Now, my output that I want my python script to generate should be like this (acc. to the INFLUX LINE PROTOCOL). You can just change the .sh file and use sed "s/^/awsec2ebs,/" or also do sed "s/^/awsec2ebs,sourcehost=$(hostname) /" (note: the space before the closing sed / character) and then you can have " around any key=value pair. I did change .py file to not use " for size and iops fields.

Anyways, if the output is something like this:

awsec2ebs,volume_id=vol-058e1d47dgh721121 create_time="2017-01-09 23:24:29.428000+00:00",availability_zone="us-east-2b",volume_type="gp2",state="in-use",size="8",iops="100",encrypted="False",snapshot_id="snap-06h1h1b91bh662avn",kms_key_id="None",InstanceId="i-0jjb1boop26f42f50",InstanceVolumeState="attached",DeleteOnTermination="True",Device="/dev/sda1",Name="[company-2b-app90] secondary",hostname="company-2b-app90-i-0jjb1boop26f42f50",high_availability="1",mirror="secondary",cluster="company",autoscale="true",role="app"

In the above final working solution, I created a measurement named awsec2ebs then gave , between this measurement and tag key volume_id and for tag value, I did NOT use any ' or " quotes and then I gave a space character (as I just wanted only one tag for now otherwise you can have more tag using command separated way and following the rules) between tag set and field set.

Finally ran the command:

$ telegraf --config-directory=/etc/telegraf --test --input-filter=exec which worked like a shenzi!

2017/03/10 03:33:54 I! Using config file: /etc/telegraf/telegraf.conf
* Plugin: inputs.exec, Collection 1
> awsec2ebs_telegraf_execplugin,volume_id=vol-058e1d47dgh721121,host=myvagrant volume_type="gp2",iops="100",kms_key_id="None",role="app",size="8",encrypted="False",InstanceId="i-0jjb1boop26f42f50",InstanceVolumeState="attached",Name="[company-2b-app90] secondary",snapshot_id="snap-06h1h1b91bh662avn",DeleteOnTermination="True",mirror="secondary",cluster="company",autoscale="true",high_availability="1",create_time="2017-01-09 23:24:29.428000+00:00",availability_zone="us-east-2b",state="in-use",Device="/dev/sda1",hostname="company-2b-app90-i-0jjb1boop26f42f50" 1489116835000000000
[vagrant@myvagrant ~] $ echo $?
0

In the above example, size is the only field which will always be a number/numeric value, so we don't need to wrap it with " but it's up to you. Recall the MOST IMPORTANT rule.. above and the error it generates.

So final python file is:

#!/usr/bin/python

#Do `sudo pip install boto3` first
import boto3

def generate(key, value, qs, qe):
    """
    Creates a nicely formatted Key(Value) item for output
    """
    return '{}={}{}{}'.format(key, qs, value, qe)

def main():
    ec2 = boto3.resource('ec2', region_name="us-west-2")
    volumes = ec2.volumes.all()

    for vol in volumes:
        # You don't need to wrap everything in `str` unless it is not a string
        # By default most things will come back as a string
        # unless they are very obviously not (complex, date time, etc)
        # but since we are printing these (and formatting them into strings)
        # the cast to string will be implicit and we don't need to make it
        # explicit

        # vol is already a fully returned volume you are essentially DOUBLING
        # your API calls when you do this
        #iv = ec2.Volume(vol.id)
        output_parts = [
            # Volume level details
            generate('volume_id', vol.volume_id, '"', '"'),
            generate('create_time', vol.create_time, '"', '"'),
            generate('availability_zone', vol.availability_zone, '"', '"'),
            generate('volume_type', vol.volume_type, '"', '"'),
            generate('state', vol.state, '"', '"'),
            generate('size', vol.size, '', ''),
            #The following vol.iops variable can be a number or None so you must wrap it with double quotes otherwise "invalid number" error will come.
            generate('iops', vol.iops, '"', '"'),
            generate('encrypted', vol.encrypted, '"', '"'),
            generate('snapshot_id', vol.snapshot_id, '"', '"'),
            generate('kms_key_id', vol.kms_key_id, '"', '"'),
        ]

        for _ in vol.attachments:
            # Will get any attachments and since it is a list
            # we should write this to handle MULTIPLE attachments
            output_parts.extend([
                generate('InstanceId', _.get('InstanceId'), '"', '"'),
                generate('InstanceVolumeState', _.get('State'), '"', '"'),
                generate('DeleteOnTermination', _.get('DeleteOnTermination'), '"', '"'),
                generate('Device', _.get('Device'), '"', '"'),
            ])

        # only process when there are tags to process
        if vol.tags:
            for _ in vol.tags:
                # Get all of the tags
                output_parts.extend([
                    generate(_.get('Key'), _.get('Value'), '"', '"'),
                ])

        # output everything at once..
        print ','.join(output_parts)

if __name__ == '__main__':
    main()

Final aws-vol-info.sh is:

#!/bin/bash

cat aws-vol-info.csv | sed "s/^/awsebsvol,host=`hostname|head -1|sed "s/[ \t][ \t]*/_/g"` /"

Final telegraf exec plugin config file is (/etc/telegraf/telegraf.d/exec-plugin-aws-info.conf) give any name with .conf:

#--- https://github.com/influxdata/telegraf/tree/master/plugins/inputs/exec

[[inputs.exec]]
  commands = ["/some/valid/path/where/csvfileexists/aws-vol-info.sh"]

  ## Timeout for each command to complete.
  timeout = "5s"

  # Data format to consume.
  # NOTE json only reads numerical measurements, strings and booleans are ignored.
  data_format = "influx"

  name_suffix = "_telegraf_exec"

Run: and everything will work now!

$ telegraf --config-directory=/etc/telegraf --test --input-filter=exec