InfluxDB: How to backfill measurement tags that were previously not measured (if possible at all)?

问题

I'm started logging data from my Smart Meter using Node-RED about a month ago, it looked like this json data (the payload is the important bit):

{
  "topic":"stat/smartmeter/all",
  "payload":"{
    \"kwh_low_tarrif\":866.696,
    \"kwh_high_tarrif\":902.156,
    \"current_tarrif\":1,
    \"current_watt_draw\":485,
    \"gas_timestamp\":1554675307000,
    \"gas_total\":326.509,
    \"kwh_combined\":1768.852
  }",
  "qos":0,
  "retain":false,
  "_topic":"stat/smartmeter/all",
  "_msgid":"db4ebc0.72b9a48"
}

The problem with this data is that I did my electrical & gas cost calculations in my Grafana dashboard:

I hardcoded the costs into the dashboard using a math function in the InfluxDB data selection:

There you can see I used the value (or price rather) of 0.230662 euro's per kWh of used electricity. Now silly me never thought about wanting to be able to run calculations over multiple years where this price would fluctuate, so once I discovered my electricity provider's public API endpoint where I could read out the prices for my specific plan, I added it to the measurements, so now the json-data looks like this:

{
  "topic":"stat/smartmeter/all",
  "payload":"{
    \"kwh_low_tarrif\":866.696,
    \"kwh_high_tarrif\":902.156,
    \"kwh_low_price\":0.230662,
    \"kwh_high_price\":0.230662,
    \"current_tarrif\":1,
    \"current_watt_draw\":485,
    \"current_kwh_price\":0.230662,
    \"gas_timestamp\":1554675307000,
    \"gas_total\":326.509,
    \"gas_price\":0.804565,
    \"kwh_combined\":1768.852
  }",
  "qos":0,
  "retain":false,
  "_topic":"stat/smartmeter/all",
  "_msgid":"db4ebc0.72b9a48"
}

The only problem (and my main questions with it) now is that:

1) How do I write a query that uses this value in the price calculation? The query I'm using now (from the screenshot above) is:

SELECT distinct("kwh_combined")  * 0.230662 FROM "smartmeter" WHERE $timeFilter GROUP BY time($__interval) fill(linear)

2) How do I backfill data? (write those electric & gas prices into the database from the beginning of my logging, adding it to the measurements I took back then)

I would much rather have the values I previously had hardcoded in my panel set into the measurements I already took instead of having to write an exception for when the measurements aren't present or 'null'.. I mean, the data itself is as static as can be since the prices haven't changed, so it can't be THAT hard, can it? Even if it need to rebuild the data, can I just re'insert it into a new data collection and ADD the fields myself?

Please tell me this is doable for InfluxDB...

I mean, in MySQL it would a simple ALTER TABLE statement with perhaps a simple insert on the records which had null values.

.. or is it unreasonable for me to ask a time-series logging system to be able to alter it's data structure of already logged data and am I asking too much of InfluxDB?

回答1:

You can add extra fields (price_1, price_2,...) to existing measurement. Insert them with exactly same timestamp and tag values to get existing data records extended with these new price columns ready to be used in grafana:

INSERT smartmeter,tagA=tagAvalue,tagB=tagBvalue,... price1=1234,price2=4321 1549983966469023105
...

Another way would be to export existing data, add extra price columns and import back.

Neither way looks simple. Using select ... into syntax seems easier to run (didn't worked for me without this ugly 0*last() + ... hack):

    SELECT 
       last(kwh_combined) as kwh_combined,
       last(other_field) as other_field,
        ...add other fields here...
       0 * last(kwh_combined) + 1234 as price1,
       0 * last(kwh_combined) + 4321 as price2 
       INTO new_smartmeter
    FROM smartmeter GROUP BY *

来源：https://stackoverflow.com/questions/55563915/influxdb-how-to-backfill-measurement-tags-that-were-previously-not-measured-if

标签

database

iot

grafana

InfluxDB

sensors