Generating range vectors from return values in Prometheus queries

Question

I have a metric varnish_main_client_req of type counter and I want to set up an alert that triggers if the rate of requests drops/raises by a certain amount in a gi

Answer 1

Solution

It's possible with the subquery-syntax (introduced in Prometheus version 2.7):

deriv(rate(varnish_main_client_req[2m])[5m:10s])

Warning: These subqueries are expensive, i.e. create very high load on Prometheus. Use recording-rules when you use these queries regularly (in alerts, etc.).

Subquery syntax

<instant_query>[<range>:<resolution>]

• instant_query: a PromQL-function which returns an instant-vector)
• range: offset (back in time) to start the first subquery
• resolution: the size of each of the subqueries.

It returns a range-vector.

In the example above, Prometheus runs rate() (= instant_query) 30 times (the first from 5 minutes ago to -4:50, ..., the last -0:10 to now). The resulting range-vector is input to the deriv()-function.

Another example (mostly available on all Prometheus instances):

deriv(rate(prometheus_http_request_duration_seconds_sum{job="prometheus"}[1m])[5m:10s])

Without the subquery-range ([5m:10s]), you'll get this error-message:

parse error at char 80: expected type range vector in call to function "deriv", got instant vector

Answer 2

Yes, you need to use a recording rule for this.

Prometheus calculates the rate of client requests over the last 2 mins and returns a derivative of the resulting values over the last 5 mins.

Herein lies the problem - at what interval should Prometheus synthesise this data?