How can I 'join' two metrics in a Prometheus query?

Question

I am using the consul exporter to ingest the health and status of my services into Prometheus. I\'d like to fire alerts when the status of services and nodes in Consul is cr

Answer 1

You can use the argument list of group_left to include extra labels from the right operand (parentheses and indents for clarity):

(
  max(consul_health_service_status{status="critical"}) 
  by (service_name,status,node) == 1
)
   + on(service_name,node) group_left(env)
(
   0 * consul_service_tags
)

The important part here is the operation + on(service_name,node) group_left(env):

• the + is "abused" as a join operator (fine since 0 * consul_service_tags always has the value 0)
• group_left(env) is the modifier that includes the extra label env from the right (consul_service_tags)