What is the maximum scrape_interval in Prometheus

Question

I used Prometheus to measure business metrics like:

# HELP items_waiting_total Total number of items in a queue
# TYPE items_waiting_total gauge
items_waitin         


        

Answer 1

It's not advisable to go above about 2 minutes. This is as staleness is 5 minutes by default (which is what's causing the gaps), and you want to allow for a failed scrape.

Answer 2

If you want to ignore gaps, it is possible to use some aggregation_over_time functions to get your DATA from Prometheus.

max_over_time(items_waiting_total[900s])

This is useful for situations where frequent gathering of DATA is expensive for collector.