Monitoring short lived python Batch Job Processes using Prometheus

Question

How can I monitor my python processes (say some script that gets triggered periodically by Cron daemon) using Prometheus?

Note that this is not a web application but a s

Answer 1

You may want to look at Prometheus' Pushgateway: whenever your script completes, it can push the metrics it collected (e.g. a histogram of how long your function calls took, total CPU utilization, peak memory utilization etc.).

You seem to be saying your script will run approximately once a second. I am hoping that means something along the lines of "once every 5 minutes for each of 300 tenants". In a case like this, you would push your metrics with something like a tenant_id label and be able to see either per-tenant or aggregated metrics.

If your script runs once a second with the same parameters/configuration, then you'll probably lose some of the metrics because multiple scripts may terminate within the same second, all push their metrics and only the last one's metrics will get collected by Prometheus (as I believe you can't set a collection interval lower than 1 second in Prometheus).

Answer 2

you can use pushgateway and prometheus_client

from prometheus_client import CollectorRegistry, Gauge, push_to_gateway

registry = CollectorRegistry()
g = Gauge('job_last_success_unixtime', 'Last time a job successfully finished', registry=registry)
g.set_to_current_time()
push_to_gateway('localhost:9091', job='job_a', registry=registry)