I have Prometheus configuration with many jobs where I am scraping metrics over HTTP. But I have one job where I need to scrape the metrics over HTTPS.
When I access:
disable selinux, then reboot server and test again.
in my case it was issue with IPv6. I have blocked IPv6 with ip6tables, but it also blocked prometheus traffic. Correct IPv6 settings solved issue for me
Increasing the timeout to 1m helped me to fix a similar issue
Probably the default scrape_timeout value is too short for you
[ scrape_timeout: <duration> | default = 10s ]
Set a bigger value for scrape_timeout.
scrape_configs:
- job_name: 'prometheus'
scrape_interval: 5m
scrape_timeout: 1m
Take a look here https://github.com/prometheus/prometheus/issues/1438
I had a similar problem, so I tried to extend my scrape_timeout but it didn't do anything - using promtool, however, explained the problem
My problematic job looked like this:
- job_name: 'slow_fella'
scrape_interval: 10s
scrape_timeout: 90s
static_configs:
- targets: ['192.168.1.152:9100']
labels:
alias: sloooow
check your config like this:
/etc/prometheus $ promtool check config prometheus.yml
Result explains the problem and indicates how to solve it:
Checking prometheus.yml
FAILED: parsing YAML file prometheus.yml: scrape timeout greater than scrape interval for scrape config with job name "slow_fella"
Just ensure that your scrape_timeout is long enough to accommodate your required scrape_interval.
This can be happened when the prometheus
server can't reach out to the scraping endpoints maybe of firewall denied rules. Just check hitting the url in a browser with <url>:9100
(here 9100
is the node_exporter service running port`) and check if you still can access?