Context Deadline Exceeded - prometheus

后端 未结 8 1400
长情又很酷
长情又很酷 2021-02-12 20:07

I have Prometheus configuration with many jobs where I am scraping metrics over HTTP. But I have one job where I need to scrape the metrics over HTTPS.

When I access:

相关标签:
8条回答
  • 2021-02-12 20:47

    disable selinux, then reboot server and test again.

    0 讨论(0)
  • 2021-02-12 20:49

    in my case it was issue with IPv6. I have blocked IPv6 with ip6tables, but it also blocked prometheus traffic. Correct IPv6 settings solved issue for me

    0 讨论(0)
  • 2021-02-12 20:56

    Increasing the timeout to 1m helped me to fix a similar issue

    0 讨论(0)
  • 2021-02-12 20:57

    Probably the default scrape_timeout value is too short for you

    [ scrape_timeout: <duration> | default = 10s ]
    

    Set a bigger value for scrape_timeout.

    scrape_configs:
      - job_name: 'prometheus'
    
        scrape_interval: 5m
        scrape_timeout: 1m
    

    Take a look here https://github.com/prometheus/prometheus/issues/1438

    0 讨论(0)
  • 2021-02-12 21:00

    I had a similar problem, so I tried to extend my scrape_timeout but it didn't do anything - using promtool, however, explained the problem

    My problematic job looked like this:

    - job_name: 'slow_fella'
      scrape_interval: 10s
      scrape_timeout: 90s
      static_configs:
      - targets: ['192.168.1.152:9100']
        labels:
          alias: sloooow    
    

    check your config like this:

    /etc/prometheus $ promtool check config prometheus.yml
    

    Result explains the problem and indicates how to solve it:

    Checking prometheus.yml
      FAILED: parsing YAML file prometheus.yml: scrape timeout greater than scrape interval for scrape config with job name "slow_fella"
    

    Just ensure that your scrape_timeout is long enough to accommodate your required scrape_interval.

    0 讨论(0)
  • 2021-02-12 21:03

    This can be happened when the prometheus server can't reach out to the scraping endpoints maybe of firewall denied rules. Just check hitting the url in a browser with <url>:9100 (here 9100 is the node_exporter service running port`) and check if you still can access?

    0 讨论(0)
提交回复
热议问题