Prometheus dns service discovery in docker swarm relabel instance

Question

My question is an addition to Prometheus dns service discovery in docker swarm.

I define the prometheus scrape targets as follows:

- job_name: 'node-exporter'
  dns_sd_configs:
  - names:
    - 'tasks.nodeexporter'
    type: 'A'
    port: 9100

This works fine but results in prometheus using the IP of the docker container as instance label.

I tried to relabel the instance label as follows:

relabel_configs:
- source_labels: [__meta_dns_name]
  target_label: instance

But doing so results in all instances of node-exporter having the same label "tasks.nodeexporter".

Is it somehow possible to relabel the instance label to something like tasks.nodexporter_1, tasks.nodeexporter_2, ...?

Answer 1

Service discovery for docker swarm setups isn’t supported well in Prometheus as there are many features missing on the swarm side.

The dns service discovery is one way to mitigate those missing features but in my opinion it isn’t a good solution and I recommend to not use it in production:

• there is no way to provide additional information e.g. using SRV records
• there is no information about how many instances should be running
• as the the dns only lists the healthy tasks, the amount of scrape targets decreases when one task is no longer considered healthy which makes it harder to alert on misbehaving containers
• when containers die and are restarted you will observe new instances as there is no information like the task slot available

Altogether these issues don’t allow this approach for being a reliable source for a monitoring system.

If you are really tied to using docker swarm, you should consider building a more sustainable solution by querying the docker api programmatically and using the file_sd service discovery mechanism of Prometheus. Please see this poc by containersolutions for reference: https://github.com/ContainerSolutions/prometheus-swarm-discovery