Does my docker images need its own consul client instances?

Question

I've got a dockerized application split in several containers (a few frontend and backend servers, load balancer, mysql, elasticsearch, etc.). The configuration of the load balancer needs to know which containers are up and so I'm registering services with Consul service discovery.

But I'm not quite sure if it's a good idea to run a consul agent on every docker container instead of using the docker host to supervise all the running docker containers and register them via Consul's HTTP-API.

Is there any best practice around I can follow?

Answer 1

You don't need to run a consul agent on every docker container, you can simply take advantage of consul by exposing it's DNS to your local. Following is not from a container but you will get the idea anyways as to what I am doing.

following is the command I am using to run my agent

consul agent -data-dir /var/lib/consul/ -config-dir /etc/consul.d/ -bind 10.X.X.X -dns-port 53 -join consul-master

Note: I have added a /etc/hosts entry for consul-master with it's IP and I have also added a nameserver for 127.0.0.1 in the /etc/resolv.conf file.

The directory /etc/consul.d/ holds my configuration file for the service. Following is an example:

{
  "service": {
    "name": "stackoverflow",
    "tags": [
      "example"
    ],
    "port": 5000
  }
}

Now once my consul agent is running, I can check on any host with consul agent (server/client) for the service via dig command or the http api request as follows:

curl http://stackoverflow.service.consul:80/api/v1/ping
{"success":true,"message":"pong"}

For DNS:

dig @127.0.0.1 -p 53 stackoverflow.service.consul

; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.62.rc1.55.amzn1 <<>> @127.0.0.1 -p 53 tracker.service.consul
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 57167
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;tracker.service.consul.        IN  A

;; ANSWER SECTION:
tracker.service.consul. 0   IN  A   X.X.X.X

;; Query time: 1 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Fri Jul  7 11:29:01 2017
;; MSG SIZE  rcvd: 56

Hope that helps and gives a clear idea of it

Answer 2

I'm not sure if there are best practices, but I found this blog post to be very helpful Automatic Docker Service Announcement with Registrator. He talks about several approaches to service registration and their benefits and shortcomings.

More directly answering your question, no, you shouldn't run a consul agent inside every container.

One option is to, run a consul agent on every host. Then you can use something like Registrator to watch for new containers starting up and shutting down and automatically update Consul. The main advantage is that your container has one job to do, run your application. Registrator also has one job to do, watch for container start/stop events and record them in Consul. Your containers could know nothing about consul and still participate on service discovery.

There is also the Autopilot pattern which suggests going in the other direction and making your application Consul aware so it can report its own health and discover its own dependencies. Most of the information I've seen on this pattern comes Joyent (like this blog post). It's worth reading for a different perspective on achieving scalability and resilience in applications.