how to define HTTP health check in a consul container for a service on the same host?

Question

We are using a consul agent on a host that also runs a service. (RabbitMQ) To verify that the service is ready we have defined a curl based health check. however, we are using the registrator to inject this check using env variable. SERVICE_CHECK_SCRIPT=curl hostname:15672/....

problem is, we've also told the consul-agent that its hostname is the same as the host. (We must have this feature since we want to see the correct hostname registered with the consul cluster.

When the consul agent runs the health check, it looks for the URL on its own container... this obviously fails... does anybody knows how to define this health check (we are using mesos to do it) so that curl will attempt to connect to the right ip?

Answer 1

You can use Registrator's HTTP health check (provided that you run Consul inside progrium/docker-consul container), for example:

ENV SERVICE_CHECK_HTTP=/howareyou
ENV SERVICE_CHECK_INTERVAL=5s

This check will run check-http script provided by docker-consul, which resolves target container's IP and port using the Docker API.

The other option is to configure Consul DNS to listen on Docker's bridge IP (which is already done in progrium/docker-consul's run script), and launch Docker daemon with the global DNS option pointing to the same IP, e.g. docker -d --bip 10.0.42.1/24 --dns 10.0.42.1 --dns 8.8.8.8.

After that Consul DNS will be available in any of your containers, including the docker-consul one, which means that you can run curl http://my-app.service.consul:12345/howareyou, or even query SRV records, e.g. with dig -t SRV, to obtain both IP and port of a service. And then you may add 10.0.42.1 to the host's resolv.conf to gain the ability to query Consul DNS not only within containers, but from the host OS too.

The preferred method, imho, is to use Registrator's SERVICE_CHECK_HTTP, but setting up Consul DNS for your containers is worth doing anyway.

Both these methods require running Consul agent in client mode on each of your application hosts, but Consul agent, when ran in this mode, uses such a tiny amount of system resources that you'll hardly notice it. And you need it anyway to perform health checks :)