DNS not working within docker containers when host uses dnsmasq and Google's DNS server are firewalled?

Question

The symptom is: the host machine has proper network access, but programs running within containers can\'t resolve DNS names (which may appear to be \"can\'t access the netwo

Answer 1

A brutal and unsafe solution is to avoid containerization of the network, and use the same network on the host and on the container. This is unsafe because this gives access to all the network resources of the host to the container, but if you do not need this isolation this may be acceptable.

To do so, just add --network host to the command-line, e.g.

$ sudo docker run -ti --network host mmoy/ubuntu-netutils /bin/bash
root@ubuntu1604:/# ping www.example.com
PING www.example.com (93.184.216.34) 56(84) bytes of data.
64 bytes from 93.184.216.34: icmp_seq=1 ttl=55 time=86.5 ms
64 bytes from 93.184.216.34: icmp_seq=2 ttl=55 time=86.5 ms

Answer 2

A clean solution is to configure docker+dnsmasq so than DNS requests from the docker container are forwarded to the dnsmasq daemon running on the host.

For that, you need to configure dnsmasq to listen to the network interface used by docker, by adding a file /etc/NetworkManager/dnsmasq.d/docker-bridge.conf:

$ cat /etc/NetworkManager/dnsmasq.d/docker-bridge.conf
listen-address=172.17.0.1

Then restart network manager to have the configuration file taken into account:

sudo service network-manager restart

Once this is done, you can add 172.17.0.1, i.e. the host's IP address from within docker, to the list of DNS servers. This can be done either using the command-line:

$ sudo docker run -ti --dns 172.17.0.1 mmoy/ubuntu-netutils bash
root@7805c7d153cc:/# ping www.example.com
PING www.example.com (93.184.216.34) 56(84) bytes of data.
64 bytes from 93.184.216.34: icmp_seq=1 ttl=54 time=86.6 ms

... or through docker's configuration file /etc/docker/daemon.json (create it if it doesn't exist):

$ cat /etc/docker/daemon.json                      
{
  "dns": [
    "172.17.0.1",
        "8.8.8.8",
        "8.8.4.4"
  ]
}

(this will fall back to Google's public DNS if dnsmasq fails)

You need to restart docker to have the configuration file taken into account:

sudo service docker restart

Then you can use docker as usual:

$ sudo docker run -ti mmoy/ubuntu-netutils bash
root@344a983908cb:/# ping www.example.com
PING www.example.com (93.184.216.34) 56(84) bytes of data.
64 bytes from 93.184.216.34: icmp_seq=1 ttl=54 time=86.3 ms

Answer 3

I just had to deal with this last night and eventually remembered that docker run has a set of options for handling it. I used --dns to specify the DNS server I want the container to use. Works like a champ and no need to hack my docker host. There are other options for the domain name and search suffixes.

Answer 4

One way is to use a user defined network for your container. In that case the container's /etc/resolv.conf will have the nameserver 127.0.0.11 (a.k.a. the Docker's embedded DNS server), which can forward DNS requests to the host's loopback address properly.

$ cat /etc/resolv.conf
nameserver 127.0.0.1
$ docker run --rm alpine cat /etc/resolv.conf
nameserver 8.8.8.8
nameserver 8.8.4.4
$ docker network create demo
557079c79ddf6be7d6def935fa0c1c3c8290a0db4649c4679b84f6363e3dd9a0
$ docker run --rm --net demo alpine cat /etc/resolv.conf
nameserver 127.0.0.11
options ndots:0    

If you use docker-compose, it will set up a custom network for your services automatically (with a file format v2+). Note, however, that while docker-compose runs containers in a user-defined network, it still builds them in the default network. To use a custom network for builds you can specify the network parameter in the build configuration (requires file format v3.4+).

Answer 5

Since the automatic DNS discovery is guilty here, you may override the default setting in docker's configuration.

First, get the IP of the DNS server dnsmasq is using with e.g.:

$ sudo kill -USR1 `pidof dnsmasq`
$ sudo tail /var/log/syslog 
[...]
Apr 24 13:20:19 host dnsmasq[2537]: server xx.yy.zz.tt1#53: queries sent 0, retried or failed 0
Apr 24 13:20:19 host dnsmasq[2537]: server xx.yy.zz.tt2#53: queries sent 0, retried or failed 0

The IP addresses correspond to the xx.yy.zz.tt placeholders above.

You can set the DNS at docker run time with the --dns option:

$ sudo docker run --dns xx.yy.zz.tt1 --dns xx.yy.zz.tt2 -ti mmoy/ubuntu-netutils bash
root@6c5d08df5dfd:/# ping www.example.com
PING www.example.com (93.184.216.34) 56(84) bytes of data.
64 bytes from 93.184.216.34: icmp_seq=1 ttl=54 time=86.6 ms
64 bytes from 93.184.216.34: icmp_seq=2 ttl=54 time=86.6 ms

One advantage of this solution is that there is no configuration file involved, hence no risk of forgetting about the configuration and running into troubles later because of a specific config: you're getting this DNS configuration if and only if you type the --dns option.

Alternatively you may set it permanently in Docker's configuration file, /etc/docker/daemon.json (create it, on the host, if it doesn't exist):

$ cat /etc/docker/daemon.json
{
    "dns": ["xx.yy.zz.tt1", "xx.yy.zz.tt2"]
}

You need to restart the docker daemon to take the daemon.json file into account:

sudo service docker restart

Then you can check the configuration:

$ sudo docker run -ti mmoy/ubuntu-netutils bash
root@56c74d3bd94b:/# cat /etc/resolv.conf 
nameserver xx.yy.zz.tt1
nameserver xx.yy.zz.tt2
root@56c74d3bd94b:/# ping www.example.com
PING www.example.com (93.184.216.34) 56(84) bytes of data.
64 bytes from 93.184.216.34: icmp_seq=1 ttl=54 time=86.5 ms

Note that this hardcodes the DNS IP in your configuration files. This is strongly discouraged if your machine is a laptop that connects to different networks, and may be problematic if your internet service provider changes the IP of the DNS servers.

Answer 6

Since dnsmasq is the issue, one option is to disable it on the host. This works, but will disable DNS caching for all applications running on the host, hence is a really bad idea if the host is used for applications other than docker.

If you're sure you want to go this way, uninstall dnsmasq, e.g. on Debian-based systems like Ubuntu, run apt remove dnsmasq.

You may then check that /etc/resolv.conf within the container points to the DNS server used by the host.