web server running inside a docker container running inside an EC2 instance responses very slowly

Question

I have a web server running inside a docker container in AWS EC2 Ubuntu instance. When I send requests to the web server, I get the response very slowly (20+ seconds most of the times, although the response time varies). It does not time-out though. The web server is a very lightweight. It is only to test, so almost does not do anything.

docker version 17.03.0-ce

docker-compose version 1.12.0-rc1

How I debugged so far

1. When sending requests to the web server running in the docker container from within the EC2 instance (url = ' http:// localhost:xxxx/api ') it is still very slow. So should not be related to sending requests from outside.

2. I run another web server inside the EC2 directly (not in a docker container), and it is not slow. It responses very fastly.

3. I run another web server inside another docker container in EC2, and it is also very slow!

4. When I send the request from inside the docker container to the web server that is running in it (at its localhost), it is also very slow!

5. I run the containers with the same command on my mac computer and the get response is not slow!

Here is one of the containers stats:

CPU %: 0.28%

MEM USAGE / LIMIT: 27.49 MiB / 992.5 MiB

MEM %: 2.77%

NET I/O: 53.7 kB / 30.5 kB

BLOCK I/O: 2.24 MB / 0 B

I understand it might be very hard to know the issue. My question is the steps to debug the cause and finally find the solution. I appreciate if you could explain your approach in detail.

Answer 1

This sounds like a name resolution problem. To debug this, you can do different things.

• You can first start a simple tcp server with nc -l -p 8000 within the docker container (which is started -p 8000:8000), and the on the host launch: nc 127.0.0.1 8000, type some charachter, see if the TCP communication works, they should appear within the container.

• Next, you can do the same as before but using "localhost" instead of 127.0.0.1

• After this, you can do the same HTTP request you did, but using 127.0.0.1 instead of localhost (this will set the request's Host: header to the same value, which the webserver might not check , or might resolve more easily).

You can also have have a look at the generated /etc/hosts and /etc/resolv.conf within the container. Maybe they do not make sense in the network context of your container.

Also, you can precisely measure the time needed for your requests, if they are near a precise second, this sounds once more like a DNS timeout (if it's 5.003, 10.200, 20.030 seconds, it's like a X seconds timeout, plus the real time needed to respond).