datadog agent not reachable from inside docker container

Question

I installed dd-agent on Amazon linux ec2. If I run my python script directly on the host machine (I used the SDK named "dogstatsd-python"), all the metrics can be sent to datadog (I logged in to datadoghq.com and saw the metrics there). the script is something like:

from statsd import statsd
statsd.connect('localhost', 8125)
statsd.increment('mymetrics')

However, I launched a docker container and run the same script from inside the container:

from statsd import statsd
statsd.connect('172.14.0.1', 8125)
statsd.increment('my metrics')

'172.14.0.1' is the IP of the host, which was extracted with command

netstat -nr | grep '^0\.0\.0\.0' | awk '{print $2}'

No metrics were sent to datadog at all.....

I'm guessing that maybe it's due to some configuration issue like "address binding". Maybe the dd-agent I installed on the host can only receive metrics from 'localhost'.

Hope someone could help me. Thank you in advance.

Answer 1

You will need to set non_local_traffic: yes in your /etc/dd-agent/datadog.conf file. Otherwise the agent will reject metrics from containers.

After setting, you will need to restart the agent for the change to take effect: sudo /etc/init.d/datadog-agent restart or sudo service datadog-agent restart

The docker-dd-agent image enables non_local_traffic: yes by default.

Answer 2

If dd-agent listens on localhost it can receive data only from localhost (127.0.0.1). Try to change the dd-agent host to 0.0.0.0 instead of localhost.

We are using docker-dd-agent and it works OOTB.

Answer 3

You don't actually want to use the IP of the host in this case. If you're running the docker dd-agent, there are two environment variables you can tap into:

statsd.connect(DOGSTATSD_PORT_8125_UDP_ADDR, DOGSTATSD_PORT_8125_UDP_PORT)

That should do the trick. If not, you should be able to find the relevant info to your problem in this section of the Datadog docs.

Also, I should point out that the only Python library that Datadog shows in their docs is datadogpy.