datadog agent not reachable from inside docker container

Question

I installed dd-agent on Amazon linux ec2. If I run my python script directly on the host machine (I used the SDK named "dogstatsd-python"), all the metrics can be sent to datadog (I logged in to datadoghq.com and saw the metrics there). the script is something like:

from statsd import statsd
statsd.connect('localhost', 8125)
statsd.increment('mymetrics')

However, I launched a docker container and run the same script from inside the container:

from statsd import statsd
statsd.connect('172.14.0.1', 8125)
statsd.increment('my metrics')

'172.14.0.1' is the IP of the host, which was extracted with command

netstat -nr | grep '^0\.0\.0\.0' | awk '{print $2}'

No metrics were sent to datadog at all.....

I'm guessing that maybe it's due to some configuration issue like "address binding". Maybe the dd-agent I installed on the host can only receive metrics from 'localhost'.

Hope someone could help me. Thank you in advance.

Answer 1

You will need to set non_local_traffic: yes in your /etc/dd-agent/datadog.conf file. Otherwise the agent will reject metrics from containers.

After setting, you will need to restart the agent for the change to take effect: sudo /etc/init.d/datadog-agent restart or sudo service datadog-agent restart

The docker-dd-agent image enables non_local_traffic: yes by default.

Answer 2

If dd-agent listens on localhost it can receive data only from localhost (127.0.0.1). Try to change the dd-agent host to 0.0.0.0 instead of localhost .

We are using docker-dd-agent and it works OOTB.

Answer 3

You don't actually want to use the IP of the host in this case. If you're running the docker dd-agent, there are two environment variables you can tap into:

statsd.connect(DOGSTATSD_PORT_8125_UDP_ADDR, DOGSTATSD_PORT_8125_UDP_PORT)

That should do the trick. If not, you should be able to find the relevant info to your problem in this section of the Datadog docs .

Also, I should point out that the only Python library that Datadog shows in their docs is datadogpy .

Answer 4

Here are instructions for Datadog version and Ubuntu Linux. The config vars have changed a bit:

Agent 7.35.2 - Commit: edd49be - Serialization version: v5.0.16 - Go version: go1.17.6

Check nano /etc/datadog-agent/datadog.yaml :

# Remember to enable this section
apt_config:
    apm_non_local_traffic: true
    dogstatsd_non_local_traffic: true

Then:

sudo systemctl restart datadog-agent

and:

sudo datadog-agent status