Troubleshooting 'Datadog Not Working': Fixing Agent Offline, Missing Metrics, and APM Connection Errors

Understanding the Error: Why is Datadog Not Working?

When engineers report that 'Datadog is not working,' the symptom can manifest in several different ways. The Datadog ecosystem is vast, relying on a localized Agent (a Go-based daemon) that collects, aggregates, and forwards metrics, logs, and traces to Datadog's cloud infrastructure. A failure in any part of this pipeline can lead to missing observability data.

The most common manifestations of this problem include:

1. The Host is Offline: The infrastructure list in the Datadog UI shows the host as '?' or 'Offline'.
2. Missing Infrastructure Metrics: CPU, memory, and disk usage stop updating.
3. Missing Integration Data: A specific service (like PostgreSQL, Redis, or NGINX) stops reporting metrics, while base host metrics continue.
4. Missing APM Traces: Distributed tracing data is absent, often accompanied by connection errors in the application logs.
5. Log Collection Failures: Application or system logs are not appearing in the Log Explorer.

Step 1: Diagnose the Datadog Agent

The absolute first step in troubleshooting any Datadog issue is to query the Agent's status. The Agent comes with a built-in CLI that performs a comprehensive health check.

Run the following command on the affected Linux host:

sudo datadog-agent status

This command outputs a wealth of information. You need to look for specific sections:

1. The Forwarder Section

Look at the 'Forwarder' section to ensure payloads are successfully reaching Datadog. If you see an error like:

Error: error connecting to the agent: dial tcp 127.0.0.1:5001: connect: connection refused

This indicates the core Agent process is not running. Check if the service is active using systemctl status datadog-agent.

If you see errors related to the API, such as:

[ERROR] Error starting the agent: no API key configured or WARN | Forwarder | Error while sending payload to https://api.datadoghq.com/api/v1/series: 403 Forbidden - Invalid API Key

This strictly points to an authentication issue. Your datadog.yaml file either lacks an API key, has a typo, or the key has been revoked in the Datadog UI.

2. The Collector Section

If specific integrations are failing, check the 'Collector' section. It lists every integration (check) configured. If an integration is misconfigured, you will see a traceback or an error like:

[ERROR] postgres: Unresolved missing requirement or Check 'nginx' failed: dial tcp 127.0.0.1:80: connect: connection refused

This indicates the Agent is running perfectly, but it cannot access the service it is trying to monitor. This is usually due to incorrect credentials in the conf.d/ YAML file, or the target service being down.

Step 2: Fix Network and Connectivity Issues

The Datadog Agent is a push-based system. It needs outbound network access to Datadog's infrastructure. If the Agent is running but no data appears, it is almost always a network issue.

Look at the main Agent log file: /var/log/datadog/agent.log. If you see:

WARN | Forwarder | Error while sending payload: dial tcp: i/o timeout

This means the Agent's outbound traffic is being blocked by a firewall, Security Group (AWS), or network proxy.

How to fix:

1. Verify your Datadog Site: Datadog has multiple regions (US1, US3, US5, EU1, US1-FED). If your account is in EU1, but your Agent is sending data to US1 (the default), it will fail. Ensure site: datadoghq.eu (or your appropriate site) is set in /etc/datadog-agent/datadog.yaml.
2. Check Firewall Rules: The Agent requires outbound access on TCP port 443 to Datadog's domains. You can test this manually using cURL:

curl -v https://api.datadoghq.com/api/v1/validate?api_key=YOUR_API_KEY

If this command hangs, you have a firewall issue. If you use a proxy, ensure the Agent is configured to use it by setting the proxy block in datadog.yaml.

Step 3: Fix APM (Trace Agent) Failures

If host metrics are working but APM traces are missing, the issue lies with the Trace Agent. The Datadog APM architecture requires your application (instrumented with a Datadog library, e.g., dd-trace-js or dd-trace-py) to send traces to the Agent over localhost on port 8126.

Common APM Errors: Application logs showing: Error: connect ECONNREFUSED 127.0.0.1:8126.

How to fix:

1. Enable APM in the Agent: By default, APM is often enabled, but it can be explicitly disabled. Ensure your datadog.yaml has: apm_config: enabled: true
2. Verify the Port: Ensure the Trace Agent is listening. Run sudo netstat -tlnp | grep 8126 or sudo ss -tlnp | grep 8126. You should see the trace-agent process bound to 0.0.0.0:8126 or 127.0.0.1:8126.
3. Containerized Environments: If your application is in a Docker container and the Agent is on the host (or in another container), 127.0.0.1 will resolve to the application's container, NOT the host. You must configure your tracer to send data to the host IP. In Docker, you can pass the host IP via an environment variable or use Datadog's Unix Domain Socket (UDS) feature for APM, which is highly recommended for Kubernetes and Docker.

Step 4: Fix Log Collection Issues

Log collection is disabled by default in the Datadog Agent. If logs are not working:

1. Enable Logs: Ensure logs_enabled: true is present in datadog.yaml.
2. Check Permissions: The Agent runs as the dd-agent user. If you configure it to read /var/log/nginx/access.log, the dd-agent user MUST have read permissions for that file and execute permissions for the directory. If it doesn't, you will see a 'Permission denied' error in agent.log. Fix: Add the dd-agent user to the appropriate group (e.g., adm or nginx), or adjust file ACLs.

Step 5: Advanced Diagnostics and The Flare

When you have verified the network, checked the configuration, and restarted the Agent (sudo systemctl restart datadog-agent), but Datadog is still not working, it's time to investigate resource constraints or contact support.

1. Check OOM Kills: If the Agent process suddenly disappears, it might have been killed by the Linux Out-Of-Memory (OOM) killer. Run dmesg -T | grep -i oom or grep -i oom /var/log/syslog. If you see the agent process listed, you need to increase the memory limit on the machine/container, or adjust the Agent's configuration to use less memory (e.g., lower the log collection rate or disable unused integrations).

2. Generate a Flare: Datadog Support will require a 'Flare'. This is a comprehensive archive of all your Agent logs, configuration files (with secrets scrubbed), and diagnostic commands. Generate it by running:

sudo datadog-agent flare

The command will ask for a ticket number (you can leave it blank) and an email address. It will securely upload the diagnostics directly to Datadog's support team, allowing them to see exactly why the Agent is failing.