Prometheus Connection Refused: Complete Troubleshooting Guide (CrashLoopBackOff, OOM, Permission Denied)

Understanding Prometheus Connection Errors

Prometheus exposes an HTTP API and UI on port 9090 by default. A connection refused error means the TCP handshake never completed — the kernel sent back a RST packet because nothing was listening on that port. This is distinct from a timeout (no response at all) and from a 403/401 (process is up but rejects the request).

Common exact errors you will see:

Get "http://prometheus:9090/api/v1/query": dial tcp 10.96.0.1:9090: connect: connection refused
ts=2024-01-15T10:23:45Z level=error msg="Opening storage failed" err="open /prometheus/queries.active: permission denied"
OOMKilled
Back-off restarting failed container prometheus in pod prometheus-0

Step 1: Determine Whether Prometheus Is Running

Kubernetes environments:

kubectl get pods -n monitoring -l app=prometheus
kubectl describe pod prometheus-0 -n monitoring   # look at Events and Last State
kubectl logs prometheus-0 -n monitoring --previous  # logs from crashed container

Look for these fields in kubectl describe pod:

• Last State: Terminated Reason: OOMKilled → memory problem
• Last State: Terminated Reason: Error with exit code 1 or 2 → config or permission error
• Restart Count: N where N > 3 → CrashLoopBackOff pattern

Bare-metal / VM environments:

systemctl status prometheus
journalctl -u prometheus -n 100 --no-pager
ps aux | grep prometheus

Step 2: Diagnose Connection Refused

Once you confirm the process state, narrow down the cause:

# Is Prometheus actually listening on port 9090?
ss -tlnp | grep 9090
# or on older systems:
netstat -tlnp | grep 9090

# Can you reach it locally?
curl -v http://localhost:9090/-/healthy

# In Kubernetes — check Service selector matches pod labels:
kubectl get svc prometheus -n monitoring -o yaml
kubectl get endpoints prometheus -n monitoring
# If Endpoints shows <none>, the Service selector is wrong

If ss shows nothing on port 9090 but the process is running, check the --web.listen-address flag:

kubectl exec -it prometheus-0 -n monitoring -- /bin/prometheus --help 2>&1 | grep listen
# Then check actual flags the process was started with:
kubectl exec -it prometheus-0 -n monitoring -- cat /proc/1/cmdline | tr '\0' ' '

The flag must be --web.listen-address=0.0.0.0:9090 (or :[port]) for the service to be reachable from other pods. If it is 127.0.0.1:9090, only loopback traffic works.

Step 3: Fix CrashLoopBackOff

CrashLoopBackOff is not a root cause — it is Kubernetes backing off restarts of a container that keeps failing. Identify why it crashes:

# Get the last 200 lines from the crashed container
kubectl logs prometheus-0 -n monitoring --previous --tail=200

Scenario A — Bad configuration file:

ts=2024-01-15T10:23:45Z level=error msg="Error loading config" file=/etc/prometheus/prometheus.yml err="yaml: line 42: mapping values are not allowed in this context"

Validate the config before applying:

promtool check config /etc/prometheus/prometheus.yml

In Kubernetes, the ConfigMap is often the source. Edit it with kubectl edit configmap prometheus-config -n monitoring and look for YAML indentation errors.

Scenario B — Storage corruption:

level=error msg="Failed to open db" err="unexpected end of JSON input"

This requires removing the corrupted WAL block:

# List TSDB blocks
ls -lah /prometheus/
# Remove WAL if corrupted (data loss for in-flight samples only)
rm -rf /prometheus/wal
# Then restart Prometheus

Step 4: Fix OOM Killed

kubectl describe pod prometheus-0 -n monitoring | grep -A5 "Last State"
# Last State: Terminated
#   Reason: OOMKilled
#   Exit Code: 137

Exit code 137 = 128 + 9 (SIGKILL). The Linux OOM killer terminated the process.

Option 1 — Raise the memory limit (fast):

# In your Deployment or StatefulSet spec:
resources:
  requests:
    memory: "2Gi"
  limits:
    memory: "4Gi"

Apply and wait for rollout: kubectl rollout status statefulset/prometheus -n monitoring

Option 2 — Reduce ingestion cardinality (sustainable):

High cardinality (millions of unique time series) is the most common cause of Prometheus OOM. Use the built-in TSDB status endpoint to find offenders:

curl http://localhost:9090/api/v1/status/tsdb | python3 -m json.tool | head -80

Look at headStats.numSeries. If it exceeds 1–2 million on a single Prometheus instance with 4 GiB RAM, you have a cardinality problem.

Drop high-cardinality labels with metric_relabel_configs:

# prometheus.yml scrape config
scrape_configs:
  - job_name: 'my-app'
    metric_relabel_configs:
      - source_labels: [request_id]   # drop unique per-request labels
        regex: '.*'
        action: labeldrop
      - source_labels: [__name__]
        regex: 'go_gc_.*'             # drop noisy Go runtime metrics
        action: drop

Option 3 — Tune retention:

# Limit retention by size instead of time
--storage.tsdb.retention.size=10GB
--storage.tsdb.retention.time=15d

Step 5: Fix Permission Denied

Prometheus runs as UID 65534 (nobody) by default. If the /prometheus data directory was created by root or another user, Prometheus cannot write to it.

ts=2024-01-15T10:23:45Z level=error caller=main.go:174 msg="Opening storage failed" err="open /prometheus/queries.active: permission denied"

Fix on bare metal:

chown -R 65534:65534 /var/lib/prometheus
# or if running as a dedicated user:
chown -R prometheus:prometheus /var/lib/prometheus
systemctl restart prometheus

Fix in Kubernetes (preferred — use securityContext):

spec:
  securityContext:
    runAsUser: 65534
    runAsNonRoot: true
    fsGroup: 65534      # ensures mounted volumes are chowned to this GID
  containers:
    - name: prometheus
      # ...

The fsGroup field is the key — Kubernetes will chown the volume mount point to that GID on pod startup, so Prometheus can write to it without running as root.

Step 6: Fix Scrape Timeouts

If Prometheus is running but you see timeout errors in the UI or logs:

level=warn msg="Scrape failed" scrape_url="http://my-app:8080/metrics" err="context deadline exceeded"

Increase the per-job scrape timeout in prometheus.yml:

global:
  scrape_timeout: 10s   # default is 10s; increase if targets are slow

scrape_configs:
  - job_name: 'slow-app'
    scrape_timeout: 30s  # job-level override
    scrape_interval: 60s

Note: scrape_timeout must always be less than or equal to scrape_interval.

Step 7: Check NetworkPolicies and Firewalls

In hardened Kubernetes clusters, NetworkPolicies can silently block traffic:

kubectl get networkpolicies -n monitoring
# Test connectivity from another pod:
kubectl run -it --rm debug --image=curlimages/curl --restart=Never -n default -- \
  curl -v http://prometheus.monitoring.svc.cluster.local:9090/-/healthy

If the curl pod gets connection refused but Prometheus is running, check if a NetworkPolicy is blocking ingress to port 9090 from the default namespace.