Cassandra 'Connection Refused' on Port 9042: Complete Troubleshooting Guide

Understanding Cassandra "Connection Refused" Errors

When a client receives Connection refused on port 9042, the TCP handshake itself is failing — Cassandra is not running, not bound to the expected address, or a firewall is dropping the SYN packet. This differs from authentication errors (TCP completes before rejecting) and read timeouts (connection succeeds but query is slow).

Common error messages:

com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed
  (tried: /10.0.0.1:9042 (TransportException: [/10.0.0.1:9042] Cannot connect))

cassandra.cluster.NoHostAvailable: ('Unable to connect to any servers',
  {'10.0.0.1': ConnectionRefusedError(111, 'Connection refused')})

Step 1: Verify Cassandra Service State

# Check service state
systemctl status cassandra --no-pager -l

# Confirm JVM process exists
ps aux | grep CassandraDaemon | grep -v grep

# Verify port 9042 is bound
ss -tlnp | grep 9042

If ss -tlnp | grep 9042 returns nothing, Cassandra is not listening. Read the last startup attempt before restarting:

tail -100 /var/log/cassandra/system.log | grep -iE "error|exception|fatal|killed"
journalctl -u cassandra -n 50 --no-pager

Step 2: Diagnose Out-of-Memory (OOM) Kills

OOM kills are the most frequent silent cause of Cassandra failures. The Linux kernel terminates the JVM without writing to Cassandra's logs.

# Kernel OOM evidence
dmesg | grep -iE "oom|killed process" | grep -i java

# GC pressure warnings
grep -E "GCInspector.*[0-9]{4,}ms" /var/log/cassandra/system.log | tail -20

A GC pause warning preceding self-halt:

WARN  [GCInspector] GCInspector.java:286 - G1 Young Generation GC in 11432ms.
G1 Eden Space: 8388608 -> 0; G1 Old Gen: 7516192768 -> 7814037504;

Pauses over 10 seconds trigger Cassandra's self-halt. Fix in /etc/cassandra/jvm.options (Java 8) or /etc/cassandra/jvm11-server.options (Java 11+):

# Rule: min(total_RAM / 4, 8GB). Never exceed 31GB.
# For a 32GB RAM server:
-Xms8G
-Xmx8G

Set -Xms equal to -Xmx to prevent resize pauses. Verify OS file descriptor limits:

ulimit -n  # Minimum 100000 for production Cassandra

Step 3: Check Native Transport Configuration

If the process is running but port 9042 is not listening:

grep -E "^(listen_address|rpc_address|broadcast_rpc_address|native_transport_port|start_native_transport)" \
  /etc/cassandra/cassandra.yaml

Correct production settings:

listen_address: 10.0.0.1         # Specific IP — never 0.0.0.0
rpc_address: 0.0.0.0             # Accept client connections on all interfaces
broadcast_rpc_address: 10.0.0.1  # IP advertised to clients
native_transport_port: 9042
start_native_transport: true

Critical mistake: listen_address: 0.0.0.0 is invalid and causes startup failure. Always use a specific IP for listen_address and pair rpc_address: 0.0.0.0 with broadcast_rpc_address.

Step 4: Firewall and Network Verification

# Test TCP reachability from the application server
nc -zv <cassandra_node_ip> 9042

# iptables (RHEL, Amazon Linux, older Ubuntu)
iptables -L INPUT -n -v --line-numbers | grep -E "9042|DROP|REJECT"

# nftables (Debian 12+, Ubuntu 22.04+)
nft list ruleset | grep -B2 -A2 9042

Required open ports:

Step 5: Slow Queries and Timeout Troubleshooting

If connections succeed but you see OperationTimedOutException or WriteTimeoutException, the problem is throughput, not connectivity.

# Check thread pool saturation — non-zero Dropped is critical
nodetool tpstats

# Examine latency percentiles
nodetool proxyhistograms

# Check compaction backlog
nodetool compactionstats

# Identify hot tables
nodetool tablestats <keyspace>.<table> | grep -E "Maximum|Mean|Local read|Local write"

Non-zero Dropped values in ReadStage, MutationStage, or ViewMutationStage mean Cassandra is shedding work. Enable slow query logging in cassandra.yaml:

slow_query_log_timeout_in_ms: 500

Then review:

grep "slow query" /var/log/cassandra/system.log | tail -20

Step 6: Data Corruption Diagnosis and Recovery

Commitlog corruption prevents startup with:

ERROR [main] CassandraDaemon.java:689 - Exception encountered during startup
org.apache.cassandra.io.FSReadError: java.io.IOException:
  Corrupt commitlog /var/lib/cassandra/commitlog/CommitLog-7-1639000000000.log

Commitlog recovery (risks loss of last few seconds of uncommitted writes):

systemctl stop cassandra
mv /var/lib/cassandra/commitlog /var/lib/cassandra/commitlog.bak.$(date +%s)
mkdir -p /var/lib/cassandra/commitlog
chown cassandra:cassandra /var/lib/cassandra/commitlog
systemctl start cassandra

For SSTable corruption discovered at runtime:

# Online scrub — drops corrupt rows, preserves valid data
nodetool scrub --skip-corrupted <keyspace> <table>

# Offline scrub for severe corruption
systemctl stop cassandra
sstablescrub --skip-corrupted <keyspace> <table>
systemctl start cassandra

# Restore replica consistency
nodetool repair -pr <keyspace>

Step 7: Validate Cluster Health

# All nodes should show UN (Up/Normal)
nodetool status

# Verify gossip propagation
nodetool gossipinfo | grep -E "STATUS|ENDPOINT_IP" | head -20

Healthy nodetool status output:

Datacenter: datacenter1
=======================
Status=Up/Down  |/ State=Normal/Leaving/Joining/Moving
--  Address      Load       Tokens  Owns  Host ID           Rack
UN  10.0.0.1    45.6 GiB   256     33.3% abc123-...-def456  rack1
UN  10.0.0.2    46.1 GiB   256     33.3% ghi789-...-jkl012  rack1
UN  10.0.0.3    44.9 GiB   256     33.4% mno345-...-pqr678  rack1

Any node showing DN (Down/Normal) is unreachable. Investigate with journalctl -u cassandra -n 200 --no-pager on that host immediately.