Resolving HAProxy Connection Refused and Performance Issues

Understanding the 'Connection Refused' Error in HAProxy

HAProxy is a high-performance, open-source load balancer and reverse proxy for TCP and HTTP-based applications. When appropriately configured, it effortlessly handles millions of concurrent connections. However, when users begin reporting Connection refused errors, or notice that HAProxy is returning 503 Service Unavailable or simply responding extremely slowly, it can quickly become a critical incident.

When you encounter a Connection refused error while accessing a service routed through HAProxy, it generally indicates a failure at the transport layer (TCP). Users typically see errors like curl: (7) Failed to connect to example.com port 80: Connection refused on the command line, or a browser displaying 503 Service Unavailable - No server is available to handle this request.

This guide provides a comprehensive, step-by-step approach to diagnosing and resolving these issues, exploring root causes ranging from basic configuration mistakes and backend outages to operating system resource exhaustion and stringent security policies.

Step 1: Diagnose the Symptom - Front-End vs. Back-End

Before applying fixes, you must determine the source of the refusal. Is HAProxy itself refusing connections from clients, or is HAProxy successfully accepting client connections but failing to connect to the backend servers, thus returning a 503 error?

1. Validate the HAProxy Process and Bind States

First, ensure HAProxy is actually running and successfully listening on the expected frontend ports. If HAProxy failed to start, the client will get a Connection refused directly from the OS network stack.

Execute the following commands to check the service status and listening ports:

sudo systemctl status haproxy
sudo ss -tulpn | grep haproxy

If HAProxy is not running or is in a failed state, test the configuration file for syntax or logic errors:

sudo haproxy -c -f /etc/haproxy/haproxy.cfg

A common reason for HAProxy failing to start is a port conflict (another process like Apache or Nginx is already bound to port 80/443), or a lack of privileges to bind to privileged ports (ports under 1024).

2. Analyze HAProxy Logs and Termination States

HAProxy logs provide the most accurate narrative of what happened to a request. By default, HAProxy logs to syslog, often found in /var/log/haproxy.log or /var/log/syslog.

Look specifically at the HTTP status codes and the session state flags (a two-character code).

grep -E "SC--|sC--|503" /var/log/haproxy.log

Understanding Termination States:

• SC--: The server or an equipment between it and HAProxy explicitly refused the TCP connection. The 'S' indicates the connection was actively being established to the server, and 'C' indicates the client unexpectedly closed the connection or the backend reset it.
• sC--: The server or an equipment between it and HAProxy explicitly refused the connection before the session was fully established.
• SD--: The connection to the server died with an error during the data transfer phase.

If you see 503 errors and SC--, HAProxy is talking to clients just fine, but the backends are unreachable.

3. Inspect Backend Health via Stats Socket

The HAProxy Runtime API (Stats Socket) allows real-time inspection of backend health. If you haven't enabled it, add this to your global section:

global
    stats socket /run/haproxy/admin.sock mode 660 level admin

Query the socket using socat:

echo "show stat" | socat unix-connect:/run/haproxy/admin.sock stdio | cut -d, -f1,2,18

Look for backends marked as DOWN. The check status column will reveal why:

• L4CON: Layer 4 Connection refused. The backend server is not listening on the configured port, or a firewall is blocking the SYN packet.
• L4TOUT: Layer 4 Timeout. The backend is blackholing traffic (often a firewall dropping packets instead of rejecting them).
• L7RSP: Layer 7 Response error. The backend responded, but with an unexpected HTTP status code (e.g., 500 instead of 200).

Step 2: Resolve Backend Availability and Routing Issues

If HAProxy is returning 503 Service Unavailable, no backend servers are healthy in the target pool. You must fix the communication between HAProxy and the application servers.

1. Verify Backend Connectivity from the Load Balancer

Log into the HAProxy server and attempt to manually connect to the backend server's IP and port using the exact same address defined in your haproxy.cfg.

curl -v -I http://<backend_ip>:<backend_port>
telnet <backend_ip> <backend_port>
nc -zv <backend_ip> <backend_port>

If nc or telnet returns Connection refused, the issue lies outside HAProxy. Check the following on the backend server:

• Is the application daemon (Node.js, Tomcat, Nginx, PHP-FPM) running?
• Is it bound to 127.0.0.1 instead of 0.0.0.0 or the specific network interface IP? If it's bound to localhost, external connections from HAProxy will be refused.

2. Check Network Firewalls and Routing

If the application is listening correctly on the backend host, check the network path.

• Local Firewalls: Check iptables, ufw, or firewalld on the backend node to ensure traffic from the HAProxy IP is permitted.
• Cloud Security Groups: In AWS, GCP, or Azure, verify that the Security Group or Network Security Group attached to the backend instances allows ingress TCP traffic on the application port from the HAProxy instance's Security Group.
• Asymmetric Routing: Ensure the return traffic from the backend server routes back through HAProxy properly, especially in complex VPC topologies.

Step 3: Fix Connection Limits and Slow Performance

If users complain that HAProxy is slow, timing out, or intermittently dropping connections under load, the system may be starved of resources or hitting configured ceilings.

1. HAProxy Maxconn Limits

HAProxy enforces connection limits at the global, frontend, and server levels. If the number of concurrent connections exceeds the frontend limit, new connections are queued or refused. If the global limit is hit, HAProxy stops accepting new sockets entirely.

Review your configuration limits:

global
    maxconn 100000

defaults
    maxconn 50000

Crucial Note: Increasing maxconn requires the operating system to allow HAProxy to open enough file descriptors (one socket = one file descriptor). Ensure system limits are raised. In systemd, edit the service file:

[Service]
LimitNOFILE=200000

2. Ephemeral Port Exhaustion

When HAProxy acts as a reverse proxy, it establishes a new TCP connection to the backend for every client request (unless HTTP keep-alive and connection pooling are strictly configured). Each outgoing connection consumes an ephemeral port on the HAProxy server. By default, Linux has about 28,000 ephemeral ports. If HAProxy opens connections faster than they close (and exit the TIME_WAIT state), you will run out of ports, leading to Cannot assign requested address or connection timeouts.

Tune your Linux sysctl parameters to expand the local port range and allow aggressive TCP connection reuse:

# Edit /etc/sysctl.d/99-haproxy.conf
net.ipv4.ip_local_port_range = 1024 65535
net.ipv4.tcp_tw_reuse = 1
net.core.somaxconn = 65535
net.ipv4.tcp_max_syn_backlog = 65535

Apply the changes:

sudo sysctl -p /etc/sysctl.d/99-haproxy.conf

Step 4: Address OS Security Policies (SELinux)

On enterprise Linux distributions like RHEL, CentOS, AlmaLinux, and Rocky Linux, Security-Enhanced Linux (SELinux) is a frequent and often silent culprit behind Connection refused and Permission denied errors.

SELinux enforces mandatory access controls. By default, the haproxy_t domain is heavily restricted. It is only allowed to bind to specific well-known ports, and it is strictly forbidden from initiating outbound connections to arbitrary ports.

1. Permitting Outbound Connections to Backends

If HAProxy is trying to connect to a backend application on a non-standard port (e.g., 8080, 8443, 9000), SELinux will block the connect() system call. HAProxy logs will show the backend as DOWN, but manual curl tests from the command line (running as unconfined_t) will succeed, causing extreme confusion.

To fix this, toggle the SELinux boolean that allows HAProxy to connect to any network port:

sudo setsebool -P haproxy_connect_any 1

2. Permitting Custom Frontend Bind Ports

If HAProxy fails to start because it cannot bind to a frontend port, and no other process is using it, SELinux is likely blocking the bind. You must label the custom port so HAProxy can use it.

Check existing labels for HTTP ports:

sudo semanage port -l | grep http_port_t

Add your custom port (e.g., 8888) to the allowed HTTP ports policy:

sudo semanage port -a -t http_port_t -p tcp 8888
sudo systemctl restart haproxy

Conclusion

Troubleshooting HAProxy requires a methodical approach. Do not guess. First, establish whether the refusal is happening at the HAProxy frontend (service down, port conflict, maxconn limits) or the backend (application down, firewall blocked, SELinux restricted). By leveraging the HAProxy configuration check, tailing syslog for connection termination states, querying the real-time stats socket, and verifying Linux network stack configurations, SREs can rapidly isolate the root cause and restore service stability.