Elasticsearch API Timeout: Diagnosing and Fixing ReadTimeoutError, SocketTimeoutException, and Request Timeout Errors

Understanding Elasticsearch API Timeouts

An Elasticsearch API timeout surfaces differently depending on where in the request chain the clock runs out. Developers typically encounter one of these exact error strings:

readtimeouterror: httpsconnectionpool(host='es-host', port=9200): read timed out. (read timeout=10)
SocketTimeoutException: 30,000 milliseconds timeout on connection http://es-host:9200
ConnectionTimeout: ConnectionTimeout caused by - ReadTimeoutError
[408 Request Timeout] or [504 Gateway Timeout] from a proxy
transportexception[failed to get node response]

The timeout can originate at three distinct layers: the client library (Python elasticsearch-py, Java High-Level REST Client, etc.), the network/proxy layer (ALB, NGINX, HAProxy), or inside Elasticsearch itself (search timeout, bulk timeout). Conflating these layers is the single most common debugging mistake.

Step 1: Identify Which Layer Is Timing Out

Check cluster health and slow-log first:

# Cluster health — are you yellow/red?
curl -s 'http://localhost:9200/_cluster/health?pretty'

# Node stats — check thread pool rejection counters
curl -s 'http://localhost:9200/_nodes/stats/thread_pool?pretty' | \
  python3 -c "import sys,json; d=json.load(sys.stdin); \
  [print(n, tp, v) for n,nd in d['nodes'].items() \
  for tp,tv in nd['thread_pool'].items() \
  for k,v in tv.items() if k in ('rejected','queue') and v>0]"

# Pending tasks
curl -s 'http://localhost:9200/_cluster/pending_tasks?pretty'

# Hot threads — what is the JVM actually doing?
curl -s 'http://localhost:9200/_nodes/hot_threads'

Enable slow logs on the problematic index:

curl -X PUT 'http://localhost:9200/my-index/_settings' -H 'Content-Type: application/json' -d '{
  "index.search.slowlog.threshold.query.warn": "5s",
  "index.search.slowlog.threshold.query.info": "2s",
  "index.search.slowlog.threshold.fetch.warn": "1s",
  "index.indexing.slowlog.threshold.index.warn": "5s"
}'

Then tail /var/log/elasticsearch/*_index_search_slowlog.log on each data node.

Check if a load balancer is killing the connection:

AWS ALB default idle timeout is 60 seconds. If your queries consistently time out at exactly 60s from the client side but Elasticsearch is still running the query, the ALB is dropping the TCP connection. Verify:

# Time an actual request end-to-end
time curl -s -w "\n\nHTTP %{http_code} | Total: %{time_total}s\n" \
  'http://localhost:9200/my-index/_search?pretty' \
  -H 'Content-Type: application/json' \
  -d '{"query":{"match_all":{}},"size":1}'

# Compare against direct node access (bypassing LB)
time curl -s 'http://ES-NODE-IP:9200/my-index/_search' \
  -H 'Content-Type: application/json' \
  -d '{"query":{"match_all":{}},"size":1}'

Step 2: Diagnose JVM GC Pauses

GC stop-the-world events pause the entire JVM, making every in-flight request appear to time out simultaneously. This is a classic pattern: you see a burst of timeouts across multiple unrelated queries at the exact same second.

# Look for GC log evidence
grep -E 'GC overhead|Stopping world|pause' /var/log/elasticsearch/gc.log | tail -50

# Node JVM stats
curl -s 'http://localhost:9200/_nodes/stats/jvm?pretty' | \
  python3 -c "
import sys, json
d = json.load(sys.stdin)
for nid, n in d['nodes'].items():
    jvm = n['jvm']
    print(n['name'])
    print('  heap_used_percent:', jvm['mem']['heap_used_percent'])
    print('  old_gc_count:', jvm['gc']['collectors']['old']['collection_count'])
    print('  old_gc_time_ms:', jvm['gc']['collectors']['old']['collection_time_in_millis'])
"

If heap_used_percent is consistently above 75% or old_gc_time_in_millis is growing rapidly, you have a heap pressure problem.

Fix for JVM heap pressure:

1. Set heap to no more than 50% of RAM, max 31GB (stays below compressed OOP threshold):

   # In /etc/elasticsearch/jvm.options
   -Xms16g
   -Xmx16g
   

2. Switch to G1GC if on JDK 9+ (default since ES 7.x):

   -XX:+UseG1GC
   -XX:G1HeapRegionSize=4m
   -XX:InitiatingHeapOccupancyPercent=30
   

3. Identify field data cache bloat:

   curl -s 'http://localhost:9200/_nodes/stats/indices/fielddata?pretty' | \
     grep -E 'memory_size|evictions'
   

Step 3: Diagnose Query-Level Timeouts

A query scanning too many shards or too many documents will exhaust the search thread pool, causing queuing and eventual timeout.

# How many shards does this index have?
curl -s 'http://localhost:9200/_cat/shards/my-index?v&h=index,shard,prirep,state,docs,store,node'

# How expensive is the query? Use profile API
curl -X POST 'http://localhost:9200/my-index/_search?pretty' \
  -H 'Content-Type: application/json' -d '{
  "profile": true,
  "query": { "YOUR_QUERY_HERE": {} }
}'

# Explain a specific document match
curl -X GET 'http://localhost:9200/my-index/_explain/DOC_ID?pretty' \
  -H 'Content-Type: application/json' -d '{
  "query": { "YOUR_QUERY_HERE": {} }
}'

Common query fixes:

• Replace wildcard: {"field": "*term*"} with an ngram analyzer at index time
• Move date range filters into filter context (cached) instead of query context (scored)
• Add routing parameter to limit shard fan-out: POST /my-index/_search?routing=tenant_id
• Set a circuit-breaker timeout at query level:

  { "timeout": "30s", "query": { ... } }
  

Set a cluster-wide search timeout as a safety net:

curl -X PUT 'http://localhost:9200/_cluster/settings' \
  -H 'Content-Type: application/json' -d '{
  "persistent": {
    "search.default_search_timeout": "30s"
  }
}'

Step 4: Fix Client-Side Timeout Configuration

Python (elasticsearch-py 8.x):

from elasticsearch import Elasticsearch

es = Elasticsearch(
    ["https://es-host:9200"],
    request_timeout=60,       # seconds before ReadTimeoutError
    retry_on_timeout=True,
    max_retries=3,
    http_compress=True,
)

Node.js (@elastic/elasticsearch):

const { Client } = require('@elastic/elasticsearch')
const client = new Client({
  node: 'https://es-host:9200',
  requestTimeout: 60000,      // milliseconds
  sniffOnStart: false,        // avoid sniff timeouts in prod
})

Java (RestHighLevelClient / 8.x JavaClient):

RestClientBuilder builder = RestClient.builder(
    new HttpHost("es-host", 9200, "https"))
  .setRequestConfigCallback(config -> config
    .setConnectTimeout(5000)
    .setSocketTimeout(60000))   // ms
  .setHttpClientConfigCallback(httpClient -> httpClient
    .setKeepAliveStrategy((response, context) -> 60_000));

Step 5: Fix Network / Proxy Timeouts

AWS ALB: Set idle timeout to 120s+ in the console or:

aws elbv2 modify-load-balancer-attributes \
  --load-balancer-arn arn:aws:elasticloadbalancing:... \
  --attributes Key=idle_timeout.timeout_seconds,Value=120

NGINX upstream:

upstream elasticsearch {
    server es-host:9200;
    keepalive 32;
}
server {
    location / {
        proxy_pass http://elasticsearch;
        proxy_read_timeout 120s;
        proxy_send_timeout 120s;
        proxy_connect_timeout 10s;
    }
}

HAProxy:

timeout connect 10s
timeout client  120s
timeout server  120s

Step 6: Validate the Fix

# Watch thread pool rejection counters in real time
watch -n 5 "curl -s 'http://localhost:9200/_cat/thread_pool?v&h=name,active,queue,rejected&s=rejected:desc' | head -15"

# Confirm GC pressure reduced
curl -s 'http://localhost:9200/_nodes/stats/jvm' | \
  python3 -c "import sys,json; d=json.load(sys.stdin); \
  [print(n['name'], n['jvm']['mem']['heap_used_percent'], '%') \
  for n in d['nodes'].values()]"

# Run a representative query with timing
for i in $(seq 1 10); do
  curl -s -w "%{time_total}\n" -o /dev/null \
    -X POST 'http://localhost:9200/my-index/_search' \
    -H 'Content-Type: application/json' \
    -d '{"query":{"match_all":{}},"size":10}'
done

If median response times drop below your SLA threshold and thread pool rejections return to zero, the fix is confirmed.