Elasticsearch API Timeout: Diagnosing and Fixing Request Timeout Errors

Understanding Elasticsearch API Timeout Errors

When a request to Elasticsearch exceeds the configured time limit, the client or server terminates the connection and surfaces one of several timeout-related errors. Understanding which layer is timing out is the critical first diagnostic step.

Common Error Messages

Depending on your client library and configuration, you will see one of these signatures:

Python elasticsearch-py client:

elasticsearch.exceptions.ConnectionTimeout: ConnectionTimeout caused by - ReadTimeoutError(HTTPConnectionPool(host='localhost', port=9200): Read timed out. (read timeout=30))

Java High-Level REST Client:

java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-0 [ACTIVE]

Kibana / curl:

{"error":{"root_cause":[{"type":"search_phase_execution_exception","reason":"all shards failed"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"my-index","node":"abc123","reason":{"type":"query_shard_exception","reason":"Request timed out"}}]}}

Server-side timeout (search timeout):

{"timed_out":true,"_shards":{"total":5,"successful":3,"skipped":0,"failed":2}}

Note the difference: a client timeout throws an exception before any response arrives. A server-side timeout returns a 200 response with "timed_out": true and partial results.

Step 1: Diagnose — Identify the Timeout Layer

Run these commands in order to narrow down the root cause.

1a. Check cluster health first

curl -s 'http://localhost:9200/_cluster/health?pretty'

A red or yellow status means shards are unassigned. Primary shard unavailability causes all writes and some reads to block until timeout. If status is red, fix this before anything else.

1b. Inspect thread pool saturation

curl -s 'http://localhost:9200/_cat/thread_pool/search,write,bulk?v&h=node_name,name,active,queue,rejected,completed'

If queue is consistently > 0 or rejected is climbing, your nodes are overwhelmed. Requests are queuing past the client timeout window.

1c. Find slow queries via the slow log

Enable the slow log temporarily:

curl -X PUT 'http://localhost:9200/my-index/_settings' -H 'Content-Type: application/json' -d '{
  "index.search.slowlog.threshold.query.warn": "2s",
  "index.search.slowlog.threshold.fetch.warn": "1s",
  "index.search.slowlog.level": "warn"
}'

Then tail the Elasticsearch log:

tail -f /var/log/elasticsearch/my-cluster_index_search_slowlog.log

1d. Check pending tasks

curl -s 'http://localhost:9200/_cluster/pending_tasks?pretty'

A large backlog of pending cluster tasks (e.g., shard assignment, mapping updates) can delay query execution.

1e. Check JVM heap pressure

curl -s 'http://localhost:9200/_nodes/stats/jvm?pretty' | python3 -c "
import json, sys
nodes = json.load(sys.stdin)['nodes']
for nid, n in nodes.items():
    heap = n['jvm']['mem']
    used_pct = heap['heap_used_percent']
    print(f"{n['name']}: heap {used_pct}%")
"

Heap usage consistently above 75% triggers frequent GC pauses, which directly cause timeouts.

Step 2: Fix — Apply the Right Remedy

Fix A: Adjust the Client-Side Timeout

This is the correct fix when your operation (bulk indexing, reindex, aggregation over large datasets) is legitimately long-running and valid.

Python:

from elasticsearch import Elasticsearch

es = Elasticsearch(
    ['http://localhost:9200'],
    request_timeout=120  # seconds
)

# Per-request override
result = es.search(index='my-index', body=query, request_timeout=60)

Node.js (@elastic/elasticsearch):

const { Client } = require('@elastic/elasticsearch')
const client = new Client({
  node: 'http://localhost:9200',
  requestTimeout: 120000  // milliseconds
})

curl:

curl --max-time 120 -X GET 'http://localhost:9200/my-index/_search' -H 'Content-Type: application/json' -d '{"query":{"match_all":{}}}'

Fix B: Set a Cluster-Wide Search Timeout

This prevents runaway queries from tying up resources. It returns partial results rather than blocking forever.

curl -X PUT 'http://localhost:9200/_cluster/settings' -H 'Content-Type: application/json' -d '{
  "persistent": {
    "search.default_search_timeout": "30s"
  }
}'

Or pass timeout per request at query time:

curl -X GET 'http://localhost:9200/my-index/_search?timeout=10s' -H 'Content-Type: application/json' -d '{"query":{"match_all":{}}}'

Fix C: Fix Unassigned Shards (Red Cluster)

# Find unassigned shards and their reason
curl -s 'http://localhost:9200/_cluster/allocation/explain?pretty'

# Retry failed shard allocation
curl -X POST 'http://localhost:9200/_cluster/reroute?retry_failed=true'

# If a node was permanently removed and you accept data loss on that shard:
curl -X POST 'http://localhost:9200/_cluster/reroute' -H 'Content-Type: application/json' -d '{
  "commands": [{
    "allocate_stale_primary": {
      "index": "my-index",
      "shard": 0,
      "node": "node-1",
      "accept_data_loss": true
    }
  }]
}'

Fix D: Relieve Thread Pool Pressure

For sustained search thread pool saturation, tune the thread pool size (requires node restart):

# elasticsearch.yml
thread_pool:
  search:
    size: 13          # default: (vCPUs * 3) / 2 + 1
    queue_size: 1000  # default: 1000

Alternatively, reduce concurrent client connections or implement request rate limiting upstream (NGINX, API gateway).

Fix E: Optimize the Query

Replace slow wildcard / fuzzy / leading-wildcard patterns with match, prefix on keyword fields, or use ngram tokenizers:

# Bad — leading wildcard scans all terms:
{"query": {"wildcard": {"title": "*cloud*"}}}

# Good — use match on analyzed field or prefix on keyword:
{"query": {"match": {"title": "cloud"}}}

Use filter context for non-scoring clauses (enables caching):

{
  "query": {
    "bool": {
      "filter": [
        {"term": {"status": "active"}},
        {"range": {"created_at": {"gte": "now-7d"}}}
      ],
      "must": [
        {"match": {"description": "kubernetes"}}
      ]
    }
  }
}

Fix F: Reduce Shard Count for Over-Sharded Indices

Over-sharding (too many small shards) causes excessive coordination overhead. The recommended shard size is 10–50 GB.

# Check current shard sizes
curl -s 'http://localhost:9200/_cat/shards?v&h=index,shard,prirep,state,docs,store,node' | sort -k6 -h

# Shrink an over-sharded index (must be read-only, all shards on one node)
curl -X PUT 'http://localhost:9200/my-index/_settings' -H 'Content-Type: application/json' -d '{
  "settings": {
    "index.routing.allocation.require._name": "node-1",
    "index.blocks.write": true
  }
}'

curl -X POST 'http://localhost:9200/my-index/_shrink/my-index-shrunk' -H 'Content-Type: application/json' -d '{
  "settings": {
    "index.number_of_shards": 1,
    "index.number_of_replicas": 1,
    "index.routing.allocation.require._name": null
  }
}'

Step 3: Verify the Fix

After applying your fix, confirm the improvement:

# Confirm cluster is green
curl -s 'http://localhost:9200/_cluster/health?wait_for_status=green&timeout=30s&pretty'

# Check response time of your query
time curl -s -X GET 'http://localhost:9200/my-index/_search' -H 'Content-Type: application/json' -d '{"query":{"match_all":{}},"size":1}' | python3 -m json.tool | grep took

# Confirm timed_out is false
curl -s 'http://localhost:9200/my-index/_search?timeout=10s' -H 'Content-Type: application/json' -d '{"query":{"match_all":{}}}' | python3 -c "import json,sys; r=json.load(sys.stdin); print('timed_out:', r.get('timed_out'))"