Elasticsearch

User preferences and memory live in ~/Clawic/data/elasticsearch/ (see setup.md on first use, memory-template.md for the file format). If you have data at an old location (~/elasticsearch/ or ~/clawic/elasticsearch/), move it to ~/Clawic/data/elasticsearch/.

When To Use

• Writing or reviewing a query DSL body, an index mapping, an analyzer, or an aggregation
• Search is wrong: no hits, wrong documents, wrong order, wrong buckets, or duplicates
• The cluster is unhealthy: yellow or red, unassigned shards, read-only indices, breakers tripping, heap pressure
• Indexing or search is slow, or bulk requests fail, stall, or get rejected
• Designing the index layout: shard counts, aliases, templates, data streams, ILM, multi-tenancy, reindex plans
• Operating it: snapshots and restores, security and API keys, monitoring, rolling upgrades, migration to or from OpenSearch
• Not for choosing a standalone vector database, or for lightweight site search where a cluster is more than the job needs (see Related Skills)

Quick Reference

Depth on demand, by phase:

• Diagnose — debugging.md why a document does not match or does not rank · errors.md error string to cause · incidents.md red cluster, read-only indices, breakers, heap · monitoring.md what to watch and alert on
• Design — mapping.md field types and mapping mechanics · analyzers.md the analysis chain, synonyms, language handling · data-modeling.md nested, parent-child, multi-tenancy, index granularity · queries.md the query DSL that matters, plus percolator reverse search · esql.md the piped query language, when to prefer it, and its row cap · relevance.md BM25, boosting, ranking evaluation · aggregations.md buckets, metrics, accuracy limits · autocomplete.md as-you-type, suggesters, highlighting · vector-search.md kNN, hybrid, RRF · geo-search.md points, shapes, distance, geo aggregations · scripting.md Painless and runtime fields
• Load — bulk-indexing.md bulk, reindex, ingest pipelines, concurrency control · clients.md REST conventions, language clients, retries and timeouts
• Operate — search-performance.md profiling, caches, deep pagination, PIT · cluster.md nodes, shards, allocation, sizing, rolling restarts · index-lifecycle.md templates, aliases, data streams, ILM, zero-downtime reindex · snapshots.md repositories, restore drills, disaster recovery · security.md authentication, API keys, RBAC, query injection · upgrades.md version paths, breaking changes, OpenSearch divergence · logs-and-timeseries.md observability workloads and retention

Core Rules

1. text matches, keyword filters. A text field is analyzed into tokens; a term query is not analyzed. "Quick Brown" indexed as text becomes [quick, brown], so term: {title: "Quick Brown"} matches nothing and returns HTTP 200 with zero hits — no error anywhere. Default shape for anything a human types AND a machine filters on: multi-field text + fields.keyword. Search title, filter and aggregate title.keyword.
2. Filter context for everything that is not ranking. Test: if dropping the clause would change WHICH documents come back but not their ORDER, it belongs in bool.filter or bool.must_not. Those skip BM25 entirely and are cached per segment; must/should score every hit. Tenant IDs, status flags, date ranges, term lists, geo boxes — always filter.
3. Mappings are append-only. New field: yes. Change an existing field's type, analyzer, or index flag: never, in any version. The only path is new index → _reindex → alias swap, so put every index that matters behind an alias on day one, before you need it (index-lifecycle.md). Corollary: dynamic: strict on anything user-facing, because a typo'd field name is otherwise silently created with a guessed type.
4. Size shards from data volume, not from node count. primary_shards = max(1, ceil(expected_primary_gb / target_shard_size_gb)) — 600 GB at the 40 GB default (target_shard_size_gb) gives 15 primaries. Primary count is fixed at creation. Too few shards means slow recovery and a ceiling on parallelism; too many means wasted heap, because every shard is a full Lucene index with fixed overhead and the cluster hard-refuses allocation past cluster.max_shards_per_node (default 1000, replicas counted). Data that keeps growing gets rollover instead of a guess (index-lifecycle.md).
5. Heap is half the RAM and never past the compressed-pointer threshold. Xms = Xmx = min(RAM/2, 31g). A 128 GB node still gets 31 GB; the other half is the OS page cache that Lucene actually reads from, and starving it costs more than the heap gains. Past roughly 32 GB the JVM drops compressed object pointers, so 33 GB of heap addresses fewer objects than 31 GB. Verify, do not assume: the startup log line reports compressed ordinary object pointers [true].
6. Bulk, never per-document, and find the ceiling by measurement. Start at bulk_batch_mb (default 10 MB of payload, not document count) with a few requests in flight, raise until throughput flatlines or es_rejected_execution_exception appears — that rejection is the signal that you found the limit, not an error to retry harder against. A _bulk call returns HTTP 200 while individual items fail: branch on the response's errors flag every single time, or you will lose documents quietly.
7. A rebuildable bulk load sets refresh_interval: -1 and number_of_replicas: 0, then restores both. Every refresh seals a segment; refreshing once per second (the default) through a four-hour load creates roughly 14,400 tiny segments that merging then has to grind back down. Adding replicas after the load copies finished segments instead of re-indexing every document. Escape hatch: keep replicas if the source cannot be replayed, since you are running without redundancy until they are back.
8. Deep pagination is search_after, not from/size. from + size must stay within index.max_result_window (default 10,000), and every shard sorts and ships from + size hits to the coordinating node, so page 500 costs each shard the same work as returning 10,000 documents. User-facing paging: search_after on a stable sort plus a point-in-time. Full exports: PIT scans or _reindex, never a paging loop (search-performance.md).
9. Aggregate and sort on keyword, never on text. terms on a text field fails with "Fielddata is disabled on text fields by default". Setting fielddata: true to silence it loads every distinct term of that field into heap, uninvertered per segment — the most common self-inflicted circuit breaker there is. The fix is always the .keyword sub-field, or index: false + doc_values: true if the field is aggregation-only.

Search Triage

Symptom-first, in order. Steps 1-3 are seconds each and eliminate the causes that look like the expensive ones.

1. Does the document exist? GET //_doc/. If not, the problem is ingest, not search (bulk-indexing.md).
2. Does the query parse the way you think? GET //_validate/query?explain=true with the body. A misplaced brace turns a filter into a no-op that silently matches everything.
3. What tokens are actually indexed? POST //_analyze with {"field": "title", "text": "Quick Brown"}. Compare against the query terms. Mismatch here explains most zero-hit reports (Core Rules 1) and every "it works for one word but not two".
4. Why did THIS document not match? GET //_explain/ with the query body — it names the clause that failed, per document.
5. Why is it ranked HERE? Same query with "explain": true, read the score breakdown top-down: field boost, then BM25 term frequency, then length norm (relevance.md).
6. Where does the time go? "profile": true — per-shard, per-clause timings. A slow should clause or a rewrite-heavy wildcard shows up as rewrite_time (search-performance.md).
7. Is it the query or the cluster? Re-run against one shard with ?preference=_shards:0. Same latency means query cost; wildly different means an unhealthy node or a cold shard (cluster.md).

Query Types That Matter

Everything else is a variation. Choose from the left column, never from habit.

Shard and Heap Arithmetic

The canonical numbers; every guide references these rather than restating them.

Worked example: 2 TB of primary log data, 40 GB target, one replica. Primaries = ceil(2000/40) = 50. Total shards = 50 × 2 = 100. At 31 GB heap per node that is well inside budget on three nodes — but as one index it would also be unrecoverable and unshrinkable, which is exactly why time-series data rolls over instead (index-lifecycle.md).

Error Strings

Match on the exception type; the message text drifts between versions. Full catalog with fixes in errors.md.

Cluster Health In One Screen

GET /_cluster/health?level=indices          # green/yellow/red and where
GET /_cat/indices?v&s=store.size:desc       # size, doc count, health per index
GET /_cat/shards?v&h=index,shard,prirep,state,unassigned.reason
GET /_cat/nodes?v&h=name,heap.percent,ram.percent,cpu,load_1m,disk.used_percent
GET /_cat/thread_pool/search,write?v&h=node_name,name,active,queue,rejected
GET /_cluster/allocation/explain            # why THIS shard is unassigned
GET /_nodes/hot_threads                     # what a pegged node is actually doing

rejected in the thread-pool output is cumulative since node start, not a rate — compare two samples a minute apart before concluding anything (monitoring.md).

Every request in this skill is written in Dev Tools / raw REST. When client is set to anything else, emit the same call in that language's client instead — clients.md carries the per-client retry, timeout, and bulk-helper branch.

Output Gates

Before emitting a mapping, a query, or an index change:

• Every string field: is it text, keyword, or a multi-field — and does each use site match that choice (Core Rules 1, 9)?
• Every non-ranking clause in bool.filter rather than bool.must (Core Rules 2)?
• New index: born from a template, behind an alias, dynamic set explicitly, default_replicas applied, and the shard count derived from the formula and a stated volume rather than copied from a neighbour (Core Rules 4)?
• Paging: within max_result_window, or search_after + PIT if not (Core Rules 8)?
• Bulk or reindex: batched, the errors flag inspected per response, and resumable from a recorded key (Core Rules 6)?
• Relevance change: measured against a judgement set with _rank_eval, not eyeballed on one query (relevance.md)?

Configuration

User-dependent variables. Defaults apply until the user states a preference; store them in ~/Clawic/data/elasticsearch/config.yaml.

Preference areas — customizable dimensions; a stated preference is recorded in config.yaml and applied from then on:

• Tooling — Kibana Dev Tools vs a client library vs raw HTTP, ingest path (Beats, Logstash, Fleet, OpenTelemetry, application code), migration tooling
• Thresholds — slow-log warn levels, ILM rollover age and size, retention per tier, heap and disk alert points, how many hits a query may return by default
• Conventions — index and alias naming, field naming (ECS vs house schema), date-format policy, whether _source filtering is applied by default
• Platform — node roles and topology, storage class, region and zone awareness, container limits, whether dedicated master nodes exist
• Risk posture — whether reindex and ILM changes run directly or come back as reviewed requests, willingness to drop replicas during loads, appetite for force_merge on live indices
• Output format — request bodies only vs request plus a plan walkthrough, how much of an explain output to narrate, whether rollback steps ship with every change
• Work order — runtime field first vs reindex first when a mapping is wrong, whether relevance changes require an offline _rank_eval run before shipping
• Integrations — monitoring stack (Kibana Stack Monitoring, Prometheus exporter, provider console), snapshot repository (S3, GCS, Azure, shared filesystem), authentication backend (native, SAML, OIDC, LDAP)
• Restrictions — features the license or platform forbids, compliance regimes that mandate field-level security or encryption, indices that must never be closed or reindexed online
• Cadence — restore-drill frequency, ILM and mapping review cycle, snapshot schedule, index-usage audits

Traps

Where Experts Disagree

• Nested vs denormalized documents. Nested keeps per-item correctness but stores each sub-object as a hidden Lucene document, so a 20-item array is 21 documents to index, delete, and merge. Boundary: query the combination of two sub-fields together → nested; only ever filter on one at a time → flatten it (data-modeling.md).
• Search-time synonyms vs reindexing. Search-time expansion is editable without a reindex but costs query CPU and skews scoring for multi-word entries; index-time is fast and frozen. Boundary: a vocabulary that still changes monthly stays at search time; a settled thesaurus in a latency-critical path earns index time.
• Sizing by shard-size rule vs by benchmark. The 10-50 GB range is a starting posture, not a law: it comes from recovery and merge behaviour, not from your query shape. Anyone running a stable workload should benchmark one shard at real volume with real queries and size from the measured knee.
• Elasticsearch as the primary store. One camp keeps a system of record elsewhere and treats the index as rebuildable; the other writes straight to Elasticsearch. The testable boundary is whether you can afford to reindex from source after an unrecoverable mapping mistake or a corrupted shard — if you cannot, it was never a search index.
• kNN alone vs hybrid retrieval. Pure vector search wins on paraphrase and loses on exact identifiers, part numbers, and rare terms that no embedding preserved. Default to hybrid with RRF and only drop BM25 if an offline evaluation says it contributes nothing (vector-search.md).

Related Skills

More Clawic skills, get them at https://clawic.com/skills/elasticsearch (install if the user confirms):

• search-engine — engine-agnostic search design and relevance evaluation; jump there when the target is not Elasticsearch
• meilisearch — lighter site and product search when a cluster is more machinery than the job needs
• vector-databases — dedicated vector stores when embeddings are the whole workload rather than one field
• rag — retrieval design for LLM pipelines built on top of an index like this one
• observability — logs, metrics, and traces strategy above the storage layer

Part of Clawic, the verified skill library. Get this skill: https://clawic.com/skills/elasticsearch.