Elasticsearch semantic_text chunking: settings, limits and getting the matching chunk

Elasticsearch semantic_text chunking splits each field value into chunks at index time, embeds every chunk, and scores the document by its best chunk. To get the matching chunk back, add a highlight on the field with "type": "semantic" and "order": "score". The highlighter returns the chunk text ranked by similarity to the query, and you control the count with number_of_fragments. Leave out order and you get up to five chunks in document order, which is the wrong default for RAG.

Everything below ran on docker.elastic.co/elasticsearch/elasticsearch:9.5.5 (Lucene 10.5.1, single node, security off, 30-day trial license) with the built-in .multilingual-e5-small model behind a custom inference endpoint. The test document is an illustrative 465-word support handbook with seven titled paragraphs. It is small on purpose, so every chunk can be read by eye.

How semantic_text chunks text

A semantic_text field stores your original string in _source and writes the chunks into a hidden nested structure under <field>.inference.chunks. Each chunk is stored as character offsets into the original text plus an embedding. The semantic_text reference describes the offsets; the embedding subfield is visible through the field mapping API.

flowchart LR
  D([Document with body text]) --> C["Chunker<br/>chunking_settings"]
  C --> E["Inference endpoint<br/>embeds each chunk"]
  E --> N["Hidden nested docs<br/>body.inference.chunks"]
  N --> V["embeddings subfield<br/>dense_vector or sparse_vector"]
  D --> S[("_source keeps<br/>the original text")]

With nothing but "type": "semantic_text" in the mapping, self-managed 9.5.5 filled in "inference_id": ".elser-2-elasticsearch". Elastic Cloud Hosted and Serverless pick different defaults by version, listed on the setup page, so pin inference_id yourself. Both built-in endpoints report the same default chunking from GET _inference:

"chunking_settings": {
  "strategy": "sentence",
  "max_chunk_size": 250,
  "sentence_overlap": 1
}

On a basic license, the first indexing request failed with current license is non-compliant for [inference]. We started a trial to continue. Check your subscription before you plan on this field type.

The mapping we used

The endpoint comes first, then a field-level chunking_settings that overrides the endpoint's settings (field-level overrides arrived in 9.1):

PUT _inference/text_embedding/qq-e5
{
  "service": "elasticsearch",
  "service_settings": {
    "model_id": ".multilingual-e5-small",
    "num_allocations": 1,
    "num_threads": 1
  }
}

PUT kb
{
  "mappings": {
    "properties": {
      "title": { "type": "text" },
      "body": {
        "type": "semantic_text",
        "inference_id": "qq-e5",
        "chunking_settings": {
          "strategy": "sentence",
          "max_chunk_size": 60,
          "sentence_overlap": 0
        }
      }
    }
  }
}

GET kb/_mapping/field/* shows what Elasticsearch generated underneath. On 9.5.5 the embedding subfield is a bfloat16 bbq_hnsw vector with rescore_vector.oversample: 3.0. If you tune its recall, our num_candidates and oversampling post applies unchanged, and bbq_hnsw vs bbq_disk covers switching it through index_options.

"body.inference.chunks.embeddings": {
  "type": "dense_vector", "element_type": "bfloat16", "dims": 384,
  "similarity": "cosine",
  "index_options": { "type": "bbq_hnsw", "m": 16, "ef_construction": 100,
                     "rescore_vector": { "oversample": 3.0 } }
}

semantic_text chunking_settings: strategies and limits

There are four strategies. sentence packs whole sentences up to max_chunk_size words. word cuts at exactly max_chunk_size words and repeats overlap words from the previous chunk. recursive (GA in 9.1) splits on a list of separators and falls back to sentences. none (also 9.1) sends the whole value as one chunk. The inference API page gives examples but no ranges, so we found them by sending bad mappings:

So word overlap runs from 1 to half of max_chunk_size. The missing upper bound on max_chunk_size matters for truncation, covered below.

For word, the chunk count is exact arithmetic. With N words, chunk size M and overlap O, a document longer than M produces 1 + ceil((N - M) / (M - O)) chunks. We checked it against "format": "chunks" output for five settings:

Every chunk is embedded at index time and stored as one hidden nested document. At 50% overlap you embed close to twice the text. Multiply the chunk count by your document count before sizing memory; the vector sizing calculator takes it from there.

Getting the matching chunk back with the semantic highlighter

The Discuss question that prompted this post, "Elasticsearch semantic text search", came from a user who could not reach individual chunks and built a custom pipeline. On 9.x that work is mostly unnecessary. The search and retrieval page documents two ways in: the semantic highlighter for the best chunks, and "format": "chunks" in fields (GA since 9.2) for all of them.

POST kb/_search
{
  "size": 3,
  "_source": ["title"],
  "query": {
    "semantic": { "field": "body", "query": "how long do I have to report a broken item?" }
  },
  "highlight": {
    "fields": {
      "body": { "type": "semantic", "number_of_fragments": 2, "order": "score" }
    }
  }
}

The response, trimmed to the first hit:

{
  "_id": "handbook",
  "_score": 0.9368892,
  "_source": { "title": "Support handbook" },
  "highlight": {
    "body": [
      "Fragile items such as glassware are packed in double boxes, and claims for them are approved without a carrier inspection.\n\nReturns and refunds. Unused items can be returned within thirty days of delivery for a full refund. Sale items can be returned for store credit only. ",
      "When a parcel is treated as lost, support opens a claim with the carrier and sends a replacement at no cost. Customers do not need to wait for the carrier to finish its investigation. Damaged items need a photo of the box and the item within forty eight hours of delivery. "
    ]
  }
}

The answer ("within forty eight hours") is in the second fragment. The first is a chunk that starts at the end of the damaged-parcels paragraph and runs into the returns paragraph, because the sentence strategy ignores paragraph breaks. Every section heading in this document ended up glued to the end of the previous chunk. Send at least two fragments to the model.

What we observed for each highlighter option:

Does chunk boundary choice change which fragment wins?

We ran seven questions against the handbook, each with one sentence that answers it, and checked whether the top fragment contained that sentence. This is one illustrative document and seven questions. It shows the failure mode and measures nothing general.

The two misses were chunks that straddled a paragraph break. The winning setups also produced larger chunks (55 to 76 words), so size and boundaries both moved. For documents with real structure, recursive is the better starting point, and it needs no code. When you already split documents upstream, none with an array of strings gives you exact control: each array element becomes one chunk, and the highlighter returns that element verbatim.

Limits: truncation, nested access and re-chunking

Truncation is silent. The model sees at most its token limit per chunk, and nothing warns you when a chunk is longer. We indexed the 465-word handbook into a none field twice, once as is and once with a sentence about volcanoes appended. Then we read both embeddings back with "_source": { "exclude_vectors": false }:

The appended sentence never reached the model. A semantic query for "when do volcanoes erupt?" scored both long documents at exactly 0.8877589. The stored chunk offsets still claimed the whole text (end_offset: 2755), and a semantic highlight on the long document returned all 2,820 characters, ending with the volcano sentence the vector never saw. Elastic's chunking strategies post states the same thing: "if the text exceeds the token limits of the model, information is simply lost". Since max_chunk_size has no upper bound, keep it well inside your model's window; E5's tokenizer config in the model metadata says max_sequence_length: 512.

You cannot put semantic_text inside nested. "sections": {"type": "nested", "properties": {"text": {"type": "semantic_text"}}} fails with semantic_text field [sections.text] cannot be nested. The hidden chunks are themselves nested documents, which is also why _cat/indices reported docs.count: 12 for an index where _count returned 2.

The hidden structure is reachable anyway. A top-level knn on body.inference.chunks.embeddings with inner_hits returned per-chunk scores and _nested.offset ordinals, and those ordinals matched the order of the "format": "chunks" array. The parent _score (0.9368892) did not equal the top inner hit's score (0.938128), so the two paths score differently. This path is an internal field layout with no documentation, so we would not build on it.

Re-chunking needs a rewrite of every document. PUT kb/_mapping with new chunking_settings succeeded. Existing documents kept their old chunks, and new documents got the new ones, so one index held both. A _update_by_query over the old documents re-ran inference and re-chunked them in place (9 sentence chunks became 9 word chunks of 100 words). Budget for a full inference pass whenever chunking changes.

When to chunk yourself

Use none with arrays when you only need to control boundaries. Move to a plain nested mapping with your own dense_vector when you need per-chunk metadata such as page number or section path, or per-chunk filters:

PUT kb-nested
{
  "mappings": {
    "properties": {
      "title": { "type": "text" },
      "chunks": {
        "type": "nested",
        "properties": {
          "ord": { "type": "integer" },
          "section": { "type": "keyword" },
          "text": { "type": "text" },
          "vector": { "type": "dense_vector", "dims": 384, "similarity": "cosine" }
        }
      }
    }
  }
}

def embed(texts, input_type):
    r = call("POST", "/_inference/text_embedding/qq-e5",
             {"input": texts, "input_type": input_type})
    return [e["embedding"] for e in r["text_embedding"]]

paras = HANDBOOK.split("\n\n")
chunks = [{"ord": i, "section": p.split(".")[0], "text": p, "vector": v}
          for i, (p, v) in enumerate(zip(paras, embed(paras, "ingest")))]
call("PUT", "/kb-nested/_doc/handbook?refresh=true",
     {"title": "Support handbook", "chunks": chunks})

Pass input_type. E5 expects passage: and query: prefixes, and the model metadata maps them to ingest and search. We got three different vectors for the same sentence with no input_type, ingest and search. semantic_text handles this for you. On the query side, query_vector_builder gave the same score (0.9419993) as a vector we built with input_type: search.

POST kb-nested/_search
{
  "_source": ["title"],
  "knn": {
    "field": "chunks.vector",
    "k": 3,
    "num_candidates": 50,
    "query_vector_builder": {
      "text_embedding": { "model_id": "qq-e5", "model_text": "how long do I have to report a broken item?" }
    },
    "inner_hits": { "size": 2, "_source": ["chunks.ord", "chunks.section", "chunks.text"] }
  }
}

The top inner hit was ord: 2, section Lost and damaged parcels, score 0.9419993, with the paragraph text in _source. The other option is one document per chunk with a parent_id keyword, which makes chunk-level filters and deletes simple but needs collapse on parent_id to return one result per source document.

Citations for RAG

A RAG answer needs the chunk text and a stable pointer to its source. With semantic_text, return the parent _id plus title or URL from _source and the highlight fragments, then cite at document level. The highlighter returns text without offsets, so if you need an exact span, match the fragment against the _inference_fields offsets or move to the nested mapping, where ord and section come back with each hit.

sequenceDiagram
  participant A as Agent or RAG app
  participant ES as Elasticsearch
  participant L as LLM
  A->>ES: semantic query plus semantic highlight, order score
  ES-->>A: parent _id, title, top 2 to 3 chunk fragments
  A->>L: question plus fragments tagged with parent _id
  L-->>A: answer citing parent _id
  A->>ES: optional fields format chunks for neighbouring context

Retrieval that returns a plausible neighbour instead of the answering chunk is one of the failures in how RAG fails in production, and an agent is only as good as its retriever. Combining these chunk scores with BM25 is covered in hybrid search scoring explained.

Decision table

Our recommendation: start with semantic_text and recursive chunking, request two or three highlight fragments with "order": "score", and switch to a nested mapping only when you need per-chunk metadata in the response. Measure top-fragment accuracy on a sample of your own questions before and after any change to chunking_settings, because each change costs a full re-inference.

Where we can help

We test chunking and citation quality on client Elasticsearch clusters before RAG systems go live. If your semantic search returns the right document but the wrong passage, see our Elasticsearch consulting service, or our RAG implementation work for the full pipeline from ingestion to cited answers.