Chunking Strategies for RAG

[Chunking](/glossary/chunking/) is the process of dividing source content into units that a retrieval system can index and return. It sounds like a preprocessing detail, but it defines what the retriever is capable of finding.

If a chunk is too small, it may match a query precisely while losing the context needed to interpret it. If it is too large, it may contain the answer but dilute the relevant passage with unrelated text. The best strategy preserves meaningful boundaries, supports the expected queries, and fits the model's [context window](/glossary/context-window/).

This guide explains the main strategies, their trade-offs, and a practical way to choose among them.

Where chunking fits in RAG

Chunking happens during knowledge preparation, before a user asks a question. A typical indexing flow parses a document, removes irrelevant artifacts, divides the remaining content, creates an [embedding](/glossary/embedding/) for each chunk, and writes the chunk plus metadata into an index.

Chunking creates the retrievable units. Embeddings make those units searchable by meaning; they do not restore structure discarded during preprocessing.

At runtime, the query is compared with these stored units. The [retrieval pipeline](/glossary/retrieval-pipeline/) can only rank what was indexed. If one important answer is split awkwardly across two chunks, neither piece may be strong enough to retrieve. If an entire handbook becomes one chunk, a match to one sentence may return far more material than the model can use effectively.

Start with the document, not a universal number

There is no universally correct chunk size. “Use 500 tokens” can be a convenient baseline, but it is not a law. A product catalog, legal contract, API reference, transcript, and troubleshooting manual have different natural units.

Before choosing a splitter, inspect:

• the source formats and their reliable structural signals;
• the typical length of a complete answer;
• whether meaning crosses headings, turns, rows, or code blocks;
• how users phrase real questions;
• the retrieval and generation models' limits;
• whether results need precise citations.

The goal is not to produce evenly sized boxes. It is to create units that remain meaningful when retrieved alone.

Five common chunking strategies

Different strategies choose boundaries from size, language, meaning, or document structure. Hybrid strategies are common in production.

Fixed-size chunks

Fixed-size chunking cuts text by a token or character count, often with overlap. It is simple, predictable, and fast. It also works when the source has little trustworthy structure.

The weakness is that the cut may land inside a sentence, list, or argument. Token-based limits are generally more aligned with model constraints than character counts, but either method ignores meaning unless combined with boundary rules.

Use fixed-size chunks as a baseline, not as proof that chunking is solved.

Sentence or paragraph chunks

This method preserves natural language boundaries. It works well when paragraphs are coherent and reasonably sized, such as short help articles.

Real documents are rarely so tidy. A one-line heading may depend on the following paragraphs; a long paragraph may contain multiple topics; a table may be parsed into meaningless fragments. Systems therefore often group adjacent sentences or paragraphs until a size budget is reached.

Recursive chunking

A recursive splitter tries larger, preferred boundaries first—such as sections, paragraphs, then sentences—and falls back to smaller separators when a unit exceeds the target. It offers more structural awareness than hard fixed-size cuts while remaining broadly applicable.

Recursive splitting is a useful general default for clean prose, but the order of separators and parser quality still matter. It does not understand the domain merely because it preserved paragraph marks.

Semantic chunking

Semantic chunking looks for topic shifts, often by comparing adjacent sentence embeddings or using a model to identify coherent segments. It can separate meaning more naturally when structure is weak.

Its costs are greater complexity, additional model calls or embedding work, and thresholds that may behave differently across domains. It can also make chunk sizes uneven. Evaluate whether the extra sophistication improves retrieval on actual questions.

Document-aware chunking

Document-aware logic understands a source type. It may keep a table with its title and headers, preserve a code function as a unit, associate FAQ answers with their questions, or group transcript turns by speaker and topic.

This strategy often produces the most usable chunks because it respects how the information was authored. It also requires format-specific parsers, tests, and fallbacks for malformed documents.

Chunk size is a retrieval trade-off

Small chunks tend to be more focused. They can improve the precision of a match and produce narrow citations. But they may omit definitions, exceptions, or references that make the passage understandable.

Large chunks preserve more surrounding material and reduce the risk of splitting an answer. But each embedding represents more topics, which can blur the semantic signal. Large results also consume more context and may place the relevant sentence among distracting text.

Chunk size controls both retrieval granularity and the amount of context delivered downstream. Overlap helps at boundaries but adds duplication.

Choose a starting range based on the expected answer span and document structure. Then test multiple values. Report size in tokens when the downstream constraint is token-based, and inspect the actual distribution rather than only an average.

What overlap does—and does not do

Overlap repeats some material at the end of one chunk and the beginning of the next. It reduces the chance that a sentence pair or short explanation disappears across a boundary.

More overlap is not automatically better. It increases index size, embedding work, and the chance that near-duplicate chunks occupy several top results. The model may see repeated evidence and mistake repetition for independent support.

Use enough overlap to preserve common boundary-spanning answers, then deduplicate or diversify retrieved results where necessary. If heavy overlap is required to make chunks coherent, a structure-aware splitter may be the better fix.

Preserve metadata and hierarchy

Every chunk should remain traceable to its source. Useful fields include document ID, title, section path, page or paragraph, version, timestamp, language, permission group, and a canonical URL.

Hierarchy can restore context without embedding an entire document as one unit. A child chunk can be used for precise matching while the system returns its larger parent section to the model. This parent-child pattern separates the unit used for search from the unit used for reading.

For example, an individual troubleshooting step may be the best retrieval target, while the full procedure—including prerequisites and warnings—is the safest context to generate from.

Special content needs special treatment

Tables

Preserve headers with rows, or convert the table into a representation that keeps column relationships. A row of values without headers is usually uninterpretable.

Lists and procedures

Keep ordered steps together when their sequence matters. If a long procedure must be split, repeat the procedure name and step range in metadata or text.

Code

Prefer functions, classes, or logical blocks over arbitrary line counts. Include language, file path, and symbol names as metadata. Avoid separating a function signature from its body.

Conversations

Preserve speaker labels and enough turns to resolve pronouns and references. Chunking every message independently may erase the question an answer responded to.

PDFs

PDF text extraction can scramble columns, headers, footnotes, and reading order. Fix parsing before tuning chunk size. A perfect splitter cannot recover structure that the parser destroyed.

A practical selection process

Treat chunking as an experiment:

1. Collect representative documents and real or carefully authored questions.
2. Mark the passages that should support each answer.
3. Implement a simple, structure-aware baseline.
4. Try a small set of chunk-size and overlap settings.
5. Retrieve a sufficiently broad candidate set.
6. Measure whether the expected evidence appears, then inspect failures.
7. Test downstream answer quality, citations, latency, and token use.
8. Segment results by document type and query type.

Retrieval metrics reveal whether relevant chunks are found. Generation evaluation reveals whether the supplied context is sufficient and usable. Both matter.

The paper commonly called “Lost in the Middle” showed that language models can use long context unevenly depending on where relevant information appears. That does not prescribe one chunk size, but it reinforces a practical point: adding more context is not equivalent to making evidence easier to use.

Diagnose the failure before changing the splitter

When an answer is wrong, classify the failure:

• Missing source: the knowledge base lacks the information.
• Parsing failure: the source was extracted incorrectly.
• Boundary failure: the answer was divided across weak chunks.
• Representation failure: embeddings did not capture the relationship.
• Retrieval failure: parameters or filters excluded the right chunk.
• Ranking failure: the right chunk appeared but too low.
• Context failure: relevant chunks were truncated or assembled poorly.
• Generation failure: the evidence was present, but the model misused it.

Only some of these are chunking problems. This classification prevents endless tuning of chunk size when the real issue is permissions, query phrasing, [reranking](/glossary/reranking/), or source quality.

Common mistakes

Avoid stripping headings before attaching them to the text they describe. Do not mix unrelated tenants or permission scopes in the same retrievable unit. Do not use the same settings for prose, code, tables, and transcripts without testing. And do not judge a strategy from a few impressive demos.

An advanced approach is not automatically a better one. Research such as RAPTOR explores recursive summaries over clustered text to retrieve at different levels of abstraction. These designs can help with broad questions across long documents, but they add indexing and evaluation complexity. A well-parsed section-based baseline may be better for a straightforward support corpus.

Recommended baseline

For ordinary documentation, begin with a document-aware parser, preserve headings, split recursively at natural boundaries, set a moderate token target, and add limited overlap. Store rich provenance. Retrieve multiple candidates and evaluate both exact evidence recall and answer quality.

From there, let failure patterns justify changes. Add parent-child retrieval when precise matches need larger reading context. Introduce semantic boundaries when topic shifts regularly defeat structural rules. Separate strategies by content type when a single policy performs poorly.

The best chunking strategy is the simplest one that reliably surfaces complete, attributable evidence for the questions your system must answer.

Sources

• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks describes a foundational retrieval-plus-generation architecture.
• Lost in the Middle: How Language Models Use Long Contexts evaluates how the position of relevant information affects long-context use.
• RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval explores retrieval over a hierarchy of text summaries at different abstraction levels.

Continue learning

Review [How RAG Works](/how-rag-works/) for the complete indexing and runtime pipeline, then compare candidate-generation methods in [Hybrid Search vs Dense vs Sparse Retrieval](/hybrid-vs-dense-vs-sparse-retrieval/).