Chunking

What is chunking?

Chunking is the process of dividing large content into smaller units that can be embedded, indexed, retrieved, and placed into a model’s context. A chunk might be a paragraph, section, group of sentences, table, code unit, or another meaningful segment.

RAG systems use chunks because complete documents are often too large and too broad to retrieve as one item.

How chunking affects retrieval

Each chunk becomes a searchable unit. When a user asks a question, the system retrieves the chunks most likely to contain relevant evidence.

Chunk size creates a trade-off:

• Small chunks can match a narrow fact precisely but may lose surrounding meaning.
• Large chunks preserve more context but can mix topics and consume more tokens.
• Overlapping chunks can preserve continuity across boundaries but create duplicates.

For example, splitting a policy exactly between a rule and its exception could produce a misleading result. A structure-aware chunker may keep the complete section together.

Common chunking strategies

Fixed-size chunking divides text by characters or tokens. Sentence- or paragraph-based chunking respects language boundaries. Structure-aware chunking follows headings, tables, code functions, or document sections. Semantic chunking tries to split when the topic changes.

No single strategy is best for every source. Legal documents, support articles, source code, and transcripts have different natural boundaries.

Chunking versus tokenization

Tokenization converts text into units that a model processes. Chunking groups content into retrieval units. A chunk contains many tokens and should be sized with the context window and embedding model limits in mind.

Chunking also differs from summarization. Chunking preserves selected source text, while summarization creates a shorter representation that may omit details.

Quality and failure modes

Poor chunking can separate titles from their content, lose table headers, duplicate passages, or combine unrelated sections. Retrieved chunks may also lack document metadata needed to judge date, authority, or access.

Builders should preserve source identifiers and structural information, test realistic questions, and measure whether the required evidence appears in retrieved results. Chunk size should be tuned using retrieval performance, not chosen only by habit.

Why it matters

Chunking determines what a retrieval system can find and what evidence a model sees. It influences relevance, context efficiency, citation quality, and answer completeness.

A strong RAG pipeline treats chunking as part of information architecture. The goal is not uniform pieces; it is self-contained units that remain understandable, searchable, and traceable to their source.