Good chunking creates units that are small enough to retrieve precisely and large enough to preserve the meaning needed for an answer. Prefer document structure and question intent over one universal character count. Carry headings, source location, version, and permissions with every chunk, then evaluate whether expected chunks appear for real queries.
Who this is for: Developers and content owners improving retrieval from manuals, policies, knowledge bases, reports, and mixed-format files.
- Split on semantic and structural boundaries before tuning a fixed size or overlap.
- Preserve context through headings and metadata without repeating unrelated text into every unit.
- Choose chunking by retrieval evidence, citation quality, and document type rather than intuition alone.
A chunk is a retrieval decision
Chunking determines which pieces can be returned independently. If a policy exception sits in a different chunk from its rule, retrieval may present the rule without the condition. If an entire chapter becomes one unit, a match to one paragraph brings a large amount of unrelated text. The useful unit is shaped by the questions people ask and the evidence needed to answer them.
Begin by inspecting document structure. Headings, sections, list groups, table boundaries, speaker turns, and code blocks often signal coherent units. Keep a stable pointer to the original location. A chunk should not become an anonymous string because readers and evaluators need to open the source and understand its surrounding context.
Preserve enough local meaning
A paragraph may depend on its title, definitions, or preceding condition. Add a concise heading path to the chunk or expand the boundary to include the necessary statement. Avoid copying a long introduction into every chunk; repeated boilerplate can dominate similarity and make many records look alike. Remove navigation and decorative text that does not help interpretation.
Overlap can protect meaning near an arbitrary boundary, but it also duplicates results and complicates citations. Use it deliberately after observing boundary misses. Structural chunking may need little overlap, while a long narrative without headings may need a sliding strategy. Deduplicate neighboring results before assembling model context so repeated sentences do not crowd out distinct evidence.
Handle tables and special formats
Tables require headers and row relationships. Embedding isolated cell text often destroys what a value means. Convert a manageable table into a structured textual representation with title, column names, row labels, and units, while retaining a link to the original. Very large tables may be better served by database queries or analysis tools than by text retrieval.
Code, transcripts, and scanned documents need their own treatment. Keep code with the function or explanation it implements. Group transcript turns around one exchange while preserving speaker names and time locations. Validate optical character recognition before indexing scans. A clean chunking strategy cannot recover text that parsing omitted or corrupted.
Evaluate and revise by failure type
Create test questions with expected source locations. Measure whether the needed unit is retrieved, whether it contains sufficient evidence, and whether the citation leads to an understandable passage. Include questions whose answer spans adjacent sections and questions that should return no answer. Inspect misses instead of immediately increasing chunk size.
Classify failures as boundary loss, excess noise, missing heading context, parsing damage, metadata filtering, or ranking. Change one preparation choice and rerun the same set. Different document families may need different strategies, which is preferable to a complicated universal splitter. Current platform file and retrieval limits can change, so consult official documentation during quarterly review.
Chunk an equipment safety manual
Technicians ask about startup checks, but warnings and exceptions sometimes appear under separate subheadings.
- Parse the manual hierarchy and identify procedure, warning, exception, table, and troubleshooting blocks with page locations.
- Create one chunk per coherent procedure, carrying its heading path and directly associated warnings rather than splitting by a blind character limit.
- Represent inspection tables with headers, row labels, acceptable values, and units, linked to the original page.
- Test technician questions, including one exception case, and check both retrieval position and whether the returned unit is sufficient to answer safely.
- Adjust only the failed boundary or metadata rule, then rerun normal and exception cases before reindexing the full corpus.
Chunk inspection checklist
Review samples from every document family before and after indexing.
- Coherence: The unit expresses one retrievable idea, procedure, exchange, or structured record.
- Context: Required heading, definition, condition, speaker, header, and unit remain understandable.
- Provenance: Source identifier, edition, section, page or anchor, and permissions are attached.
- Noise: Navigation, repeated headers, unrelated boilerplate, and parsing artifacts are removed.
- Evidence: Representative queries retrieve a sufficient unit and a reviewer can inspect its origin.
Common mistakes
- Applying one token or character size to policies, tables, transcripts, and code without inspecting their structure.
- Adding large overlap everywhere, which duplicates evidence and displaces distinct passages from model context.
- Discarding section paths and page locations after embedding, leaving answers with weak or unusable citations.
Try one
A retrieved paragraph says employees may work remotely two days, but the preceding heading limits the rule to one office. How should chunking change?
The correction should carry the office heading path with the paragraph or combine the condition and rule into one coherent unit. It should preserve office metadata for filtering and add tests for each office plus a cross-office query. Simply increasing every chunk is poorly targeted because it may add noise elsewhere. Evaluation should confirm that the result names the scope and cites the exact policy location.
Sources
- OpenAI retrieval guideOfficial guidance for indexing, searching, and retrieving relevant source material.
- OpenAI embeddings guideOfficial explanation of embeddings and common semantic search uses.