The best chunking strategy for a RAG system is the one that wins on your own documents and your own questions. A simple baseline, either fixed-size chunks or page-level chunks, is the right place to start. You move to structure-aware or semantic methods only when a test shows the baseline failing. This guide covers the main strategies, what the published evidence says about each, how to set size and overlap, and how to run a test you can defend to your team.
TL;DR
- Chunking splits documents into pieces that are indexed and retrieved. Chunks that are too small lose context, and chunks that are too large add noise and cost.
- Start simple: fixed-size chunks with overlap, or page-level chunks for PDFs and reports. Add complexity only when a measured problem justifies it.
- Semantic and proposition-based chunking cost more to run, and the research does not show they win consistently.
- Add metadata such as title, summary and keywords to every chunk, and choose the method by the file's structure.
- Judge any change with a fixed set of real questions, scored on groundedness, completeness, utilization and relevancy.
Table of Contents
- What chunking does to your answers
- The main RAG chunking strategies
- Which one to start with
- Setting chunk size and overlap
- Keeping context, headings, tables and metadata
- How to evaluate a chunking strategy
- Common chunking mistakes
- Our perspective on chunking
- Where to get help with a RAG build
- FAQ
- Sources
What chunking does to your answers
A RAG system does not read your whole document library when someone asks a question. It retrieves a few chunks and hands them to the language model. If the right passage was never retrieved, the model either guesses or says it does not know. Chunking decides what a "passage" even is.
Microsoft's architecture guidance describes the goal as breaking documents into appropriately sized chunks that hold semantically relevant content, so that the model receives the relevant information and little else. The same guidance warns that small chunks may lack context, while oversized chunks can be expensive and overwhelm token limits.
There is a practical reason this matters. Language models have finite context windows, and UK government guidance on RAG notes that chunking manages those limits while balancing semantic accuracy, context preservation and processing efficiency. Three trade-offs pull against each other, and every strategy below is a different way of settling them.

Here is what that looks like in a business setting. A clause in a vendor contract says "Termination requires 60 days' notice." If your chunker cut the page so that the clause landed in one chunk and the heading "Master Services Agreement, Section 9" landed in another, a question about the agreement's notice period may retrieve the clause with no sign of which contract it belongs to. The failure looks like a model problem, but the cause is upstream.
The main RAG chunking strategies
Most approaches fall into three families. A course from Anthropic's academy lists them as size-based (strings of equal length), structure-based (headers, paragraphs or sections) and semantic-based (grouping related sentences or sections using NLP). Two newer variations sit on top of those.
| Strategy | How it works | Where it fits | Main cost |
|---|---|---|---|
| Fixed-size | Cuts text every N tokens or characters, usually with overlap | Uniform content, first prototypes, unknown formats | Can cut through a sentence or a table |
| Recursive / structure-based | Splits on headings, paragraphs, then sentences, until chunks fit a limit | Manuals, policies, wikis, anything with clear headings | Needs consistent formatting |
| Page-level | One chunk per page | PDFs, reports, filings, anything cited by page | Pages vary in length and topic |
| Semantic | Groups neighboring sentences by embedding similarity | Long prose with shifting topics | Extra embedding compute |
| Proposition-based | Uses an LLM to break text into atomic facts | Fact lookup over dense text | Many more chunks, high LLM cost |
| Contextualized (late) | Embeds the whole document with a long-context model, then segments | Documents where meaning spans sections | Needs a long-context embedding model |
The last two rows come from a 2026 taxonomy paper on arXiv. It describes late chunking as preserving relationships across chunks by processing the full document before segmenting. It also reports that proposition-based chunking improves factual retrieval but multiplies the number of chunks and brings high computational cost through LLM usage.
Fixed-size chunking
This is the crudest method and the easiest to reason about. You pick a size, cut, and optionally overlap the pieces. Pinecone's guide recommends fixed-size chunking as a starting path in most cases, then iterating if it proves insufficient. It works best on fairly uniform content. It struggles when one document mixes narrative text, tables and bullet lists.
Structure-based and document-aware chunking
If your files have headings, sections or numbered clauses, splitting on those boundaries keeps a thought together. Microsoft's guidance says to choose the approach based on the file's structure. Content-dependent splitting also helps with conversational material: Cohere's documentation notes that for transcripts or meeting notes it can keep one speaker's turn together in a single chunk.
Page-level chunking
NVIDIA tested several strategies across multiple datasets and recommends page-level chunking as a default starting point for diverse documents. Its reasoning: performance was consistent, and citations are easier than with arbitrary token boundaries. A user can be told "see page 14" instead of "see characters 41,200 to 42,100." For regulated or audit-heavy work, that traceability can matter as much as the retrieval score.
Semantic chunking
Semantic chunking is popular because it sounds smarter. It splits where the topic shifts, using embeddings of sentences and their neighbors. The evidence is less flattering than the pitch. A paper in the ACL Anthology asks whether semantic chunking is worth its computational cost and finds that it does not give consistent gains over simpler fixed-size methods. The authors advise weighing the overhead against measurable improvement. Semantic chunking may still help on your data, but treat it as a hypothesis to test, not an upgrade to assume.
Which one to start with
Use this order of operations:
- Unknown or messy formats: start with fixed-size chunks. One course on chunking calls character-based splitting the safest bet for unknown document formats.
- PDFs, filings, reports, slide decks: start with page-level chunks, and add a smaller-chunk variant as a comparison.
- Well-structured manuals, policies, knowledge bases: split on headings and paragraphs, with a size cap.
- Transcripts and chat logs: split by speaker turn or topic segment.
- Everything else: run the baseline, look at the failures, and only then try semantic or proposition-based methods.
Microsoft makes a point worth keeping on a sticky note: any chunking approach might suit any document type, so observe the trade-offs on your specific data instead of leaning on generic recommendations. NVIDIA's own results show why. Its best chunk size differed by dataset: FinanceBench did best at 1,024 tokens, the Earnings dataset at 512, and smaller chunks of 256 to 512 tokens worked well for varied collections. No single number won everywhere.
Setting chunk size and overlap
Chunk size is a dial between precision and context. Small chunks match a narrow question precisely but may strip away the sentence that explains what "it" refers to. Large chunks carry context but dilute the match and fill the model's prompt with material it does not need.
The guidance from Microsoft is to choose fixed size and overlap through exploratory analysis, since results vary by document type. A workable process:
- Pick three sizes that bracket your expectation, for example 256, 512 and 1,024 tokens.
- For each, test with no overlap and with a modest overlap. Keep the overlap a fraction of the chunk size, not a large share of it.
- Run the same question set against every combination (see the evaluation section).
- Keep the smallest configuration that does not lose answers.
Overlap is insurance against bad cuts. Cohere's guidance says it can preserve context and soften the weaknesses of content-independent splitting, though it creates redundancy. Redundancy means more storage, more embedding spend, and sometimes near-duplicate chunks competing in the top results. If your retrieval returns the same passage three times, overlap is the first thing to reduce.
One technical detail: when you do fixed-size parsing, Microsoft recommends BERT-based tokenization instead of raw character counts so chunks line up better with semantic units. Count in tokens the way your embedding model does, not in characters.

Keeping context, headings, tables and metadata
Business documents are rarely clean prose. They have headers, footers, tables, numbered clauses and cross-references. Chunking that ignores this produces chunks that are technically correct and practically useless.
Carry the heading with the chunk. Prepend the document title and section heading to each chunk's text or store them as fields. A chunk that begins "Section 9: Termination" is far easier to retrieve and to interpret than one that begins "Either party may…"
Add metadata. Microsoft recommends enriching chunks with fields such as title, summary and keywords. Beyond those, business teams usually benefit from fields like document type, owner, effective date and department, because they let you filter before you search. These are our suggestions, not a standard: choose fields that match the questions your staff actually ask.
Keep tables whole. A table split mid-row is the classic silent failure. Where possible, keep a table as one chunk, or convert each row into a short sentence that repeats the column headers.
Respect versions. If a policy exists in three revisions, make the effective date a filterable field. Otherwise the system may happily quote the superseded one.
Record the source location. Store page numbers or section IDs so the answer can cite where it came from. This is also what lets a human reviewer check the answer quickly, which is the point of a human-in-the-loop design.
How to evaluate a chunking strategy
Opinions about chunking are cheap. Measurements are not. Microsoft lays out a structured evaluation process: gather representative test documents and queries, run multiple experiments, assess metrics such as groundedness, completeness, utilization and relevancy, then document and aggregate the results.
In practice:
- Build a question set. Collect 30 to 100 real questions from the people who will use the system, each paired with the document and passage that answers it. Include the awkward ones: questions that span two sections, questions about a table, questions the documents cannot answer.
- Freeze everything except chunking. Same embedding model, same retriever, same prompt, same language model. Change one variable per run.
- Score retrieval and answers separately. Did the right passage appear in the top results? Then, was the final answer grounded in what was retrieved, complete, and relevant? Separating the two tells you whether chunking or generation is at fault.
- Track cost and latency. More chunks mean more embedding calls and a bigger index. Larger chunks mean more tokens per query. Write these numbers down beside the quality scores.
- Read the failures. Aggregate scores hide patterns. Twenty minutes reading the ten worst answers will often show you the cause.
How much can the choice matter? Chroma's research found differences of up to 9% in recall between common strategies. That is meaningful, but also a reminder that chunking is one lever among several. If your recall is already high and answers are still wrong, look at the prompt, the reranking step or the source documents themselves.
Common chunking mistakes
- One setting for every document type. Contracts, spreadsheets exported to PDF, and support transcripts need different treatment. Microsoft's advice is to experiment with approaches suited to each document's type, structure and content.
- Chunks too small to stand alone. Poor chunking, whether too small and context-free or too large and noisy, can lead to inaccurate or irrelevant model responses, a point made in the Microsoft guidance and echoed in the UK government's RAG overview.
- Adopting semantic or proposition-based chunking because it sounds advanced. Both add compute and complexity. Adopt them when your test shows a gain that covers the cost.
- Never re-testing. New document types, a new embedding model or a new use case can change the answer. Keep the question set and re-run it.
- Ignoring the source documents. If the underlying files are outdated, duplicated or contradictory, no chunker will fix that.
- No path to human review. Production systems need monitoring, a way to roll back a change, and a person checking sensitive answers. Treat chunking changes as releases, not tweaks.
Our perspective on chunking
We rarely see small and midsize teams struggle because they picked the wrong chunking algorithm. They struggle because nobody agreed on what "a good answer" looks like, so every change is judged by whoever tried it last.
Our approach is to find where an AI agent creates real leverage, rank those candidates by return and risk instead of hype, and build the work into tools your team already uses. Chunking is a small part of that. It is a decision you make with a baseline, a real question set and a rollback plan, not a setting you tune once and forget.
We also think the plan should belong to you. The test questions, the results and the reasoning behind your chunking choice should be written in plain language and owned by your company, so that whoever maintains the system next year can see why it was built that way. Where documents are sensitive, such as legal intake or patient operations, we put human-in-the-loop checkpoints in from the first day, not after the first mistake.
jaras
Where to get help with a RAG build
If your team is weighing a document-search or internal-assistant project but does not have a technology lead to own the decisions, Mindpod Technologies starts with a free technology assessment and turns it into a prioritized, plain-language plan you own. Our AI Governance work covers monitoring, rollback and human review for AI systems that go to production, and our Custom Apps work builds software around your real workflow.

Bring your document types and the ten questions you most want answered. That is enough to decide what to test first.
FAQ
What is the best chunk size for RAG?
There is no universal best. In NVIDIA's tests the strongest size varied by dataset, with 1,024 tokens best on one and 512 on another, and 256 to 512 tokens working well for varied collections. Test two or three sizes against your own questions and keep the smallest one that does not lose answers.
Should I use semantic chunking?
Only if your tests show it beating a simpler baseline. Research in the ACL Anthology found no consistent advantage over fixed-size chunking, and semantic methods add embedding cost. Start with fixed-size or page-level chunks and move up if specific failures call for it.
Do I need overlap between chunks?
Usually a modest amount helps, because it keeps a sentence that straddles a boundary from being lost. The trade-off is redundancy: more storage and sometimes duplicate results. Treat overlap as a setting to test, not a fixed rule.
How do I know if chunking is my problem or something else?
Score retrieval and answer quality separately. If the right passage is not retrieved, look at chunking, metadata and the embedding model. If it is retrieved but the answer is still wrong, look at the prompt, the model and the source content.
Sources
- Develop a RAG Solution on Azure: Chunking Phase (Microsoft Learn)
- Design and Develop a RAG Solution on Azure (Microsoft Learn)
- AI Insights: RAG Systems (GOV.UK)
- Text chunking strategies (Claude academy)
- Beyond Chunk-Then-Embed: A Comprehensive Taxonomy (arXiv)
- Finding the Best Chunking Strategy for Accurate AI Responses (NVIDIA)
- Is Semantic Chunking Worth the Computational Cost? (ACL Anthology)
- Chunking Strategies for LLM Applications (Pinecone)
- Effective Chunking Strategies for RAG (Cohere)
- Evaluating Chunking Strategies for Retrieval (Chroma)
