Four offline heuristics tell you which chunking scheme will actually retrieve the right passage — before you spend a week indexing a vector database. A RAG pipeline that returns the right facts depends almost entirely on how a document is split into chunks, yet most teams pick a chunk size from blog posts and ship it. The AI RAG Chunk Quality Scorer scores up to three candidate splitting schemes for the same document across four offline metrics — coherence, coverage, overlap health, and chunk-size consistency — and picks the best one with no model calls, no embeddings, and no GPU.
Why Chunk Quality Matters More Than the Embedding Model
Vector search ranks passages by cosine similarity in the embedding space, but a chunk that cuts a sentence in half, repeats itself across two neighbors, or wanders between three topics pulls the retrieval toward noise no matter how good the embedding model is. The classic failure looks like this: the user asks “What is the return window for international orders?” and the retriever returns a chunk that starts mid-sentence with “30 days for domestic orders…” because that chunk contains both topics and its embedding sits closer to the query than the clean dedicated chunk sitting right next to it.
The fix is not a better model — it is a better splitter. Smaller chunks raise precision but lose context, larger chunks carry context but dilute relevance, and overlap between chunks preserves context across boundaries at the cost of duplicate storage. There is no single right answer because the right answer depends on the document, the query distribution, and the embedding model. What you need is a way to compare candidate schemes on the same document before you commit.
Coherence: How Cleanly Chunks Snap to Sentence Boundaries
Coherence measures how cleanly chunk boundaries snap to sentence and paragraph delimiters. A 256-token scheme running a sentence-aware splitter should score near 100; the same scheme in fixed mode drops sharply because fixed slicing cuts mid-sentence and produces chunks that begin and end in the middle of a clause. Coherence is the metric most directly tied to retrieval precision because the embedding of a mid-sentence fragment is biased toward whatever grammatical context survives the cut. A chunk that starts with “…due to the architectural constraints of the legacy pipeline” carries the embedding of a clause fragment, not a statement, and cosine similarity will pull the chunk toward architectural queries even when the actual answer is about the legacy pipeline.
Coverage: How Focused Each Chunk Is on One Topic
Coverage measures how focused each chunk is on a single topic. It looks at unique key-term concentration per chunk — high coverage means each chunk dominates one term cluster, low coverage means each chunk is a grab-bag. The scorer uses a key-term frequency analysis that does not need a model: rare nouns, named entities, and high-tf-idf tokens are weighted higher than function words. A chunk that mentions “shipping”, “refund”, “policy”, and “tracking” all four times is low-coverage because no single topic dominates; a chunk that mentions “shipping” eight times and “policy” twice is high-coverage because shipping is the topic.
Overlap Health: The Right Amount of Cross-Boundary Context
Overlap health measures whether adjacent chunks share enough context to preserve cross-boundary meaning without bloating the index. A 12.5% overlap is generally healthy; a 50% overlap doubles storage and slows retrieval without proportional recall gain. The scorer penalizes both low overlap (boundary drift) and excessive overlap (index bloat). The interesting case is the inflection point: most documents have an overlap percentage below which boundary recall collapses and above which retrieval quality plateaus. The scorer surfaces this by reporting coverage stability across the overlap spectrum — if coverage at 25% overlap is the same as coverage at 50% overlap on your document, you are paying for storage you do not need.
Chunk-Size Consistency: The Standard Deviation Question
Chunk-size consistency measures the standard deviation of chunk length in the chosen unit. Real documents have uneven sections — a header followed by three short paragraphs followed by a code block — and fixed-size splitting produces wildly uneven chunks. Consistency above 90 means the chunker is working with the document’s natural rhythm; consistency below 70 means you should consider a sentence-aware or paragraph-aware method. The metric matters because embedding models have an effective context window, and a chunk of 12 tokens sitting next to a chunk of 1,024 tokens in the same index creates an unbalanced retrieval surface where short chunks dominate short queries and long chunks dominate long queries in ways that do not reflect content relevance.
How the Comparison Mode Works
The scorer accepts three candidate schemes (A, B, C) with independent size, overlap, method, and unit settings. For each scheme, it scores all four metrics independently, then ranks the three by a weighted total — coherence and coverage weighted heavier because they correlate most strongly with retrieval precision on standard eval sets. This side-by-side mode is the practical answer to “should I use 256 or 512 tokens?” because the right answer changes with the document. A 1024-token scheme can beat both on a long-form policy document with multi-paragraph answers; a 256-token scheme can beat both on a glossary or FAQ where each entry is itself a complete answer. The scorer’s output table shows not just which scheme wins but where each scheme loses — a 512-token scheme with 95 coherence but 62 coverage is telling you it is splitting paragraphs across chunks; a 256-token scheme with 95 coverage but 70 coherence is telling you it is cutting sentences.
When Fixed-Size Beats Sentence-Aware
The counter-intuitive case for fixed-size splitting is a document written in short, formulaic sentences where the sentence boundaries carry no semantic weight. A compliance checklist or a glossary entry is one sentence per line; snapping to sentences is identical to fixed-size at the same target, but the fixed-size metric for coherence will report higher because there is nothing to cut. The scorer catches this: a 256-token fixed scheme on a sentence-per-line document scores 100 on coherence and 95 on coverage, beating a sentence-aware scheme that pays the overhead of detecting sentence boundaries in a stream of single-line statements. The general rule is that fixed-size wins when the source text is already uniform and sentence-aware wins when the source text varies in paragraph length. The counting unit you choose — characters, words, or tokens — also matters: tokens is the default because it is the closest proxy for what the embedding model and the LLM downstream actually see, but tokens are not equal across tokenizers, so a 256-token chunk in GPT-style BPE is not the same as a 256-token chunk in a SentencePiece model. If you are committing to a specific embedding model, score in tokens and use that model’s tokenizer; if you are picking between sentence-aware and paragraph-aware methods at a coarse level, score in words to keep the comparison readable.
Putting It Together
For a first pass, score three schemes that bracket the obvious choice: 128, 512, 1024 tokens, all sentence-aware, all 12% overlap. The output will tell you which end of the spectrum your document lives on and whether the obvious 512 is actually optimal. For the second pass, take the winner and vary overlap — try 0%, 12%, 25% — to find the smallest overlap that preserves boundary recall without bloating storage. The whole exercise takes a few minutes and the Elysia Tools version of the scorer runs offline in the browser, so you can iterate without API cost. Once you have a winner, re-embed the document with the chosen scheme and re-run your retrieval eval — the delta on Recall@5 is usually the most informative number your eval set will produce all quarter.
Explore more tools at elysiatools.com.
