RAG Beyond Toy Demos: Chunking, Reranking, and the Evaluation Problem Nobody Solved

A retrieval-augmented generation demo is one of the most dishonest artifacts in software. You load a dozen PDFs, embed them, wire an off-the-shelf vector store to a chat model, and within an afternoon it answers questions fluently and looks like magic. Then you point it at ten thousand documents from a real organization and it confidently cites the wrong policy, misses the one paragraph that mattered, and invents a number that appears nowhere in the corpus — and you discover that the easy part was the only part you built. The hard parts of production RAG are not the embedding model or the vector database; those are commodities, and the retrieval internals are well understood. The hard parts are chunking the corpus so the right span is even retrievable, transforming the user’s question into something searchable, ranking aggressively enough that the answer lands in the model’s context window, grounding the generation so it cites rather than confabulates, and — the part nobody has cleanly solved — measuring whether any of it actually works. This post is about those parts: the engineering between the toy and the system, and why the demo lied to you.

The Demo That Lies

The toy demo works because it operates in a regime where retrieval cannot fail. With twelve documents, even mediocre embeddings return the relevant chunk in the top results, because there is almost nothing to compete with it. The model then has the answer sitting in its context and paraphrases it well. Every component is masked by the small scale.

Production breaks each masked assumption at once. With millions of chunks, the relevant span now competes with thousands of near-duplicates and semantically adjacent distractors, so top-k retrieval routinely misses it. Real questions are underspecified, full of pronouns and implicit context, so the embedding of the raw question points at the wrong neighborhood entirely. Real documents have tables, headers, footnotes, and cross-references that naive chunking shreds into meaningless fragments. And the failure mode is silent: the model still produces a fluent, confident answer, because that is what language models do whether or not the retrieval succeeded. A RAG system does not crash when retrieval fails; it lies. That property — confident failure — is why you cannot ship on vibes and must build the unglamorous machinery below.

Chunking Is the Retrieval You Never Tuned

The single highest-leverage decision in a RAG pipeline is how you split documents, and it is the one most teams make by accident with a default chunk_size=1000. Chunking determines what is retrievable at all: if the answer spans a boundary, or sits in a chunk dominated by unrelated text, no reranker or clever prompt recovers it. You are tuning retrieval whether you think about it or not.

The trade-off is fundamental. Small chunks embed precisely — a 200-token chunk has a focused meaning, so its vector is sharp and matches narrow queries — but they fragment context, and the retrieved span may lack the surrounding information the model needs to answer. Large chunks preserve context but dilute the embedding: a 2,000-token chunk covering five topics has a muddy average vector that matches everything weakly and nothing strongly. There is no universal right answer, only a fit to your corpus and queries.

Two techniques earn their keep. First, overlap — carrying the last sentence or two of one chunk into the next — cheaply prevents boundary answers from vanishing. Second, contextual retrieval: before embedding a chunk, prepend a short LLM-generated description of where it sits (“This is from the FY2025 refund policy, section on international orders”), so an otherwise ambiguous fragment becomes findable. A recursive splitter with structural awareness is the pragmatic default:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=512,          # tokens, not characters, if your tokenizer allows
    chunk_overlap=64,        # ~12% overlap to survive boundary answers
    separators=["\n## ", "\n### ", "\n\n", "\n", ". ", " "],  # respect structure first
)
chunks = splitter.split_text(doc)
# Attach metadata you will filter and cite on later:
records = [{"text": c, "source": doc_id, "section": detect_section(c)} for c in chunks]

Carry metadata — source, section, page, timestamp — on every chunk from the start. You will need it for filtering, for citations, and for the day someone asks “why did it say that,” and retrofitting it means re-indexing the whole corpus.

Query Transformation: Fixing the Question Before You Search

The user’s question is rarely a good search query. It contains pronouns referring to earlier turns, omits context the user assumes, or phrases things in language that does not match the documents (“can I get my money back” vs. a corpus that says “refund eligibility”). Embedding the raw question and searching is the most common reason a production system retrieves nothing useful. The fix is to transform the question into one or more better queries before retrieval.

Three transformations cover most of the value. Rewriting resolves references and context using conversation history, turning “what about for international ones?” into “what is the refund policy for international orders?”. Multi-query expansion generates several paraphrases and unions their results, widening recall when a single phrasing is unlucky. HyDE (Hypothetical Document Embeddings) asks the model to draft a hypothetical answer and embeds that, exploiting the fact that an answer is lexically and semantically closer to the source document than the question is. For complex questions, decomposition splits a multi-hop question into sub-questions retrieved independently.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

# Multi-query expansion: one question -> several search queries
EXPAND = """Rewrite the user question into 3 diverse search queries that would
find relevant documents. Resolve pronouns using the chat history. Return one per line.

History: {history}
Question: {question}"""

queries = llm(EXPAND.format(history=history, question=q)).splitlines()
hits = {}
for query in queries:
    for chunk in vector_search(embed(query), k=20):
        hits[chunk.id] = chunk          # de-dupe by id across queries
candidates = list(hits.values())        # union, then rank (next section)

Every transformation costs an LLM call and latency, and they can hurt: aggressive rewriting can drift from the user’s intent, and HyDE fails when the model has no idea what the answer looks like. Measure before adopting — which requires the evaluation harness this post builds toward.

Retrieve Wide, Rank Narrow

The architecture that actually ships separates recall from precision into two stages. First retrieve a wide candidate set cheaply — top 50 or 100 — optimizing to not miss the answer. Then rerank that set with an expensive, accurate model and keep only the few that fit the context window, optimizing for what the generator actually sees. This two-stage shape exists because the cheap retriever (a bi-encoder, comparing pre-computed vectors) is fast but coarse, while a cross-encoder reranker reads the query and each candidate together and scores their actual relevance — far more accurate, far too slow to run over millions of chunks, perfect over a hundred.

        RETRIEVE WIDE, RANK NARROW

   query --> [rewrite/expand] --+--> dense (vector)  --\
                                |                        >-- fuse (RRF) --> top 100
                                +--> sparse (BM25)   --/         |
                                                                 v
                                                       [cross-encoder rerank]
                                                                 |
                                                          keep top 5-8
                                                                 |
                                                   [pack context + cite] --> LLM --> answer

The first stage should be hybrid: union a dense vector search (semantic) with a sparse BM25 search (exact terms, names, IDs, error codes that embeddings handle poorly), fused by reciprocal rank fusion. The mechanics of dense, sparse, and HNSW indexing are covered in the embeddings and vector search post; the production point is that you almost always want both, because each catches what the other misses. Then rerank:

1
2
3
4
5
6
7

from sentence_transformers import CrossEncoder
reranker = CrossEncoder("BAAI/bge-reranker-v2-m3")

pairs = [(q, c.text) for c in candidates]          # candidates from hybrid retrieval
scores = reranker.predict(pairs)                   # query x candidate, scored jointly
ranked = [c for _, c in sorted(zip(scores, candidates), key=lambda x: -x[0])]
context = ranked[:6]                               # only what fits and stays relevant

Reranking is frequently the highest-ROI addition to a struggling pipeline, because it compensates for mediocre first-stage retrieval and for embedding-model choices that matter far less once a good reranker sits behind them. And resist the urge to stuff all 100 candidates into a long context: models attend poorly to the middle of a long prompt (the “lost in the middle” effect), and every extra token is KV-cache cost and latency. Fewer, better chunks beat more, worse ones.

Grounding and Citation Faithfulness

A correct retrieval is wasted if the model ignores it or embellishes it. Faithfulness is the property that every claim in the answer is supported by the retrieved context, and it is distinct from correctness: an answer can be factually true and still unfaithful if it draws on the model’s parametric memory rather than the documents — which is exactly the behavior RAG exists to prevent, because that memory is stale and unattributable.

The engineering levers are prompt discipline and citation enforcement. Instruct the model to answer only from the provided context, to say “the documents do not contain this” when they do not, and to cite the source of each claim. Then make citation structural rather than hoped-for: tag each context chunk with an ID and require the model to reference those IDs, so you can verify post-hoc that cited spans actually support the sentences attributed to them.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

PROMPT = """Answer using ONLY the sources below. Cite the source id in [brackets]
after each claim. If the sources do not contain the answer, say so explicitly.

{sources}

Question: {question}"""

sources = "\n".join(f"[{c.id}] ({c.source}) {c.text}" for c in context)
answer = llm(PROMPT.format(sources=sources, question=q))
# Verification pass: for each cited [id], check the sentence is entailed by chunk[id].

The honest limit: you cannot fully prevent an LLM from going off-script with prompting alone. A verification pass — a second model or an NLI check confirming each sentence is entailed by its cited chunk — catches the residue, at the cost of another call. Whether you need it depends on stakes, which is, again, a measurement question.

The Evaluation Problem Nobody Solved

Here is the part teams skip and regret: you cannot improve what you cannot measure, and RAG is genuinely hard to measure because failure is distributed across stages. Did the answer go wrong because retrieval missed the chunk, because the reranker buried it, or because the generator ignored a chunk it had? A single end-to-end “is the answer good” score cannot tell you, so you instrument each stage.

Split metrics into retrieval quality and generation quality. Retrieval is evaluated against known-relevant chunks with classic information-retrieval measures. Generation is evaluated for faithfulness and relevance, increasingly with an LLM acting as judge.

Frameworks like RAGAS operationalize these, several using an LLM-as-judge under the hood:

1
2
3
4
5
6
7
8

from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_recall, context_precision

result = evaluate(
    dataset,   # questions, retrieved contexts, generated answers, ground-truth answers
    metrics=[faithfulness, answer_relevancy, context_recall, context_precision],
)
print(result)  # per-metric scores you can track across pipeline changes

Two honest caveats. First, LLM-as-judge is itself a model with biases — it favors longer answers, can be inconsistent run to run, and must be spot-checked against human labels, not trusted blindly. Second, none of this works without a golden evaluation set: a few hundred real questions with verified answers and the chunks that support them, built by hand from your actual corpus. That set is tedious to create and is the single most valuable asset in the project, because it converts “the demo feels worse today” into “context recall dropped from 0.81 to 0.62 when we changed the chunker.” Build it before you tune anything; every optimization above is a guess until you can score it.

A Production Architecture That Ships

Assemble the pieces and the system stops being a single inference call and becomes a pipeline with caching, fallbacks, and observability. Cache embeddings and rerank scores; queries repeat. Cache full answers behind a semantic cache so identical-in-meaning questions skip the whole pipeline. Add a fallback path for when retrieval returns nothing above a relevance threshold — better to say “I do not have information on that” than to answer from thin air. And log everything: the rewritten queries, the retrieved IDs and scores, the reranked set, and the final answer, because the only way to debug “why did it say that” is to replay the exact retrieval.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

def rag(question, history):
    if hit := semantic_cache.get(question):            # 1. semantic answer cache
        return hit
    queries = expand(question, history)                # 2. query transformation
    candidates = hybrid_retrieve(queries, k=100)       # 3. dense + sparse + RRF
    ranked = rerank(question, candidates)[:6]          # 4. cross-encoder, narrow
    if ranked[0].score < RELEVANCE_FLOOR:              # 5. honest fallback
        return "I don't have information on that."
    answer = generate(question, ranked)                # 6. grounded, cited
    log(question, queries, ranked, answer)             # 7. replayable trace
    semantic_cache.set(question, answer)
    return answer

None of these steps is exotic, and that is the point: production RAG is ordinary software engineering — caching, fallbacks, logging, measurement — wrapped around a few model calls, not a clever prompt. The teams that ship are the ones who treated it that way, building from the from-scratch pipeline and local document pipeline toward an instrumented system, rather than the ones who shipped the notebook and waited for the bug reports. The vector store you pick — pgvector, Qdrant, or Chroma — matters far less than whether you built the machinery around it.

Verdict

The distance between a RAG demo and a RAG product is almost entirely the unglamorous middle: chunk so the answer is retrievable, transform the question so the search hits, retrieve wide and rerank narrow so the right spans reach the model, ground and cite so the output is verifiable rather than merely fluent, and — above all — build a golden evaluation set so every change is measured instead of guessed. Skip the evaluation harness and you are not engineering a system, you are decorating one and hoping; the confident-failure property of RAG guarantees you will not notice the regression until a user does. Start with a recursive structural chunker and overlap, add hybrid retrieval and a cross-encoder reranker (the highest-ROI single change in most struggling pipelines), enforce grounded citations, and stand up retrieval and faithfulness metrics on real questions before touching anything else. Do that and RAG is a tractable engineering problem with honest trade-offs. Skip it and you have a very expensive way to generate plausible-sounding wrong answers.

Sources

• RAGAS — evaluation framework for RAG (documentation)
• HyDE — Precise Zero-Shot Dense Retrieval without Relevance Labels (Gao et al., 2022)
• Lost in the Middle: How Language Models Use Long Contexts (Liu et al., 2023)
• Anthropic — Introducing Contextual Retrieval
• Reciprocal Rank Fusion (Cormack et al., 2009)
• BEIR — heterogeneous benchmark for retrieval (Thakur et al., 2021)
• BGE reranker — FlagEmbedding (BAAI)
• LangChain — text splitters documentation
• Cohere Rerank — documentation
• sentence-transformers — Cross-Encoders