We built a RAG-based documentation assistant for a client's internal knowledge base, demoed it to their leadership team with five carefully chosen questions, and it nailed every single one. Confident, accurate, cited the right source documents. Two weeks after launching it to their actual support team, the honest accuracy on real questions was closer to 60%. The gap between those two numbers is basically the whole story of what production RAG actually requires beyond a LangChain quickstart tutorial.

The chunking decision we didn't think hard enough about

Our first pass chunked every document into fixed 500-token blocks with a small overlap, because that's the default in most RAG tutorials and it's genuinely fine for a demo. In production, it quietly broke retrieval on any question whose answer spanned a chunk boundary. A policy document explaining a refund process had the eligibility criteria in one chunk and the actual dollar limit in the next chunk over, and depending on exactly how the question was phrased, the retriever would sometimes grab only one of the two, and the model would confidently answer with half the information.

The fix was chunking along semantic boundaries — actual document sections, headers, and paragraphs — rather than a fixed token count, even though it made chunk sizes inconsistent and, honestly, more annoying to reason about upfront. We also started overlapping at the chunk level more generously specifically around section headers, since that's where related information tends to cluster. Retrieval quality on multi-part questions improved noticeably once chunks respected the document's own logical structure instead of an arbitrary token count that had nothing to do with the content's meaning.

Retrieval isn't the same as relevance

The vector search was doing exactly what it's designed to do — finding chunks with high semantic similarity to the question. The problem was that "semantically similar" and "actually answers the question" are not the same thing, and the gap between them is where a lot of RAG systems quietly fail. A question about "how do I cancel my subscription" would retrieve a chunk about the subscription pricing tiers, because pricing and cancellation live in similar semantic space, even though it had nothing useful to say about cancellation itself.

// What we added: a lightweight reranking step after initial retrieval
async function retrieveAndRerank(query, topK = 20, finalK = 4) {
  const candidates = await vectorStore.similaritySearch(query, topK);

  const reranked = await rerankerModel.rerank({
    query,
    documents: candidates.map(c => c.pageContent),
  });

  return reranked.slice(0, finalK).map(r => candidates[r.index]);
}

Retrieving a wider initial set of candidates and then reranking them with a model actually trained for relevance scoring, before handing only the top few to the generation step, fixed a meaningful chunk of the wrong-answer cases. It costs an extra API call and adds latency, and we still think it's worth it for anything beyond a toy demo, because the alternative is a confidently wrong answer, which is worse than a slow correct one.

The failure mode nobody demos: confident wrong answers on missing information

The scariest bug wasn't retrieval quality, it was what happened when the knowledge base genuinely didn't contain the answer. Left to its own devices, the model would often generate a plausible-sounding answer anyway, extrapolating from adjacent context rather than saying "I don't know." For a documentation assistant, a confidently wrong answer is worse than no answer at all, because the user has no reason to doubt it.

We fixed this with an explicit instruction in the generation prompt to only answer from the provided context and to say so clearly when the context doesn't contain a real answer, combined with a similarity score threshold on retrieval — if the best-matching chunk's relevance score falls below a level we tuned against real failure cases, the system says it doesn't have enough information rather than guessing, and offers to route the question to a human. That threshold took real tuning against actual production queries, not the demo's five hand-picked questions, to land somewhere useful.

What we measure now that we didn't at first

We track three things continuously in production that never came up during the demo phase: retrieval hit rate against a growing set of real user questions with known correct source documents, the rate of "I don't know" responses (too high means retrieval is too conservative, too low means it's probably hallucinating on gaps), and thumbs-down feedback correlated with which document chunks were retrieved for that answer. That last one is what actually tells you which parts of the knowledge base need better chunking or better source documents, not guesswork.

The real lesson

A RAG demo tests whether the pipeline works at all. Production tests whether it works on the actual, messy, unpredictable range of things real users ask, phrased in ways you didn't anticipate, about edge cases your five demo questions never touched. Budget real time for chunking strategy, reranking, and honest "I don't know" handling. That's where production RAG systems are actually won or lost, not in the initial proof of concept.