Hybrid RAG in production: what the tutorials leave out

Most RAG tutorials end at the moment the demo works: embed a few PDFs, run a cosine search, hand the top five chunks to a model, done. Production starts where that demo ends. This post covers the three decisions that matter most once real users, real documents and real latency budgets show up.

1. Retrieval is a ranking problem, not a similarity problem

Dense vectors are excellent at meaning and poor at exactness. Ask a purely dense index for ERR_CONN_RESET or an invoice number and it will happily return “network issues” in general. Lexical search (BM25) has the opposite profile.

The fix is boring and effective: run both, then fuse.

from rank_bm25 import BM25Okapi

def hybrid_search(query: str, k: int = 20, alpha: float = 0.5) -> list[Chunk]:
    dense = vector_index.search(embed(query), k=k)        # [(chunk_id, score)]
    sparse = bm25.get_top_n(tokenize(query), corpus, n=k)  # [(chunk_id, score)]

    # Reciprocal Rank Fusion: robust to differently-scaled scores.
    fused: dict[str, float] = {}
    for rank, (cid, _) in enumerate(dense):
        fused[cid] = fused.get(cid, 0) + alpha / (60 + rank)
    for rank, (cid, _) in enumerate(sparse):
        fused[cid] = fused.get(cid, 0) + (1 - alpha) / (60 + rank)

    return [chunks[cid] for cid, _ in sorted(fused.items(), key=lambda x: -x[1])[:k]]

Reciprocal Rank Fusion (RRF) deserves its popularity. It needs no score normalisation, has one constant that barely matters, and it degrades gracefully when one retriever returns garbage.

Treat the retriever as a candidate generator. Its job is recall. Precision belongs to the next stage.

2. Rerank before you generate

Twenty fused candidates are too many for the model and too noisy for the user. A cross-encoder reranker reads query and chunk together and produces a far better ordering than either embedding did alone.

Stage	Input	Output	Typical latency
Hybrid retrieval	query	20–50 candidates	30–80 ms
Cross-encoder rerank	query × candidates	top 5–8	60–150 ms
Generation	prompt + context	answer	1–4 s

Reranking is where most of the quality gain lives. It is also cheap compared with generation, so there is rarely a good reason to skip it.

A few rules that have held up:

• Keep chunks self-contained: prepend the document title and section path to every chunk before embedding.
• Rerank on the full chunk text, not on the embedding summary.
• Pass the model fewer, better chunks. Eight good chunks beat twenty mixed ones every time.

3. Build the evaluation loop before the feature

Without an evaluation set you cannot tell whether a change helped. You will “feel” that it did. That feeling is wrong roughly half the time.

Start small. Fifty question/answer pairs with the source passage marked is enough to catch regressions. Then measure two things separately:

interface EvalResult {
  recallAtK: number; // did the right passage make it into the context?
  faithfulness: number; // does the answer stick to that context?
}

export async function evaluate(cases: EvalCase[], k = 8): Promise<EvalResult> {
  let hits = 0;
  let faithful = 0;

  for (const c of cases) {
    const ctx = await retrieve(c.question, k);
    if (ctx.some((chunk) => chunk.id === c.sourceId)) hits++;

    const answer = await generate(c.question, ctx);
    if (await judge(answer, ctx)) faithful++;
  }

  return { recallAtK: hits / cases.length, faithfulness: faithful / cases.length };
}

Separating retrieval recall from answer faithfulness tells you where a failure comes from. Low recall means the index or chunking is wrong. Low faithfulness with high recall means the prompt or the model is the problem. Conflating the two is how teams spend a week tuning a prompt when the chunker was dropping tables.

What this looks like in practice

1. Hybrid retrieval with RRF, k = 30.
2. Cross-encoder rerank down to 8.
3. Context assembled with titles and section paths.
4. Every change gated by the evaluation set in CI.

None of this is novel. All of it is routinely skipped. The systems that hold up in production are the ones where somebody did the boring parts.