What RAG Pipelines Get Wrong About Retrieval

The chunking assumption nobody questions

Fixed-size chunking (500 tokens, some overlap, done) is the default in almost every RAG tutorial. It's also frequently wrong for the content it's applied to. A legal clause and a changelog entry do not have the same natural unit of meaning, and forcing both through the same chunking strategy quietly degrades retrieval quality for one of them.

The fix isn't a smarter chunker — it's chunking that respects document structure: headings, list boundaries, and table rows as first-class units instead of arbitrary token windows.

Similarity search is not relevance

Cosine similarity tells you two vectors point in a similar direction. It does not tell you one document actually answers the question in another. This distinction shows up constantly in production:

• A query about "deployment rollback" retrieves a chunk about "deployment automation" because the embeddings are close, not because the content overlaps.
• Rare, high-signal terms (a specific error code, a proper noun) get diluted in a dense embedding the same way common words do.

Hybrid search — combining dense vector similarity with sparse keyword matching (BM25-style) — closes most of this gap, and it's a much cheaper fix than fine-tuning an embedding model. The combination step is a handful of lines once you already have both retrievers:

def hybrid_search(query: str, k: int = 20) -> list[Chunk]:
    dense_hits = vector_index.search(embed(query), top_k=k)
    sparse_hits = bm25_index.search(query, top_k=k)

    # Reciprocal rank fusion: cheap, no tuned weights, hard to get badly wrong
    scores: dict[str, float] = {}
    for rank, hit in enumerate(dense_hits):
        scores[hit.id] = scores.get(hit.id, 0) + 1 / (60 + rank)
    for rank, hit in enumerate(sparse_hits):
        scores[hit.id] = scores.get(hit.id, 0) + 1 / (60 + rank)

    ranked_ids = sorted(scores, key=scores.get, reverse=True)[:k]
    return [chunk_store[i] for i in ranked_ids]

Reciprocal rank fusion is deliberately dumb — no learned weights, nothing to overfit to a validation set that doesn't represent production queries. It's usually the first thing worth trying before reaching for anything more elaborate.

Reranking is the highest-leverage thing most teams skip

A second-pass reranker over the top-k retrieved chunks, scored specifically against the query, catches the cases where "similar" and "relevant" diverge. It adds latency. It's worth it far more often than teams expect, especially for anything where a wrong answer is worse than no answer.

The pattern I keep coming back to: spend your engineering effort on retrieval quality before you spend it on prompt engineering. A great prompt over bad context still produces a bad answer.