The demo was flawless. You fed it your product documentation, asked a question a real user would ask, and the answer came back precise and fluent. Then you shipped it. A week later a support ticket arrived: the model had confidently cited a pricing tier you discontinued three months ago. The retrieval had pulled the most plausibly similar document β not the most correct one.
That gap, between plausible and correct, is where RAG lives. Understanding it properly requires looking at what RAG actually does, step by step, and why naive implementations break in predictable ways.
What you are choosing between
Before the pipeline, the alternatives. Retrieval-augmented generation (Lewis et al., 2020) is not the only way to give a model access to a corpus.
Long-context stuffing drops everything β or a large slice of everything β directly into the modelβs context. It is the simplest architecture: no retrieval infrastructure, no chunking decisions, no index to maintain. It works well when the corpus is small and stable. It fails at scale: token cost rises linearly with corpus size, and attention dilutes as context grows longer. A model asked to synthesise a 200-page document whose answer lives on page 7 can miss it.
Keyword search / BM25 is the classic information-retrieval approach: match tokens, rank by term frequency and inverse document frequency. It is fast, interpretable, and surprisingly hard to beat on exact-term lookups. If your users search for invoice number INV-20240312-007, BM25 will find it. A dense embedding will blur that string into a region of meaning-space where it is surrounded by other invoice documents β close, but perhaps not the right one.
GraphRAG (a design popularised by Microsoft Research) stores the corpus as a knowledge graph and traverses edges at query time instead of doing vector lookup. It handles multi-hop questions well: βwho approved the contract signed by the person who reported to the VP of sales in Q3?β can be answered by following edges. It adds significant infrastructure and graph maintenance overhead.
Naive vector (single-vector) RAG is the simplest dense approach: chunk the corpus, embed the chunks, retrieve the top-k by cosine similarity, drop them in context. This is what most tutorials show. It is where most production failures begin.
The real pipeline
The production-grade approach β the one that addresses the most common failure modes β looks like this.
Chunk. Split your corpus into pieces the model can use. Chunk size is a real decision: too small and you lose context; too large and you retrieve a lot of noise around the answer. Overlapping windows help. Metadata tagging at chunk time (document date, section, source) pays dividends later.
Embed. Run each chunk through an embedding model to produce a dense vector. The embedding model is not a footnote: different models have different strengths across domains. A general embedding model trained on web text may underperform on legal or clinical corpora.
Retrieve β dense + BM25 in parallel. At query time, embed the query and retrieve top-k by cosine similarity (dense retrieval). Simultaneously, run BM25 over the same corpus. Dense retrieval finds semantically related chunks even when the userβs words differ from the documentβs words. BM25 finds exact tokens β model names, IDs, proper nouns β that dense retrieval blurs.
Rerank. Merge the two candidate lists and pass them to a cross-encoder reranker. A cross-encoder sees the query and a candidate together and scores their relevance jointly β it is slower than embedding similarity but significantly more accurate. The reranker does the triage that neither retrieval pass can do alone.
Ground. Drop the top reranked chunks into the modelβs context with an instruction to answer from what is provided and to say when it cannot. Grounding instructions matter: without them, the model will mix retrieved content with its parametric knowledge, and the seam is invisible.
This is the architecture the original RAG paper (Lewis et al., 2020) pointed toward and that production deployments have converged on. Hybrid retrieval plus reranking addresses the failure modes that pure vector RAG cannot.
Knowledge is not memory
One distinction worth stating plainly, because conflating them causes real design errors.
Knowledge is facts in a corpus β documents, records, policies, code. RAG retrieves knowledge: at query time it fetches relevant facts from a store and passes them to the model as context. The store does not grow because of the conversation; it grows because someone updates the corpus.
Memory is the running state of a relationship β what was said in prior turns, what a user prefers, what happened in a past session. Memory is tracked across time; knowledge is indexed across documents.
They need different stores. A knowledge base is a retrieval index (vector store, search index, graph). Memory is a session log, a preference record, a summary of past interactions. Routing a memory question to the knowledge retrieval pipeline β or vice versa β is a category error. For a deeper look at the memory side, see how agents remember.
Where even good RAG breaks
Hybrid retrieval with reranking solves the most common failures. It does not solve all of them.
Multi-hop questions. βWhat did the Q2 report say about the supplier mentioned in the renewal clause of the contract the CFO signed last year?β requires chaining: find the contract β find the clause β find the supplier β find the Q2 report β retrieve the relevant passage. Single-pass retrieval, dense or BM25, retrieves one set of chunks. It cannot follow a chain of reasoning across documents. GraphRAG was designed for exactly this; it is worth the infrastructure cost when multi-hop is the dominant query pattern.
Exact-term and ID lookups. Dense embeddings are trained to represent meaning. A product SKU or invoice ID is a token sequence with no semantic neighbourhood worth exploiting. BM25 is better here β but even BM25 can fail if the token is rare or if the index was built with aggressive normalisation. For high-precision ID lookups, a dedicated structured lookup (a database query, not a vector search) is more reliable.
Freshness. A retrieval index is only as current as its last update. If your corpus is updated at midnight and a user asks at 11 pm about an event that happened at 6 pm, the index does not have it. This is not a retrieval failure; it is an architecture failure. Streaming ingestion, incremental indexing, and cache invalidation are operational concerns that RAG does not solve by itself.
Attention dilution at top-k. Passing the top 20 retrieved chunks into context does not guarantee the model uses all of them equally. Research community findings (reported, not Felesh results) show models attend more strongly to content at the beginning and end of context than to the middle β sometimes called the βlost in the middleβ effect. A small, high-quality top-k is usually better than a large noisy one.
The choice, and where it loses
The principled default for a domain knowledge retrieval system is hybrid dense + BM25 with a rerank stage. It handles the widest range of query types, is robust to vocabulary mismatch, and does not require graph infrastructure or a very large context window.
Where it loses: it is not simple. You are running two retrieval passes instead of one, maintaining both an embedding index and a BM25 index, and adding a reranker in the serving path. Latency increases. Operational surface grows. If your corpus is small, stable, and fits in context, long-context stuffing with a single call is faster to build and cheaper to run. If your users ask mostly exact-term questions against a well-structured corpus, a search index plus BM25 alone may be enough.
The architecture that answered beautifully in the demo was probably naive vector RAG. The one that holds in production is usually not.
For the adaptation decision β when RAG is the right tool versus fine-tuning or prompt engineering β see fine-tune, RAG, or prompt. For memory architecture β the other store, the one RAG does not touch β see how agents remember.