RAG in Production: A Failure Taxonomy
A practical taxonomy of RAG failure modes as of January 2024: retrieval misses, stale indexes, lost in the middle, and citation drift. We define each class, cite the measurements behind it, and show why retrieval and generation must be evaluated separately — with recall@k on one side and faithfulness metrics on the other.
The Demo Works and Production Does Not
Retrieval-augmented generation is almost four years old as a term — Lewis et al. coined it in May 2020 — and has been an industry default for roughly one year. In 2023, RAG became the preferred way to ground language models in private documents. The demos work reliably. Production exposes failures those demos conceal, and the incident report almost always contains the same word: hallucination.
That label diagnoses the model too quickly. In our project work at Blue IT Systems, most RAG failures begin before the model sees a single token: during ingestion, indexing, or retrieval. This article describes four recurring failure modes, cites the measurements behind them, and derives one central discipline: measure retrieval separately from generation.
A Taxonomy of Four Failure Modes
A RAG pipeline has at least four stages: ingestion, indexing, retrieval, and generation. Each stage fails differently, and a wrong answer is a symptom, not a diagnosis. The table assigns each failure mode to the stage that produces it. The assignment matters because the fixes do not transfer across stages.
| Failure mode | Stage | Symptom |
|---|---|---|
| Retrieval miss | Retrieval | The correct passage exists in the corpus but is not in the top-k |
| Stale index | Ingestion and indexing | The answer was correct at index time and is wrong today |
| Lost in the middle | Generation | The correct passage is in the context but is ignored |
| Citation drift | Generation | The cited source does not support the cited statement |
Retrieval Misses Come First
A retrieval miss means the relevant passage is indexed but does not reach the top-k. The generator then answers from parametric memory or refuses. Embedding leaderboard scores do not predict this: BEIR (Thakur et al., 2021) evaluated 10 retrieval systems on 18 datasets and found that dense retrievers trained on MS MARCO often fall behind plain BM25 under domain shift, while BM25 remains a robust zero-shot baseline.
The practical consequence is straightforward: benchmark the retriever on your corpus, not on its leaderboard. Hybrid retrieval — BM25 plus dense — with a cross-encoder reranker can recover much of the gap. It cannot repair chunking that splits an answer across boundaries, nor can it find an answer absent from the corpus.
Stale Indexes Answer Yesterday's Questions
A vector index is a snapshot. Embeddings are computed once; the source documents keep changing. A stale index therefore serves passages that were true at index time — a price list from March, a policy replaced in June. The failure is silent: retrieval metrics stay green because the passage is still relevant to the query. It is merely wrong.
The mitigations are unglamorous: incremental ingestion keyed on source timestamps, freshness budgets per source, and deletion propagation, so that removed documents also leave the index. One trap deserves naming: embedding vectors from different models are not comparable. Changing the embedding model means re-embedding the entire corpus. Plan that cost before it becomes an outage.
Lost in the Middle
Liu et al. (arXiv 2307.03172, July 2023) showed that language models use long contexts unevenly. Across GPT-3.5-Turbo, Claude 1.3, MPT-30B-Instruct, and LongChat-13B, accuracy on multi-document question answering follows a U-shaped curve: highest when the relevant passage is first or last, lowest in the middle. For GPT-3.5-Turbo, performance drops by more than 20 percent; with 20 or 30 documents, middle placement scores below the closed-book baseline of 56.1 percent.
For RAG this inverts an intuition: retrieving more documents can make answers worse. Retrieved-but-ignored is a real failure mode, and it is invisible in retrieval metrics. The lever is ordering — place the strongest passages first or last. A reranker gives you that control; a raw similarity sort does not.
Citation Drift Undermines Trust
Citation drift is the gap between what a system cites and what the citation supports. Liu, Zhang, and Liang (arXiv 2304.09848, April 2023) audited four generative search engines — Bing Chat, NeevaAI, perplexity.ai, and YouChat. On average, only 51.5 percent of generated sentences were fully supported by their citations, and only 74.5 percent of citations supported their sentence. Worse, citation precision correlated negatively with perceived utility (r = −0.96).
The lesson: citations are generated text, not provenance. A footnote does not verify itself. If your product shows sources, verify support explicitly — an entailment check between statement and cited passage — and report citation precision and citation recall as first-class metrics.
Measure Retrieval Separately From Generation
An end-to-end quality score confounds two systems. Split them. Retrieval is measured offline against labeled pairs of question and gold passage: recall@k, MRR, nDCG@10. These metrics are deterministic, cost nothing per run, and belong in CI next to the unit tests. Generation is measured with the retrieved context held fixed: faithfulness, answer relevance, and context relevance, as formalized by RAGAS (Es et al., September 2023) using an LLM as a reference-free judge.
The split also reveals where further investment still pays. In an open-domain QA case study, Liu et al. found that reader performance saturates well before retriever recall — past a certain point, more retrieval no longer improves the answer. The trade-offs remain explicit: LLM-judged metrics are noisy and consume tokens; retrieval metrics are exact but require labeling work. Production systems need both, as separate signals.
What We Expect for 2024
As of January 2024, context windows are growing fast: GPT-4 Turbo advertises 128K tokens, Claude 2.1 200K — both announced in November 2023. We do not expect long context to replace retrieval. Cost and latency scale with input tokens, and lost-in-the-middle shows that a longer window is not the same as a used window. Retrieval stays; what changes is how rigorously it must be measured.
For 2024, we expect three shifts: hybrid retrieval with reranking moves from optional optimization to default; retrieval evaluation sets become versioned repository artifacts like code; and citation verification moves from the UI into the pipeline. Some of these January 2024 predictions will age badly. One claim will not: a system you can measure only end to end is a system you cannot debug deliberately.
Sources
- Lewis et al. — Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (arXiv, May 2020)
- Thakur et al. — BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models (arXiv, April 2021)
- Liu, Zhang, Liang — Evaluating Verifiability in Generative Search Engines (arXiv, April 2023)
- Liu et al. — Lost in the Middle: How Language Models Use Long Contexts (arXiv, July 2023)
- Es et al. — Ragas: Automated Evaluation of Retrieval Augmented Generation (arXiv, September 2023)