04 Jan 2025 AI Engineering5 min read

Agentic RAG: When Retrieval Loops Beat One-Shot Lookup

Agentic RAG replaces one-shot retrieval with a bounded search loop. We assess the evidence from IRCoT, Self-RAG, and CRAG, show when iterative search genuinely helps, and explain why routing, a sound index, and hard cost limits must come before the loop.

The Limits of One-Shot Retrieval

Classic RAG follows a fixed pipeline: embed the query, retrieve the top-k chunks, generate an answer. It gets one retrieval, one generation, and no second chance. That is often sufficient for factoid questions against a clean corpus. On multi-hop questions, however, it fails systematically because the correct second query emerges only from the first result. The embedding of the complete question rarely matches the document containing the next step.

The failure is troubling not only because it occurs, but because it remains invisible. When the first retrieval returns noise, the model still generates a fluent, confident, unsupported answer. A pipeline with no follow-up or second retrieval cannot recover from a poor first result. Agentic RAG adds precisely that option — at a measurable cost.

Documents are chunked, embedded and indexed — vectors plus keywords. 1/4

What Makes RAG Agentic

Anthropic's engineering guide from 19 December 2024 draws a useful line: workflows follow predefined code paths, agents direct their own process. Agentic RAG applies this distinction to retrieval. The model decides whether to retrieve, what query to issue, whether the results are sufficient, and when to stop. Retrieval becomes a tool the model calls, not a stage it passes through.

The principle predates the label. ReAct interleaved reasoning steps with actions in October 2022, IRCoT alternated chain-of-thought with retrieval in December 2022, and FLARE retrieved during generation when token confidence fell in May 2023. What changed by early 2025 is not the idea but its practicality: current models call tools reliably enough for these loops to run in production code rather than only in research harnesses.

Query Decomposition in Practice

Query decomposition splits a compound question into sub-queries and retrieves for each. "Which of our two suppliers had the higher audit score in 2023?" becomes two lookups and a comparison. The numbers are concrete: IRCoT with GPT-3 (code-davinci-002) improved retrieval recall by 11 to 21 points and downstream QA by up to 15 F1 points on HotpotQA, 2WikiMultihopQA, MuSiQue, and IIRC, while cutting factual errors in reasoning chains by up to 50 percent.

Decomposition has a narrow scope. It helps when the question is compound or multi-hop. It does nothing when the corpus lacks the answer, and it adds the latency of one LLM call plus several retrievals to every request — including simple ones, unless a router filters those out first.

Iterative Search With Self-Critique

Self-critique closes the loop. Self-RAG (October 2023) trained models to emit reflection tokens: retrieve on demand, then grade each passage for relevance and each generated segment for support. Its 7B and 13B models outperformed ChatGPT on open-domain QA and fact verification. CRAG (January 2024) added a lightweight retrieval evaluator that scores retrieved documents and triggers corrective actions, including a web-search fallback, when confidence is low.

Production teams rarely retrain a model for this pattern. The common implementation is a graded loop: retrieve chunks, have an LLM assess them against the question, rewrite the query when they fail, and search again. Anthropic describes the same structure as an evaluator-optimizer workflow. Two to three iterations resolve most recoverable failures. If the loop continues, the underlying issue is usually the corpus or index rather than the query wording.

When a Loop Beats a Single Lookup

A loop pays off chiefly under four conditions: the next query depends on intermediate results, the question and documents use different vocabulary, evidence is distributed across heterogeneous sources, or a wrong answer is costly. For factoid lookups against a well-indexed corpus, one-shot retrieval with a reranker remains cheaper and faster. Even three iterations can multiply latency by roughly three to ten times.

A loop does not repair a weak index. If chunking, embeddings, or filters are broken, every iteration retrieves the same noise, and the critique step correctly rejects it forever. Fix single-shot retrieval quality first — hybrid search, reranking, better chunking. In our projects at Blue IT Systems, the loop is the last optimization we add, not the first.

Whether that cost is justified can be measured directly. Run the same evaluation set through the one-shot and looped paths, then compare answer quality, token use, and latency per query. If the loop improves accuracy less than a reranker upgrade you have not yet shipped, deploy the reranker first.

Cost Control in Retrieval Loops

Loops multiply token spend. Each iteration re-sends the accumulated context, so cost grows faster than linearly with iteration count. Without limits, a pathological query can cost fifty times the median. Cost control is therefore part of the architecture, not an afterthought.

The main levers are caps, model tiering, and caching. Grading and query rewriting do not need a frontier model: GPT-4o mini costs $0.15 per million input tokens and $0.60 per million output tokens (July 2024), against $2.50 and $10.00 for GPT-4o. Prompt caching cuts the price of re-sent context: Anthropic bills cache reads at 10 percent of the base input price, OpenAI discounts cached input by 50 percent.

Log token counts per query and per iteration from day one. A weekly percentile report shows which queries loop and why. In our experience, a small share of queries drives most loop cost; a router that sends easy questions down the one-shot path removes most of that spend without measurable quality loss.

Outlook for 2025

From the vantage point of early January 2025, three developments are likely to shape the year. First, reasoning models such as OpenAI's o1 will absorb some of the planning currently encoded in loop logic; the loop will shrink, not disappear. Second, frameworks such as LangGraph and LlamaIndex workflows are making graded retrieval loops standard components, turning per-iteration telemetry from custom work into a default. Third, Anthropic's Model Context Protocol points toward standardized access to tools and data sources — exactly the foundation retrieval loops require.

We therefore expect most production RAG systems to contain at least one model-controlled retrieval decision by the end of 2025. Fully autonomous research agents will remain the exception in regulated European environments. Falling iteration costs do not change the engineering rule: hard stopping conditions remain mandatory. The successful teams will not be those with the longest loops, but those that measure their value as carefully as they implement them.