When retrieval breaks, generation cannot save you
In 2026, 73% of RAG failures occur in retrieval, not generation. No amount of prompt or model tuning can undo it: once retrieval fetches the wrong evidence, the generation stage has no way to recover. The center of gravity for quality work must move from the prompt to retrieval. If most failures originate in retrieval, that is where observability and recovery branches belong.
The naive pipeline is not production
The consensus today is that a single chunk-embed-cosine-prompt pipeline is a prototype, not production. It embeds the query once and pushes the top-k cosine matches straight into the prompt. When the query is ambiguous, the evidence is scattered across several documents, or the top-k results are actually irrelevant, there is not a single decision point to catch it.
Agentic RAG treats retrieval as multi-step decision-making
Agentic RAG treats retrieval not as a single lookup but as multi-step decision-making: planning, a re-retrieval judgment, and inspection of intermediate evidence. It decomposes the query into a retrieval plan, assesses whether the first-round results are sufficient, rewrites the query and re-retrieves when they are not, and only hands off to generation after re-checking that the gathered evidence is enough to answer.
A full guide: from planning to operations
In planning, fix retrieval-quality targets as numbers. For example: evidence-sufficiency pass rate at or above 95%, retrieval p95 latency at or below 1.5s, at most 2 re-retrievals per query, and 100% source-match between cited evidence and the answer. Without numeric targets you cannot decide which stage failed, and the boundaries for re-retrieval and abort stay blurry.
Failure patterns concentrate in retrieval. The common ones: groundless retrieval where all top-k results are irrelevant, scattered evidence spread across documents that one lookup cannot cover, and ambiguous queries that retrieve along the wrong axis. Build three recovery branches. If the sufficiency score is below threshold, rewrite the query and re-retrieve; if sufficiency is still not reached within the cap of 2 re-retrievals, escalate to human review or drop to safe narrowing, answering only the confident part and explicitly stating the rest is unknown. If retrieval fails repeatedly or latency exceeds its cap, trigger the abort condition and do not attempt generation.
In operations, keep standard logs. Record the query, the retrieval plan, each round's top-k document IDs and scores, the sufficiency verdict, the re-retrieval count, and the final cited sources in one format, so you can reconstruct which stage failed after the fact. Apply PII masking rules to both retrieval inputs and storage so personal data never enters logs or the search index. Target zero groundless-answer violations, and tally daily any case that reached generation without evidence.
The continuous-improvement loop classifies failure logs by retrieval stage each week. Frequent groundless retrieval points to indexing and chunking strategy; frequent re-retrieval points to query-rewrite rules; frequent human escalation points to the sufficiency threshold. Validate improvements against trends in pass rate, p95 latency, and average re-retrieval count, and check that lifting one metric does not push another past its cap.
Executive summary
In short, 73% of RAG failures come from retrieval, so shift the center of gravity there. Replace the naive single pipeline with multi-step decision-making across planning, re-retrieval, and intermediate-evidence inspection; make numeric targets like a 95% pass rate and 1.5s p95 explicit alongside recovery branches for re-retrieval, human review, safe narrowing, and abort; and use standard logs with PII masking to reconstruct failures while a weekly loop keeps tuning the thresholds.