When should you skip RAG and just retrieve the answer?
You bolted retrieval onto your LLM, and answers still drift from reality. The uncomfortable finding from field reports is that the model is rarely the reason RAG fails.
RAG is plumbing, not intelligence
Forbes reports that many RAG systems fail, and the model is rarely the reason. The instinct to upgrade to a newer model treats the symptom, not the pipe. RAG, short for retrieval-augmented generation, means fetching your documents first and having the model answer from them. TechTarget notes its appeal: better factual grounding, fine-grained access control, and no retraining when your content changes. The hard part is everything around the model.
- Retrieval irrelevance: the wrong chunks reach the model
- Debugging complexity: failures span retrieval and generation
Grounding is a dial, not a switch
Constraining outputs to retrieved documents reduces fabrication but does not remove it. TechTarget lists eight RAG limitations, including retrieval irrelevance, latency bottlenecks, operational complexity, and residual hallucination. That last one matters for mode selection: constraining outputs to retrieved documents reduces fabrication but does not remove it. This is why how much generation you buy matters as much as which model you pick. Fewer model calls mean fewer chances to drift from the document.
- Latency and performance bottlenecks grow with retrieval load
- Operational and infrastructure complexity rises over time
AWS turns mode choice into one setting
Retrieval-only mode uses zero model calls and simply returns passages. That is now a first-class option. Per Unite.AI, Amazon Bedrock's AgenticRetrieveStream lets a single maxAgentIteration value choose retrieval-only, single-shot RAG, or multi-step agentic search. Single-shot RAG cuts model calls to one generation. Unite.AI also notes AWS offers no measured latency or cost comparison, so buyers must benchmark themselves. Lowering the iteration ceiling stops the agent sooner, which can reduce accuracy on complex queries.
Match the mode to the query
The AWS update turns RAG into a dial, not a binary. Simple factual lookups may need retrieval-only: return the document, skip generation. Mid-complexity questions fit single-shot RAG, one retrieval plus one answer. Only tangled, multi-hop questions justify agentic search, where the system runs multiple retrieval steps and decides the next query itself. The failure mode is defaulting every query to the most expensive mode and blaming the model when quality disappoints.
What this does not solve
Choosing modes does not fix bad retrieval, the failure Forbes and TechTarget both place at the center. Residual hallucination and debugging complexity persist in every mode, per TechTarget. And without vendor benchmarks, per Unite.AI, you still own the measurement work. Mode selection optimizes cost and latency, not correctness of what you retrieve.
What percentage of your queries could return raw passages instead of generated answers? Estimate it from a week of query logs and justify the number.
Which RAG failures in your stack are retrieval problems you have mislabeled as model problems?
Take ten query logs and label each one retrieval-only, single-shot, or agentic, then check where you are overpaying.
Sources
Forbes: RAG is an engineering problem
Recommended for you
- LLM Engineering
Why does your LLM feature fail silently in production?
Your LLM feature has no error signal when it gets worse. Drift, hidden hallucinations, and missing monitoring let decay hide inside plausible outputs.
- LLM Engineering
When should you fine-tune instead of prompting?
You patched your prompt for the tenth time and output still drifts. Fine-tuning helps, but only after evals and prompting stop moving the needle.
- LLM Engineering
Why does prompt engineering still leave you without a product?
A clever prompt produces one good output. A product must survive users you never met, and the prompt cannot carry that weight alone.
- LLM Engineering
Why do your LLM bills stay high after you picked cheap models?
You switched to smaller models and costs barely moved. The problem is not which model you picked, it is that no one decides which request deserves which model.
Enjoyed this article?
Subscribe for new articles. No spam. Unsubscribe anytime.