Back to blog

When should you skip RAG and just retrieve the answer?

Thắng Đoàn
Thắng Đoàn

You bolted retrieval onto your LLM, and answers still drift from reality. The uncomfortable finding from field reports is that the model is rarely the reason RAG fails.

RAG is plumbing, not intelligence

Forbes reports that many RAG systems fail, and the model is rarely the reason. The instinct to upgrade to a newer model treats the symptom, not the pipe. RAG, short for retrieval-augmented generation, means fetching your documents first and having the model answer from them. TechTarget notes its appeal: better factual grounding, fine-grained access control, and no retraining when your content changes. The hard part is everything around the model.

  • Retrieval irrelevance: the wrong chunks reach the model
  • Debugging complexity: failures span retrieval and generation

Grounding is a dial, not a switch

Constraining outputs to retrieved documents reduces fabrication but does not remove it. TechTarget lists eight RAG limitations, including retrieval irrelevance, latency bottlenecks, operational complexity, and residual hallucination. That last one matters for mode selection: constraining outputs to retrieved documents reduces fabrication but does not remove it. This is why how much generation you buy matters as much as which model you pick. Fewer model calls mean fewer chances to drift from the document.

  • Latency and performance bottlenecks grow with retrieval load
  • Operational and infrastructure complexity rises over time

AWS turns mode choice into one setting

Retrieval-only mode uses zero model calls and simply returns passages. That is now a first-class option. Per Unite.AI, Amazon Bedrock's AgenticRetrieveStream lets a single maxAgentIteration value choose retrieval-only, single-shot RAG, or multi-step agentic search. Single-shot RAG cuts model calls to one generation. Unite.AI also notes AWS offers no measured latency or cost comparison, so buyers must benchmark themselves. Lowering the iteration ceiling stops the agent sooner, which can reduce accuracy on complex queries.

Match the mode to the query

The AWS update turns RAG into a dial, not a binary. Simple factual lookups may need retrieval-only: return the document, skip generation. Mid-complexity questions fit single-shot RAG, one retrieval plus one answer. Only tangled, multi-hop questions justify agentic search, where the system runs multiple retrieval steps and decides the next query itself. The failure mode is defaulting every query to the most expensive mode and blaming the model when quality disappoints.

What this does not solve

Choosing modes does not fix bad retrieval, the failure Forbes and TechTarget both place at the center. Residual hallucination and debugging complexity persist in every mode, per TechTarget. And without vendor benchmarks, per Unite.AI, you still own the measurement work. Mode selection optimizes cost and latency, not correctness of what you retrieve.

What percentage of your queries could return raw passages instead of generated answers? Estimate it from a week of query logs and justify the number.
Which RAG failures in your stack are retrieval problems you have mislabeled as model problems?

Take ten query logs and label each one retrieval-only, single-shot, or agentic, then check where you are overpaying.

Sources

Forbes: RAG is an engineering problem

TechTarget: Limitations and challenges of RAG

Unite.AI: AWS adds single-shot RAG to Bedrock

Share:

Recommended for you

Enjoyed this article?

Subscribe for new articles. No spam. Unsubscribe anytime.

By subscribing you agree to receive the newsletter. No spam, and you can unsubscribe anytime.