Why do your LLM bills stay high after you picked cheap models?
Your inference bill barely moved after the downgrade to smaller models. The real lever is not which model you use, it is deciding per request which model that task actually needs.
Routing is a per-request decision, not a model choice
The ideal model for any given task depends on the complexity of the task, as InfoWorld explains. Picking the most appropriate model automatically based on the request and the context is called model routing. A router needs three things: a pool of available models, the criteria used for selecting a model, and an objective to optimize, such as cost, quality, or system performance. Vendors including Databricks, Microsoft, NVIDIA, and Snowflake each offer different implementations of this scheme.
- Router inputs: model pool, selection criteria, optimization objective
- Frontier models work well but run very expensive for coding and other tasks
You do not need an LLM to make the routing call
A counter-intuitive option is using a small, purpose-built decision model instead of a general-purpose LLM for routing. TechTarget reports on Jev, a decision model from TypeSafe AI that drew early interest from developers facing AI cost and governance challenges. Jev is designed for structured decision-making within software, rather than for text generation. It can only return nulls (how true something is), rubric scores, or multiple-choice picks. Decision models make quick determinations that developers can use to guide LLM behavior.
Sometimes the best route is away from the LLM entirely
Cutting RAG inference costs 6x starts with deciding what never reaches the LLM. Routing is not only about choosing between models. A VentureBeat practitioner account describes RAG systems in regulated settings that cut inference costs 6x by deciding what never reaches the LLM at all. Routing every ambiguous case to the language model falls apart when a compliance officer asks why a specific decision was made six months ago. The fix is a cascade architecture: easy cases follow a deterministic path, and only genuinely ambiguous ones reach the LLM. In these systems, the real cost of a wrong answer is a decision that cannot survive scrutiny, not a bad chatbot reply.
- Cascade: deterministic path for easy cases, LLM only for ambiguity
- In regulated settings, every routed decision must stay explainable under audit
What this does not solve
Routing optimizes for a measurable objective, and InfoWorld warns of the Pareto trap: improving cost can quietly degrade quality on the tasks your router misjudges. Routing also cannot fix an unclear objective. If you have not defined what quality means for your workload, per InfoWorld's framing of pool, criteria, and objective, the router will optimize the wrong thing confidently. None of these approaches removes the governance question of who is accountable when a routed decision goes wrong.
Which 20 percent of your requests actually need a frontier model, and could you prove it from logs?
If an auditor asked why a specific request was routed to a specific model, could you answer?
Pull last week's inference logs and sort requests by token cost. Flag the expensive ones a smaller model could plausibly handle.
Sources
InfoWorld: Using model routing to lower inference costs
TechTarget: Jev decision model touted as quicker, cheaper LLM alternative
Recommended for you
- LLM Engineering
Why does your LLM feature fail silently in production?
Your LLM feature has no error signal when it gets worse. Drift, hidden hallucinations, and missing monitoring let decay hide inside plausible outputs.
- LLM Engineering
When should you fine-tune instead of prompting?
You patched your prompt for the tenth time and output still drifts. Fine-tuning helps, but only after evals and prompting stop moving the needle.
- LLM Engineering
Why does prompt engineering still leave you without a product?
A clever prompt produces one good output. A product must survive users you never met, and the prompt cannot carry that weight alone.
- LLM Engineering
Why does your LLM feel slow when the latency chart looks fine?
Your p95 dashboard says the platform is healthy, but users still complain. LLM latency has two phases, and your monitoring probably only sees one.
Enjoyed this article?
Subscribe for new articles. No spam. Unsubscribe anytime.