Adaptive retrieval · benchmarked

Stop paying for the wrong retrieval strategy.

Most RAG pipelines run every query through the same expensive path. Smart Retrieval Router classifies each query and sends it to the cheapest backend that can answer it — dense, sparse, hybrid, or live web. Lower cost, lower latency, same quality.

Try the live demoStar on GitHub
−70%
retrieval cost
−40%
p95 latency
4
backends, 1 router
Live routing
$

What is RRF?

ROUTERclassifySparseBM259 msDenseVectors42 msHybridRRF61 msTavilyWeb380 ms
Routed toSparse · BM25
9 ms$0.00002

Built on a proven, production-grade stack

FastAPIPythonLangChainFAISSSentenceTransformersBM25TavilyStreamlitFastAPIPythonLangChainFAISSSentenceTransformersBM25TavilyStreamlit
The problem

Every query pays the maximum price

Traditional RAG treats a one-word lookup and a nuanced research question identically — routing both through the most expensive path. At scale, that assumption quietly burns compute, latency, and budget.

Traditional RAG cost & latency up
  1. 01
    Every query
    no matter how simple
  2. 02
    Always dense / hybrid
    one fixed pipeline
  3. 03
    Full embedding compute
    vectors + reranking, every time

≈100% cost on every request

vs
Smart Retrieval Router cost down
  1. 01
    Every query
    classified in microseconds
  2. 02
    Cheapest capable backend
    chosen per query
  3. 03
    One targeted lookup
    no wasted compute

≈30% cost — quality retained

Architecture

One decision changes the entire pipeline

The router sits between the request and retrieval. It adds microseconds of classification and removes entire lookups — the rest of your stack stays exactly the same.

User query

Arrives at the FastAPI endpoint

Query router

Heuristic classifier scores intent

DenseBM25HybridTavily

Retrieved context

Cheapest capable backend runs once

LLM

Grounds the response in context

Final answer

Accurate, cited, cost-optimized

Stateless routing · zero changes to your index or LLM · drop-in FastAPI service

Retrieval backends

Four strategies. One picks per query.

Each backend is excellent at something and wasteful at everything else. The router knows which is which — so you get the right tool without paying for all four.

Keyword

Sparse · BM25

Classic lexical scoring. Near-zero compute, unbeatable on exact terms, IDs, and rare tokens.

  • Exact keywords
  • Codes & identifiers
  • Acronyms like RRF
Cost
Lowest
Latency
~9 ms
Semantic

Dense · Vectors

Sentence-transformer embeddings over FAISS. Understands meaning and paraphrase, not just tokens.

  • Conceptual questions
  • Paraphrased intent
  • “Explain how…”
Cost
Medium
Latency
~42 ms
Fusion

Hybrid · RRF

Reciprocal Rank Fusion over dense + sparse. Highest recall when a query mixes exact terms and meaning.

  • Mixed intent
  • Long, specific queries
  • High-recall needs
Cost
High
Latency
~61 ms
Live

Tavily · Web

Live internet retrieval for anything time-sensitive or outside the local corpus. Fresh, but priced per call.

  • Breaking news
  • Prices & stocks
  • Post-cutoff facts
Cost
Metered
Latency
~380 ms
How routing works

From query to answer in five steps

No model call decides the route. A fast, transparent heuristic reads the query and commits — you can trace exactly why every request went where it did.

  1. 01

    Query arrives

    A request hits the FastAPI endpoint. Nothing is embedded yet — no compute spent.

  2. 02

    Heuristic classifier

    Lightweight signals — length, entities, temporality, token rarity — score the query in microseconds.

  3. 03

    Backend selected

    The router picks the cheapest backend predicted to answer accurately, and only that one.

  4. 04

    Retrieve context

    The chosen backend returns ranked passages. One lookup, not four.

  5. 05

    Generate answer

    The LLM grounds its response in the retrieved context and returns the final answer.

Benchmarks

Measured, not promised

Every claim ships with a reproducible harness. These are illustrative results from a 2,000-query evaluation set across all four backends and the router.

benchmark_dashboard.py
Retrieval cost
−70%
p95 latency
−40%
Quality retained
98%
Skip embeddings
41%

Cost per 1,000 queries

normalized · lower is better

Always Hybrid
100%
Always Dense
74%
Smart Router
30%

Traffic routed by backend

share of 2,000 queries

Sparse
34%
Dense
41%
Hybrid
18%
Tavily
7%

Median latency by backend

milliseconds per query

BM25
9ms
Dense
42ms
Hybrid
61ms
Tavily
380ms

Answer quality (LLM-judged)

faithfulness + relevance, 0–100

BM25
78
Dense
89
Hybrid
93
Router
91
Interactive demo

Watch the router think

Type a query or pick an example. The same heuristic that runs in production classifies it in the browser and shows you exactly where it would go — and what it would cost.

Try:
queryWhat is RRF?
ROUTERclassifySparseBM259 msDenseVectors42 msHybridRRF61 msTavilyWeb380 ms
Router decision
Sparse · BM25
latency
9 ms
cost / query
$0.00002

Why: Short, exact-term lookup for a specific acronym. Embeddings add cost, not accuracy.

Grounded answer

Reciprocal Rank Fusion combines multiple ranked lists into one, scoring each document by 1/(k + rank) across lists — the basis of the hybrid backend.

Capabilities

Everything an infra team needs to ship confidently

From the routing core to the evaluation harness and dashboard — the system is built to be measured, extended, and run in production.

Adaptive query routing

Every query is classified and dispatched to a single backend — no blanket pipeline.

Cost-aware selection

The router optimizes for spend, preferring the cheapest backend that clears the quality bar.

Dense vector search

Sentence-transformer embeddings over FAISS for semantic recall.

Sparse BM25

Lexical retrieval for exact terms, identifiers, and rare tokens.

Hybrid RRF fusion

Reciprocal Rank Fusion blends dense and sparse for maximum recall.

Live web search

Tavily handles time-sensitive queries the local corpus can't answer.

FastAPI service

A typed, production-ready HTTP API you can drop in front of any RAG stack.

Automatic benchmarking

A Parquet benchmark harness replays query sets across every backend.

LLM-based evaluation

Answers are scored for faithfulness and relevance, not just retrieval hits.

Streamlit dashboard

Interactive cost, latency, and quality analysis across backends.

Response caching

Repeat and near-duplicate queries short-circuit retrieval entirely.

Horizontally scalable

Stateless routing means you scale the API independently of the indexes.

Developer experience

A router you can read in one screen

No opaque model deciding your infrastructure spend. The classifier is a handful of transparent heuristics you can audit, extend, and unit-test — wrapped in a typed FastAPI service.

  • Drop-in FastAPI endpoint
  • Deterministic, traceable routing
  • Swap or add backends without touching callers
app/router.py
1from fastapi import FastAPI
2from router import QueryRouter, Backend
3 
4app = FastAPI()
5router = QueryRouter()
6 
7@app.post("/retrieve")
8async def retrieve(query: str) -> Answer:
9    # 1 · classify the query — microseconds, no embeddings
10    backend = router.classify(query)
11 
12    # 2 · run only the chosen backend
13    context = await backend.search(query, k=8)
14 
15    # 3 · ground the LLM in the retrieved context
16    return await llm.answer(query, context)
17 
18 
19def classify(self, q: str) -> Backend:
20    if self.is_time_sensitive(q):
21        return Backend.TAVILY      # live web search
22    if self.is_keyword_like(q):
23        return Backend.BM25        # cheapest path
24    if self.is_mixed_intent(q):
25        return Backend.HYBRID      # RRF fusion
26    return Backend.DENSE           # semantic default

The bottom line

Same answers. A fraction of the bill.

−0%
Lower retrieval cost
vs. always-hybrid baseline
−0%
Lower p95 latency
fewer redundant lookups
0
Retrieval modes
dense · sparse · hybrid · web
0%
Quality retained
LLM-judged vs. baseline

100% Python · open source · reproducible benchmark harness included

Ship smarter retrieval

Ready to optimize your RAG pipeline?

Star the repo, run the benchmark on your own query set, or reach out to talk through routing for production RAG.

Star on GitHubView demoContact