From Cloud Qdrant RAG to On-Device RAG: What Actually Transfers and What Doesn't

Short answer: The pipeline shape transfers cleanly — you still chunk, embed, do ANN retrieval, and generate. What does not transfer is scale and slack: your index size is capped by unified memory plus disk instead of a shardable cluster, your embedder shrinks to something that fits and runs locally (costing recall), and the cheap hosted re-ranker call disappears, leaving the context window as your hard budget.

I ran a Qdrant-backed search pipeline in production — vector similarity plus payload pre-filters, hydrate from Postgres, post-sort. Here is what I learned porting that mental model to a fully on-device design.

The shape transfers, the budgets don't

Chunk → embed → retrieve → rerank → generate survives the move. The four things that break are index scale (no cloud sharding), embedder size (must fit locally), the re-ranking budget (no free hosted call), and the fact that on-device removes network round-trips but caps index size at unified-memory and disk realities. Everything below is a consequence of those four.

Embeddings: the swap that quietly costs recall

Hosted models like text-embedding-3-small give you 512–1536 dims with strong quality and zero local footprint. On-device, you pick from a spectrum: Apple's zero-dependency Natural Language embeddings at one end, MLX or ModernBERT-class models at the other. VecturaKit exposes exactly this range, and the tradeoff is real — smaller local embedders lose recall, especially on paraphrase and multi-hop queries.

Two things that bit me:

• Dimension mismatch. You cannot reuse a cloud-built index with a new embedder. Different model, different vector space — you re-embed the entire corpus.
• Recall regression is silent. The pipeline still returns results; they're just worse. Build a small labeled query set and measure recall@k before and after the swap. Do not eyeball it.

Vector store: Qdrant features you'll miss

Qdrant gives you HNSW with rich payload filtering, multi-value MatchAny filters, and horizontal sharding. On-device stores like VecturaKit and ObjectBox provide genuine local ANN — fast and good for thousands to a few million vectors — but not Qdrant-scale sharding or heavily-filtered indexes.

Practical consequences:

• Complex pre-filtering (the payload MatchValue/MatchAny combinations you lean on in Qdrant) is thinner or absent. Expect to filter in application code after ANN, which means over-fetching candidates.
• There is no sharding escape hatch. When the index outgrows memory, you either quantize, prune, or move to online/streaming indexing. EdgeRAG and MobileRAG (arXiv 2412.21023, 2507.01079) do exactly this — build indexes online to stay inside device memory and energy budgets rather than holding everything resident.

Retrieval budget: no free re-ranker, context is the wall

In the cloud, a re-rank call is a rounding error — fire off 100 candidates to a hosted cross-encoder and keep the top 10. On-device that call competes for the same memory and compute as your generator. Re-ranking is still possible, but it is no longer free.

So the discipline inverts. Instead of "retrieve wide, rerank hard," you retrieve a small, high-precision candidate set and spend the context window carefully — because the context window, not an external ranking service, is now the hard constraint. Keep candidate sets small (single-digit to low-tens), and if you rerank, use a tiny model and cap the batch.

Cloud:      ANN(top 200) → hosted rerank(200→10) → LLM
On-device:  ANN(top 20)  → optional local rerank(20→6, shared budget) → local LLM

Migration checklist

• Benchmark recall first. Labeled query set, recall@k with the cloud embedder as baseline. This is your regression gate.
• Pick the embedder against the budget. Apple NL if quality allows; MLX/ModernBERT-class if you need it and can afford the load time and memory.
• Re-embed everything. New model = new vector space = full rebuild.
• Move filters into app code. Assume Qdrant-style payload filtering won't fully port; over-fetch then filter.
• Size the index to memory. If it doesn't fit, choose quantization, pruning, or online indexing (EdgeRAG/MobileRAG) — decide before shipping, not after OOM.
• Shrink the retrieval budget. Small candidate sets, optional tiny reranker, context window as the governing constraint.
• Delete network assumptions. No round-trips is a latency win; no elastic scale is the cost.

Bottom line: on-device RAG is your cloud pipeline with the slack removed — the algorithm survives, the abundance doesn't.

Related: more writing.