Writing

#rag #data-ingestion #chunking

Why Tables Are the #1 Silent Failure in Production RAG (and How I Fixed Mine)

Tables break RAG because flattening them destroys row/column relationships; keep them as HTML and attach headers to every row to fix it.

#rag #production #retrieval

Why RAG fails in production — and where

When RAG breaks, the model usually isn't the problem — retrieval is, about 73% of the time. Here's the real failure map and what to fix first.

#qdrant #vector-search #rag

Why Payload Filtering Is Qdrant's Killer Feature (and How Filterable HNSW Actually Works)

Qdrant applies payload filters during HNSW graph traversal, not after — so filtered queries stay fast and always return a full top-k.

Why I don't use LangChain in production RAG

Frameworks are great for a demo. In production, the abstraction you can't see into costs more than it saves.

#rag #llm #production

#retrieval #rag #hybrid-search

Why Dense Retrieval Fails on Acronyms, IDs, and Code (and How Hybrid Fixes It)

Vector search misses exact terms like SKUs and error codes because embeddings blur rare tokens; a sparse channel restores exact-match signal.

When NOT to use RAG (RAG vs fine-tuning vs prompting)

RAG became the default reflex for every LLM problem. Often it's the wrong tool. A straight decision framework for RAG vs fine-tuning vs just prompting.

#rag #llm #architecture

#fastapi #streaming #llm-latency

Streaming LLM Responses in FastAPI: SSE, Backpressure, and the Latency Traps Nobody Warns You About

Stream LLM tokens in FastAPI with SSE the moment they arrive, kill buffering layers, and cancel on disconnect so you stop paying for unseen tokens.

#llm-evals #mlx #apple-silicon

Running LLM Evals On-Device With MLX: Zero-Cost Batch Judging on a Mac

Yes — a quantized 3-8B judge running on MLX on Apple Silicon handles objective eval batches at effectively zero marginal cost.

#embeddings #core-ml #ios

On-Device Semantic Search on iPhone with Core ML and EmbeddingGemma

Run semantic search fully on-device on iOS: convert EmbeddingGemma to Core ML, store vectors with SimilaritySearchKit, quantize to fit memory.

OCR Confidence Scores: Using Them to Gate What Enters Your RAG Index

Use per-recognition OCR confidence as a metadata gate: drop, re-OCR, or flag low-confidence spans before they pollute your RAG index.

#rag #ocr #ingestion

#rag #debugging #observability

The single most useful log line in a RAG system

If you log one thing in your RAG pipeline, log the final assembled prompt. It's the fastest path from 'bad answer' to root cause.

#local-llm #privacy #on-device

Is Local AI Actually Private? A Practical Threat Model for On-Device LLMs

Running an LLM locally removes network egress, but telemetry, model downloads, and RAG indexes still leak — here's what to check.

#rag #ingestion #data-engineering

Designing an Idempotent Document Ingestion Pipeline (Content Hashing, DLQs, Reprocessing)

Make a RAG ingestion pipeline idempotent by keying every stage on a SHA-256 content hash, isolating failures in DLQs, and reprocessing deterministically.

#hybrid-search #embeddings #rag

Hybrid Search: When Pure Vector Embeddings Aren't Enough

Add sparse/keyword search to dense embeddings only when your eval shows lexical misses on IDs, rare terms, acronyms, code, or names.

#rag #debugging #production

How to debug a RAG pipeline that returns wrong answers

A step-by-step debug order for bad RAG answers — start with the assembled prompt, not the model. Most bugs are retrieval, not generation.

#rag #on-device #vector-search

From Cloud Qdrant RAG to On-Device RAG: What Actually Transfers and What Doesn't

Moving a production cloud RAG pipeline on-device: the shape transfers, but index scale, embedder size, and re-ranking budgets do not.

#agentic-search #qdrant #rag

Building Agentic Search on Qdrant: A Production Architecture Walkthrough

How to build a production agentic search system on Qdrant: single-shot vector+filter tool calls, Postgres hydration, and progressive relaxation.

On-Device OCR for RAG: Apple Vision vs Tesseract for macOS Document Pipelines

On macOS, Apple Vision beats Tesseract for RAG ingestion — faster, more accurate on real scans, free, offline. Tesseract wins on Linux CI.

#ocr #rag #ingestion

Reference & guides

#rag #evaluation #observability

Separating Retrieval Failures From Generation Failures in RAG Evaluation

Score retrieval and generation separately: the pair of metrics localizes whether a bad RAG answer is a retriever or an LLM problem.

Self-RAG vs Corrective RAG (CRAG): Which Self-Correcting Pattern Should You Use?

CRAG fixes retrieval quality before generation; Self-RAG uses fine-tuned reflection tokens to critique its own output. Here's when to use each.

#rag #agentic #llm

#llm #json #structured-outputs

How to Get Reliable JSON From LLMs: JSON Mode vs Structured Outputs vs Constrained Decoding

Structured Outputs with grammar-constrained decoding is the most reliable way to get valid JSON from an LLM. Here's how the three mechanisms compare.

Reference-Free vs Reference-Based LLM Evaluation: When You Actually Need Ground Truth

Reference-based evals compare output to a known correct answer; reference-free evals score intrinsic properties like grounding without ground truth.

#llm-eval #rag #ragas

Prompt Versioning in Production: Git vs Prompt Management Tools, and the Eval Gate That Prevents Silent Breaks

Version prompts in Git for small eng teams, move to a prompt tool when non-engineers contribute, and gate every change on an eval suite.

#prompts #llm-ops #evals

#llm #prompt-caching #cost-optimization

Prompt Caching Explained: How to Cut LLM Cost by up to 90% Without Losing Quality

Prompt caching reuses precomputed KV state for identical prompt prefixes, cutting cost up to 90% and latency up to 85% with no quality change.

July 8, 2026/2 min/reference

A reference architecture for production RAG

The stages every production RAG system needs — indexing, query-time retrieval, and always-on evaluation — and how to order them.

#rag #production #architecture

#openai #cost-optimization #llm-infra

The OpenAI Batch API: When 50% Cheaper Tokens Are Worth the 24-Hour Wait

Use the OpenAI Batch API for non-real-time bulk jobs to get 50% cheaper tokens with a 24h SLA and a separate rate-limit pool.

#rag #retrieval #metadata

Metadata Is the Unsung Hero of RAG Accuracy: What to Extract and How to Filter

Attach structural, content, and contextual metadata to chunks, then use it for filtered multi-axis retrieval to sharpen RAG accuracy.

#embeddings #semantic-search #retrieval

Matryoshka Embeddings in Production: One Vector, Many Sizes

Matryoshka Representation Learning packs meaning into early dimensions so you can truncate one embedding to many sizes for fast, cheap retrieval.

#llm #reliability #scaling

LLM Fallback Strategies: Building Multi-Model Routing That Survives Outages and Rate Limits

Build a production LLM fallback strategy: a cost-ordered chain of models with health checks, backoff, and cross-provider routing.

#rag #pdf-parsing #ingestion

Layout-Aware vs VLM Parsing: When to Pay for a Vision Model to Read Your PDFs

Use VLM parsing for scanned, complex, or merged-cell PDFs; use layout parsers for clean digital docs to save cost and latency.

July 8, 2026/2 min/reference

Hybrid search for RAG: combining BM25 and dense vectors with RRF

Why you can't average keyword and vector scores — and how Reciprocal Rank Fusion combines them for an 8–15% accuracy gain.

#rag #retrieval #search

July 8, 2026/2 min/reference

How to reduce RAG hallucinations: a practical checklist

Most RAG hallucinations are retrieval bugs in disguise. A concrete checklist to cut them — grounding, retrieval quality, and faithfulness checks.

#rag #hallucination #production

#embeddings #vector-db #rag

How to Choose an Embedding Model and Dimension for Your Vector Database

Pick an embedding model and dimension by trading MTEB quality against RAM, latency, and re-embedding cost — with a decision tree.

#rag #retrieval #chunking

How to choose chunk size and overlap for RAG

Start at 512 tokens with light overlap, then tune by query type. A practical, benchmark-backed guide to chunk size for retrieval.

#embeddings #vector-search #similarity

Cosine vs Dot Product vs Euclidean: Which Similarity Metric Should You Use?

On normalized embeddings all three metrics rank identically, so pick dot product for speed; magnitude only matters with unnormalized vectors.

Corrective RAG (CRAG) Explained: Self-Healing Retrieval Without Fine-Tuning

Corrective RAG inserts a retrieval evaluator between vector search and the LLM that scores documents and routes to proceed, filter, or re-retrieve.

#rag #agentic #retrieval

Core ML vs MLX for On-Device LLMs: ANE, GPU, and When Each Wins

MLX runs on-device LLMs on the GPU for throughput; Core ML can place them on the ANE to keep the GPU free — here's how to choose.

#mlx #core ml #on-device

Cleaning Extracted Text Before Embedding: Headers, Footers, Hyphenation, and Noise

Strip repeating headers/footers, de-hyphenate line breaks, and normalize unicode before chunking so RAG embeddings stay clean.

#rag #pdf #ingestion

#rag #chunking #retrieval

Chunk Size and Overlap for RAG: A Reference with Sensible Defaults

Use structure-defined chunks first; when you must set a size, 256-512 tokens with 10-20% overlap is a sane default, then tune.

#rag #evaluation #observability

Building a Golden Dataset for RAG Evaluation From Your Production Traces

Mine failing production traces, store input plus retrieved chunks plus expected behavior, and version 50-100 focused cases in-repo.