/4 min
Running LLM Evals On-Device With MLX: Zero-Cost Batch Judging on a Mac
Yes — a quantized 3-8B judge running on MLX on Apple Silicon handles objective eval batches at effectively zero marginal cost.
Topic
On-Device & Local LLMs
MLX, Core ML, quantization, and running inference locally on Apple Silicon.
/4 min
On-Device Semantic Search on iPhone with Core ML and EmbeddingGemma
Run semantic search fully on-device on iOS: convert EmbeddingGemma to Core ML, store vectors with SimilaritySearchKit, quantize to fit memory.
/4 min
Is Local AI Actually Private? A Practical Threat Model for On-Device LLMs
Running an LLM locally removes network egress, but telemetry, model downloads, and RAG indexes still leak — here's what to check.
/3 min/reference
Core ML vs MLX for On-Device LLMs: ANE, GPU, and When Each Wins
MLX runs on-device LLMs on the GPU for throughput; Core ML can place them on the ANE to keep the GPU free — here's how to choose.