On-Device Semantic Search on iPhone with Core ML and EmbeddingGemma

Short answer: Convert a small embedding model like EmbeddingGemma (308M params) to Core ML with coremltools, run it on the Neural Engine to turn text into vectors, and store those vectors in a lightweight Swift index such as SimilaritySearchKit. Everything — embedding, indexing, and cosine search — happens on the phone, with no network call at query time.

Why on-device: privacy, offline, zero server cost

Three reasons pushed me off a server-side embedding API for a personal-corpus search feature:

• Privacy. Notes, messages, and documents never leave the device. Nothing to log, breach, or subpoena.
• Offline. Search works on a plane or a subway. No latency floor from a round-trip.
• Zero marginal cost. No per-request embedding bill, no GPU box to keep warm.

The constraints are just as real. You have a fixed memory budget — a background-eligible app that spikes past a few hundred MB gets jetsammed by iOS. Battery matters: running a model in a tight loop on the GPU will heat the phone and drain it. And you want the Apple Neural Engine (ANE), not the CPU, doing the matrix math. Apple already ships Core ML for exactly this: it powers Photos semantic search and the on-device model behind Apple Intelligence, and it's ANE-first, which is the battery-friendly path.

Pick a small embedding model and convert it

EmbeddingGemma is the natural fit: 308M parameters, purpose-built by Google for on-device RAG and semantic search. It's small enough to fit and quality is competitive with much larger encoders on retrieval benchmarks.

Conversion is a coremltools job. Trace the model, convert, and target the ANE-friendly mlprogram format with float16 weights:

import coremltools as ct
import torch
 
# traced_model: torch.jit.trace of the encoder over a fixed seq length
mlmodel = ct.convert(
    traced_model,
    inputs=[
        ct.TensorType(name="input_ids", shape=(1, 128), dtype=int),
        ct.TensorType(name="attention_mask", shape=(1, 128), dtype=int),
    ],
    minimum_deployment_target=ct.target.iOS17,
    compute_units=ct.ComputeUnit.CPU_AND_NE,  # keep it off the GPU
    compute_precision=ct.precision.FLOAT16,
)
mlmodel.save("EmbeddingGemma.mlpackage")

Fixed sequence length matters — the ANE dislikes dynamic shapes, so pad/truncate to a constant (128 tokens covers most short docs). Bundle the tokenizer vocab too; you tokenize in Swift before the Core ML call.

Store and search vectors locally

You don't need a real vector database on a phone. For a few thousand to low tens of thousands of items, a flat cosine scan is fast and simple.

• SimilaritySearchKit is the shortcut: it gives you on-device text embeddings plus search in a few lines of Swift, with pluggable index backends. Good for prototyping and for corpora up to ~10-20k items.
• A compact custom store is what I ended up with: a contiguous [Float] (or [Int8]) buffer of all vectors, an Accelerate/vDSP dot-product loop for scoring, and a parallel array of IDs. It's maybe 40 lines and gives you full control over memory layout.

For anything larger, layer an approximate index (HNSW) on top — but measure first, because brute force over 10k × 256-dim int8 vectors is sub-millisecond.

Truncate and quantize to fit the budget

Two levers keep you inside the memory budget:

• Truncate dimensions. EmbeddingGemma supports Matryoshka representation, so you can slice the 768-dim vector down to 256 (or 128) and keep most of the retrieval quality. That's a 3x cut in stored bytes and scan time for free.
• Quantize vectors to int8. Store each dimension as a signed byte instead of a 4-byte float — another 4x reduction. At 256 dims × int8, each vector is 256 bytes, so 10k documents is ~2.5 MB of index. Normalize before quantizing so cosine stays a plain dot product.

Quantize the model weights too (float16 at minimum, palettized int4 if size is tight) so the .mlpackage itself doesn't dominate your app download.

Benchmarking latency and battery on real hardware

Simulator numbers are meaningless here — the ANE only exists on device. Measure on real hardware:

• Latency: wrap the Core ML prediction call and the search scan in signposts, view them in Instruments' Points of Interest track. On an A17/M-class chip, expect low tens of milliseconds to embed one short query.
• Compute unit: open the .mlpackage in Xcode's Core ML Performance Report to confirm ops actually land on the ANE, not fall back to CPU. Fallbacks are where latency and battery quietly regress.
• Battery: use the Energy Log in Instruments and the Xcode organizer's energy metrics. Batch-embed your corpus once (indexing), not per-query, and never re-embed on every keystroke — debounce.

Bottom line: a 308M model in Core ML, Matryoshka-truncated and int8-quantized, gives you private, offline semantic search that fits a phone's memory and battery budget.

Related: more writing.