Matryoshka Embeddings in Production: One Vector, Many Sizes

Short answer: Matryoshka Representation Learning (MRL) trains an embedding so that its early dimensions carry most of the semantic signal, which means you can truncate any vector to a shorter prefix and it stays coherent. In retrieval you exploit this by running a small truncated index for fast approximate nearest-neighbor (ANN) recall, then re-scoring the top candidates with the full-length vectors — one embedding, two sizes, no re-embedding.

What Matryoshka Representation Learning actually is

Standard embedding models spread information roughly evenly across all dimensions, so chopping a 1536-dim vector down to 256 dims destroys it. MRL changes the training objective: the loss is computed at multiple nested prefix lengths (say 64, 128, 256, 512, ... dims) at once, forcing the model to pack coarse, high-value semantics into the leading dimensions and use the tail only for refinement. Like nested Russian dolls, every prefix is a valid, usable embedding on its own.

The retention is better than intuition suggests. A 768-dim Matryoshka model truncated all the way down to 64 dims kept 98.37% of its STS benchmark performance. And because the early dimensions are so information-dense, MRL at 128 dims often matches a standard, independently-trained embedding at 512 dims — a 4x storage cut for equivalent quality.

The two-stage retrieval pattern

This is where MRL pays off in production. You store the full vectors once, but serve most of the work from a truncated index.

• Stage 1 (recall): Build the ANN index on truncated vectors — 256 dims is a strong default. Query it to pull a candidate set, typically 100-200 candidates.
• Stage 2 (re-score): Take those candidates, compute exact similarity against the full-length (e.g. 1536-dim) vectors, and return the re-ranked top-K.

Stage 1 is where latency and memory live, so making it 6x smaller is a huge win. Stage 2 only touches a couple hundred vectors, so using full precision there is cheap and restores any ranking accuracy the truncation cost you.

Which models support it, and how to verify

These are trained with MRL, so their vectors are safe to truncate:

• OpenAI text-embedding-3-small / -large — pass the dimensions parameter to get a truncated (and re-normalized) vector directly.
• Google Gemini Embedding — supports an output_dimensionality parameter.
• Nomic Embed — documents supported truncation sizes.
• Voyage — offers MRL-trained models with an output dimension option.

If a model isn't documented as MRL, verify empirically: embed a few hundred pairs, truncate to your target size, re-normalize, and compare ranking quality (e.g. Spearman on a small STS set) against the full vectors. If it collapses, the model wasn't MRL-trained and you'd need to re-embed with one that was.

Storage and latency math

Say you have 10M documents at 1536 dims, float32:

• Full index: 10M × 1536 × 4 bytes ≈ 61.4 GB.
• 256-dim index: 10M × 256 × 4 bytes ≈ 10.2 GB — a 6x reduction.

That difference decides whether the hot index fits in RAM on one box or spills to disk. ANN query cost scales roughly with the dimension count too, so distance computations get ~6x cheaper per candidate. You keep the 61 GB of full vectors on cheaper storage and only read the ~200 you need for re-scoring — a bounded, small fetch.

Gotchas

• Re-normalize after truncating. Dropping dimensions changes the vector's norm. Cosine similarity and dot-product both assume unit vectors here, so divide by the new L2 norm before comparing. (APIs that return a truncated vector via a dimensions param already do this; if you slice manually, you don't get it for free.)
• Never mix truncated and full vectors in one index. They live in different geometric spaces — a 256-dim prefix and a full 1536-dim vector aren't comparable. Keep the recall index and the re-score store as two separate collections.
• Pick the truncation size by measuring, not by vibes. Sweep 128 / 256 / 512 and plot recall@k vs. size on your own data; the STS numbers above are a guide, not a guarantee for your domain.

Bottom line: MRL lets you ship a small, fast recall index and a full-precision re-ranker from a single embedding call — measure your truncation size, and always re-normalize.

Related: more writing.