Matryoshka Embeddings: Shorter Vectors in text-embedding-3
OpenAI’s text-embedding-3 models, released on 25 January 2024, can shorten vectors deliberately through the dimensions parameter. We explain the underlying technique, Matryoshka Representation Learning, verify the benchmark figures, and calculate storage savings at scale. We also show why moving from ada-002 still requires re-embedding the entire corpus.
Why Embedding Dimensions Became a Cost Problem
Every retrieval system built on embeddings stores one vector per chunk of text. The dimension of that vector is fixed by the model. text-embedding-ada-002, OpenAI's standard since December 2022, produces 1536 dimensions — 6,144 bytes per vector in float32. At one hundred million chunks that is 614 GB of raw vectors, before any index structure. HNSW-style indexes keep vectors in RAM, so dimension count drives memory cost, query latency, and hardware sizing in roughly linear fashion.
Until January, teams could not tailor the dimension to their use case. Post-hoc reduction methods such as PCA exist, but they add a fitted transformation that must be versioned and applied consistently to every query. They also degrade retrieval quality in ways that are difficult to predict in advance. Most teams therefore paid for 1536 dimensions regardless of actual need. The bill appeared elsewhere: larger indexes, more RAM, and slower distance computations.
What OpenAI Shipped on 25 January
On 25 January 2024 OpenAI released two embedding models. text-embedding-3-small produces 1536 dimensions and costs $0.00002 per 1,000 tokens — five times cheaper than ada-002 at $0.0001. Its MTEB average rises from 61.0 to 62.3, and the multilingual MIRACL average from 31.4 to 44.0. text-embedding-3-large produces up to 3072 dimensions at $0.00013 per 1,000 tokens, scoring 64.6 on MTEB and 54.9 on MIRACL.
The structural change is less the benchmark delta than the new dimensions API parameter. Both models can return shortened vectors, and OpenAI states that they were trained with a technique enabling a controlled trade-off between performance and cost. The announcement mentions that technique only briefly. It is Matryoshka Representation Learning, published in May 2022 — long before the product launch.
Matryoshka Representation Learning in Brief
MRL was published by Kusupati et al. (arXiv:2205.13147) and presented at NeurIPS 2022. The problem it solves: conventional training diffuses information across all dimensions of an embedding, so truncating a vector destroys its geometry. MRL adds training losses on nested prefixes of the vector — typically halving steps such as 64, 128, 256 up to the full dimension — so that the first m dimensions already form a usable embedding on their own.
The paper reports embeddings up to 14 times smaller at the same ImageNet-1K classification accuracy, with no additional cost at inference time. Information also interpolates: dimensions between the explicitly trained sizes remain meaningful. The structure is coarse-to-fine, like the nested dolls the method is named after — early dimensions carry the most general semantics, later dimensions add successively finer detail.
Shortening Vectors With Minimal Loss
With text-embedding-3 you either pass the dimensions parameter in the API call or truncate the full vector yourself. If you truncate, re-normalize the result to unit length: OpenAI's vectors ship normalized, and indexes that use the inner product as a cosine substitute rely on that property. According to OpenAI's published figures, text-embedding-3-large shortened to 256 dimensions still scores 62.0 on MTEB — above the full 1536-dimensional ada-002 at 61.0 and with twelve times fewer floats per vector.
The limits matter just as much. Token pricing does not change because billing is based on input tokens, regardless of output size. Shortened vectors also do not make different model spaces comparable. And the quality loss is small but measurable: 64.6 at 3072 dimensions versus 62.0 at 256. Whether that difference matters depends on your retrieval task, not on the benchmark.
The Storage Math at Scale
Float32 storage costs 4 bytes per dimension, and at corpus scale the arithmetic is unforgiving. The table combines OpenAI's published MTEB averages with raw vector storage for a corpus of 100 million chunks. Index overhead, replicas, and backups come on top of these numbers in every case.
Memory is usually the binding constraint, not disk. An HNSW index over 3072-dimensional float32 vectors needs the full 1.23 TB in RAM plus graph overhead; at 256 dimensions the same corpus fits in 102 GB. The MRL paper also demonstrates adaptive retrieval: shortlist candidates with a short prefix, then re-rank the shortlist with full vectors — reporting up to 14x wall-clock speedups at comparable accuracy.
| Model | Dimensions | MTEB avg | Bytes per vector (float32) | Storage for 100M vectors |
|---|---|---|---|---|
| text-embedding-ada-002 | 1536 | 61.0 | 6,144 | 614 GB |
| text-embedding-3-small | 1536 | 62.3 | 6,144 | 614 GB |
| text-embedding-3-small | 512 | 61.6 | 2,048 | 205 GB |
| text-embedding-3-large | 3072 | 64.6 | 12,288 | 1.23 TB |
| text-embedding-3-large | 1024 | 64.1 | 4,096 | 410 GB |
| text-embedding-3-large | 256 | 62.0 | 1,024 | 102 GB |
Migration Means Full Re-Indexing
Embedding spaces of different models are mutually incompatible. A query embedded with text-embedding-3-small cannot search documents embedded with ada-002, and shortened vectors of the new models do not become comparable to old ones either; there is no conversion function between spaces. Adopting the new models therefore means re-embedding every chunk in the corpus and rebuilding every index. There is no shortcut — and the two new models span different spaces even from each other.
The API bill is the smaller part: one billion tokens cost $20 with 3-small and $130 with 3-large. The operational work is more expensive — batch pipelines under rate limits, a dual-write phase for systems that must remain online, and a retrieval evaluation on your own data before switching over. MRL does provide one practical option: store the full 3072 dimensions once, and shorter indexes can later be derived by truncation without another API call.
Measure before you migrate. A few hundred annotated query-document pairs from your own domain say more than any leaderboard. Run them against the old index and the new one, at full and shortened dimensions, and let the deltas decide the configuration. In our projects this evaluation set is the most reusable artifact of a migration.
Outlook From February 2024
We expect Matryoshka training to become a standard property of embedding models rather than an OpenAI differentiator. The open-source side moved within three weeks: Nomic released nomic-embed-text-v1.5 on 14 February 2024, trained with MRL and resizable from 768 down to 64 dimensions, with open weights and open training data. At 256 dimensions it scores 61.04 on MTEB — less than a point below its full size.
Two developments seem likely from here. Vector databases will add native support for multi-resolution indexes and query-time truncation. MRL will also be combined with scalar and binary quantization, which targets the 4 bytes per dimension rather than the dimension count — multiplying the savings. Storage should therefore stop dictating model choice. What will remain is the work no launch post can do for you: measuring retrieval quality on your own corpus.
Sources
- OpenAI: New embedding models and API updates (25 January 2024)
- Kusupati et al.: Matryoshka Representation Learning, arXiv:2205.13147, NeurIPS 2022 (26 May 2022)
- Muennighoff et al.: MTEB — Massive Text Embedding Benchmark, arXiv:2210.07316 (13 October 2022)
- Nussbaum et al.: Nomic Embed — Training a Reproducible Long Context Text Embedder, arXiv:2402.01613 (2 February 2024)
- Nomic AI: nomic-embed-text-v1.5 model card (14 February 2024)