The Math of Embedding Similarity: Cosine vs. Dot Product vs. Euclidean in Production

# The Math of Embedding Similarity: Cosine vs. Dot Product vs. Euclidean in Production


Ask a RAG team what decides whether their system is good and you will hear a lot of things: the embedding model, the chunking strategy, the reranker, the prompt. Almost nobody says *the distance metric*. It sits in the config file as a one-line default — `metric=”cosine”` — inherited from a tutorial, never questioned, never benchmarked.

That is an expensive mistake. The metric defines the *neighborhood* of every query and sets the recall ceiling for everything downstream: a reranker can reorder a candidate list, but it cannot surface a chunk the metric never retrieved. When retrieval silently returns plausible-but-wrong neighbors, the failure looks like model quality — but it is usually geometry.

Here is the uncomfortable part: for most modern embedding pipelines, cosine, dot product, and Euclidean distance are *not* interchangeable — their rankings only converge under a specific condition most teams never verify. This post walks through the math of each metric, when they agree and when they diverge, how CPUs actually execute these computations, and how to pick a metric for the embedding families you are most likely to run in production: OpenAI, Cohere, and BGE.

## Cosine Similarity: Direction Is All That Matters

Every vector embedding lives in ℝ^d, but cosine similarity deliberately discards almost all of that space. It keeps exactly one property: the *angle* between two vectors.

$$\text{cosine}(a, b) = \frac{a \cdot b}{\|a\| \|b\|} = \frac{\sum_{i=1}^{d} a_i b_i}{\sqrt{\sum a_i^2} \sqrt{\sum b_i^2}}$$

The denominator is the product of the two L2 norms. Geometrically, you are projecting both vectors onto the unit hypersphere — the surface of a d-dimensional ball — and then measuring their alignment there. The result is bounded in [-1, 1]: 1.0 means the vectors point in the identical direction, 0 means they are orthogonal, and negative values mean they point away from each other. The associated *angular distance* is simply arccos(cosine), which is a true distance metric on the hypersphere (it satisfies the triangle inequality, unlike raw cosine similarity).

Why does this dominate LLM search? Because of what the denominator *removes*: magnitude. Language-model embeddings are trained so that paraphrase, topical, and semantic relationships live in *direction*. Magnitude, meanwhile, absorbs all the noise you do not want ranking on — document length, token frequency, vocabulary density, idiosyncratic writing style. Two passages that mean the same thing rarely have the same norm, but they reliably sit at similar angles from the origin. Cosine is the metric that says “I do not care how long your document is, only what it is about.” That is exactly the invariance sentence transformers and retrieval-focused LLM embeddings were trained to respect, which is why cosine has become the default in LangChain, LlamaIndex, and nearly every vector database’s documentation.

One sharp edge: most vector databases define “cosine distance” as 1 − cosine(a, b), bounded in [0, 2]. It is a monotone transform that preserves ranking, but it is not arccos — it is not a true metric and cannot be used for triangle-inequality pruning. If you threshold, know which flavor your engine implements. And because real cosine scores cluster in a narrow band near 0.7–0.95, small differences carry large ranking significance; a fixed cutoff like “0.8 = relevant” is almost always wrong across domains.

## Dot Product: When Magnitude Carries Signal

The dot product is the raw, unnormalized inner product:

$$a \cdot b = \sum_{i=1}^{d} a_i b_i = \|a\| \|b\| \cos(\theta)$$

It is unbounded — it can range from −∞ to +∞ — and it rewards vectors that are both *aligned* and *long*. That coupling is precisely the point when magnitude is learned to encode something real.

The canonical case is collaborative filtering. Matrix-factorization recommenders learn user and item vectors whose dot product predicts interaction strength, and there a vector’s norm is learned signal, not noise: popular items and active users carry larger magnitudes, and the length bonus is exactly the popularity prior the loss function optimized for. Normalize those vectors away and ranking quality measurably degrades. The same logic extends to ad ranking and e-commerce scoring, where evidence strength (clicks, purchases, confidence) is deliberately folded into magnitude so the score becomes `strength × topical alignment`.

Dot product has a second virtue: for many retrieval models it is the *native* score. Dual encoders trained with InfoNCE or in-batch negatives optimize a softmax over inner products; cosine is just the L2-normalized variant of that quantity. When a model card says “trained with cosine similarity,” what was actually maximized was almost always an inner product after normalization.

The trap: if magnitude is *not* meaningful — the common case with raw text embeddings, where norms track verbosity — dot product silently biases results toward long, generic passages. The fix is normalization, which also brings a free performance win we will get to shortly.

## Euclidean Distance (L2): The Geometry of the Space You Actually Built

Euclidean distance is the most intuitive metric and, for unnormalized embeddings, the most dangerous:

$$\|a – b\|_2 = \sqrt{\sum_{i=1}^{d} (a_i – b_i)^2}$$

Unlike cosine and dot product, L2 is a genuine metric over the full space — symmetric, non-negative, zero only for identical vectors, and it satisfies the triangle inequality. That makes it the natural choice for algorithms that *assume* metric geometry: k-means, k-medoids, and metric indexes that prune with triangle-inequality bounds all rely on centroid arithmetic only L2 provides cleanly.

But here is the failure mode. Consider u = (3, 4), v = (6, 8), and w = (4, 3). Vector v is an exact scalar multiple of u — the *same direction*, cosine similarity 1.0 — just doubled in length. Vector w has the same norm as u (both length 5) but points a different way, cosine similarity 0.96. Run the metrics:

– Cosine: v ranks above w (1.0 > 0.96) — semantically correct.
– Dot product: u·v = 50, u·w = 24 — v wins, but only because it is longer.
– Euclidean: ‖u − v‖ = 5, but ‖u − w‖ = √2 ≈ 1.41 — **L2 says w is closer**, even though w is the *less* similar direction.

That inversion is the classic unnormalized-L2 trap: it conflates “different magnitude” with “different meaning.” In text retrieval, where norms are driven by document length and term statistics, L2 on raw embeddings happily ranks a scaled copy of your topic below an unrelated vector that happens to share its norm. Use L2 and you must normalize first — and once you do, the geometry collapses into something very convenient.

For unit vectors, a simple identity connects all three metrics:

$$\|a – b\|^2 = \|a\|^2 + \|b\|^2 – 2(a \cdot b) \xrightarrow{\|a\|=\|b\|=1} 2(1 – \cos\theta)$$

On the unit hypersphere, squared Euclidean distance is a monotone function of cosine similarity. **Ranking is identical.** L2 says 0 is identical and 2 is maximally dissimilar; cosine says the same thing in reverse. Which means the metric choice on normalized embeddings is never a quality decision — it is a performance and calibration decision. That is the single most useful fact in this entire post, and it is the key to the decision framework below.

## Hardware Optimization: SIMD, AVX-512, and the Real Cost of Distance

Similarity search at production scale is not a storage problem, it is an *arithmetic* problem. Every query against a flat index is an O(N·d) scan: for each of N stored vectors you perform d multiply-accumulates. Search 1 million 1024-dimensional vectors and you stream ~4 GB of floats and issue ~2 billion fused multiply-adds — per query, before any HNSW graph traversal. The SIMD width of your CPU decides whether that takes tens of milliseconds or hundreds.

Modern x86 CPUs have been vectorizing this exact workload for two decades, through progressively wider SIMD (single-instruction, multiple-data) units:

– **SSE** (128-bit): processes 4 float32 values per instruction.
– **AVX / AVX2** (256-bit): 8 float32 values per instruction; AVX2 adds fused multiply-add (FMA), where one instruction performs `a*b+c` — the atomic operation of every distance kernel.
– **AVX-512** (512-bit): 16 float32 values per instruction, with two FMA units per core on many server parts — up to ~32 float32 FLOPs per clock cycle, versus 2 for a scalar loop. On paper that is a 16× throughput advantage over scalar code.
– **AVX-512 VNNI / AMX** (newer Xeons): packed 8-bit integer or bf16 dot products, relevant when you quantize embeddings to int8/bf16, multiplying effective throughput further.

The catch is that compute is rarely the only bottleneck. Scanning 4 GB per query at realistic memory bandwidth (~50 GB/s per socket) puts a ~80 ms floor on the naive flat scan that no vector width removes — which is exactly why real engines use IVF to shrink the candidate set and HNSW to replace brute force with graph traversal, and why cache blocking, prefetching, and data layout matter as much as SIMD width. In practice, on one modern core, AVX-512/FMA brings the *compute* cost of a large similarity scan down to roughly a tenth of scalar execution — then memory becomes the wall you actually hit.

This is why the normalization trick matters beyond correctness. Pre-normalize every vector at index time, normalize the query once, and cosine similarity *is* a plain inner product — every engine then runs each comparison as a single fused dot product with no per-vector square root or division. That drops you straight onto the tightest, most SIMD-friendly kernel the hardware offers. Libraries differ in how aggressively they exploit this: FAISS ships hand-vectorized kernels with runtime CPU dispatch, hnswlib and USearch use SSE/AVX/AVX-512 paths (USearch auto-detects and dispatches), and pgvector and Milvus’s Knowhere do the same natively. Whichever you choose, measure on your own data and CPU — AVX-512 can carry a frequency penalty on some parts, and dispatch bugs occasionally fall back to scalar silently.

## The Decision Framework: OpenAI vs. Cohere vs. BGE

With the math in place, the metric question reduces to three checks: **Is the embedding normalized? Is magnitude meaningful? What does the model card actually recommend?** Here is how the three dominant embedding families answer:

| Embedding family | Typical dims | Normalized by default? | Recommended metric | Production note |
|—|—|—|—|—|
| **OpenAI** text-embedding-3-small / 3-large | 1536 / 3072 (Matryoshka-reducible to 256–1024) | No — API returns raw vectors | Cosine (per OpenAI docs) | Normalize client-side, then inner product ranks identically and runs faster |
| **Cohere** embed-english-v3.0 / embed-multilingual-v3.0 | 1024 | **Yes** — unit-length output | Dot product | Since vectors are unit-normed, dot product ≡ cosine; use inner-product kernels |
| **BGE** bge-base/-large-en-v1.5, bge-m3 | 768 / 1024 / 1024 | No | Cosine (per model card) | Prepend the retrieval instruction prefix; L2-normalize, then inner product in FAISS/Milvus |

**OpenAI.** The `text-embedding-3` family is not normalized at the API boundary, and OpenAI’s own guidance recommends cosine similarity. Two consequences follow. First, store raw vectors with dot product and you inherit a length bias you never asked for. Second, the Matryoshka `dimensions` parameter lets you truncate to, say, 512 dims for cheaper storage and faster scans — since truncation preserves relative ordering reasonably well, re-rank the top few hundred with the full-dimension cosine.

**Cohere.** The v3 models return unit-length embeddings by construction — the API normalizes for you. That single fact dissolves the whole debate: on unit vectors, dot product and cosine produce *identical rankings*, so search with the plain inner product and let your engine run its fastest kernel. If a teammate “helpfully” re-normalizes Cohere vectors, nothing breaks — just check you are not dividing twice or storing unnormalized copies in a mixed index.

**BGE.** BAAI’s models (including bge-m3) are trained with cosine-oriented objectives and the model card says cosine — but the model card *also* tells you to prepend `”Represent this sentence for searching relevant passages:”` to short queries, and that instruction is the higher-leverage knob. For ANN search, the standard production pattern is: encode with the instruction, L2-normalize both sides, then index with inner product — you keep cosine’s exact ranking while unlocking the fastest kernels.

If you are not on one of these families: (1) check whether the training objective or model card implies normalized embeddings or magnitude-meaningful scores; (2) measure ranking agreement between cosine and inner product on ~1,000 held-out queries — and if a reranker or threshold relies on absolute scores, agreement on *order* is not enough; (3) pick the fastest kernel among the metrics that agree, and never let a library default decide for you. Benchmark on *your* chunks and query distribution with recall@k and MRR — top-1 accuracy on a toy eval set will lie every time.

## The Bottom Line

The distance metric is not a config detail; it is a modeling assumption about what your embeddings mean. Cosine says direction encodes semantics and magnitude is noise. Dot product says magnitude encodes signal — popularity, confidence, engagement. Euclidean says the space is metric in the raw-coordinate sense, which is only true after normalization. On unit vectors, all three agree on ranking and the decision becomes purely about kernel speed and score calibration. Off the unit sphere, they disagree — silently, and in ways that poison retrieval quality without ever raising an error.

At OnlineCloud, we build production retrieval systems for teams who are done guessing: evaluated RAG pipelines, vector indexes sized and tuned to real hardware, and agent architectures wired into your actual databases. If you want your embedding stack audited — or you are shipping a retrieval system next quarter and want the metric, index, and eval strategy decided by engineers who have done it in production — we should talk.

**[Start a conversation with OnlineCloud](https://www.onlinecloud.dev/)** — free initial consultation, US-based senior engineers only, no offshore handoffs, response within 24 hours at [contact@onlinecloud.dev](mailto:contact@onlinecloud.dev).