LUNAROPS · OPERATIONAL UPLINK 100% UPTIME 1,247d POSTS 893 JEFF.MOON@LUNAROPS.DEV UTC --:--:--

Embedding Models and Vector Search, Honestly

20 min read AI & Machine Learning
embeddingsvector-searchraghnswanninformation-retrievalmachine-learning
Contents

The single most expensive mistake in production retrieval is treating embedding choice as the lever that determines quality. It is not. An embedding model is a lossy compression function that projects text into a fixed-dimensional space where geometric proximity approximates semantic relatedness, and once you put a cross-encoder reranker behind it, the gap between a mediocre embedding and a state-of-the-art one collapses to single-digit points of nDCG. Where quality actually lives is in the division of labor: a fast, recall-oriented first stage (dense, sparse, or both) that pulls a few hundred candidates out of millions, followed by a slow, precision-oriented second stage that reorders the top fifty. Everything else — HNSW versus IVF-PQ, cosine versus dot product, 768 dimensions versus 1536 — is a set of trade-offs about latency, memory, and recall, not about whether your system understands meaning. This post is about getting those trade-offs right, with real numbers and real failure modes, and about being honest regarding where each component earns its keep.

What An Embedding Actually Represents

An embedding is the output of a learned function f: text -> R^d trained so that the dot product (or cosine) between two vectors correlates with some notion of relatedness defined by the training objective. That last clause is the part everyone skips, and it is the part that explains every weird retrieval result you have ever debugged. The geometry is not “meaning” in the abstract. It is meaning as defined by the contrastive loss the model was trained on, over the data distribution it was trained on. A model trained on (query, relevant passage) pairs from web search will place a question near its answer. A model trained on (sentence, paraphrase) pairs will place paraphrases together but may scatter a question away from its answer because they are lexically and structurally dissimilar. Same input text, completely different geometry, because the objective was different.

The dimensions themselves do not encode human-interpretable features in any clean way. There is no “tense” axis or “sentiment” axis you can read off. What the training does is arrange the space so that the directions and distances that matter for the loss are preserved. Modern bi-encoders use a contrastive objective — typically InfoNCE — that pulls positive pairs together and pushes in-batch negatives apart. The temperature parameter and the negative-sampling strategy (especially hard negatives mined from a first-pass retriever) matter enormously to the final geometry. This is why two 768-dimensional models can have wildly different retrieval behavior despite identical dimensionality: the dimensionality is the size of the canvas, the training data and loss are the painting.

Cosine similarity dominates as the metric for a concrete reason, not by convention. Most sentence-embedding models are trained with normalized vectors or with a loss invariant to magnitude, so the direction carries the signal and the length is noise (often correlated with token count or frequency, things you do not want to rank on). Cosine throws the magnitude away. If your vectors are L2-normalized, cosine similarity and dot product are monotonically equivalent and ||a - b||^2 = 2 - 2(a·b), so Euclidean, dot, and cosine all induce the same ranking. The practical takeaway: normalize at index time, then you can use the cheapest metric your index supports and stop worrying about it. The trap is mixing a model trained for cosine with an index configured for raw L2 on un-normalized vectors — you will silently rank by a blend of direction and length and your recall will quietly degrade in ways that never throw an error.

Embedding space (schematic, d reduced to 2 for sanity)

        "how do I reset my password"
                  *  <- query
                 / \
     0.91 cos   /   \  0.88 cos
               /     \
      "reset login   "forgot password
       credentials"   recovery flow"
          *              *
                                       "annual revenue grew 12%"
                                              *  <- 0.04 cos, far away
   The query sits in a neighborhood of its
   semantic relatives. ANN search = "find the
   nearest few stars without checking all of them."

The honest limitation: embeddings compress, and compression loses. A single 768-d vector cannot represent every facet of a 512-token passage. It encodes a smeared average. Specific entities, rare terms, exact numbers, and negations are exactly where dense embeddings are weakest, because the contrastive objective rewards topical similarity over lexical precision. “The drug is safe” and “the drug is not safe” embed close together. This is not a bug you can tune away; it is structural, and it is the entire reason sparse retrieval refuses to die.

Dense vs Sparse vs Hybrid: The Honest Version

Dense bi-encoders are what people mean by “embeddings”: encode query and document independently into R^d, compare with cosine. They generalize to paraphrase and synonym, they handle “stuff that means the same thing in different words,” and they are the reason semantic search feels magical compared to keyword search. Their weakness is the mirror image of their strength: they are weak on exact lexical matches, rare tokens, identifiers, error codes, product SKUs, and out-of-domain vocabulary the model never saw during training.

Sparse retrieval is the old guard and it is still excellent. BM25 is a bag-of-words scoring function over an inverted index: term frequency saturated by a k1 parameter, normalized by document length via b, weighted by inverse document frequency. It has no notion of semantics whatsoever, which is precisely why it never embarrasses itself on exact matches. If a user searches for ERR_CONNECTION_REFUSED or CVE-2024-3094, BM25 finds the document containing that exact string every time; a dense model may or may not, depending on whether that token survived its tokenizer and training. The modern evolution is learned sparse retrieval — SPLADE being the canonical example — which uses a transformer (MLM head over BERT) to predict a sparse vocabulary-sized vector with term expansion. SPLADE gets you BM25’s exact-match robustness plus learned synonym expansion (“car” lighting up “automobile”), while still producing a sparse vector you can serve from an inverted index. It is the best of both worlds for a single retriever, at the cost of a more expensive indexing-time encode and larger postings.

Hybrid is the combination, and the honest framing is: hybrid helps most when your corpus has both natural language and hard lexical anchors (code, IDs, names, jargon), and helps least on clean prose where dense already wins. The standard fusion method is Reciprocal Rank Fusion, which is beautiful because it ignores raw scores entirely (dense cosine and BM25 scores are not comparable and not calibrated) and fuses on rank position:

RRF(d) = sum over retrievers r of  1 / (k + rank_r(d))      k typically 60

A document ranked 1st by dense and 3rd by sparse beats one ranked 2nd and 50th. RRF is unreasonably effective and requires zero tuning beyond k. The alternative — min-max normalizing scores and taking a weighted sum — is more powerful when you can calibrate per-corpus but is fragile across query types.

Retriever Strength Weakness Index Typical cost
Dense bi-encoder Paraphrase, synonym, semantic Exact match, rare tokens, negation ANN (HNSW/IVF) High RAM, GPU at index time
BM25 sparse Exact lexical, rare tokens, zero training No semantics, vocabulary mismatch Inverted index Cheap, CPU only
SPLADE learned-sparse Exact match + learned expansion Expensive encode, large postings Inverted index Moderate, GPU at index time
Hybrid (RRF) Robust across query types Two pipelines, more ops surface Both Sum of both

The dishonest pitch is “always use hybrid.” The honest reality is that hybrid doubles your infrastructure (two indices, two encoders, a fusion step) and the win on the BEIR benchmark suite is real but uneven — large on datasets with entity-heavy queries, marginal on clean Q&A. If your corpus is internal documentation full of service names, config keys, and stack traces, hybrid is worth every line of code. If it is a tidy FAQ, dense alone plus a reranker will likely match it for a fraction of the operational weight. If you want the tooling-level comparison of where to actually run these indices, the companion post on pgvector, Qdrant, and Chroma covers that ground; here we stay on the algorithms.

Why HNSW and IVF-PQ Won ANN

Exhaustive (flat) search is O(N·d) per query: compute the distance to every vector. For a million 768-d vectors that is roughly 768 million multiply-adds per query, which a well-optimized SIMD kernel does in a few milliseconds, and flat search returns exact nearest neighbors. People forget this: if you have under ~100k–500k vectors, flat search is often the right answer. It is exact, it has zero index-build time, it has zero recall loss, and it is trivially correct. You graduate to approximate nearest neighbor (ANN) only when N gets large enough that linear scan blows your latency budget. ANN trades a bounded amount of recall for orders-of-magnitude speedup. The two dominant families are graph-based (HNSW) and partition-plus-compression (IVF-PQ), and they win for different reasons.

HNSW (Hierarchical Navigable Small World) builds a multi-layer proximity graph. The bottom layer connects every node to its near neighbors; each higher layer is an exponentially sparser subset that acts as an express lane. Search starts at the top, greedily walks toward the query, drops a layer, repeats. The genius is logarithmic-ish hop count: you traverse a “small world” where any two nodes are a few hops apart.

HNSW: multi-layer navigable small world

Layer 2   A -------------------- F            (sparse, long jumps)
          |                      |
Layer 1   A ----- C ----- D ---- F            (medium density)
          |       |       |      |
Layer 0   A-B-C-D-E-F-G-H-I-J-K-L-M           (every node, dense)
                      ^
        query enters at top, greedily descends,
        ef_search controls how wide the beam is at layer 0

The key knob is ef_search (size of the dynamic candidate list during the layer-0 beam search). Higher ef_search = more nodes visited = higher recall = higher latency, almost linearly. Build-time knobs are M (edges per node, controls graph degree and memory) and ef_construction (build-time beam width, controls graph quality). HNSW’s strength is excellent recall at low latency. Its honest weakness is memory: you store the full vectors plus the graph (each node holds up to M neighbor IDs per layer, and M is typically 16–64). For a million 768-d float32 vectors that is ~3 GB of raw vectors plus ~0.5–1 GB of graph — and it is RAM-resident, because graph traversal with random access patterns is death on disk. HNSW does not gracefully spill to SSD.

IVF-PQ (Inverted File with Product Quantization) attacks both the latency and the memory problem differently. IVF clusters the space into nlist Voronoi cells via k-means; at query time you only search the nprobe cells nearest the query, skipping the rest. That is the speedup. PQ then compresses each vector: split the d-dimensional vector into m subvectors, run k-means (usually 256 centroids = 8 bits) in each subspace, and store each vector as m bytes of centroid IDs. A 768-d float32 vector (3072 bytes) becomes, say, m=96 bytes — a 32x compression. Distances are computed approximately via precomputed lookup tables over the centroids (asymmetric distance computation). IVF-PQ’s strength is that it fits enormous datasets in RAM — a billion vectors at ~16–64 bytes each is feasible where flat float32 would need terabytes. Its honest weakness is recall: quantization throws away precision, and you usually need a re-ranking pass that re-scores the top candidates with full-precision vectors to recover quality.

Property Flat (exact) HNSW IVF-PQ
Recall@10 (typical) 1.00 0.95–0.99 0.80–0.95 (PQ-dependent)
Query latency (1M vecs) ~5–20 ms ~0.5–3 ms ~1–5 ms
Memory per vector (768-d) 3072 B 3072 B + graph (~3.5–4 KB total) 16–96 B (compressed)
Build time none slow (graph construction) moderate (k-means + PQ)
Main tuning knob none ef_search, M nprobe, nlist, m
Scales to ~1M ~10–100M (RAM-bound) ~1B+ (compression)
Updates trivial incremental insert OK, delete awkward retrain-ish for big shifts

The decision rule, honestly: under ~1M vectors and you care about recall, use HNSW (or flat if even smaller). Above ~50–100M vectors or when RAM is the binding constraint, IVF-PQ earns its accuracy loss. In between, it is a genuine judgment call and you should benchmark on your vectors, because recall is data-distribution-dependent and nobody else’s numbers transfer.

Tuning Knobs That Actually Move The Needle

For HNSW, the only runtime knob is ef_search, and the relationship to recall is monotone and roughly logarithmic in its payoff: going from ef_search=40 to ef_search=100 might lift recall@10 from 0.94 to 0.98 while doubling latency; going from 100 to 400 might buy you 0.98 to 0.992 while quadrupling latency again. You are buying the long tail of recall at a steep latency price. The discipline is to fix a recall target (say 0.97), sweep ef_search, and pick the smallest value that hits it on a held-out query set. Build-time M is a memory/recall trade: higher M improves recall ceiling but inflates RAM linearly. For most workloads M=16 to M=32 is the sweet spot; pushing M=64 is for when you need the last point of recall and have the RAM.

For IVF, nprobe is the recall dial: nprobe=1 searches only the single nearest cell (fast, low recall), nprobe=nlist degenerates to exhaustive search (slow, exact within the PQ approximation). A common heuristic is nlist ≈ sqrt(N) and nprobe in the 8–64 range, tuned to recall target. PQ’s m (number of subquantizers) is the memory/accuracy dial — more subquantizers means finer compression and better recall but larger codes. The combination of IVF-PQ + a full-precision re-rank of the top-k (FAISS’s IndexIVFPQ followed by a flat re-score, or the IndexRefineFlat wrapper) is the standard production recipe: cheap approximate retrieval to get candidates, exact re-scoring to fix the ordering.

Here is a concrete FAISS build with both families, so the knobs are not abstract:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

import faiss
import numpy as np

d = 768
xb = np.random.random((1_000_000, d)).astype("float32")
faiss.normalize_L2(xb)  # cosine == inner product on normalized vectors

# --- HNSW ---
hnsw = faiss.IndexHNSWFlat(d, 32)          # M = 32 edges per node
hnsw.hnsw.efConstruction = 200             # build-time beam width
hnsw.add(xb)
hnsw.hnsw.efSearch = 80                    # runtime recall/latency knob

# --- IVF-PQ ---
nlist = 4096                               # ~ sqrt(N) Voronoi cells
m = 96                                     # PQ subquantizers -> 96-byte codes
quantizer = faiss.IndexFlatIP(d)
ivfpq = faiss.IndexIVFPQ(quantizer, d, nlist, m, 8)  # 8 bits/subq = 256 centroids
ivfpq.train(xb)                            # k-means for cells AND PQ codebooks
ivfpq.add(xb)
ivfpq.nprobe = 32                          # cells to probe at query time

q = xb[:5]
D_h, I_h = hnsw.search(q, 10)
D_i, I_i = ivfpq.search(q, 10)

And the same intent expressed in pgvector, where the index lives next to your relational data and you tune the postgres-side equivalents:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE docs (
  id        bigserial PRIMARY KEY,
  body      text,
  embedding vector(768)
);

-- HNSW index; m and ef_construction are build-time, ef_search is per-session
CREATE INDEX ON docs USING hnsw (embedding vector_cosine_ops)
  WITH (m = 16, ef_construction = 200);

SET hnsw.ef_search = 80;           -- recall/latency knob, per connection

-- IVF-Flat alternative (pgvector's partition index; lists ~ rows/1000)
CREATE INDEX ON docs USING ivfflat (embedding vector_cosine_ops)
  WITH (lists = 1000);
SET ivfflat.probes = 32;           -- the nprobe equivalent

SELECT id, body
FROM docs
ORDER BY embedding <=> '[...query vector...]'   -- <=> is cosine distance
LIMIT 200;                          -- pull a wide first stage for reranking

Note the LIMIT 200, not LIMIT 10. That is the single most important line in the SQL, and it is the bridge to the part of the pipeline that actually controls quality.

Where Ranking Quality Actually Lives

Here is the claim that reorganizes most people’s mental model: in a two-stage retrieve-then-rerank pipeline, the first stage’s job is recall, not precision. It does not need to put the best document first. It needs to put the best document somewhere in the top 100–200. The reranker’s job is precision: take those 100–200 candidates and reorder them so the genuinely best ones float to the top 5–10 that go into the LLM context. Once you internalize that split, the obsession with squeezing two more nDCG points out of your embedding model looks misallocated, because the reranker is doing the precision work.

A bi-encoder embeds query and document separately and compares vectors — fast, because you precompute all document embeddings once, offline. A cross-encoder takes the query and a candidate document together as a single input to a transformer and outputs a relevance score, attending across both at once. This is dramatically more accurate because the model can see the actual interaction between query terms and document terms — but it is O(candidates) forward passes per query at runtime, so you can only afford it on a short list. You cannot cross-encode a million documents per query; you can cross-encode the 200 your cheap first stage handed you. That asymmetry is the whole architecture.

RAG retrieval pipeline: cheap-and-wide -> expensive-and-narrow

   query
     |
     v
 [ bi-encoder embed ]            [ BM25 / SPLADE ]
     |                                |
     v                                v
 [ ANN top-200 ]   --- RRF fuse ---  [ top-200 ]
                       |
                       v
              ~200 candidates  (recall stage: get the right doc IN here)
                       |
                       v
            [ CROSS-ENCODER rerank ]   <- precision lives here
                       |
                       v
                 top 5-10 reordered
                       |
                       v
              [ LLM generation ]

The empirical punchline from the BEIR line of work and countless production reports: adding a cross-encoder reranker on top of a mediocre first-stage retriever beats a state-of-the-art first-stage retriever with no reranker, on most datasets. The reranker is the great equalizer. This is why “which embedding model should I use” is the wrong first question. The right first question is “do I have a reranker,” and the answer should usually be yes. The cost is latency: a cross-encoder over 100 candidates adds 50–300 ms depending on model size and batching, which is real but usually affordable in a RAG flow where the LLM generation downstream takes seconds anyway.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

from sentence_transformers import SentenceTransformer, CrossEncoder
import numpy as np

# Stage 1: bi-encoder, run offline over the corpus, online for the query
bi = SentenceTransformer("BAAI/bge-base-en-v1.5")
query = "how do I rotate TLS certs without downtime"
q_emb = bi.encode(query, normalize_embeddings=True)

# (ANN search elsewhere returns the candidate passages; shown inline here)
candidates = [
    "Rolling cert rotation: stage the new cert, reload, then retire the old.",
    "TLS handshake overview and cipher suite negotiation.",
    "Zero-downtime certificate rotation via dual-cert serving and SIGHUP reload.",
    "How to generate a self-signed certificate with openssl.",
]

# Stage 2: cross-encoder rerank — query+doc together, true precision stage
ce = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
pairs = [(query, c) for c in candidates]
scores = ce.predict(pairs)
order = np.argsort(scores)[::-1]
reranked = [candidates[i] for i in order]
# 'Zero-downtime ... dual-cert serving' now outranks the generic
# 'rolling rotation' line, even if the bi-encoder ranked them the other way.

The honest caveats on rerankers. First, they only fix ordering within the candidate set — if the right document is not in your top-200 from stage one, no reranker can save you. This is why first-stage recall still matters and why hybrid retrieval (better recall) and reranking (better precision) are complementary, not competing. Second, reranker quality is itself domain-sensitive; the off-the-shelf MS MARCO rerankers are tuned on web search and can underperform on legal, biomedical, or code corpora, where a fine-tuned reranker pays off more than a fancier embedding ever would. Third, latency is real and compounds with candidate count, so you tune the rerank depth (rerank 50, not 500) the same way you tune ef_search. The deeper lesson connects to the broader RAG architecture — for the end-to-end build see RAG from scratch and the local RAG pipeline walkthrough — but the retrieval-quality story is fully captured by this one rule: spend your engineering budget on the two-stage split before you spend it on embedding shopping.

A Note On Embedding Dimensions, Quantization, And Cost

Dimensionality is a cost lever, not a quality lever, past a point. Going from a 384-d to a 768-d model usually buys a few points of retrieval quality; going from 768-d to 1536-d buys less and sometimes nothing, while doubling your RAM and your ANN graph size. Matryoshka representation learning changes this calculus: models trained with the MRL objective let you truncate the vector (use the first 256 of 768 dimensions) with graceful degradation, so you can store short vectors for the first stage and full vectors for re-scoring. That is a genuinely useful free lunch when the model supports it.

Scalar and binary quantization of the embeddings themselves (distinct from PQ inside the index) is the other big cost lever. Int8 scalar quantization of a float32 vector is a 4x memory cut for typically under 1 point of recall loss — almost always worth taking. Binary quantization (1 bit per dimension, Hamming distance) is a 32x cut that is shockingly serviceable for a first stage if you re-rank the top candidates with full-precision vectors; on its own it loses real recall. The pattern rhymes with everything else in this post: aggressive compression for the wide-and-cheap stage, full precision for the narrow re-scoring stage. If you want the model-weight side of the quantization story rather than the embedding side, the GPTQ/AWQ/GGUF deep dive covers that; here the relevant fact is just that quantized vectors are usually a free 4x and sometimes a careful 32x.

Verdict

Be honest about what each layer does. An embedding model is a lossy projection whose geometry encodes the relatedness its training objective defined — nothing more mystical than that, and its weaknesses on exact matches, negation, and rare tokens are structural, not tunable. Cosine wins because models are trained to put signal in direction and noise in magnitude; normalize your vectors and the metric question disappears. Dense retrieval gives you semantics, sparse gives you lexical precision, and hybrid via RRF is worth the doubled infrastructure exactly when your corpus has hard lexical anchors and not much otherwise. HNSW dominates up to tens of millions of vectors when you have the RAM and care about recall; IVF-PQ dominates at billion-scale or when RAM is the constraint, at a quantization-recall cost you claw back with a full-precision re-score. Tune ef_search and nprobe to a fixed recall target, never to a vibe.

But the load-bearing conclusion is the division of labor. First stage optimizes recall and goes wide and cheap; the cross-encoder reranker optimizes precision and goes narrow and expensive; and the reranker, not the embedding model, is where ranking quality actually lives. A reranker on a mediocre retriever beats a great retriever with none. So if you have one afternoon to spend on your RAG quality, do not spend it benchmarking embedding models. Spend it adding a reranker, widening your first-stage LIMIT to a few hundred, and measuring recall before precision. Then, and only then, go shopping for a better embedding — and you will probably find it barely moves the needle, which is exactly the point.

Sources

Comments