# Building a Custom Vector Search Engine in Rust: Performance Benchmarks
Every serious vector search project starts with the same seduction: grab the nearest database that ships an “extension,” call `CREATE EXTENSION`, and you’re searching in an afternoon. The extension is free, it speaks SQL, and your team already runs the database. What nobody tells you is that you have also signed up for a p99 latency ceiling you cannot tune, cannot prefetch around, and cannot thread. At a billion vectors under real concurrency, that ceiling is where products start to feel slow — and no amount of index tuning inside someone else’s extension will move it.
This is the story of why we stopped renting a search index inside an OLTP database and built our own HNSW engine in Rust — how the proximity graph went together with `ndarray`, what AVX-512 and Rayon actually bought us on the tail, and where we landed benchmarking against pgvector and FAISS on a billion vectors.
## The p99 Ceiling You Inherit With Generic Vector Extensions
Let’s be precise about what “generic vector extension” means and where it pinches. A Postgres extension like pgvector is not a search engine; it is a *guest* that borrows the host’s execution model. Your ANN query becomes a plan node in the Postgres executor. It competes with your OLTP workload for the same buffer pool, the same WAL, the same memory allocator, and — most painfully — the same per-backend execution context.
HNSW is a pointer-chasing workload: each level of the graph is a random walk over node neighborhoods, where every hop is a dependent load into a cache line you only discover after the previous hop resolves. That pattern rewards *dedicated* structures: memory you own, a layout you control, prefetches you issue. Inside an extension, the graph lives in pages that are subject to the buffer manager’s eviction policy, because Postgres does not know your index is hot — it knows only that pages get touched. Under checkpoint pressure or a bulk load elsewhere in the database, your graph pages get written out, and the next search takes a page fault at p99.
Then there is concurrency. Postgres gives you one backend per connection and no shared-memory compute. At 64 concurrent search clients, the graph walk’s memory-level parallelism per query collapses — everyone is thrashing the same TLB and the same LLC. The aggregate throughput story is fine; the *tail* story is not. We watched p99 drift from tens of milliseconds to hundreds under mixed load, with latency spikes correlating to vacuum, checkpoints, and nothing at all. That is the ceiling: you can buy more RAM, but you cannot buy control. A generic extension optimizes for “works everywhere,” which is the opposite of “fastest possible tail on *your* data, on *your* hardware, for *your* workload.”
FAISS, meanwhile, is not an extension — it is a C++ library — but it is also not a serving system. It gives you the kernel, then leaves you to invent the threading model, the memory lifecycle, the graceful degradation under load. That gap — kernel performance without serving control, or serving control without kernel performance — is exactly the space a custom Rust engine occupies.
## Why Rust?
If we were going to own the hot path, we were going to own everything on it. That pointed at Rust for four concrete reasons.
**Memory safety without a GC.** A p99 tail is, more than anything, a statement about unpredictable pauses. Garbage collectors stop the world; `malloc` sometimes takes a lock. Rust gives you deterministic ownership and `no_std`-adjacent discipline — you know exactly when memory is allocated and freed, and you can route all hot-path allocation through a pre-sized arena. We measure allocations per query in single digits.
**Zero-cost abstractions that survive monomorphization.** The distance kernel is generic over the storage type — `f32` for float, `i8` for quantized — and Rust monomorphizes that generic into separate, specialized functions. There is no virtual dispatch on the inner loop, no interface box, no indirection the optimizer cannot see through. A C++ template gives you the same thing; Rust gives you it with `cargo`-managed feature flags and an ecosystem that treats it as normal.
**First-class SIMD and safe-enough intrinsics.** `core::arch` exposes `_mm512` intrinsics directly, `is_x86_feature_detected!` gives runtime CPU dispatch without build-time guessing, and the portable SIMD story keeps improving. You write one AVX-512 kernel, one AVX2 fallback, and the runtime picks per machine.
**Rayon.** Parallelism on the serving path in Rust is a `.par_iter()` away with work-stealing that does not collapse under load — a sentence you cannot write about most languages without qualification.
Rust is not magic. C++ with `-march=native` and careful allocator choice matches most of this. But Rust matches it while making the concurrency and memory errors that destroy p99 *at compile time*, and that trade — equivalent performance with a fraction of the debugging tax — is why the engine is Rust.
## Building the HNSW Graph With ndarray
HNSW is conceptually simple: a multi-layer proximity graph where upper layers are sparser and longer-range, and layer 0 contains every node. Search descends greedily from the top; each layer is a best-first walk with a bounded candidate pool. The two operations that dominate are (a) neighbor selection during insert and (b) distance evaluation everywhere. Both are matrix problems in disguise, which is where `ndarray` earns its place.
We store all vectors as rows of a single `Array2
“`rust
use ndarray::{Array2, s};
use std::cmp::Reverse;
use std::collections::BinaryHeap;
struct HnswIndex {
vectors: Array2
layers: Vec
m: usize, // max outgoing edges per node
ef_construction: usize,
}
struct Layer {
// adjacency: node_id -> Vec
neighbors: Vec
}
“`
Insert is where HNSW earns its reputation for subtle bugs. The classic algorithm adds the new node to layer 0, finds its `ef_construction` nearest neighbors at each level via a greedy walk, then runs the *heuristic neighbor selection* — which deliberately keeps diverse neighbors rather than the closest ones, because a proximity graph of only-closest links degrades to a linked list. We implemented that as a loop over `vectors.rows()` with `ndarray`’s `dot`, then applied the distance-and-diversity prune against the running candidate set:
“`rust
fn search_layer(&self, query: &[f32], entry: u32, ef: usize) -> BinaryHeap
let mut visited = vec![false; self.vectors.nrows()];
let mut best = BinaryHeap::new(); // max-heap by (distance) for pruning
let mut candidates = BinaryHeap::new(); // min-heap for exploration
// … greedy descent with distance computed as a row-view dot product …
while let Some((dist, node)) = candidates.pop() {
if dist > best.peek().map(|x| x.0).unwrap_or(f32::MAX) { break; }
for &nb in &self.layers[0].neighbors[node as usize] {
if visited[nb as usize] { continue; }
visited[nb as usize] = true;
let d = 1.0 – self.vectors.row(nb as usize).dot(query); // normalized: cosine ≡ IP
if best.len() < ef || d < best.peek().unwrap().0 {
candidates.push((d, nb));
best.push(Reverse((d, nb)));
if best.len() > ef { best.pop(); }
}
}
}
best
}
“`
`ndarray`’s real payoff shows up outside the graph walk: building the scalar-quantization codebooks (per-dimension min/max via `fold_axis`), normalizing a million embeddings with a fused `mapv` pass, and — critically — batch-scoring candidate lists as real matrix multiplies when `ef_search` is large. Graph traversal is pointer chasing and stays scalar; the *arithmetic* is SIMD-friendly and belongs in `ndarray`’s lane. Getting that split right is half the performance story.
## Benchmarking: Rust vs. pgvector vs. FAISS at a Billion Vectors
Benchmarks lie by omission, so here is the whole rig. We used the Deep1B corpus — 1 billion 96-dimensional learned descriptors, the de facto standard for billion-scale ANN work — with recall@10 against the published ground truth. Hardware: a single 2-socket box with Sapphire Rapids Xeons (AVX-512, 60 physical cores total), 1 TB DDR5. Every engine stored vectors int8-quantized (SQ8) to keep the working set comparable: ~96 GB of vectors plus graph. pgvector ran on the same hardware as a dedicated Postgres 16 instance. FAISS used `IndexHNSWSQ` with 64 threads. Our engine used the same HNSW parameters (M=32, `efConstruction`=200, `efSearch`=64) and 60 Rayon threads. Queries: 128 concurrent clients, 10,000-query sample, p50/p99 reported per query.
| Engine | Storage | Recall@10 | p50 | p99 | QPS @ 128 clients |
|—|—|—|—|—|—|
| **pgvector** (HNSW, SQ8) | 96 GB + WAL/graph | 0.89 | 24 ms | 512 ms | 410 |
| **FAISS** (IndexHNSWSQ) | 96 GB + graph | 0.93 | 6.1 ms | 41 ms | 1,150 |
| **Rust engine** (pre-tune) | 96 GB + graph | 0.93 | 7.4 ms | 68 ms | 980 |
| **Rust engine** (AVX-512 + Rayon batch) | 96 GB + graph | 0.93 | 4.2 ms | 11 ms | 3,700 |
Read pgvector’s row carefully. Its *median* is respectable — 24 ms for a billion-vector ANN query from inside Postgres is genuinely not bad. Its p99 is where the extension tax shows up: 512 ms is a 21× blowup over the median. Buffer-manager evictions, WAL churn from graph updates, allocator contention across 128 backends — the tail is not search time, it is *co-tenancy time*. If your product SLA is “p99 under 50 ms,” pgvector fails by an order of magnitude no matter how many replicas you throw at it, because each replica pays the same structural tax.
FAISS is the honest baseline: with 64 threads hammering a shared index it delivers 41 ms p99. Our pre-tuned engine was actually *slower* on p50 — expected, we were a young implementation — but even then the tail was better behaved. After the optimization pass in the next section, the gap stopped being about kernels (FAISS’s distance kernels and ours are within noise of each other) and became about the serving layer: we control the threading, we batch across queries, we pin memory, and we refuse to let any single query’s graph walk monopolize a socket. That is what turned 68 ms of tail into 11 ms — and it is the exact thing a library or an extension cannot sell you.
Honesty corner: if your corpus is a few million vectors, pgvector is the right call and this post is academic. At billion scale, the engine *is* the product, and renting one you cannot tune is a p99 liability.
## Optimization: Rayon and AVX-512 on the Distance Kernel
The post-tune numbers came from two moves: vectorize the arithmetic, then parallelize the *right* granularity.
**AVX-512 on the dot product.** A 96-dimension int8 dot product is 96 multiply-accumulates per comparison — trivially SIMD-shaped. The AVX-512 kernel processes 64 int8 lanes per instruction via `_mm512_maddubs_epi16` (unsigned×signed multiply-add pairs) followed by `_mm512_sad_epu8` to widen and accumulate:
“`rust
#[target_feature(enable = “avx512f,avx512bw”)]
unsafe fn dot_i8_avx512(a: *const i8, b: *const i8, len: usize) -> i32 {
let mut acc = _mm512_setzero_si512();
let mut i = 0;
while i + 64 <= len {
let va = _mm512_loadu_si512(a.add(i) as *const __m512i);
let vb = _mm512_loadu_si512(b.add(i) as *const __m512i);
// maddubs: pairwise multiply-accumulate of u8 * i8 -> i16
acc = _mm512_add_epi32(acc, _mm512_sad_epu8(_mm512_maddubs_epi16(vb, va), _mm512_setzero_si512()));
i += 64;
}
// horizontal sum of acc, then scalar tail…
hsum_epi32(acc) + scalar_tail(a, b, i, len)
}
“`
Two practical notes from the trenches. First, `#[target_feature]` with `unsafe` is the *only* safe way to compile AVX-512 in the current stable toolchain without poisoning the whole crate with `-C target-feature` — and it keeps a clean scalar fallback for non-AVX-512 machines. Second, AVX-512 on server Xeons can downclock the core under heavy 512-bit load; because our comparisons are memory-latency-bound far more than compute-bound, the FMA throughput rarely matters. What matters is issuing 64-wide loads that fill cache lines in one pass. If you are ever tempted to obsess over FLOPs in an ANN kernel, don’t — obsess over memory-level parallelism.
**Rayon at the right granularity.** The naive parallelization — `par_iter` over the candidate set inside one query’s graph walk — is wrong: graph walks are latency-critical, lock-free, and short, and cross-thread work-stealing per hop costs more than it saves. The two places parallelism actually pays are *construction* and *batch serving*. During index build, neighbor selection for different nodes is embarrassingly parallel, so `par_iter_mut()` over the layer-0 node range turns a 14-hour build into 3.5. During serving, we batch concurrent queries — 128 clients arriving on 128 async tasks funnel into a Rayon pool where each worker owns a thread-local query buffer and the AVX-512 kernel never contends for a shared scratch space. Query throughput scales nearly linearly to the socket count, and the tail stays flat because no single query can starve the pool.
That batching layer — not the kernel, not the graph — is the real deliverable of writing your own engine.
## The OnlineCloud Stack: Low-Latency Search for E-commerce Intelligence
This engine did not come from a “let’s build a vector database” itch. It came from a product requirement: e-commerce intelligence at OnlineCloud needs semantic search over product catalogs that can grow to hundreds of millions of SKUs — embeddings for titles, descriptions, images, and behavioral signals — queried from recommendation, personalization, and agentic shopping surfaces where every millisecond is a conversion decision.
The production shape is the unglamorous one that makes the benchmarks matter. Embeddings (catalog + query-side) are refreshed by the applied-ML pipeline and normalized into the `Array2` codebook. The Rust engine runs as a lean gRPC service on dedicated instances — no co-tenancy, memory pinned, graph preloaded at boot — and is wrapped as a typed MCP tool, so the agentic surfaces (order-status bots, recommendation agents, the e-commerce intelligence layer) call `catalog_search(query, k, filters)` with the same discipline as any other tool: rate limits, audit logs, and a hard latency budget. On AWS, the whole thing sits behind the same VPC isolation and FinOps discipline OnlineCloud applies to every AI workload. The engine answers at p99 ≈ 11 ms on a billion vectors; the agents that call it never see a database checkpoint again.
We learned the ceiling the expensive way so you do not have to: generic vector extensions are a compatibility layer, not a latency strategy. At the scale where your p99 is your product, you want the hot path in a language that lets you own it — and a team that has already built it.
—
**Ready to ship search that holds its p99 under real load?** OnlineCloud is a US-based engineering studio that builds MCP-native, evaluated, production-observable systems — including custom vector search for e-commerce intelligence — and ships them in 6–10 weeks, not quarters. We would rather argue about your `efSearch` than sell you a deck. [Contact us](https://www.onlinecloud.dev/) or write to contact@onlinecloud.dev — free initial consultation, response within 24 hours.
Leave a Reply