Local Vector Search for Homelab RAG: pgvector vs Qdrant vs Chroma

Running a private retrieval-augmented generation stack means keeping every piece on your own hardware: the language model, the embedding model, and the vector store that ties them together. Most tutorials assume you will reach for a managed API or a hosted vector database, but if your threat model includes data leaving the machine — or you simply refuse to pay per-token for something a modest server can do — you need a stack that works completely offline. That is what this post is about.

The companion piece Vector Databases in Production covers the at-scale story: managed Pinecone, fully cloud Weaviate, production-grade Qdrant clusters, and pgvector inside managed Postgres services. This post takes the opposite angle. We are talking homelab hardware — a workstation, a repurposed server, a NUC, maybe a single-GPU box — and we want to choose the right local vector store for that context. The three realistic candidates are Chroma (dead-simple, embedded, Python-native), pgvector (vector search inside the PostgreSQL instance you already run), and Qdrant (purpose-built Rust engine, containerised, the choice when the other two start creaking). Everything — embedding generation included — stays on the box.

What Vector Search Is and Why RAG Needs It

When you embed a chunk of text with a model like nomic-embed-text or BAAI/bge-m3, you get back a fixed-length array of floating-point numbers — a vector that encodes the semantic content of that text as a point in a high-dimensional space. Two chunks about the same topic land near each other; two chunks about unrelated subjects land far apart. A vector store lets you index millions of these points and then, given a query vector, return the k nearest neighbours in milliseconds.

The similarity metric you use matters. Cosine similarity normalises for vector magnitude and is the most common choice for text embeddings. Dot product is faster and equivalent to cosine when vectors are already unit-normalised. Euclidean (L2) distance is used less often for text. Exact nearest-neighbour search over large collections is prohibitively slow, so every production-grade store uses an approximate algorithm. The dominant one is HNSW (Hierarchical Navigable Small World): a multi-layer proximity graph built at index time that trades a small, tunable recall loss for O(log n) query time. All three stores covered here support HNSW.

RAG plugs vector search into the LLM loop: at query time you embed the user’s question, retrieve the top-k most relevant passages, inject them into the prompt as context, and let the model synthesise an answer. If you want to go deeper on the pipeline itself, the posts Building a Local RAG Pipeline and RAG from Scratch cover chunking strategy, reranking, and full runnable implementations in detail. Here we focus on the vector store layer specifically.

Generating Embeddings on Your Own Hardware

Before picking a store, sort out the embedding model, because model choice dominates retrieval quality far more than which store you run.

For a homelab setup the two best paths are:

Ollama’s embedding endpoint. If you are already running local LLMs with Ollama, pull an embedding model alongside your chat model and hit its HTTP API. nomic-embed-text produces 768-dimensional vectors, fits comfortably on CPU, and outperforms OpenAI text-embedding-ada-002 on both short and long-context tasks despite its small footprint. Pull it once:

1

ollama pull nomic-embed-text

Then generate embeddings from Python without any external dependency:

1
2
3
4
5
6
7
8
9

import httpx

def embed(text: str) -> list[float]:
    resp = httpx.post(
        "http://localhost:11434/api/embeddings",
        json={"model": "nomic-embed-text", "prompt": text},
    )
    resp.raise_for_status()
    return resp.json()["embedding"]

sentence-transformers directly. For offline-first setups where you do not want even a local HTTP hop, load the model directly:

1
2
3
4
5
6

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("nomic-ai/nomic-embed-text-v1", trust_remote_code=True)

def embed(text: str) -> list[float]:
    return model.encode(text, normalize_embeddings=True).tolist()

For higher recall on mixed-language corpora or longer documents, BAAI/bge-m3 (568M params) is worth the extra VRAM — it supports dense, sparse, and multi-vector retrieval from a single model. mxbai-embed-large-v2 is a lighter English-only alternative with good quantisation behaviour on CPU. All three are permissively licensed (MIT or Apache 2.0) and weights are available on Hugging Face for fully air-gapped pulls.

Now, on to the stores.

Chroma: The Embedded Starting Point

Chroma is the easiest vector store to get running. It ships as a Python package, stores everything to a local directory via SQLite and a simple file layout, and requires no external process. You install it, point it at a path, and start ingesting. That simplicity is genuinely valuable when you are prototyping a RAG workflow and do not want to context-switch into infrastructure.

1

pip install chromadb

A persistent local collection in about fifteen lines:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

import chromadb
from chromadb.utils import embedding_functions

# Persist to disk — survives process restarts
client = chromadb.PersistentClient(path="./chroma_db")

# Use a local sentence-transformers model; no API key needed
ef = embedding_functions.SentenceTransformerEmbeddingFunction(
    model_name="nomic-ai/nomic-embed-text-v1"
)

collection = client.get_or_create_collection(
    name="docs",
    embedding_function=ef,
    metadata={"hnsw:space": "cosine"},
)

# Ingest
collection.add(
    documents=["PostgreSQL is a relational database.", "Qdrant is written in Rust."],
    metadatas=[{"source": "wiki"}, {"source": "blog"}],
    ids=["doc-1", "doc-2"],
)

# Query
results = collection.query(query_texts=["fast vector search engine"], n_results=2)
print(results["documents"])

Chroma handles embedding automatically when you attach an embedding function to the collection, which keeps the ingest loop very clean. You can also supply pre-computed vectors directly via the embeddings= argument if you are managing embedding generation yourself (recommended once you move beyond prototyping, since it gives you more control over the model and batching).

Metadata filtering works with a where= dict that supports $eq, $ne, $in, $gt, $gte, $lt, $lte, and $and/$or combinators:

1
2
3
4
5

results = collection.query(
    query_texts=["vector indexing"],
    n_results=5,
    where={"source": {"$eq": "blog"}},
)

Chroma also ships a client-server mode (a lightweight FastAPI server) and, as of 2026, Chroma Cloud is generally available for teams who want a managed option. For local use, though, the embedded mode is the story.

When Chroma stops being enough. Chroma is backed by SQLite for persistence, which means it is single-writer and does not handle heavy concurrent ingest well. At collections of a few hundred thousand documents it begins to feel slow on HNSW rebuilds, and metadata filtering against large collections can scan more than you would expect. The embedding function abstraction is convenient but can obscure performance — if you are ingesting a million chunks you want batched, async embedding outside Chroma’s control. There is also no native sparse-vector or hybrid (dense + keyword) search. For a personal knowledge base, a private documentation corpus, or a proof-of-concept pipeline under a few hundred thousand vectors, Chroma is the right choice. Beyond that, graduate to one of the other two.

pgvector: Vector Search Inside Your Existing Postgres

If you already operate a PostgreSQL instance — and if you run a homelab seriously enough to care about this post, there is a decent chance you do — pgvector adds vector search as a first-class Postgres extension. You get vector columns, HNSW indexes, and similarity operators sitting right next to your normal relational data, transactions, foreign keys, and row-level security. That “one fewer service” argument is real.

pgvector 0.8.2 (released February 2026, patching a parallel HNSW index build issue) is the current stable release. The simplest way to run it locally is the official Docker image:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

# docker-compose.yml
services:
  postgres:
    image: pgvector/pgvector:pg17
    environment:
      POSTGRES_USER: raguser
      POSTGRES_PASSWORD: ragpass
      POSTGRES_DB: ragdb
    ports:
      - "5432:5432"
    volumes:
      - pgdata:/var/lib/postgresql/data

volumes:
  pgdata:

1

docker compose up -d

Enable the extension and create a table:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE documents (
    id        BIGSERIAL PRIMARY KEY,
    content   TEXT NOT NULL,
    source    TEXT,
    embedding VECTOR(768)          -- match your model's output dimension
);

-- Build an HNSW index; tune m and ef_construction for your recall/speed target
CREATE INDEX ON documents
    USING hnsw (embedding vector_cosine_ops)
    WITH (m = 16, ef_construction = 64);

Ingest and query from Python with psycopg:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

import psycopg
import numpy as np

DSN = "postgresql://raguser:ragpass@localhost:5432/ragdb"

def ingest(content: str, source: str, vec: list[float]) -> None:
    with psycopg.connect(DSN) as conn:
        conn.execute(
            "INSERT INTO documents (content, source, embedding) VALUES (%s, %s, %s)",
            (content, source, vec),
        )

def search(query_vec: list[float], k: int = 5, source_filter: str | None = None):
    sql = """
        SELECT content, source, 1 - (embedding <=> %s::vector) AS score
        FROM documents
        {where}
        ORDER BY embedding <=> %s::vector
        LIMIT %s
    """.format(where="WHERE source = %s" if source_filter else "")

    params = [query_vec, query_vec, k]
    if source_filter:
        params.insert(2, source_filter)

    with psycopg.connect(DSN) as conn:
        return conn.execute(sql, params).fetchall()

The <=> operator is cosine distance; <#> is negative dot product; <-> is L2. Because your metadata lives in normal Postgres columns you can build any index Postgres supports — B-tree, GIN for full-text, partial indexes — and combine them with vector search in a single query. This is a significant advantage over stores that treat metadata filtering as a second-class afterthought.

HNSW tuning. The m parameter controls the number of edges per node in the graph (default 16; raising it improves recall at the cost of more memory and slower build time). ef_construction controls the search width during index build (default 64; higher values give better recall but slower builds). At query time, set SET hnsw.ef_search = 100; (or higher) to trade latency for recall on filtered queries. Version 0.8.0 introduced hnsw.iterative_scan, which re-scans the graph when filtering would otherwise leave you with too few results — critical for highly selective WHERE clauses.

Honest downsides. HNSW index builds in Postgres are single-threaded by default (parallel builds were added in 0.7.x but the parallel build had a CVE addressed in 0.8.2, so read the release notes). Index maintenance on bulk inserts can temporarily degrade query recall until the index is fully updated. There is no native sparse-vector support — you can fake keyword search with Postgres full-text search alongside the vector column, but joining the results is left to your query logic. For a homelab, though, the operational simplicity of one fewer service and the power of SQL for metadata filtering make pgvector an excellent choice for collections up to a few million vectors.

Qdrant: When You Want the Purpose-Built Engine

Qdrant is a vector search engine written entirely in Rust, and it shows. It handles millions of vectors on modest hardware, its payload (metadata) filtering is deeply integrated into the HNSW traversal rather than bolted on as a post-filter, and it natively supports hybrid search by storing both dense and sparse vectors in the same collection and fusing results at query time. The trade-off is that it is another service to operate and tune.

The current stable release is v1.17.1. Run it as a container:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

# docker-compose.yml
services:
  qdrant:
    image: qdrant/qdrant:v1.17.1
    ports:
      - "6333:6333"   # REST API
      - "6334:6334"   # gRPC
    volumes:
      - qdrant_storage:/qdrant/storage

volumes:
  qdrant_storage:

1

docker compose up -d

Create a collection and ingest with the Python client:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

from qdrant_client import QdrantClient
from qdrant_client.models import (
    Distance, VectorParams, PointStruct
)

client = QdrantClient(host="localhost", port=6333)

client.recreate_collection(
    collection_name="docs",
    vectors_config=VectorParams(size=768, distance=Distance.COSINE),
)

# Ingest — payload is arbitrary JSON, indexed automatically
client.upsert(
    collection_name="docs",
    points=[
        PointStruct(
            id=1,
            vector=embed("PostgreSQL is a relational database."),
            payload={"source": "wiki", "year": 2024},
        ),
        PointStruct(
            id=2,
            vector=embed("Qdrant is written in Rust."),
            payload={"source": "blog", "year": 2026},
        ),
    ],
)

Query with payload filtering — filters are applied inside the HNSW traversal, not after:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

from qdrant_client.models import Filter, FieldCondition, MatchValue, SearchRequest

results = client.search(
    collection_name="docs",
    query_vector=embed("fast vector search engine"),
    query_filter=Filter(
        must=[FieldCondition(key="source", match=MatchValue(value="blog"))]
    ),
    limit=5,
)

Hybrid search with dense and sparse vectors. Qdrant’s native sparse-vector support lets you store BM25 or SPLADE sparse vectors alongside dense embeddings and fuse them with Reciprocal Rank Fusion at query time. With BAAI/bge-m3 you can generate both from a single model pass:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

from qdrant_client.models import (
    SparseVector, SparseVectorParams, NamedVector,
    NamedSparseVector, Prefetch, Query, FusionQuery, Fusion
)

# Collection with both dense and sparse named vectors
client.recreate_collection(
    collection_name="hybrid_docs",
    vectors_config={"dense": VectorParams(size=1024, distance=Distance.COSINE)},
    sparse_vectors_config={"sparse": SparseVectorParams()},
)

# Hybrid query — RRF fusion over dense + sparse results
results = client.query_points(
    collection_name="hybrid_docs",
    prefetch=[
        Prefetch(query=dense_vec, using="dense", limit=20),
        Prefetch(query=SparseVector(indices=sparse_indices, values=sparse_values),
                 using="sparse", limit=20),
    ],
    query=FusionQuery(fusion=Fusion.RRF),
    limit=5,
)

Qdrant also supports named vectors (store multiple embedding spaces in one collection), quantisation (scalar and binary, to shrink memory footprint significantly), on-disk indexing, and a rich Web UI accessible at http://localhost:6333/dashboard. For homelab use these features become relevant when you are indexing tens of millions of vectors or want one collection to answer both semantic and keyword queries.

Honest downsides. Qdrant is another process to monitor, back up, and upgrade. The Python client API has changed substantially across major versions — check the client version matches the server version. Memory usage scales with collection size and the m parameter of your HNSW index; a 1M-vector collection at m=16 and 768 dimensions will consume roughly 3–5 GB of RAM for the in-memory graph. The Rust-based configuration can feel opaque compared to familiar Postgres EXPLAIN ANALYZE output when you are debugging slow queries.

End-to-End Offline RAG: Wiring It Together

The full loop looks like this regardless of which store you choose:

                    ┌─────────────────────────────────────────────┐
                    │              INGEST PATH                    │
                    │                                             │
  Documents ──────► │  Chunk  ──► Embed (local model) ──► Store  │
                    │                                             │
                    └─────────────────────────────────────────────┘

                    ┌─────────────────────────────────────────────┐
                    │              QUERY PATH                     │
                    │                                             │
  User Query ─────► │  Embed query ──► Retrieve top-k chunks     │
                    │       │                                     │
                    │       ▼                                     │
                    │  Build prompt: system + context + question  │
                    │       │                                     │
                    │       ▼                                     │
                    │  Local LLM generates answer                 │
                    │       │                                     │
                    └───────┼─────────────────────────────────────┘
                            │
                            ▼
                       Answer to user
                    (nothing left the box)

The local LLM leg is covered in depth in Local LLMs with Ollama and Jan Local LLM Runner. The retrieval orchestration — chunking, context assembly, prompt templating — is in Building a Local RAG Pipeline. Here is a minimal self-contained retrieval function that works with any of the three stores via a common interface:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# shared interface — swap the backend behind `search()`
def rag_answer(question: str, search_fn, llm_fn, k: int = 4) -> str:
    query_vec = embed(question)
    chunks = search_fn(query_vec, k=k)
    context = "\n\n---\n\n".join(chunks)
    prompt = (
        f"Answer the question using only the context below.\n\n"
        f"Context:\n{context}\n\n"
        f"Question: {question}"
    )
    return llm_fn(prompt)

For the LLM call you can hit the Ollama /api/generate endpoint directly with httpx, use LangChain’s OllamaLLM, or use any other local inference wrapper — the vector store choice is completely orthogonal to the LLM choice.

If you are thinking about caching repeated queries to avoid redundant embedding and retrieval calls, Semantic Caching for LLM Cost and Latency covers the pattern in detail.

Comparison: Resource Footprint and When to Pick Each

Honest Trade-offs and What Actually Matters

Embedding model quality dominates everything else. You can switch vector stores in a day; switching embedding models requires re-ingesting your entire corpus. Spend more time on model selection — context window, dimensionality, language support — than on store selection. For most English homelab corpora, nomic-embed-text (768-dim, CPU-friendly) is the pragmatic starting point. For multilingual or technical content, BAAI/bge-m3 (1024-dim, benefits from a GPU) is materially better.

Chroma’s simplicity is a real asset, not a compromise. For a personal knowledge base of a few thousand to a few hundred thousand documents, Chroma’s embedded mode is perfectly adequate. The ability to iterate on chunking strategy, embedding model, and retrieval logic without thinking about infrastructure is genuinely valuable. Its limitations — no hybrid search, single-writer SQLite, weak performance at scale — only bite you when you push past its design envelope.

pgvector’s killer feature is what you already have. If you already run Postgres for other services in your homelab, adding CREATE EXTENSION vector; and a vector column to an existing table is the lowest possible friction. The ability to join vector results with relational data in a single query — “find the top-5 most relevant documents where user_id = 42 and created_at > '2025-01-01'” — is something neither Chroma nor Qdrant can match without a second round-trip. The downsides are real (no sparse vectors, index tuning requires some Postgres expertise, HNSW build times on large tables) but for a homelab with existing Postgres investment they are manageable.

Qdrant’s payload filtering is genuinely better. Chroma and most stores apply metadata filters after the approximate nearest-neighbour search, which means with a very selective filter you may end up with far fewer results than k (or none). Qdrant integrates payload conditions into the HNSW graph traversal itself, so the filter does not shrink the candidate pool unexpectedly. If your RAG system filters heavily by date ranges, categories, or ownership, this matters. Qdrant is also the right answer when you want hybrid search without gluing together two separate queries.

Running as a managed service vs. self-hosted. For the cloud/managed at-scale story — Pinecone, managed Weaviate, hosted Qdrant Cloud, Neon pgvector — see the Vector Databases in Production companion post. The operational calculus is entirely different once you are beyond a homelab and need SLAs.

Verdict

Start with Chroma if you are building your first local RAG pipeline or experimenting with chunking strategies. The embedded mode means zero infrastructure overhead, and you can always migrate to another store once you know what your workload actually looks like.

Reach for pgvector if you already run PostgreSQL for other homelab services. One fewer service to operate, full SQL for metadata filtering, and the Postgres ecosystem for backups and monitoring are compelling advantages that outweigh the lack of sparse-vector support for most use cases.

Move to Qdrant when you hit Chroma’s concurrency or scale ceiling, when you need hybrid dense-plus-sparse search, or when your filtering patterns require the precision of intra-HNSW payload filtering. It is a more capable engine with a correspondingly higher operational bar, but on a capable homelab machine it runs happily as a single container alongside your local LLM.

Sources

• chromadb on PyPI — version history and release notes
• pgvector 0.8.2 release announcement — PostgreSQL.org
• Qdrant Quickstart — local Docker deployment
• Qdrant Hybrid Queries documentation — dense + sparse fusion
• nomic-embed-text on Ollama library
• Best Open-Weight Embedding Models 2026 — Presenc AI
• pgvector key features, tutorial, and pros and cons — Instaclustr 2026 guide
• Build a Modern RAG Pipeline in 2026: Docling + Qdrant Hybrid — Medium