4. Vector databases¶

Intermediate · 12 min read

An ANN index (previous topic) answers one question: "which vectors are closest to this one?" An application needs much more: store the text and metadata with each vector, update and delete records, filter by tenant, date or category, combine with keyword search, survive restarts, and scale across machines. That is what a vector database adds.

Capability	Index library (FAISS, hnswlib)	Vector database
Nearest-neighbour search	✅	✅
Store text + metadata with vectors	❌ (you keep a separate mapping)	✅
Insert / update / delete by ID	limited	✅
Metadata filtering during search	❌	✅
Persistence, backups, replication	you build it	✅
Multi-tenancy (namespaces, collections)	❌	✅
Hybrid (keyword + vector) search	❌	many do
API, auth, monitoring	❌	✅

A library is fine for a notebook, a static dataset or an embedded app. For a RAG system that changes and serves users, use a database.

4.1 Records: id + vector + metadata¶

Every record looks roughly like this:

record = {
    "id": "refund-policy.md#chunk-3",                 # stable, deterministic — see topic 6
    "values": [0.012, -0.044, 0.091],                 # the embedding (shortened)
    "metadata": {
        "doc_id": "refund-policy.md",
        "text": "Refunds are processed within 5 working days of approval.",
        "tenant_id": "acme",
        "category": "refunds",
        "updated_at": "2026-09-30",
        "embedding_model": "text-embedding-3-small@1536",
    },
}
print(sorted(record["metadata"]))

Output

['category', 'doc_id', 'embedding_model', 'tenant_id', 'text', 'updated_at']

Metadata does two jobs: it comes back with the results (the text to put in the prompt, the source for citations), and it filters the search.

4.2 Metadata filtering: pre- vs post-filtering¶

"Top 5 chunks about refunds, for tenant acme, updated this year" needs vector similarity and filters. How the database combines them matters:

import numpy as np

rng = np.random.default_rng(0)
N = 10_000
vectors = rng.normal(size=(N, 32)).astype(np.float32)
vectors /= np.linalg.norm(vectors, axis=1, keepdims=True)
tenant = rng.choice(["acme", "globex", "initech", "umbrella", "hooli"], size=N, p=[0.02, 0.38, 0.2, 0.2, 0.2])

query = vectors[0] + 0.1 * rng.normal(size=32).astype(np.float32)
query /= np.linalg.norm(query)
scores = vectors @ query

def post_filter(k=5, candidates=50):
    top = np.argsort(-scores)[:candidates]                     # ANN search first…
    return [i for i in top if tenant[i] == "acme"][:k]         # …then drop other tenants

def pre_filter(k=5):
    allowed = np.where(tenant == "acme")[0]                    # filter first…
    return allowed[np.argsort(-scores[allowed])[:k]]           # …then search only within it

print("acme records:", int((tenant == "acme").sum()), "of", N)
print("post-filter results:", len(post_filter()))
print("pre-filter results: ", len(pre_filter()))

Output

acme records: 205 of 10000
post-filter results: 0
pre-filter results:  5

Post-filtering fetched the 50 nearest vectors, but none of them belonged to the small tenant — so the user gets zero results instead of five. Pre-filtering (or filtered search inside the index, which modern databases do) always returns k results when they exist. When evaluating a database, check how it filters, and test with selective filters (small tenants, rare categories).

Index the fields you filter on

Most databases need you to declare or index metadata fields used in filters (Qdrant payload indexes, Pinecone's metadata indexing settings, a B-tree index in PostgreSQL). Unindexed filters are slow or unsupported.

4.3 Multi-tenancy¶

A SaaS RAG product must never return company A's documents to company B. Options:

Approach	How	Pros	Cons
Metadata filter	`tenant_id` on every record; filter on every query	simple, one index	one missing filter = data leak; noisy neighbours
Namespace / partition per tenant	Pinecone namespaces, Qdrant/Milvus partitions, Weaviate tenants	strong isolation inside one index; easy per-tenant delete	many small tenants → overhead in some systems
Collection / index per tenant	separate collection per customer	strongest isolation, per-tenant settings	operational overhead at thousands of tenants
Database row-level security	pgvector + PostgreSQL RLS policies	isolation enforced by the database	PostgreSQL only

Whatever you choose, the tenant must come from the authenticated user, applied on the server — never from a value the LLM or the client can change.

4.4 Hybrid search: dense + sparse¶

Dense vectors capture meaning; keyword (sparse) search catches exact terms like error codes and SKUs (see NLP for GenAI). Many vector databases now store a sparse vector (BM25 or a learned sparse model like SPLADE) next to the dense one and combine scores in one query:

def hybrid(dense: np.ndarray, sparse: np.ndarray, alpha: float) -> np.ndarray:
    """alpha=1 → pure vector search, alpha=0 → pure keyword search. Scores are min-max normalised first."""
    norm = lambda s: (s - s.min()) / (s.max() - s.min() + 1e-9)
    return alpha * norm(dense) + (1 - alpha) * norm(sparse)

docs = ["Error E-4012: payment gateway timeout", "Payments failing at checkout", "How to reset your password"]
dense_scores = np.array([0.52, 0.81, 0.10])     # the embedding model prefers the paraphrase…
sparse_scores = np.array([7.9, 0.0, 0.0])       # …BM25 sees the exact error code
for alpha in (1.0, 0.5, 0.0):
    best = int(np.argmax(hybrid(dense_scores, sparse_scores, alpha)))
    print(f"alpha={alpha}: {docs[best]}")

Output

alpha=1.0: Payments failing at checkout
alpha=0.5: Error E-4012: payment gateway timeout
alpha=0.0: Error E-4012: payment gateway timeout

Weighted score fusion (above) needs the scores normalised; reciprocal rank fusion uses only ranks and needs no tuning. Both are common; evaluate alpha (or RRF) on your eval set.

4.5 The main options¶

	Type	Strengths	Consider when
Pinecone	fully managed (serverless)	zero ops, namespaces, metadata filters, hybrid, scales automatically	you want managed infrastructure and fast time to production
pgvector (PostgreSQL)	extension	one database for app data + vectors, SQL joins, transactions, row-level security	you already run Postgres; up to tens of millions of vectors
Qdrant	open source + cloud	fast filtered search, payload indexes, quantisation, sparse vectors	you want open source with strong filtering
Weaviate	open source + cloud	built-in hybrid search, multi-tenancy, vectorizer modules	hybrid search and many tenants out of the box
Milvus / Zilliz	open source + cloud	very large scale (billions), many index types incl. GPU	huge collections
Chroma	open source, embedded or server	simplest developer experience, runs in-process	prototypes, local apps, small production workloads
Elasticsearch / OpenSearch	search engine with vectors	mature keyword search + vectors + aggregations	you already use them for search
Redis, MongoDB Atlas, Azure AI Search, Vertex AI Vector Search	vectors inside platforms you may already use	fewer moving parts	you're already on that platform
FAISS	library	fastest raw ANN, GPU support	building your own system or offline batch jobs

4.6 How to choose¶

def suggest(n_vectors: int, already_postgres: bool, want_managed: bool, need_self_host: bool) -> str:
    if n_vectors < 50_000 and not want_managed:
        return "Chroma (embedded) or plain NumPy/FAISS — keep it simple"
    if already_postgres and n_vectors < 20_000_000:
        return "pgvector — one database, SQL filters and joins, RLS for tenants"
    if want_managed and not need_self_host:
        return "Pinecone (or another managed service)"
    if n_vectors > 500_000_000:
        return "Milvus / Zilliz or another distributed engine"
    return "Qdrant or Weaviate (self-hosted or their cloud)"

print(suggest(20_000, False, False, False))
print(suggest(3_000_000, True, False, False))
print(suggest(30_000_000, False, True, False))
print(suggest(30_000_000, False, False, True))

Output

Chroma (embedded) or plain NumPy/FAISS — keep it simple
pgvector — one database, SQL filters and joins, RLS for tenants
Pinecone (or another managed service)
Qdrant or Weaviate (self-hosted or their cloud)

The rule of thumb behind the code: start with what you already run, and move to a dedicated vector database when scale, filtering performance or operations demand it. Retrieval quality depends far more on chunking, embeddings and hybrid search than on which database you pick.

Interview questions¶

Why use a vector database instead of FAISS?

FAISS is an index library: fast similarity search, but no metadata storage, filtering, updates by ID, persistence, replication, access control or multi-tenancy. A vector database provides those around an ANN index, which a production RAG system needs.

What's the problem with post-filtering?

If you retrieve the top-k vectors and then apply metadata filters, selective filters can remove most or all results. Pre-filtering or filter-aware ANN search returns k results that satisfy the filter. Test databases with selective filters.

How do you implement multi-tenancy in a RAG system?

Isolate each tenant's data with namespaces/partitions, separate collections, or a tenant_id filter (with row-level security in pgvector). Always derive the tenant from the authenticated user on the server, and test that one tenant's queries can never return another's data.

pgvector or a dedicated vector database?

pgvector if you already use PostgreSQL and the collection is moderate: one system, transactions, joins, RLS. A dedicated database (Pinecone, Qdrant, Weaviate, Milvus) for very large collections, heavy filtered search at high QPS, built-in hybrid search, or when you want a managed service.

Practice¶

Change the acme share in 4.2 to 20%. How many results does post-filtering return now?
For your project, fill in suggest(...) with real numbers. Do you agree with the answer?

Next: Hands-on: Chroma, Pinecone, pgvector — the same operations in four databases.