3. Case: document Q&A at scale¶

Advanced · 6 min read

The prompt: "Design a document Q&A product for 2,000 business customers. Together they have 50 million documents in Google Drive, SharePoint and Confluence. Users ask questions and get cited answers, and must only see documents they have access to."

A RAG demo over 50 PDFs takes an afternoon. At 50 million documents and 2,000 tenants, the hard parts are ingestion, index size, tenant isolation, permissions and freshness. (Background: Production RAG architecture.)

3.1 Requirements¶

Functional	Non-functional
Connect Drive, SharePoint, Confluence; sync edits and deletions	Answers in under 5 s (p95)
Cited answers with page links	Document permissions enforced on every query
Filter by source, date, folder	New documents searchable within 15 minutes

3.2 The headline estimate¶

tokens = 50_000_000 * 6 * 500                   # documents x pages x tokens per page
chunks = tokens / (400 * 0.9)                   # ~400-token chunks with 10% overlap
for label, bytes_per_dim in (("float32", 4), ("int8", 1)):
    print(f"{chunks / 1e6:,.0f}M vectors x 1024 dims, {label}: {chunks * 1024 * bytes_per_dim / 1e9:,.0f} GB")

Output

417M vectors x 1024 dims, float32: 1,707 GB
417M vectors x 1024 dims, int8: 427 GB

Index memory, not the LLM, is the first design driver here.

3.3 Architecture¶

flowchart LR
    subgraph Ingestion
        CN[Connectors<br/>webhooks + polling] --> Q[[Queue]] --> W[Workers: parse · OCR ·<br/>chunk · embed]
        W --> VX[(Vector index<br/>by tenant)]
        W --> MD[(Metadata: ACLs,<br/>versions, hashes)]
    end
    subgraph Query
        U[User] --> API[API: tenant + user groups]
        API --> RT[Hybrid search<br/>+ permission filter + rerank] --> LLM[LLM with citations] --> U
        RT --> VX
    end

The key insight: permissions are enforced in retrieval, before anything reaches the LLM. Each chunk carries its document's access list, synced from the source, and every query filters by tenant and the user's groups. A permission change that isn't synced quickly is a data leak.

In the full GenAI System Design guide (PDF)

Full estimates: embedding cost, daily re-embedding, ingestion throughput and worker counts
Three tenant-isolation models and when to use each
Document permissions end to end: ACL sync, group resolution, leak tests
Freshness and deletion sync, retrieval quality at scale, and cost controls
Upgrading the embedding model for 50M documents with zero downtime
Trade-offs, failure modes, alerts and model answers to the follow-ups below

See the full guide in the Store → · Get notified when it goes on sale ↓

Interview follow-ups¶

How do you make sure users only see documents they have access to?
The index won't fit in memory. What do you do?
How do you upgrade the embedding model for 50 million documents?

Next: 4. Case: coding assistant Back to: GenAI System Design overview · Notes overview