6. Trade-offs and capacity estimates¶
Advanced · 6 min read
Every GenAI design comes down to the same dozen decisions, made differently depending on the numbers.
6.1 The recurring decisions¶
| Decision | Choose the first when… | Choose the second when… |
|---|---|---|
| API vs self-hosted model | Quality first, spiky traffic, small team | Data can't leave, steady high volume |
| RAG vs fine-tuning | Facts change; citations; permissions | Style, format, or a narrow task |
| RAG vs long context | Large corpus, many queries | A few documents, one-off analysis |
| Workflow vs agent | Steps are known; audit needed | Steps vary per task |
| One large model vs routing | Low volume, quality-critical | High volume with many easy requests |
| Streaming vs async job | Chat | Long tasks: reports, agents, batches |
| Vector-only vs hybrid search | Paraphrase-heavy questions | Codes, names, IDs (usually: hybrid) |
| pgvector vs dedicated vector DB | Already on Postgres, millions of vectors | Hundreds of millions, heavy filtering |
6.2 Core capacity formulas¶
| Quantity | Formula |
|---|---|
| Peak requests/s | requests per day ÷ 86,400 × 2–3 |
| Tokens per minute | peak requests/s × 60 × (input + output tokens) |
| Cost per request | input tokens × input price + output tokens × output price |
| Vector memory | chunks × dimensions × bytes per value |
| GPU weights | parameters × bytes per parameter |
6.3 Numbers to remember¶
| Rule of thumb | Value |
|---|---|
| 1 token | ≈ 4 English characters ≈ 0.75 words |
| Output vs input price | Output usually 3–5× more per token |
| Chat target | First token under ~1 s, always streaming |
| Retrieval | Retrieve 20–50, rerank, send 3–8 chunks |
| Reliability compounds | 95% per step ≈ 60% over 10 steps |
| Headroom | Plan for ≤ 80% of any limit |
In the full GenAI System Design guide (PDF)
- All 12 recurring decisions, each with reasoning, numbers and an example answer
- Latency budgets: why a 10-step agent takes ~26 s, and how to cut it
- Provider-limit and GPU-sizing calculators with worked examples
- The complete formula sheet and numbers to remember
- How to present a trade-off so it scores, with sample phrasing
- Model answers to the interview questions below
See the full guide in the Store → · Get notified when it goes on sale ↓
Interview questions¶
- How do you decide between RAG, fine-tuning and long context?
- Your agent takes 30 seconds to answer. What can you do?
- How do you size an LLM system for peak load?
The full GenAI System Design guide: 35 pages, 8 chapters. The interview framework, four complete case studies, practice problems and model answers. Get one email when it goes on sale.
Back to: GenAI System Design overview · Notes overview