1. An approach for interviews¶

Advanced · 7 min read

GenAI system design rounds look like classic ones ("design a support chatbot for 10 million users") but are judged on different things. Interviewers want to hear how you'll know the answers are good, what each request costs, how long the user waits, and what stops the system from doing something harmful.

1.1 How GenAI rounds differ¶

Classic system design	GenAI system design adds
Correctness is deterministic	Quality is statistical: you need an evaluation plan
Latency is milliseconds	LLM calls take seconds: streaming and async work
Cost is servers	Cost is per token, and can dominate the budget
Security is auth and validation	Plus prompt injection, data leakage, unsafe actions
Data model and storage	Plus retrieval: chunking, embeddings, indexes, freshness

1.2 The 7-step framework (45–60 minutes)¶

#	Step	Time
1	Clarify requirements, including the quality bar, latency and budget	5 min
2	Estimate requests, tokens and cost	5 min
3	High-level architecture	8 min
4	Data and retrieval	8 min
5	Model strategy: API vs self-host, routing, fine-tuning	6 min
6	Evaluation and guardrails	6 min
7	Scale, reliability, cost and wrap-up	7 min

1.3 Estimate tokens and cost, out loud¶

def estimate(name, daily_users, requests_per_user, in_tokens, out_tokens, price_in=2.50, price_out=10.00):
    """Prices are $ per 1M tokens (placeholders: use the real model's prices)."""
    per_day = daily_users * requests_per_user
    peak_rps = per_day / 86_400 * 3
    cost_req = (in_tokens * price_in + out_tokens * price_out) / 1e6
    print(f"{name}: peak {peak_rps:,.0f} req/s | ${cost_req:.4f} per request | ${cost_req * per_day * 30:,.0f}/month")

estimate("Support chatbot: 200k users/day x 4 messages", 200_000, 4, 3000, 300)

Output

Support chatbot: 200k users/day x 4 messages: peak 28 req/s | $0.0105 per request | $252,000/month

A quarter of a million dollars a month is a design problem, not a budget line. That one number is what justifies routing, caching or a smaller model, and interviewers want to see you get there yourself.

In the full GenAI System Design guide (PDF)

The complete framework with what to say at each step, and a 30-minute version
The full list of clarifying questions, and why each one changes the design
The estimator with model routing, peak tokens per minute and provider-limit checks
A reference architecture you can redraw from memory, layer by layer
The deep-dive menu: what interviewers probe for each type of system
Scoring rubric and the mistakes that cost candidates the round
Model answers to the interview questions below

See the full guide in the Store → · Get notified when it goes on sale ↓

Interview questions¶

How is designing a GenAI system different from a classic system design?
What estimates would you make at the start of an LLM system design?

Next: 2. Case: customer support chatbot Back to: GenAI System Design overview · Notes overview