2. Case: customer support chatbot¶

Advanced · 6 min read

The prompt: "Design a support assistant for an e-commerce company with 200,000 daily active users. It should answer policy questions, check order status, start refunds, and hand off to a human when needed."

The most common GenAI design question. It combines RAG (policies), tools (orders and refunds), safety (money moves) and a human in the loop.

2.1 Requirements¶

Functional	Non-functional
Answer policy questions with citations	First token under 1 s
Order status and refunds for the logged-in user	A customer sees only their own orders
Hand off to a human with a summary	Refunds above ₹2,000 need human approval
English and Hindi	Cost per conversation far below a human-handled ticket

2.2 The headline estimate¶

conversations = 200_000 * 0.10                 # 10% of daily users open a chat
turns = conversations * 5
cost_per_turn = (3_500 * 2.50 + 250 * 10.00) / 1e6      # large model for every turn (placeholder prices)
print(f"{conversations:,.0f} conversations/day, {turns:,.0f} turns/day")
print(f"all turns on the large model: ${cost_per_turn * turns * 30:,.0f}/month")

Output

20,000 conversations/day, 100,000 turns/day
all turns on the large model: $33,750/month

The design question then becomes: how much of that can go to a cheaper path without hurting quality?

2.3 Architecture¶

flowchart LR
    C[Web / app chat] --> G[API gateway<br/>auth · rate limits]
    G --> O[Conversation orchestrator]
    O --> R{Router<br/>small model}
    R -->|policy question| RAG[RAG over help centre<br/>with citations]
    R -->|my order| AG[Order agent<br/>status · address · refund]
    R -->|complex / upset| H[Handoff to a human<br/>with summary]
    AG --> P{Policy engine}
    P -->|allow| API[(Order & payment APIs)]
    P -->|large refund| A[Human approval]

The key insight: this is a workflow with a router, not one big agent. Most turns are simple (order status, a policy question) and go to cheap, testable paths; only some need the large model; and money decisions are made by a policy engine in code, never by the model.

In the full GenAI System Design guide (PDF)

Full estimates: routing savings, peak tokens per minute, cost per conversation vs a human ticket
Every component explained: router, RAG, order agent, policy engine, memory, handoff, observability
Deep dives: identity and permissions, when to hand off, Hindi and mixed language, evaluation
Trade-off table with the alternatives and when to choose them
Failure modes and the alerts for each
Model answers to the interview follow-ups below

See the full guide in the Store → · Get notified when it goes on sale ↓

Interview follow-ups¶

How do you stop the bot from refunding the wrong order?
How would you measure whether the chatbot is worth it?
Traffic doubles during a sale. What breaks first?

Next: 3. Case: document Q&A at scale Back to: GenAI System Design overview · Notes overview