4. Case: coding assistant¶
Advanced · 6 min read
The prompt: "Design an AI coding assistant for 5,000 developers: inline completions in the IDE, chat about the codebase, and an agent mode that can make multi-file changes. The company's code must not be used to train anyone's models."
4.1 Requirements¶
| Functional | Non-functional |
|---|---|
| Inline completions as you type | Completion p95 under ~400 ms |
| Chat that understands the repository | Chat first token under 1.5 s |
| Agent: change several files, run tests, open a PR | Every agent change reviewed by a human |
| Python, TypeScript, Java, Go | Code never retained or used for training by vendors |
4.2 The headline estimate¶
per_day = 5_000 * 0.6 * 4 * 200 # devs x active share x coding hours x completions per hour
peak_rps = per_day / (8 * 3600) * 2.5
print(f"{per_day:,.0f} completions/day, peak ~{peak_rps:,.0f} requests/s, each needing an answer in a few hundred ms")
4.3 Architecture¶
flowchart LR
IDE[IDE plugin<br/>debounce · cancel] -->|completion| CS[Completion service<br/>small code model, self-hosted]
IDE -->|chat / agent| CH[Chat & agent service]
CH --> RI[Repo context engine<br/>symbols + code search]
CH --> LM[Large model<br/>zero-retention API or self-hosted]
CH --> SB[Sandbox: edit, run tests] --> PR[Pull request for review]The key insight: this is two systems with opposite needs. Completions are tiny, constant and must feel instant, so they suit a small self-hosted model near the developers. Chat and agent requests are rarer and need the strongest model. Designing them as one path makes completions too slow or chat too weak.
In the full GenAI System Design guide (PDF)
- GPU sizing for the completion fleet, and the API-vs-self-host comparison
- Eight techniques that make completions fast (FIM, speculative decoding, prefix caching and more)
- How to fit repository context into a token budget: AST chunks, symbol graph, ranking
- Agent mode in a sandbox, step by step, ending in a reviewed pull request
- Privacy and secret scanning, evaluation (acceptance and retention rates), trade-offs and failure modes
- Model answers to the interview follow-ups below
See the full guide in the Store → · Get notified when it goes on sale ↓
Interview follow-ups¶
- Why separate completions from chat in the architecture?
- How do you give the model context about a large codebase?
- How do you measure if the assistant is actually helping?
The full GenAI System Design guide: 35 pages, 8 chapters. The interview framework, four complete case studies, practice problems and model answers. Get one email when it goes on sale.
Next: 5. Case: multi-agent workflow Back to: GenAI System Design overview · Notes overview