LLMOps / Production¶
Intermediate → Advanced · 7 topics
Most candidates can build a GenAI demo. Far fewer can say how they'd ship it: package it, run it reliably, see what it's doing, stop it from getting worse or more expensive, and keep it secure. That gap shows up in interviews as soon as someone asks "and how would you run this in production?". This series answers that question, topic by topic.
Free preview of the full LLMOps guide
Each page here covers the key ideas with a working example. The complete guide (a PDF in the Store) adds the full code, step-by-step setups, worked examples, checklists and model interview answers. See the full guide in the Store.
| # | Topic | Sub-topics |
|---|---|---|
| 1 | Docker and deployment | A production Dockerfile · config and secrets · health checks · Compose with Redis · where to deploy · zero-downtime releases |
| 2 | Serving open models | API vs self-hosting · GPU memory maths · vLLM, Ollama, llama.cpp, TGI · throughput and concurrency · an OpenAI-compatible endpoint |
| 3 | Caching and rate limits | Exact and semantic caches · prompt caching · token-based rate limits per user · handling provider 429s · fallbacks |
| 4 | Observability with Langfuse | Traces and spans from scratch · what to record · Langfuse tracing · user feedback and scores · dashboards and alerts |
| 5 | Running evals in CI | An eval runner · baselines and noise margins · a release gate · GitHub Actions · online evals on live traffic |
| 6 | Cost control | Cost per request, user and feature · budgets and alerts · model routing · the biggest savings levers |
| 7 | Security | Secrets · redacting logs · output handling · data exfiltration · abuse and denial-of-wallet · a security checklist |
flowchart LR
C[Code + prompts] --> CI[CI: tests + evals<br/>release gate] --> D[Deploy<br/>Docker · canary]
D --> P[Production<br/>gateway · caches · limits]
P --> O[Observability<br/>traces · costs · feedback]
O -->|failures become test cases| CIBefore you start
These pages build on FastAPI for GenAI (the service being deployed), Context, memory & cost, Evaluation & guardrails and the RAG production checklist.
Back to: Notes overview