6. Cost control¶
Intermediate · 6 min read
LLM costs grow with usage in a way normal hosting doesn't: every request is billed by the token, one chatty feature or one heavy user can double the bill, and an agent stuck in a loop spends money every second. Controlling cost means knowing where it goes, limiting it before it surprises you, and reducing it without hurting quality.
6.1 Know where the money goes¶
Log every LLM call with user, feature, model and token counts, then add it up:
PRICES = {"small": (0.15, 0.60), "large": (2.50, 10.00)} # $ per 1M input/output tokens (placeholders)
USAGE = [("asha", "chat", "large", 3200, 400), ("ravi", "chat", "large", 2900, 350),
("ravi", "summary", "small", 6000, 300), ("bot7", "agent", "large", 18000, 900),
("bot7", "agent", "large", 21000, 950), ("meera", "summary", "small", 5500, 280)]
by_feature, total = {}, 0.0
for user, feature, model, tin, tout in USAGE:
cost = (tin * PRICES[model][0] + tout * PRICES[model][1]) / 1e6
by_feature[feature] = by_feature.get(feature, 0) + cost
total += cost
for feature, cost in sorted(by_feature.items(), key=lambda kv: -kv[1]):
print(f"{feature:8} ${cost:.4f} {cost / total:4.0%} of spend")
One agent feature, used by one account, is most of the spend. Without per-feature and per-user attribution, all you'd see is "the bill went up".
6.2 Limits at every level¶
| Level | Example limit |
|---|---|
| Per request | max_tokens, retrieved-token cap, agent step and cost budget |
| Per user / API key | Tokens per minute and per day; plan-based quotas |
| Per feature | Daily budget; degrade to a smaller model when it's hit |
| Whole account | Provider spend limit; alerts at 50 / 80 / 100% of the monthly budget |
6.3 The biggest savings levers¶
| Lever | Typical effect |
|---|---|
| Rerank and send fewer chunks | Large cut in input tokens, often better answers |
| Shorter outputs | Output is the expensive side |
| Prompt caching (stable prefix first) | Cheaper input, faster first token |
| Route easy requests to a small model | Usually the biggest single saving |
| Response caching | Free answers on repeat questions |
| Batch API for offline jobs | Often about half the price |
Applied together, these levers commonly cut a bill by 80% or more. The full guide shows the calculation step by step.
In the full LLMOps guide (PDF)
- A complete cost attribution script: per feature, per user, per conversation
- Budget forecasting with warnings and alerts, in code
- A worked example taking a real-sized app from 100% to 16% of its cost, lever by lever
- Which levers trade quality for cost, and how to prove they don't with evals
- A cost checklist for design reviews, before a feature ships
- Model answers to the interview questions below
See the full guide in the Store → · Get notified when it goes on sale ↓
Interview questions¶
- How would you reduce the LLM cost of a RAG chatbot?
- How do you stop one user from running up your LLM bill?
- How do you attribute LLM cost to features and customers?
The full LLMOps guide: 36 pages, 9 chapters. Deployment, serving, caching, observability, evals in CI, cost and security, with complete code and model interview answers. Get one email when it goes on sale.
Next: 7. Security Back to: LLMOps overview · Notes overview