3. Caching and rate limits¶

Intermediate · 7 min read

Two cheap controls do more for an LLM app's cost and reliability than most model changes. Caching avoids paying twice for the same answer. Rate limits stop one user, bug or attacker from using up your provider quota and budget.

3.1 Where caching fits¶

Cache	Saves	Risk
Exact response cache	The whole LLM call, for identical requests	Stale answers after content changes
Semantic cache	The whole call, for similar questions	A wrong answer to a question that only looks similar
Provider prompt caching	Input cost and time to first token	None, if the prompt prefix is stable
Embedding / retrieval cache	Embedding calls and searches	Stale after re-indexing

3.2 An exact cache with a safe key¶

The cache key must include everything that changes the answer: model, prompt version, parameters, the normalised input and the tenant's scope. Leave out the scope and one customer sees another customer's cached answer.

import hashlib
import json

def cache_key(model, prompt_version, params, user_input, scope) -> str:
    normalised = " ".join(user_input.lower().split())
    raw = json.dumps([model, prompt_version, params, normalised, scope], sort_keys=True)
    return hashlib.sha256(raw.encode()).hexdigest()[:16]

base = {"model": "gpt-4o-mini", "prompt_version": "support-v7", "params": {"temperature": 0}}
a = cache_key(user_input="What is your refund policy?", scope="tenant-acme", **base)
b = cache_key(user_input="what is your   REFUND policy?", scope="tenant-acme", **base)
c = cache_key(user_input="What is your refund policy?", scope="tenant-globex", **base)
print("same question, re-typed  ->", "HIT" if a == b else "MISS")
print("same question, other tenant ->", "HIT" if a == c else "MISS (correct: never share across tenants)")

Output

same question, re-typed  -> HIT
same question, other tenant -> MISS (correct: never share across tenants)

3.3 The semantic-cache trap¶

A semantic cache returns a stored answer when a new question is close to a cached one. It catches paraphrases, but "How do I cancel my order?" and "How do I cancel my subscription?" are close in wording and different in meaning. A loose threshold serves a confident, wrong answer. If you use one, keep the threshold high, scope it per tenant, and limit it to FAQ-style questions.

3.4 Rate limits counted in tokens¶

One request can cost 100 tokens or 100,000, so counting requests isn't enough. Limit requests and tokens per user per minute, keep the counters in Redis so every instance shares them, and reply 429 with a Retry-After value. Add a global ceiling just under your provider's quota.

In the full LLMOps guide (PDF)

A complete response cache with TTL, invalidation and Redis code
The semantic-cache experiment with real scores, and how to pick a safe threshold
Prompt-caching rules that actually get cache hits, with before/after costs
A working sliding-window token rate limiter, step by step
Handling provider 429s: Retry-After, backoff, fallback models and the Batch API
Model answers to the interview questions below

See the full guide in the Store → · Get notified when it goes on sale ↓

Interview questions¶

What should be in the cache key for an LLM response cache?
What are the risks of semantic caching?
How would you rate-limit an LLM API?

Next: 4. Observability with Langfuse Back to: LLMOps overview · Notes overview