Skip to content

4. Memory

Intermediate · 12 min read

LLMs are stateless: every call starts from nothing, and the model knows only what's in its context window. Agent "memory" is everything you choose to put back into that window: recent messages, facts about the user, lessons from past tasks. Get it wrong and the agent forgets the user's name in turn 10, or remembers a stale preference forever.

4.1 Kinds of memory

Kind Holds Lives in Example
Short-term (working) This conversation or task The context window, trimmed or summarised "Earlier you asked about order A100"
Long-term semantic Facts about the user or world A database or vector store, per user "Prefers UPI; works in Bengaluru"
Episodic What happened in past tasks Stored summaries of past runs "Last time the CSV import failed on dates; parse them first"
Procedural How to behave System prompt, tool descriptions, rules "Always confirm before refunds"
State / checkpoint Where a run is A checkpointer keyed by thread ID Resume a paused run after human approval

4.2 Short-term: keep the window under control

The conversation grows every turn. Keep the system prompt, the latest turns in full, and a summary of older ones (more strategies in Context, memory & cost):

def count_tokens(text: str) -> int:
    return max(1, len(text) // 4)                      # rough rule: ~4 characters per token

def build_context(system: str, history: list[dict], summary: str, budget: int = 120, keep_last: int = 4) -> list[dict]:
    recent = history[-keep_last:]
    messages = [{"role": "system", "content": system}]
    if summary:
        messages.append({"role": "system", "content": f"Summary of earlier conversation: {summary}"})
    messages += recent
    while sum(count_tokens(m["content"]) for m in messages) > budget and len(messages) > 3:
        messages.pop(2 if summary else 1)              # drop the oldest kept turn, never the system prompt
    return messages

history = [{"role": "user" if i % 2 == 0 else "assistant", "content": text} for i, text in enumerate([
    "Hi, I'm Asha. I ordered a laptop last week, order A100, and I wanted to check on it before the weekend.",
    "Thanks Asha! Order A100, a Laptop Pro, was packed on Monday and shipped yesterday from our Pune warehouse.",
    "Great. Is it insured, and will I need to sign for it when it's delivered?",
    "Yes, it's insured for the full value, and the courier will ask for a signature and an OTP on delivery.",
    "When will it arrive?", "Friday, by 7 pm.", "Can I change the address?", "Yes, until it's out for delivery.",
    "Change it to my office in Whitefield.", "Done: delivery address updated to Whitefield."])]
summary = "Asha's order A100 (laptop) shipped; arrives Friday by 7 pm."   # written by a cheap summariser model

context = build_context("You are Acme's support agent.", history, summary)
for m in context:
    print(f"{m['role']:9} | {m['content']}")
print("tokens ≈", sum(count_tokens(m["content"]) for m in context), "instead of",
      sum(count_tokens(m["content"]) for m in history))
Output
system    | You are Acme's support agent.
system    | Summary of earlier conversation: Asha's order A100 (laptop) shipped; arrives Friday by 7 pm.
user      | Can I change the address?
assistant | Yes, until it's out for delivery.
user      | Change it to my office in Whitefield.
assistant | Done: delivery address updated to Whitefield.
tokens ≈ 64 instead of 137

The summary keeps the facts the agent still needs ("Asha", "A100", "Friday") without carrying every old message. Re-summarise every few turns, and never summarise away anything the agent was told to remember exactly (IDs, amounts, addresses): keep those as structured state instead.

4.3 Long-term memory: facts that survive the session

Long-term memory is a small store of facts per user, written when the agent learns something worth keeping and retrieved by relevance at the start of each turn, like RAG over the user's own history:

import math
import re
from collections import Counter

def embed(text: str) -> Counter:                       # stand-in for an embedding model
    return Counter(w.rstrip("s") for w in re.findall(r"[a-z]+", text.lower()) if len(w) > 2)

def cosine(a: Counter, b: Counter) -> float:
    dot = sum(a[t] * b[t] for t in a)
    norm = math.sqrt(sum(v * v for v in a.values())) * math.sqrt(sum(v * v for v in b.values()))
    return dot / norm if norm else 0.0

class MemoryStore:
    def __init__(self):
        self.facts: dict[tuple[str, str], str] = {}    # (user, topic) -> fact; one fact per topic

    def remember(self, user: str, topic: str, fact: str) -> None:
        old = self.facts.get((user, topic))
        self.facts[(user, topic)] = fact               # a new fact on the same topic REPLACES the old one
        print(f"remember [{topic}] {fact}" + (f"   (replaces: {old})" if old else ""))

    def recall(self, user: str, query: str, k: int = 2) -> list[str]:
        q = embed(query)
        scored = [(cosine(q, embed(f"{t} {f}")), f) for (u, t), f in self.facts.items() if u == user]
        return [f for score, f in sorted(scored, reverse=True)[:k] if score > 0]

memory = MemoryStore()
memory.remember("asha", "payment", "Prefers paying by UPI.")
memory.remember("asha", "location", "Works in Whitefield, Bengaluru; deliveries go to the office.")
memory.remember("asha", "device", "Owns a Laptop Pro bought in 2026.")
memory.remember("asha", "payment", "Now prefers paying by credit card.")
memory.remember("ravi", "payment", "Prefers cash on delivery.")

print("\nrecall for 'Which payment method should I use?':")
for fact in memory.recall("asha", "Which payment method should I use?"):
    print("  -", fact)
Output
remember [payment] Prefers paying by UPI.
remember [location] Works in Whitefield, Bengaluru; deliveries go to the office.
remember [device] Owns a Laptop Pro bought in 2026.
remember [payment] Now prefers paying by credit card.   (replaces: Prefers paying by UPI.)
remember [payment] Prefers cash on delivery.

recall for 'Which payment method should I use?':
  - Now prefers paying by credit card.

Only the current preference comes back: not the replaced UPI fact, and not Ravi's. (The word-matching stand-in only matches whole words, so the query says "payment". A real embedding model also matches "how should I pay?".)

Three details that matter in production:

  • Facts change. Store them under a topic (or let an LLM decide "update or add"), so "now prefers card" replaces "prefers UPI" instead of both being recalled and contradicting each other.
  • Memory is per user. Ravi's preference never appears for Asha: filter by user ID on the server, the same rule as tenant isolation in RAG.
  • Recall only what's relevant. Injecting every stored fact into every prompt costs tokens and distracts the model.

4.4 Episodic memory: learn from past runs

After a task, store a short lesson: what was tried, what failed, what worked. Retrieve similar episodes before the next similar task and include them as hints:

episodes = [
    {"task": "import sales CSV into the database", "lesson": "Dates were DD/MM/YYYY; parse them explicitly."},
    {"task": "summarise weekly support tickets", "lesson": "Group by product first; the summary was too long otherwise."},
    {"task": "import customer CSV into CRM", "lesson": "Emails had trailing spaces; strip and lowercase before matching."},
]

def similar_episodes(task: str, k: int = 2) -> list[str]:
    q = embed(task)
    ranked = sorted(episodes, key=lambda e: cosine(q, embed(e["task"])), reverse=True)
    return [f"{e['task']} -> {e['lesson']}" for e in ranked[:k] if cosine(q, embed(e["task"])) > 0]

print("Hints for 'import orders CSV into the database':")
for hint in similar_episodes("import orders CSV into the database"):
    print("  -", hint)
Output
Hints for 'import orders CSV into the database':
  - import sales CSV into the database -> Dates were DD/MM/YYYY; parse them explicitly.
  - import customer CSV into CRM -> Emails had trailing spaces; strip and lowercase before matching.

4.5 What to remember, and what not to

Remember Don't remember
Stable preferences the user stated One-off moods or guesses about the user
Facts the user asked you to remember Passwords, card numbers, OTPs, government IDs
Corrections ("my name is spelled Aasha") Sensitive data (health, finances) without explicit consent
Lessons from failed and successful tasks Anything from untrusted content (web pages, emails): it could be an injection

Give users a way to see and delete what's stored about them, and apply your data-retention and deletion rules to memory exactly as to any other personal data.

4.6 Memory in LangGraph

LangGraph separates the two kinds: a checkpointer for short-term, per-thread state (see 3.5), and a store for long-term memory shared across threads, organised by namespace:

# no-run — pip install langgraph
from langgraph.checkpoint.memory import InMemorySaver
from langgraph.store.memory import InMemoryStore

store = InMemoryStore()                                   # Postgres/Redis-backed stores for production
store.put(("memories", "asha"), "payment", {"fact": "Now prefers paying by credit card."})
item = store.get(("memories", "asha"), "payment")
print(item.value["fact"])

graph = builder.compile(checkpointer=InMemorySaver(), store=store)   # nodes can read/write the store
graph.invoke({"messages": [...]}, config={"configurable": {"thread_id": "t-42", "user_id": "asha"}})

Interview questions

What types of memory does an agent need?

Short-term memory (the current conversation, trimmed or summarised to fit the context window), long-term semantic memory (facts about the user, stored per user and retrieved by relevance), episodic memory (lessons from past tasks), procedural memory (instructions and rules) and run state (checkpoints for resuming). They differ in lifetime, storage and how they're retrieved.

How do you handle a long conversation that exceeds the context window?

Keep the system prompt and the latest turns verbatim, replace older turns with a rolling summary written by a cheap model, and keep exact values (IDs, amounts) as structured state rather than in the summary. Retrieve long-term facts by relevance instead of including everything.

How do you keep long-term memory correct and safe?

Store facts by user and topic so updates replace stale facts; retrieve only relevant ones; filter by user on the server; never store secrets or sensitive data without consent; don't write memories from untrusted content; let users view and delete their memories; and apply retention policies.

Practice

  • Lower budget in 4.2 until the summary is the only older context left. What does the agent lose?
  • Add a memory for Ravi and confirm recall("asha", ...) never returns it.
  • Add an episode for "import products CSV" and see which hints the import task gets now.

Next: 5. Tools and MCP Back to: AI Agents overview · Notes overview