4. Observability with Langfuse¶

Intermediate · 7 min read

When a user says "the bot gave me a wrong answer yesterday", logs that say POST /chat 200 4.2s don't help. You need the trace: the question, the rewritten query, the chunks retrieved, the exact prompt, the answer, and the tokens, cost and latency of each step. Langfuse is an open-source tool that records exactly that.

4.1 What a trace looks like¶

A trace is one request. It contains spans (steps like retrieval) and generations (LLM calls, with model, tokens and cost):

TRACE rag-answer  user=u_42  prompt_version=support-v7  tokens=2350  cost=$0.00048
  request         1100 ms
    rewrite        100 ms  gpt-4o-mini  250 tokens  $0.00005
    retrieve       300 ms  chunks=5  top_score=0.81
    generate       300 ms  gpt-4o-mini  2100 tokens  $0.00043

With that, "why was this answer wrong?" takes minutes, not days: you can see whether retrieval found the right chunks, or whether the model ignored them.

4.2 What to record¶

Record	Why
User/tenant, session, app and prompt versions	Find a specific bad answer; compare versions
Each LLM call: model, prompt, output, tokens, cost, latency	See exactly what the model saw and said
Retrieval: queries, chunk IDs, scores	Tell retrieval failures from generation failures
Tool calls: arguments, results, errors	Debug and audit agents
User feedback and evaluation scores	Find and fix bad answers

4.3 Tracing with Langfuse¶

With the Langfuse Python SDK, a decorator turns functions into spans, and a drop-in OpenAI wrapper records each LLM call with tokens and cost:

# no-run — pip install langfuse openai ; Langfuse Python SDK v3
from langfuse import observe
from langfuse.openai import openai            # every call becomes a traced generation

@observe()
def answer(question: str) -> str:
    chunks = retrieve(question)                # retrieve() is also decorated with @observe()
    reply = openai.chat.completions.create(model="gpt-4o-mini", messages=build_messages(question, chunks))
    return reply.choices[0].message.content

Attach the user's thumbs up/down to the trace, filter for the bad ones, and turn each failure into a test case. That loop is how quality improves week after week.

In the full LLMOps guide (PDF)

Build a tracer from scratch (nested spans, tokens, cost) to understand what tools do underneath
Full Langfuse setup: cloud or self-hosted with Docker, users, sessions, tags and versions
Recording user feedback and evaluation scores on traces, with code
LangChain / LangGraph tracing with the callback handler
The dashboard and alerting set-up: what to watch and when to page someone
Keeping personal data out of traces: masking, sampling and retention
Model answers to the interview questions below

See the full guide in the Store → · Get notified when it goes on sale ↓

Interview questions¶

What would you log for each request in an LLM application?
What is the difference between a trace and a log?
How do you use observability to improve answer quality over time?

Next: 5. Running evals in CI Back to: LLMOps overview · Notes overview