6. Citations

Intermediate · 11 min read

An answer with citations can be checked. The user clicks through to the source, a reviewer sees which document caused a bad answer, and your code can verify that each claim is backed by a retrieved chunk. Citations turn a RAG bot from "trust me" into "here's where it says so". In regulated domains (legal, finance, health), they are usually a hard requirement.

6.1 Label every source in the prompt

The model can only cite what it can refer to. Give each retrieved chunk a short, stable label, and keep a map from label to metadata in your code. Short numbers ([1], [2]) are easier for models than long IDs, and they look good in a UI.

retrieved = [   # chunks from the retriever, with metadata from the loader
    {"id": "refunds.pdf#p2c0", "title": "Refund policy", "page": 2, "url": "https://docs.acme.com/refunds",
     "text": "Refunds are processed within 5 working days of approval."},
    {"id": "refunds.pdf#p2c1", "title": "Refund policy", "page": 2, "url": "https://docs.acme.com/refunds",
     "text": "Damaged items get a full refund including shipping."},
    {"id": "shipping.md#3", "title": "Shipping FAQ", "page": None, "url": "https://docs.acme.com/shipping",
     "text": "Express shipping takes 1 day and costs 9 dollars."},
]

def build_sources(chunks: list[dict]) -> tuple[str, dict[int, dict]]:
    """Return the sources block for the prompt and a label -> chunk map for rendering and validation."""
    labels = {n: c for n, c in enumerate(chunks, start=1)}
    block = "\n".join(f'<source n="{n}" title="{c["title"]}">\n{c["text"]}\n</source>' for n, c in labels.items())
    return block, labels

sources_block, labels = build_sources(retrieved)
print(sources_block)

Output

<source n="1" title="Refund policy">
Refunds are processed within 5 working days of approval.
</source>
<source n="2" title="Refund policy">
Damaged items get a full refund including shipping.
</source>
<source n="3" title="Shipping FAQ">
Express shipping takes 1 day and costs 9 dollars.
</source>

The system prompt states the rule precisely:

Answer using ONLY the numbered sources. After every sentence that states a fact, add the number of the
source(s) that support it in square brackets, e.g. [1] or [1][3]. Never cite a source that doesn't support
the sentence. If the sources don't contain the answer, say "I don't know based on the documents." and cite nothing.

6.2 Turn markers into clickable sources

The model returns text with markers. Your code, not the model, turns those into links. That way URLs and page numbers come from your metadata and can't be hallucinated.

import re

answer = ("Refunds are processed within 5 working days of approval [1]. "
          "If the item arrived damaged, you also get the shipping cost back [2].")

def render(answer: str, labels: dict[int, dict]) -> str:
    cited = sorted({int(n) for n in re.findall(r"\[(\d+)\]", answer)})
    renumber = {old: new for new, old in enumerate(cited, start=1)}       # show [1], [2], ... in order of use
    body = re.sub(r"\[(\d+)\]", lambda m: f"[{renumber[int(m.group(1))]}]", answer)
    lines = []
    for old in cited:
        c = labels[old]
        page = f", page {c['page']}" if c["page"] else ""
        lines.append(f"[{renumber[old]}] {c['title']}{page}: {c['url']}")
    return body + "\n\nSources:\n" + "\n".join(lines)

print(render(answer, labels))

Output

Refunds are processed within 5 working days of approval [1]. If the item arrived damaged, you also get the shipping cost back [2].

Sources:
[1] Refund policy, page 2: https://docs.acme.com/refunds
[2] Refund policy, page 2: https://docs.acme.com/refunds

Only the sources that were actually cited are listed. Source 3 (shipping) was retrieved but not used, and listing it would wrongly suggest it supports the answer.

6.3 Validate citations automatically

Models get citations wrong in three ways: they cite a number that doesn't exist, they state facts with no citation, and they cite a source that doesn't say what the sentence claims. The first two are easy to check in code. The third needs an approximate check, here word overlap, or in production an NLI model or LLM judge.

STOP = {"a", "an", "the", "is", "are", "to", "of", "and", "in", "it", "you", "your", "if", "also", "get", "back",
        "within", "for", "with"}

def content_words(text: str) -> set[str]:
    return {w.rstrip("s") for w in re.findall(r"[a-z0-9]+", text.lower()) if w not in STOP}

def check_citations(answer: str, labels: dict[int, dict], min_support: float = 0.5) -> list[str]:
    problems = []
    for sentence in re.split(r"(?<=[.!?])\s+", answer.strip()):
        numbers = [int(n) for n in re.findall(r"\[(\d+)\]", sentence)]
        claim = re.sub(r"\s*\[\d+\]", "", sentence)
        if not numbers:
            problems.append(f"NO CITATION   {claim.strip()!r}")
            continue
        for n in numbers:
            if n not in labels:
                problems.append(f"INVALID [{n}]   {claim.strip()!r}")
                continue
            words = content_words(claim)
            support = len(words & content_words(labels[n]["text"])) / max(len(words), 1)
            if support < min_support:
                problems.append(f"WEAK [{n}] {support:.0%}  {claim.strip()!r}")
    return problems

bad_answer = ("Refunds are processed within 5 working days of approval [1]. "
              "Express shipping is free for damaged items [2]. "
              "Refunds are paid in cash at any store [1]. "
              "Refunds over 500 dollars need a manager [4]. "
              "We ship worldwide.")
for problem in check_citations(bad_answer, labels):
    print(problem)

Output

WEAK [1] 17%  'Refunds are paid in cash at any store.'
INVALID [4]   'Refunds over 500 dollars need a manager.'
NO CITATION   'We ship worldwide.'

It caught the cash refund (only "refund" overlaps), the citation to a source that doesn't exist, and the uncited claim. But look at what it missed: "Express shipping is free for damaged items [2]" is false, yet it shares "shipping", "damaged" and "item" with source 2, so it scores 60% and passes. Word overlap catches lazy mistakes, not plausible-sounding wrong ones. That's why production systems use an NLI model or an LLM judge for support checks.

What to do when a check fails:

• Invalid number: strip the marker and log it. If it happens often, your prompt or model needs work.
• Missing citation on a factual sentence: regenerate, or show the sentence with a "not verified" style.
• Weak support: run a stronger check (an LLM judge asking "Does the source support this sentence? yes/no"), then regenerate or drop the sentence.

Count these failures in production. The share of answers with any citation problem is a cheap, continuous signal for faithfulness.

6.4 Quote-level citations

For high-stakes domains, ask the model to return the exact quote from the source that supports each claim. Quotes can be checked mechanically: either the text is in the source or it isn't. They can also be highlighted in the UI.

import json

model_output = json.loads("""{
  "answer": "Refunds take 5 working days after approval, and damaged items are refunded in full.",
  "citations": [
    {"source": 1, "quote": "Refunds are processed within 5 working days of approval."},
    {"source": 2, "quote": "Damaged items get a full refund, including shipping."}
  ]
}""")   # produced with structured output (a JSON schema or a Pydantic model)

normalise = lambda s: re.sub(r"[^a-z0-9]+", " ", s.lower()).strip()
for cite in model_output["citations"]:
    exact = cite["quote"] in labels[cite["source"]]["text"]
    loose = normalise(cite["quote"]) in normalise(labels[cite["source"]]["text"])
    print(f"source {cite['source']}: exact={exact!s:5} after normalising={loose!s:5}  {cite['quote']!r}")

Output

source 1: exact=True  after normalising=True   'Refunds are processed within 5 working days of approval.'
source 2: exact=False after normalising=True   'Damaged items get a full refund, including shipping.'

The second quote has an extra comma, so an exact match fails while the normalised one passes. Normalise whitespace and punctuation before comparing. A quote that fails even after normalising was made up, so treat the claim as unsupported.

# no-run — ask for citations as structured output (OpenAI SDK + Pydantic)
from pydantic import BaseModel

class Citation(BaseModel):
    source: int        # the source number from the prompt
    quote: str         # exact supporting text copied from that source

class CitedAnswer(BaseModel):
    answer: str
    citations: list[Citation]

response = client.beta.chat.completions.parse(model="gpt-4o-mini", messages=messages, response_format=CitedAnswer)
result: CitedAnswer = response.choices[0].message.parsed

Some providers have native citation features: you pass documents, and the API returns answer spans linked to exact source passages. Where available, they are more reliable than prompt-only citations.

6.5 Showing citations in the UI

• Put markers inline and the sources list below. On hover or click, show the chunk text with the quote highlighted.
• Link to the exact place: the page anchor, the PDF page (file.pdf#page=2), or the heading anchor.
• Show the document's date, so users can see when a source is old.
• Apply the permissions check again when the user opens a source link. Citing a document must not leak one the user can't open.
• With streaming, markers arrive token by token. Render [1] once the closing bracket arrives, and fill in the sources list at the end.

Interview questions

How do you implement citations in a RAG system?

Give each retrieved chunk a short label in the prompt and keep a label-to-metadata map in code. Instruct the model to cite labels after each factual sentence, and to say "I don't know" without citations when the answer isn't there. Parse the markers, list only the sources actually cited, and build links from your own metadata, never from model output. Optionally use structured output with exact quotes, or a provider's native citation feature.

How can you verify that citations are correct?

Check mechanically that cited labels exist and that factual sentences have citations. Check support by matching exact quotes against the source text, or by asking an NLI model or LLM judge whether the source entails the sentence. Regenerate or flag failures, and track the citation-error rate as a production faithfulness signal.

Why should URLs in citations come from metadata, not the LLM?

LLMs can hallucinate plausible but wrong URLs or page numbers. If the model only outputs a source label and your code maps it to stored metadata, every link is real. You can also re-check permissions and freshness when you render it.

Practice

• Extend render to handle grouped markers like [1, 3].
• In check_citations, also flag answers that say "I don't know" and include a citation.
• Make model_output include a fabricated quote and confirm both checks catch it.

Next: 7. Evaluation with RAGAS Back to: RAG overview · Notes overview