2. Loading & parsing documents¶

Intermediate · 12 min read

Garbage in, garbage out applies to RAG more than anywhere else. If a table turns into a soup of numbers, or every chunk starts with "Confidential — Page 3 of 40", no embedding model or prompt can rescue the answer. Parsing is the least glamorous stage of the pipeline and the one that most often decides quality.

2.1 What "loading" has to produce¶

Whatever the file type, every loader should output the same thing: clean text plus metadata. Everything downstream (chunking, citations, filters, deletions) depends on that metadata, so capture it here, at the start.

class Document:
    def __init__(self, text: str, source: str, metadata: dict | None = None):
        self.text = text
        self.source = source                # file path or URL, used for citations and deletion
        self.metadata = metadata or {}      # title, page, section, updated_at, tenant, doc_type ...

    def __repr__(self) -> str:
        return f"Document(source={self.source!r}, metadata={self.metadata}, text={self.text[:40]!r})"

doc = Document(text="Refunds are processed within 5 working days.", source="policies/refunds.md",
               metadata={"title": "Refund policy", "updated_at": "2026-09-01", "tenant": "acme"})
print(doc)

Output

Document(source='policies/refunds.md', metadata={'title': 'Refund policy', 'updated_at': '2026-09-01', 'tenant': 'acme'}, text='Refunds are processed within 5 working d')

Metadata field	Used for
`source`, `title`, `page`, `section`	Citations users can click
`updated_at`, `version`	Freshness filters, sync and deleting stale chunks
`tenant`, `allowed_groups`	Permission filtering at retrieval (security)
`doc_type`, `language`, `product`	Metadata filters and query routing
`content_hash`	Skip re-embedding unchanged content

2.2 One loader per format¶

Real corpora are mixed: Markdown docs, HTML help pages, CSV exports, PDFs. Dispatch on the file type, and give each loader one job: get the meaningful text out and leave the noise behind.

import csv
import re
from html.parser import HTMLParser
from pathlib import Path

# Sample files (written to a temporary folder)
Path("faq.md").write_text("# Shipping FAQ\n\nExpress shipping takes 1 day.\n\n## Countries\n\nWe ship to India and the UK.\n")
Path("help.html").write_text("""<html><head><title>Warranty</title><style>p {color: red}</style></head>
<body><nav>Home | Products | Login</nav><h1>Warranty</h1><p>All laptops have a 1 year warranty.</p>
<script>trackVisitor();</script><footer>© 2026 Acme</footer></body></html>""")
Path("prices.csv").write_text("product,price,stock\nLaptop Pro,1200,yes\nMouse,25,no\n")

class TextExtractor(HTMLParser):
    SKIP = {"script", "style", "nav", "footer", "header"}       # page furniture, not content
    def __init__(self):
        super().__init__()
        self.parts, self.skipping, self.title, self._in_title = [], 0, "", False
    def handle_starttag(self, tag, attrs):
        self.skipping += tag in self.SKIP
        self._in_title = tag == "title"
    def handle_endtag(self, tag):
        self.skipping -= tag in self.SKIP
        self._in_title = False
    def handle_data(self, data):
        if self._in_title:
            self.title = data.strip()
        elif not self.skipping and data.strip():
            self.parts.append(data.strip())

def load_markdown(path: Path) -> Document:
    text = path.read_text()
    title = re.search(r"^# (.+)$", text, re.M)
    return Document(text, str(path), {"title": title.group(1) if title else path.stem, "doc_type": "markdown"})

def load_html(path: Path) -> Document:
    parser = TextExtractor()
    parser.feed(path.read_text())
    return Document("\n".join(parser.parts), str(path), {"title": parser.title, "doc_type": "html"})

def load_csv(path: Path) -> list[Document]:
    # One document per row, written as "column: value" so each row reads on its own
    with path.open() as f:
        return [Document("; ".join(f"{k}: {v}" for k, v in row.items()), f"{path}#row{i}", {"doc_type": "csv"})
                for i, row in enumerate(csv.DictReader(f), start=1)]

LOADERS = {".md": load_markdown, ".html": load_html, ".csv": load_csv}

def load(path: Path) -> list[Document]:
    loader = LOADERS.get(path.suffix)
    if loader is None:
        raise ValueError(f"No loader for {path.suffix}")
    result = loader(path)
    return result if isinstance(result, list) else [result]

for name in ["faq.md", "help.html", "prices.csv"]:
    for d in load(Path(name)):
        print(f"{d.source:16} {d.metadata}\n    {d.text!r}")

Output

faq.md           {'title': 'Shipping FAQ', 'doc_type': 'markdown'}
    '# Shipping FAQ\n\nExpress shipping takes 1 day.\n\n## Countries\n\nWe ship to India and the UK.\n'
help.html        {'title': 'Warranty', 'doc_type': 'html'}
    'Warranty\nAll laptops have a 1 year warranty.'
prices.csv#row1  {'doc_type': 'csv'}
    'product: Laptop Pro; price: 1200; stock: yes'
prices.csv#row2  {'doc_type': 'csv'}
    'product: Mouse; price: 25; stock: no'

Three details matter here:

The HTML loader drops <nav>, <script>, <style> and <footer>. Otherwise every page's chunks would start with "Home | Products | Login", and all pages would look similar to the embedding model.
The Markdown loader keeps the headings. Structure-aware chunking uses them later.
The CSV loader writes one document per row, with column names repeated. price: 1200 means something to an embedding model. A bare 1200 on its own doesn't.

2.3 PDFs: the hard case¶

PDF is a format for printing, not for storing text. A PDF page is a set of positioned glyphs, so a parser has to guess the reading order, columns, tables and paragraphs. Common problems:

Problem	What you see in the extracted text	Fix
Repeated headers and footers	"ACME CONFIDENTIAL" and "Page 3 of 40" in every chunk	Remove lines that repeat on most pages
Hyphenation and line breaks	`pro-\ncessed` and sentences broken mid-line	Join hyphenated words and re-flow lines
Two columns	Lines from both columns interleaved	Layout-aware parser
Tables	Numbers with no column names	Table extraction, then rows as text
Scanned pages	Empty string (there's no text layer, only an image)	OCR or a vision model

Headers, footers and hyphenation are easy to fix yourself once you have the text of each page:

from collections import Counter

pages = [   # what a PDF text extractor typically returns, one string per page
    "ACME CONFIDENTIAL\nRefund Policy\nRefunds are pro-\ncessed within 5 working days\nof approval.\nPage 1 of 3",
    "ACME CONFIDENTIAL\nDamaged items get a full refund\nincluding shipping.\nPage 2 of 3",
    "",                                                     # a scanned page: no text layer at all
]

def clean_pdf_pages(pages: list[str]) -> tuple[list[str], list[int]]:
    non_empty = [p for p in pages if p.strip()]
    # A line (with digits replaced) that appears on most pages is a header or footer
    normalise = lambda line: re.sub(r"\d+", "#", line.strip())
    counts = Counter(normalise(line) for p in non_empty for line in set(p.splitlines()))
    repeated = {line for line, n in counts.items() if n >= max(2, 0.6 * len(non_empty))}

    cleaned, needs_ocr = [], []
    for number, page in enumerate(pages, start=1):
        if not page.strip():
            needs_ocr.append(number)
            cleaned.append("")
            continue
        lines = [l for l in page.splitlines() if normalise(l) not in repeated]
        text = "\n".join(lines)
        text = re.sub(r"(\w)-\n(\w)", r"\1\2", text)        # pro-\ncessed -> processed
        text = re.sub(r"(?<![.:!?])\n(?=[a-z])", " ", text)  # re-join a sentence broken across lines
        cleaned.append(text)
    return cleaned, needs_ocr

cleaned, needs_ocr = clean_pdf_pages(pages)
for number, text in enumerate(cleaned, start=1):
    print(f"--- page {number} ---\n{text}")
print("Pages that need OCR:", needs_ocr)

Output

--- page 1 ---
Refund Policy
Refunds are processed within 5 working days of approval.
--- page 2 ---
Damaged items get a full refund including shipping.
--- page 3 ---

Pages that need OCR: [3]

Keep the page number with every chunk you make from a PDF. "See page 2 of the Refund Policy" is a much better citation than a file name.

To get the page texts in the first place, use a PDF library:

# no-run — pip install pymupdf   (pypdf is a lighter pure-Python alternative)
import pymupdf

def pdf_pages(path: str) -> list[str]:
    with pymupdf.open(path) as pdf:
        return [page.get_text("text", sort=True) for page in pdf]   # sort=True: top-to-bottom, left-to-right

2.4 Tables: keep the column names¶

A flattened table is one of the most common silent failures. Laptop Pro 1200 yes Mouse 25 no can't answer "Is the mouse in stock?", because nothing links no to stock. Two representations work well:

rows = [{"product": "Laptop Pro", "price": "1200", "stock": "yes"},
        {"product": "Mouse", "price": "25", "stock": "no"}]

def table_to_markdown(rows: list[dict]) -> str:
    cols = list(rows[0])
    lines = ["| " + " | ".join(cols) + " |", "|" + "---|" * len(cols)]
    lines += ["| " + " | ".join(r[c] for c in cols) + " |" for r in rows]
    return "\n".join(lines)

def table_to_sentences(rows: list[dict], key: str) -> list[str]:
    return [f"{r[key]}: " + ", ".join(f"{c} is {v}" for c, v in r.items() if c != key) + "." for r in rows]

print(table_to_markdown(rows))
print()
print("\n".join(table_to_sentences(rows, key="product")))

Output

| product | price | stock |
|---|---|---|
| Laptop Pro | 1200 | yes |
| Mouse | 25 | no |

Laptop Pro: price is 1200, stock is yes.
Mouse: price is 25, stock is no.

Markdown tables suit small tables that fit in one chunk. LLMs read them well.
One sentence per row suits large tables. Each row becomes its own retrievable chunk that makes sense alone.
For big numeric tables (sales, logs), RAG is often the wrong tool. Load the data into a database and use text-to-SQL instead.

2.5 Scans, slides and complex layouts¶

Content	Approach
Clean digital PDFs, Word, HTML	Simple text extraction (PyMuPDF, pypdf, python-docx, BeautifulSoup)
Complex layouts: columns, tables, forms	Layout-aware parsers such as Docling, Unstructured, LlamaParse, Azure Document Intelligence, AWS Textract
Scanned pages and photos	OCR (Tesseract, a cloud OCR service) or a vision LLM
Charts, diagrams, slides	A vision LLM that writes a text description, indexed alongside the slide's own text

A vision model can turn a page image straight into structured Markdown (see multimodal models):

# no-run — render the scanned page as an image, then ask a vision model to transcribe it
import base64
import pymupdf
from openai import OpenAI

page = pymupdf.open("policy.pdf")[2]
png = base64.b64encode(page.get_pixmap(dpi=150).tobytes("png")).decode()
response = OpenAI().chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": [
        {"type": "text", "text": "Transcribe this page to Markdown. Keep headings and tables. "
                                 "Describe any chart in one sentence. Output only the Markdown."},
        {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{png}"}},
    ]}],
)
markdown = response.choices[0].message.content

Vision parsing costs far more per page than text extraction. Use it only for the pages that need it: the needs_ocr pages from 2.3, or pages where text extraction finds tables or images.

2.6 Parsing checklist¶

Every document has source, title, updated_at and permission metadata. PDFs keep the page.
Navigation, cookie banners, scripts, headers and footers are removed.
Hyphenation and broken lines are repaired.
Tables keep their column names (Markdown or one sentence per row).
Pages with no text are detected and sent to OCR, not silently indexed as empty.
Spot-check: print the parsed text of 10 random documents and read it. Nothing beats this.

Interview questions¶

What are the common problems when parsing PDFs for RAG?

PDFs store positioned glyphs, not text structure. Common problems: repeated headers and footers polluting chunks, hyphenation and broken lines, multi-column reading order, tables losing their column names, and scanned pages with no text layer. Fixes: remove repeated lines, re-flow text, use layout-aware parsers for complex documents, extract tables into Markdown or row sentences, and OCR or a vision model for scans. Keep page numbers for citations.

How would you handle tables in a RAG pipeline?

Extract them with a table-aware parser, then keep each row's meaning by repeating the column names. Small tables go in as Markdown inside one chunk. Large tables become one sentence or record per row. For large numeric or analytical tables, store them in a database and answer with text-to-SQL instead of vector search.

Why does metadata matter so much in RAG?

It powers citations (source, page), freshness and sync (updated_at, content hash), security (tenant, allowed groups filtered at retrieval), and filtering or routing (doc type, product, language). Capture it at load time. Adding it later means re-ingesting everything.

Practice¶

Add a .txt loader to LOADERS that uses the first line as the title.
Extend clean_pdf_pages to store the page number in each page's metadata, and build Document objects from it.
Run pymupdf on a real PDF from your own files and compare the raw and cleaned text.

Next: 3. Chunking for RAG Back to: RAG overview · Notes overview