Extract text from PDFs, Word files and web pages¶
Beginner · 12 min · tested with pypdf 6, python-docx 1.2, beautifulsoup4 4
Every RAG project starts by turning documents into text, and the quality of that text caps the quality of every answer. Garbage in (footers repeated on every chunk, words split across lines, navigation menus) means garbage retrieved. This tutorial extracts and cleans the three formats you'll meet most.
1. PDFs with pypdf¶
from pypdf import PdfReader
reader = PdfReader("refund-policy.pdf")
print(len(reader.pages), "pages")
for number, page in enumerate(reader.pages, start=1):
print(f"--- page {number} ---")
print(page.extract_text())
2 pages
--- page 1 ---
ShopKart Refund Policy
1. Refunds are processed within 5 working
days of approval.
2. Damaged items get a full refund.
ShopKart - page 1
--- page 2 ---
3. Gift cards are non-refundable.
Page 2 of 2
ShopKart - page 2
Two problems show up straight away, and they're in almost every real PDF:
- A footer ("ShopKart - page 1") repeated on every page. Left in, it pollutes every chunk.
- A sentence broken across lines ("5 working / days"), because PDFs store positioned lines, not paragraphs.
Section 4 cleans both. Keep the page number of each piece of text: you need it to show citations ("Source: refund-policy.pdf, page 1").
2. Word files with python-docx¶
from docx import Document
doc = Document("onboarding.docx")
parts = [p.text for p in doc.paragraphs if p.text.strip()]
for table in doc.tables: # tables are NOT included in doc.paragraphs
for row in table.rows:
parts.append(" | ".join(cell.text for cell in row.cells))
print("\n".join(parts))
Onboarding guide
Welcome to the support team.
Escalate refunds over Rs 5,000 to a manager.
Tool | Use
Zendesk | Tickets
The classic mistake is reading only doc.paragraphs and silently losing every table, which is often where the
important facts are. Writing table rows as a | b keeps them readable for the model. (python-docx reads
.docx only; convert old .doc files first.) Headings come through as normal paragraphs; check
p.style.name.startswith("Heading") if you want to split documents by section.
3. Web pages with BeautifulSoup¶
import requests
from bs4 import BeautifulSoup
html = requests.get("https://example.com/faq", timeout=15).text
soup = BeautifulSoup(html, "html.parser")
for tag in soup(["script", "style", "nav", "header", "footer", "aside", "form"]):
tag.decompose() # remove menus, scripts and boilerplate
main = soup.find("main") or soup.find("article") or soup.body
print(soup.title.get_text())
print(main.get_text(separator="\n", strip=True))
separator="\n" stops words from neighbouring elements running together, and strip=True drops blank lines.
Without removing nav and footer, every page of a site would start with "Home | Shop | Cart". For JavaScript-heavy
sites where requests returns an empty shell, use a headless browser (Playwright) to render the page first. Only
scrape sites whose terms allow it, and respect robots.txt.
4. Clean the text¶
A small cleaning function fixes the usual PDF damage. Tune the footer patterns to your documents:
import re
def clean(text: str) -> str:
text = re.sub(r"ShopKart - page \d+|Page \d+ of \d+", "", text) # repeated headers and footers
text = re.sub(r"(\w)-\n(\w)", r"\1\2", text) # re-join hyphenated words
text = re.sub(r"(?<![.\n:])\n(?!\n|\d+\.)", " ", text) # join lines broken mid-sentence
text = re.sub(r"[ \t]+", " ", text) # collapse spaces
text = re.sub(r"\n{3,}", "\n\n", text) # at most one blank line
return text.strip()
print(clean(reader.pages[0].extract_text()))
ShopKart Refund Policy
1. Refunds are processed within 5 working days of approval.
2. Damaged items get a full refund.
The footer is gone and the broken sentence is whole again, while list items keep their own lines. Always print a few pages before and after cleaning: cleaning rules that are too greedy can merge headings into paragraphs.
5. Keep metadata for citations¶
Return text with where it came from, so your RAG app can cite sources and you can debug bad answers:
from pathlib import Path
def load_pdf(path: str) -> list[dict]:
reader = PdfReader(path)
return [
{"text": clean(page.extract_text()), "source": Path(path).name, "page": number}
for number, page in enumerate(reader.pages, start=1)
if page.extract_text().strip()
]
pages = load_pdf("refund-policy.pdf")
print(pages[0])
# → {'text': 'ShopKart Refund Policy\n1. Refunds are processed within 5 working days of approval.\n2. Damaged items get a full refund.', 'source': 'refund-policy.pdf', 'page': 1}
Then chunk each page's text and carry source and page into every chunk's metadata (see
Chunking for RAG).
6. When the simple tools aren't enough¶
| Problem | Symptom | Use |
|---|---|---|
| Scanned PDF (an image of text) | extract_text() returns an empty string |
OCR: Tesseract (pytesseract), or a cloud OCR / document AI service |
| Complex layouts, multi-column, tables | Columns interleaved, tables flattened into soup | pymupdf (fast, good layout), pdfplumber (tables), Docling or Unstructured (layout-aware) |
| Many formats at once (PPTX, XLSX, emails) | One parser per format | Unstructured, Docling, or LlamaIndex / LangChain loaders |
| Very important documents | Any of the above | A vision-capable LLM reading page images, checked against the text version |
Check for scanned pages automatically: if a page yields fewer than ~20 characters of text, send it to OCR. More in Loading & parsing documents.
Checklist¶
- Each format has its own loader; Word tables are included.
- Repeated headers and footers are removed; broken lines are re-joined.
- Every piece of text keeps its source file and page number.
- Empty pages are detected and sent to OCR.
- You've read the extracted text of a few real documents before building anything on it.