2. Tokenization¶

Beginner → Intermediate · 11 min read

LLMs don't read characters or words — they read tokens: frequent pieces of text mapped to integer IDs. Tokens are what you pay for, what fills the context window, and the reason models are bad at counting letters. Every GenAI engineer should be able to count them.

2.1 Three ways to split text¶

import re

text = "Tokenizers split unbelievably long words."

chars = list(text)
words = re.findall(r"\w+|[^\w\s]", text)
print(len(chars), "characters")
print(len(words), "words:", words)

Output

41 characters
6 words: ['Tokenizers', 'split', 'unbelievably', 'long', 'words', '.']

Level	Vocabulary	Problem
Characters	tiny (~100s)	sequences become very long; each piece carries little meaning
Words	huge (millions)	any new word, typo or name is "unknown"
Sub-words	~50k–200k	the middle ground: common words = 1 token, rare words = a few pieces

All modern LLMs use sub-word tokenizers — mostly BPE (byte-pair encoding).

2.2 BPE from scratch¶

BPE starts from characters and repeatedly merges the most frequent adjacent pair into a new token. Here it is in 20 lines:

from collections import Counter

corpus = ["low", "lower", "lowest", "newer", "newest", "wider", "widest"] * 3
words = Counter(tuple(w) + ("</w>",) for w in corpus)        # </w> marks the end of a word

def most_frequent_pair(words: Counter) -> tuple[str, str]:
    pairs = Counter()
    for symbols, freq in words.items():
        for a, b in zip(symbols, symbols[1:]):
            pairs[a, b] += freq
    return max(pairs, key=pairs.get)

def merge(words: Counter, pair: tuple[str, str]) -> Counter:
    merged = Counter()
    for symbols, freq in words.items():
        out, i = [], 0
        while i < len(symbols):
            if symbols[i:i + 2] == pair:
                out.append(symbols[i] + symbols[i + 1]); i += 2
            else:
                out.append(symbols[i]); i += 1
        merged[tuple(out)] += freq
    return merged

for step in range(1, 7):
    pair = most_frequent_pair(words)
    words = merge(words, pair)
    print(step, pair, "→", "".join(pair))

print(sorted({" ".join(w) for w in words}))

Output

1 ('w', 'e') → we
2 ('l', 'o') → lo
3 ('r', '</w>') → r</w>
4 ('s', 't') → st
5 ('st', '</w>') → st</w>
6 ('lo', 'we') → lowe
['lo w </w>', 'lowe r</w>', 'lowe st</w>', 'n e we r</w>', 'n e we st</w>', 'w i d e r</w>', 'w i d e st</w>']

After six merges the words are built from learned pieces such as lo, lowe, we, st</w> and r</w> (the end of "lower", "newer", "wider"). Real tokenizers do this over terabytes of text with ~100k merges, and work on bytes, so any text — emoji, Hindi, code — can be encoded with no "unknown" token.

2.3 Real tokens with `tiktoken`¶

tiktoken is OpenAI's tokenizer library. o200k_base is the encoding used by GPT-4o-family models:

import tiktoken

enc = tiktoken.get_encoding("o200k_base")

ids = enc.encode("Tokenizers split unbelievably long words.")
print(ids)
print([enc.decode([i]) for i in ids])
print(len(ids), "tokens")

Output

[4421, 24223, 12648, 180692, 1701, 6391, 13]
['Token', 'izers', ' split', ' unbelievably', ' long', ' words', '.']
7 tokens

Notice: the leading space is part of the token (' split'), and a long but common word can still be one token.

What costs more tokens¶

samples = {
    "English": "Please summarise this document in three bullet points.",
    "Hindi": "कृपया इस दस्तावेज़ का तीन बिंदुओं में सारांश दें।",
    "Code": "def add(a: int, b: int) -> int:\n    return a + b",
    "JSON": '{"name": "Priya", "skills": ["python", "rag", "agents"]}',
    "Numbers": "3.14159265358979 2718281828 1234567890",
}
for name, s in samples.items():
    n = len(enc.encode(s))
    print(f"{name:8} {len(s):3} chars → {n:2} tokens  ({len(s) / n:.1f} chars/token)")

Output

English   54 chars → 10 tokens  (5.4 chars/token)
Hindi     49 chars → 19 tokens  (2.6 chars/token)
Code      48 chars → 18 tokens  (2.7 chars/token)
JSON      56 chars → 20 tokens  (2.8 chars/token)
Numbers   38 chars → 17 tokens  (2.2 chars/token)

Rules of thumb

English ≈ 4 characters or ¾ of a word per token. Non-English languages, code, JSON and numbers use noticeably more tokens for the same length — so the same prompt costs more in Hindi than in English. Other providers (Anthropic, Google, Llama) use their own tokenizers; counts differ by 10–30%. For exact numbers use the provider's token-counting endpoint or the usage field in each response.

2.4 Why tokens explain odd LLM behaviour¶

for w in ["strawberry", " strawberry", "Strawberry", "STRAWBERRY"]:
    print(repr(w), [enc.decode([i]) for i in enc.encode(w)])

Output

'strawberry' ['st', 'raw', 'berry']
' strawberry' [' strawberry']
'Strawberry' ['Str', 'aw', 'berry']
'STRAWBERRY' ['ST', 'RAW', 'B', 'ERRY']

The model never sees the letters r, r, r — it sees a few opaque IDs, split differently for each casing. That is why "how many r's in strawberry" was hard, why spelling and character-level tasks are unreliable, and why the same word with or without a leading space is a different token.

2.5 Counting chat messages and estimating cost¶

def count_chat_tokens(messages: list[dict], enc=enc) -> int:
    # each message has a few tokens of formatting overhead; ~3 per message + 3 to prime the reply
    return sum(3 + len(enc.encode(m["content"])) for m in messages) + 3

messages = [
    {"role": "system", "content": "You are a helpful assistant for an e-commerce store."},
    {"role": "user", "content": "What is your refund policy for damaged items?"},
]
prompt_tokens = count_chat_tokens(messages)
print(prompt_tokens, "prompt tokens")

PRICE_PER_1M = {"input": 0.15, "output": 0.60}          # USD — example prices, check your provider
def cost_usd(prompt_toks: int, completion_toks: int) -> float:
    return (prompt_toks * PRICE_PER_1M["input"] + completion_toks * PRICE_PER_1M["output"]) / 1_000_000

per_request = cost_usd(prompt_tokens + 1500, 300)        # + 1,500 tokens of retrieved RAG context
print(f"${per_request:.6f} per request → ${per_request * 100_000:,.2f} per 100k requests")

Output

29 prompt tokens
$0.000409 per request → $40.94 per 100k requests

Notice where the cost comes from: the retrieved context dwarfs the question. Fewer, better chunks are the biggest cost lever in RAG.

2.6 Fitting the context window¶

Truncate a document to a token budget¶

def truncate_to_tokens(text: str, max_tokens: int) -> str:
    ids = enc.encode(text)
    return text if len(ids) <= max_tokens else enc.decode(ids[:max_tokens])

doc = "Refunds are processed within five working days of approval. " * 50
short = truncate_to_tokens(doc, 20)
print(len(enc.encode(doc)), "→", len(enc.encode(short)), "tokens")
print(short)

Output

551 → 20 tokens
Refunds are processed within five working days of approval. Refunds are processed within five working days of

Cutting on token IDs never produces a broken character, unlike cutting a string at a byte offset.

Keep the newest chat turns that fit¶

def trim_history(system: dict, history: list[dict], budget: int) -> list[dict]:
    kept, used = [], count_chat_tokens([system])
    for m in reversed(history):                       # newest first
        cost = 3 + len(enc.encode(m["content"]))
        if used + cost > budget:
            break
        kept.append(m)
        used += cost
    return [system, *reversed(kept)]

system = {"role": "system", "content": "Be concise."}
history = [{"role": "user" if i % 2 == 0 else "assistant", "content": f"message number {i} " * 5}
           for i in range(10)]
trimmed = trim_history(system, history, budget=120)
print(len(history), "→", len(trimmed) - 1, "messages kept:", [m["content"].split()[2] for m in trimmed[1:]])

Output

10 → 4 messages kept: ['6', '7', '8', '9']

More advanced strategies: summarise old turns into one message, or store them in a vector store and retrieve only the relevant ones (long-term memory for agents).

2.7 Special tokens¶

Tokenizers reserve IDs for control markers — end of text, start of a message, tool-call boundaries. By default tiktoken refuses to encode them from user input, which protects you from users injecting fake control tokens:

try:
    enc.encode("hello <|endoftext|>")
except ValueError as e:
    print("refused:", str(e)[:60], "…")
print(len(enc.encode("hello <|endoftext|>", disallowed_special=())), "tokens when treated as plain text")

Output

refused: Encountered text corresponding to disallowed special token ' …
8 tokens when treated as plain text

Interview questions¶

What is a token and why does it matter?

The unit an LLM reads and writes: a frequent sub-word piece mapped to an integer ID (≈ 4 English characters). Tokens determine cost (billed per input/output token), latency (output is generated one token at a time), and what fits in the context window.

How does BPE work?

Start with characters (or bytes) as the vocabulary. Count adjacent pairs across the corpus, merge the most frequent pair into a new token, repeat until the vocabulary reaches the target size. To encode new text, apply the learned merges in order. Byte-level BPE can encode any input with no unknown token.

Why are LLMs bad at counting letters or reversing words?

They see token IDs, not characters. "strawberry" may be 2–3 tokens with no explicit letters, so character-level operations require knowledge the model only has indirectly.

Practice¶

Count the tokens of your longest system prompt. What does it cost per 1M requests?
Change trim_history so it always keeps the first user message (often the task statement).

Next: Text representation — turning text into numbers you can compare.

2. Tokenization¶

2.1 Three ways to split text¶

2.2 BPE from scratch¶

2.3 Real tokens with tiktoken¶

What costs more tokens¶

2.4 Why tokens explain odd LLM behaviour¶

2.5 Counting chat messages and estimating cost¶

2.6 Fitting the context window¶

Truncate a document to a token budget¶

Keep the newest chat turns that fit¶

2.7 Special tokens¶

Interview questions¶

Practice¶

2.3 Real tokens with `tiktoken`¶