4. Prompt chaining & routing¶

Intermediate · 13 min read

One prompt that "reads the email, classifies it, extracts the order number, checks the policy and writes a reply" does five jobs at once, and when it fails you can't tell which job broke. A chain splits the work into small prompts, each with one job, connected by your code. Each step is easier, testable and can use a different model.

4.1 One big prompt vs a chain¶

	One prompt	Chain
Quality on multi-part tasks	Steps get skipped or blended	Each step gets full attention
Debugging	Which part failed?	Inspect each step's output
Cost and latency	One call	Several calls (but each can be smaller or cheaper)
Control	The model decides the order	Your code decides, and can stop or branch

Chain when a task has distinct stages, when you need to check something between stages, or when stages need different models.

4.2 A simple chain with a fake model¶

Every example on this page uses a small fake model, so it runs offline. In production, llm() would call your provider.

import json
import re

def llm(prompt: str) -> str:
    """A fake model: returns canned outputs, chosen by which step's prompt it receives."""
    if prompt.startswith("CLASSIFY"):
        return json.dumps({"intent": "refund_status"})
    if prompt.startswith("EXTRACT"):
        found = re.search(r"#(\d+)", prompt)
        return json.dumps({"order_id": found.group(1) if found else None})
    if prompt.startswith("REPLY"):
        return "Hi Asha, your refund for order 4471 was approved on 3 Oct and will reach you within 5 working days."
    return ""

email = "Hi, I returned order #4471 last week. When will I get my money back? - Asha"

intent = json.loads(llm(f"CLASSIFY the intent of this email. JSON only.\n<email>{email}</email>"))["intent"]
order = json.loads(llm(f"EXTRACT the order id. JSON only, null if missing.\n<email>{email}</email>"))["order_id"]
status = {"4471": "refund approved on 3 Oct"}.get(order, "not found")      # a normal function, not the LLM
reply = llm(f"REPLY to the customer in 2 sentences.\nIntent: {intent}\nOrder status: {status}\n<email>{email}</email>")

print("intent:", intent, "| order:", order, "| status:", status)
print("reply :", reply)

Output

intent: refund_status | order: 4471 | status: refund approved on 3 Oct
reply : Hi Asha, your refund for order 4471 was approved on 3 Oct and will reach you within 5 working days.

Notice the middle step: the order lookup is ordinary code, not a model call. Chains let you put deterministic work (database lookups, calculations, policy checks) where it belongs.

4.3 Gates: check before you continue¶

Between steps, validate the output and stop or branch if it's wrong. This is where chains beat one big prompt.

def gate_order_id(order_id) -> str | None:
    if order_id is None:
        return "ask_customer_for_order_id"
    if not re.fullmatch(r"\d{4,8}", str(order_id)):
        return "invalid_order_id"
    return None

for candidate in ["4471", None, "44-71"]:
    print(f"{candidate!s:6} -> {gate_order_id(candidate) or 'continue'}")

Output

4471   -> continue
None   -> ask_customer_for_order_id
44-71  -> invalid_order_id

Typical gates: JSON parses and matches the schema; extracted IDs exist in the database; a draft passes a safety or policy check; confidence is above a threshold.

4.4 Routing: send each input to the right prompt or model¶

A router is a cheap first step that decides where an input goes: a specialised prompt, a cheaper or stronger model, a human, or a fixed reply.

ROUTES = {
    "refund_status": {"prompt": "refund_reply_v3", "model": "small"},
    "complaint":     {"prompt": "complaint_reply_v2", "model": "large"},
    "legal":         {"prompt": None, "model": None, "handoff": "human"},
}

def route(intent: str) -> dict:
    return ROUTES.get(intent, {"prompt": "general_reply_v1", "model": "small"})

for intent in ["refund_status", "legal", "something_new"]:
    print(f"{intent:14} -> {route(intent)}")

Output

refund_status  -> {'prompt': 'refund_reply_v3', 'model': 'small'}
legal          -> {'prompt': None, 'model': None, 'handoff': 'human'}
something_new  -> {'prompt': 'general_reply_v1', 'model': 'small'}

Routing is also the main cost lever: most traffic goes to a small model, and only hard or sensitive cases reach the large one or a human.

4.5 Map-reduce for long inputs¶

When an input is too long for one good call (a 2-hour transcript, a 300-page report), map a prompt over chunks in parallel, then reduce the results with a final prompt:

def chunks(text: str, size: int) -> list[str]:
    words = text.split()
    return [" ".join(words[i:i + size]) for i in range(0, len(words), size)]

def summarise_chunk(chunk: str) -> str:            # map step (fake): first 6 words as a "summary"
    return " ".join(chunk.split()[:6]).rstrip(".:") + " …"

transcript = ("Priya: pricing launch moves to November. Ravi: we need legal sign-off first. "
              "Asha: I'll draft the FAQ by Friday. Ravi: budget approved for two more engineers. ") * 3
parts = [summarise_chunk(c) for c in chunks(transcript, 20)]
reduce_prompt = "Combine these partial notes into decisions and action items:\n" + "\n".join(f"- {p}" for p in parts)
print(f"{len(transcript.split())} words -> {len(parts)} chunks")
print(reduce_prompt)

Output

78 words -> 4 chunks
Combine these partial notes into decisions and action items:
- Priya: pricing launch moves to November …
- budget approved for two more engineers …
- draft the FAQ by Friday. Ravi …
- need legal sign-off first. Asha: I'll …

The map steps run in parallel, so latency stays close to one call. Keep timestamps or chunk IDs in each partial result so the final answer can point back to the source.

4.6 Chains vs agents¶

	Chain (workflow)	Agent
Who decides the steps?	Your code, fixed in advance	The model, at run time
Predictability	High; easy to test	Lower; needs more evals and limits
Best for	Known processes: triage, extraction, reports	Open-ended tasks where steps depend on what's found

Start with a chain. Move to an agent only when the steps genuinely can't be known in advance.

Interview questions¶

When would you split a prompt into a chain?

When the task has distinct stages, needs validation between stages, or stages need different models or deterministic code. Chains make each step simpler and testable, and failures easy to locate, at the cost of more calls and some latency.

What is a router in an LLM application?

A cheap first step, often a small model or classifier, that decides where each input goes: a specialised prompt, a cheaper or stronger model, a fixed answer or a human. It improves quality by specialising prompts and cuts cost by sending most traffic to small models.

How do you process a document longer than the context window?

Map-reduce: split it into chunks, run a prompt over each chunk in parallel, then combine the partial results in a final prompt. Keep chunk IDs or timestamps so the result is traceable. Alternatives are RAG over the chunks or a refine chain that updates a running summary.

Practice¶

Add a gate after CLASSIFY that sends unknown intents to general_reply_v1.
Change chunks to overlap by 5 words, and explain when overlap helps.

Next: 5. Structured output prompting Back to: Prompt Engineering overview · Notes overview