2. LlamaIndex¶

Intermediate · 12 min read

Where LangChain started as a general toolkit for LLM apps, LlamaIndex started from one question: how do you connect an LLM to your data? Its strengths are the RAG-specific parts: ingestion, chunking into nodes, indexes, retrieval strategies and query engines. Many of the advanced RAG techniques from the RAG series exist in LlamaIndex as a single class.

2.1 The core concepts¶

LlamaIndex concept	What it is	In the RAG series
`Document`	A loaded file or record: text + metadata	The document record
`Node`	A chunk of a document, with metadata and relationships (source, previous, next, parent, child)	Chunks
Reader	Loads files into Documents (`SimpleDirectoryReader`, plus hundreds on LlamaHub)	Loaders
Node parser	Splits Documents into Nodes (`SentenceSplitter`, `MarkdownNodeParser`, `HierarchicalNodeParser` ...)	Chunking
Index	A structure over nodes, usually `VectorStoreIndex`	Embeddings + vector DB
Retriever	Gets the relevant nodes for a query	Retrieval
Node postprocessor	Reranks, filters or replaces retrieved nodes	Reranking
Query engine	Retriever + postprocessors + response synthesis: question in, answer out	The whole query pipeline
Chat engine	A query engine with conversation memory and query condensing	Query rewriting
`Settings`	Global defaults: LLM, embedding model, chunk size	Your config

2.2 A RAG app in five lines¶

# no-run — pip install llama-index   (uses OpenAI by default; reads OPENAI_API_KEY)
from llama_index.core import SimpleDirectoryReader, VectorStoreIndex

documents = SimpleDirectoryReader("data").load_data()          # PDFs, Markdown, DOCX, ... in the folder
index = VectorStoreIndex.from_documents(documents)               # chunk + embed + store (in memory)
query_engine = index.as_query_engine(similarity_top_k=3)
response = query_engine.query("How long do refunds take?")

print(response)                                                  # the answer
for node in response.source_nodes:                               # the chunks it used, for citations
    print(f"{node.score:.2f}  {node.metadata.get('file_name')}  {node.text[:60]!r}")

Those five lines hide every decision from the RAG series: chunk size, embedding model, prompt, k. The defaults are reasonable for a demo. Production means setting each of them yourself (2.4).

2.3 Nodes and relationships, from scratch¶

What makes a LlamaIndex Node more than a string is its relationships: each node knows its source document and its neighbours. Advanced retrievers use these links, for example to fetch the next chunk or merge children into their parent. Here's the idea in plain Python:

import re

def to_nodes(doc_id: str, text: str, sentences_per_node: int = 2) -> list[dict]:
    sentences = re.split(r"(?<=\.)\s+", text.strip())
    nodes = [{"id": f"{doc_id}-n{i // sentences_per_node}",
              "text": " ".join(sentences[i:i + sentences_per_node]),
              "relationships": {"source": doc_id}}
             for i in range(0, len(sentences), sentences_per_node)]
    for prev, nxt in zip(nodes, nodes[1:]):                  # link neighbours both ways
        prev["relationships"]["next"] = nxt["id"]
        nxt["relationships"]["previous"] = prev["id"]
    return nodes

text = ("Express shipping takes 1 day. It costs 9 dollars. Order before 2 pm for same-day dispatch. "
        "Standard shipping takes 3 to 5 days. It is free over 50 dollars.")
nodes = {n["id"]: n for n in to_nodes("shipping.md", text)}
for n in nodes.values():
    print(n["id"], n["relationships"])
    print("   ", n["text"])

hit = nodes["shipping.md-n0"]                                # suppose retrieval matched this node
print("\nhit + next node:", hit["text"], nodes[hit["relationships"]["next"]]["text"])

Output

shipping.md-n0 {'source': 'shipping.md', 'next': 'shipping.md-n1'}
    Express shipping takes 1 day. It costs 9 dollars.
shipping.md-n1 {'source': 'shipping.md', 'previous': 'shipping.md-n0', 'next': 'shipping.md-n2'}
    Order before 2 pm for same-day dispatch. Standard shipping takes 3 to 5 days.
shipping.md-n2 {'source': 'shipping.md', 'previous': 'shipping.md-n1'}
    It is free over 50 dollars.

hit + next node: Express shipping takes 1 day. It costs 9 dollars. Order before 2 pm for same-day dispatch. Standard shipping takes 3 to 5 days.

Following the next link brings back the 2 pm cut-off that the hit alone would miss. Notice the chunking flaw too: n1 mixes express and standard shipping, and n2 says "It is free" with no subject. Fixed sentence counts ignore structure, which is why the chunking page splits by section and adds headers.

The real TextNode stores the same links in node.relationships, using NodeRelationship.SOURCE, PREVIOUS, NEXT, PARENT and CHILD.

2.4 Customising the pipeline¶

Production RAG means choosing the models, the chunker and a persistent vector store yourself:

# no-run — pip install llama-index llama-index-vector-stores-chroma chromadb
import chromadb
from llama_index.core import Settings, SimpleDirectoryReader, StorageContext, VectorStoreIndex
from llama_index.core.ingestion import IngestionPipeline
from llama_index.core.node_parser import SentenceSplitter
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.llms.openai import OpenAI
from llama_index.vector_stores.chroma import ChromaVectorStore

Settings.llm = OpenAI(model="gpt-4o-mini", temperature=0)
Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small")

collection = chromadb.PersistentClient(path="./chroma_db").get_or_create_collection("docs")
vector_store = ChromaVectorStore(chroma_collection=collection)

pipeline = IngestionPipeline(
    transformations=[SentenceSplitter(chunk_size=512, chunk_overlap=64), Settings.embed_model],
    vector_store=vector_store,                     # nodes go straight into Chroma
)
documents = SimpleDirectoryReader("data", filename_as_id=True).load_data()   # stable IDs, so re-runs upsert
pipeline.run(documents=documents)

index = VectorStoreIndex.from_vector_store(vector_store)   # reconnect later without re-ingesting

An IngestionPipeline given a docstore also skips unchanged documents, using hashes. That's the incremental sync from the vector DB notes, built in.

2.5 Advanced retrieval in a few lines¶

Here's where LlamaIndex shines. Techniques from the RAG series map to ready-made components:

RerankingSentence windowAuto-merging (parent–child)

# no-run — retrieve 20, rerank to 3 (pip install llama-index-postprocessor-sbert-rerank or use core's version)
from llama_index.core.postprocessor import SentenceTransformerRerank

rerank = SentenceTransformerRerank(model="cross-encoder/ms-marco-MiniLM-L-6-v2", top_n=3)
query_engine = index.as_query_engine(similarity_top_k=20, node_postprocessors=[rerank])

# no-run — embed single sentences, send each hit with 3 sentences either side
from llama_index.core.node_parser import SentenceWindowNodeParser
from llama_index.core.postprocessor import MetadataReplacementPostProcessor

parser = SentenceWindowNodeParser.from_defaults(window_size=3, window_metadata_key="window")
index = VectorStoreIndex(parser.get_nodes_from_documents(documents))
query_engine = index.as_query_engine(
    similarity_top_k=5,
    node_postprocessors=[MetadataReplacementPostProcessor(target_metadata_key="window")],
)

# no-run — index small leaf nodes; if enough siblings are retrieved, return their parent instead
from llama_index.core import StorageContext, VectorStoreIndex
from llama_index.core.node_parser import HierarchicalNodeParser, get_leaf_nodes
from llama_index.core.query_engine import RetrieverQueryEngine
from llama_index.core.retrievers import AutoMergingRetriever

nodes = HierarchicalNodeParser.from_defaults(chunk_sizes=[2048, 512, 128]).get_nodes_from_documents(documents)
storage = StorageContext.from_defaults()
storage.docstore.add_documents(nodes)                          # parents live in the docstore
index = VectorStoreIndex(get_leaf_nodes(nodes), storage_context=storage)   # only leaves are embedded

retriever = AutoMergingRetriever(index.as_retriever(similarity_top_k=12), storage)
query_engine = RetrieverQueryEngine.from_args(retriever)

These are the parent–child and reranking ideas, each a few lines of configuration. LlamaIndex also has hybrid search (with vector stores that support it), metadata filters, query transformations (including HyDE), and a property-graph index for graph RAG.

2.6 Chat engines¶

A query engine answers one question. A chat engine keeps history and rewrites follow-ups. The condense_plus_context mode is exactly the query-rewriting pattern: condense the history and the new message into a standalone question, retrieve, then answer with the history.

# no-run
chat_engine = index.as_chat_engine(
    chat_mode="condense_plus_context",
    system_prompt="Answer only from the documents. Say you don't know if they don't cover it.",
)
print(chat_engine.chat("How long is the warranty on phones?"))
print(chat_engine.chat("What about laptops?"))        # rewritten to "How long is the warranty on laptops?"

2.7 Agents and workflows¶

To let an agent decide when to search, wrap a query engine as a tool:

# no-run
from llama_index.core.agent.workflow import FunctionAgent
from llama_index.core.tools import FunctionTool, QueryEngineTool

policies = QueryEngineTool.from_defaults(query_engine, name="search_policies",
                                         description="Search company policies: refunds, shipping, warranty.")

def get_order_status(order_id: str) -> str:
    """Look up the shipping status of an order by its ID."""
    return {"A100": "Shipped, arriving Friday"}.get(order_id, "Order not found")

agent = FunctionAgent(tools=[policies, FunctionTool.from_defaults(get_order_status)],
                      llm=Settings.llm, system_prompt="You are a support assistant.")
response = await agent.run("Where is order A100, and how long do refunds take?")

Workflows are LlamaIndex's way to write custom multi-step pipelines. Steps are async functions that receive and emit typed events, and the workflow routes events to whichever step accepts them:

# no-run
from llama_index.core.workflow import Event, StartEvent, StopEvent, Workflow, step

class Retrieved(Event):
    nodes: list

class SimpleRAG(Workflow):
    @step
    async def retrieve(self, ev: StartEvent) -> Retrieved:
        return Retrieved(nodes=index.as_retriever(similarity_top_k=5).retrieve(ev.query))

    @step
    async def answer(self, ev: Retrieved) -> StopEvent:
        context = "\n\n".join(n.text for n in ev.nodes)
        reply = await Settings.llm.acomplete(f"Answer from this context only:\n{context}\n\nQuestion: ...")
        return StopEvent(result=str(reply))

result = await SimpleRAG(timeout=60).run(query="How long do refunds take?")

Workflows and LangGraph solve the same problem, explicit control flow for multi-step LLM apps, with different styles. LangGraph uses a graph of nodes over shared state. Workflows use steps connected by events.

2.8 LlamaIndex or LangChain?¶

	LlamaIndex	LangChain (+ LangGraph)
Centre of gravity	Data and retrieval: RAG over documents	General LLM apps, tools and agents
Advanced RAG built in	Many retrievers and postprocessors (auto-merging, sentence window, rerankers)	Fewer built in. You compose more yourself
Agents and control flow	`FunctionAgent`, Workflows	`create_agent`, LangGraph (more mature for complex agents)
Integrations	LlamaHub: readers, vector stores, LLMs	The largest integration library
Document parsing	LlamaParse (hosted) for complex PDFs	Community loaders, Unstructured, Docling
Pick it when	The app is mainly "chat with / search our documents"	The app is mainly agents, tools and multi-step workflows

They aren't mutually exclusive. A common setup is LlamaIndex for ingestion and retrieval, exposed as a tool inside a LangGraph agent.

Interview questions¶

What is the difference between a Document and a Node in LlamaIndex?

A Document is a whole loaded source (file, page, record) with metadata. A Node is a chunk of a Document, the unit that's embedded, retrieved and cited. It carries metadata and relationships to its source document, its neighbours and its parent or child nodes. Advanced retrievers use those relationships: sentence windows use neighbours, auto-merging uses parents.

What is a query engine versus a chat engine?

A query engine is stateless: retrieve, postprocess (rerank or filter), then synthesise an answer for one question. A chat engine adds conversation memory. In condense-plus-context mode it rewrites each follow-up into a standalone question using the history before retrieving, then answers with the history in context.

When would you choose LlamaIndex over LangChain?

When the core of the app is retrieval over your own data: many document types, advanced retrieval strategies (auto-merging, sentence windows, rerankers) and query engines with sources. LangChain and LangGraph are stronger for tool-heavy agents and complex control flow. Teams often combine them.

Practice¶

Change sentences_per_node to 1 in to_nodes and print the hit with both of its neighbours.
Add a parent relationship: group every two nodes under a parent ID and look up a hit's parent.
Sketch which LlamaIndex classes you'd use for the RAG production checklist items in 8.1–8.3.

Next: 3. LangGraph Back to: Frameworks overview · Notes overview