Run an LLM locally with Ollama¶

Beginner · 15 min · no API key, no cost, works offline

Cloud APIs cost money per token and send your data to someone else's servers. Ollama runs open models (Llama, Qwen, Gemma, Mistral and more) on your own machine. It's perfect for learning, for experiments you'd rather not pay for, and for data that can't leave your laptop. The quality is below the best cloud models, but small local models handle summarising, extraction and simple RAG surprisingly well.

1. Install¶

WindowsmacOSLinux

Download the installer from ollama.com/download, or:

winget install Ollama.Ollama

Download the app from ollama.com/download, or brew install ollama.

curl -fsSL https://ollama.com/install.sh | sh

Ollama runs a small server in the background on http://localhost:11434. Check it's up:

ollama --version
curl http://localhost:11434          # → Ollama is running

2. Pick a model that fits your machine¶

Models are downloaded once and run from memory. The size you can run depends on your RAM (or GPU memory if you have an NVIDIA or Apple Silicon GPU):

Your machine	Model size to try	Examples
8 GB RAM, no GPU	1-4B parameters	`llama3.2:3b`, `qwen3:1.7b`, `gemma3:1b`
16 GB RAM or a 6-8 GB GPU	7-9B	`qwen3:8b`, `llama3.1:8b`, `gemma3:4b`
32 GB+ RAM or a 16-24 GB GPU	12-32B	`gemma3:12b`, `qwen3:14b`, `qwen3:32b`

Rough rule: a model needs about 0.6 GB of memory per billion parameters at the default 4-bit quantization, plus room for the context. Model names change often; browse ollama.com/library for what's current.

ollama pull llama3.2:3b          # download (about 2 GB)
ollama run llama3.2:3b           # chat in the terminal; /bye to exit
ollama list                      # models on disk
ollama ps                        # models loaded in memory right now
ollama rm llama3.2:3b            # free the disk space

The first answer is slow while the model loads into memory; after that it stays loaded for a few minutes.

3. Call it from Python (OpenAI-compatible)¶

Ollama speaks the same API as OpenAI, so the code you already have works by changing two lines:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:11434/v1",   # your local Ollama server
    api_key="ollama",                       # required by the SDK, ignored by Ollama
)

reply = client.chat.completions.create(
    model="llama3.2:3b",
    messages=[
        {"role": "system", "content": "You are a concise assistant."},
        {"role": "user", "content": "Explain RAG in two sentences."},
    ],
    temperature=0.2,
)
print(reply.choices[0].message.content)

Streaming (stream=True) and JSON mode (response_format={"type": "json_object"}) work the same way, so the streaming and structured JSON tutorials apply as-is. Small models follow instructions less reliably than large ones, so validate their JSON carefully.

Or use the ollama package¶

pip install ollama

import ollama

response = ollama.chat(
    model="llama3.2:3b",
    messages=[{"role": "user", "content": "Give me three names for a chai shop."}],
)
print(response.message.content)

4. Local embeddings for RAG¶

Ollama also runs embedding models, so a whole RAG pipeline can run offline:

ollama pull nomic-embed-text

emb = client.embeddings.create(model="nomic-embed-text",
                               input=["Refunds take 5 days", "Delivery takes 3 days"])
vectors = [d.embedding for d in emb.data]
print(len(vectors), len(vectors[0]))          # 2 vectors of 768 numbers each

Store them in Chroma or pgvector exactly as you would cloud embeddings (see Hands-on vector DBs). Don't mix: documents and queries must be embedded with the same model.

5. Switch between local and cloud with one setting¶

Read the base URL and model from the environment (see Manage API keys and settings), and the same code runs against Ollama on your laptop and a cloud model in production:

.env (local)

LLM_BASE_URL=http://localhost:11434/v1
LLM_API_KEY=ollama
LLM_MODEL=llama3.2:3b

import os
from dotenv import load_dotenv
from openai import OpenAI

load_dotenv()
client = OpenAI(base_url=os.getenv("LLM_BASE_URL"), api_key=os.getenv("LLM_API_KEY"))
MODEL = os.getenv("LLM_MODEL")

In production, set LLM_BASE_URL to the cloud provider (or leave it unset for OpenAI's default) and change the model name. No code changes.

Troubleshooting¶

Problem	Fix
`Connection refused` on port 11434	Start Ollama (open the app, or run `ollama serve`)
Very slow answers	The model is too big for your RAM and is swapping: use a smaller one; check `ollama ps`
`model not found`	`ollama pull <name>` first; the name must match `ollama list` exactly, including the tag
Answers ignore the start of long documents	Ollama's default context window is small: start Ollama with `OLLAMA_CONTEXT_LENGTH=8192`, or pass `options={"num_ctx": 8192}` with the `ollama` package
Calling Ollama from Docker	Use `http://host.docker.internal:11434/v1` instead of `localhost`

Checklist¶

Ollama is running and a model that fits your RAM is pulled.
Your app reads the base URL, key and model from settings, so local and cloud are one switch.
You've compared the local model's answers with a cloud model on a few real examples.
Embedding model is the same for documents and queries.

Next: Extract text from PDFs, Word files and web pages →