2. Serving open models¶

Advanced · 7 min read

Calling an API is the easiest way to use an LLM. Self-hosting an open-weight model (Llama, Qwen, Mistral, Gemma) gives you control over data, cost at high volume and customisation, at the price of running GPUs. (Quantisation basics are in Transformers for GenAI.)

2.1 API or self-host?¶

Factor	API	Self-hosted open model
Quality	Usually the frontier	Close for many focused tasks; fine-tunable
Data control	Data leaves your network	Stays in your infrastructure
Cost shape	Pay per token; nothing when idle	Pay per GPU-hour, busy or idle
Effort	None	Capacity planning, autoscaling, upgrades
Best for	Most apps, spiky traffic	Strict data rules, steady high volume, fine-tuned models

A GPU costs the same whether it serves one request or ten thousand, so self-hosting pays off only at steady, high volume, and only if a GPU can actually carry that load.

2.2 GPU memory: weights + KV cache¶

A GPU must hold the model weights plus the KV cache for every token of every request in flight:

MODELS = {"Llama-3.1-8B": {"params_b": 8.0, "layers": 32, "kv_heads": 8, "head_dim": 128},
          "Llama-3.1-70B": {"params_b": 70.6, "layers": 80, "kv_heads": 8, "head_dim": 128}}

def fit(model: str, gpu_gb: float, weight_bytes: float = 2, context: int = 8192) -> str:
    m = MODELS[model]
    weights_gb = m["params_b"] * weight_bytes
    kv_per_request_gb = 2 * m["layers"] * m["kv_heads"] * m["head_dim"] * 2 * context / 1e9
    free_gb = gpu_gb * 0.90 - weights_gb - 1.5
    if free_gb <= 0:
        return f"weights {weights_gb:5.1f} GB -> does NOT fit"
    return f"weights {weights_gb:5.1f} GB -> ~{int(free_gb // kv_per_request_gb)} full-length requests at once"

for model, gpu, wbytes, label in [("Llama-3.1-8B", 24, 2, "fp16 on 24 GB"), ("Llama-3.1-8B", 80, 2, "fp16 on 80 GB"),
                                  ("Llama-3.1-70B", 80, 2, "fp16 on 80 GB"), ("Llama-3.1-70B", 80, 0.5, "4-bit on 80 GB")]:
    print(f"{model:14} {label:15} {fit(model, gpu, wbytes)}")

Output

Llama-3.1-8B   fp16 on 24 GB   weights  16.0 GB -> ~3 full-length requests at once
Llama-3.1-8B   fp16 on 80 GB   weights  16.0 GB -> ~50 full-length requests at once
Llama-3.1-70B  fp16 on 80 GB   weights 141.2 GB -> does NOT fit
Llama-3.1-70B  4-bit on 80 GB  weights  35.3 GB -> ~13 full-length requests at once

The two points interviewers look for: the KV cache, not the weights, limits how many users a GPU serves, and quantisation is what makes large models fit on fewer GPUs.

2.3 Serving engines¶

Engine	Best for
vLLM	Production GPU serving: continuous batching, PagedAttention, OpenAI-compatible API
SGLang / TGI / TensorRT-LLM	Production alternatives with different strengths
Ollama	Local development and small servers; one command to run a model
llama.cpp	CPU, Apple Silicon and edge devices with quantised GGUF models

Both vLLM and Ollama expose the OpenAI API format, so switching your app to a self-hosted model is mostly a change of base_url and model name.

In the full LLMOps guide (PDF)

The API vs self-host break-even calculator, with GPU counts and engineering cost included
KV-cache maths explained line by line, multi-GPU sizing and why 70B fp16 needs four GPUs
Step-by-step vLLM and Ollama setup (Docker commands, flags, OpenAI-compatible client code)
Continuous batching, PagedAttention and speculative decoding explained simply
The model-server metrics to watch, and how to autoscale GPUs on queue length
Model answers to the interview questions below

See the full guide in the Store → · Get notified when it goes on sale ↓

Interview questions¶

When would you self-host an open model instead of using an API?
How do you estimate the GPU memory needed to serve a model?
Why is vLLM faster than a naive generation loop?

Next: 3. Caching and rate limits Back to: LLMOps overview · Notes overview