1. Docker and deployment¶

Intermediate · 7 min read

"It works on my laptop" isn't a deployment. A container packages your app, its exact dependencies and its start command into one image that runs the same everywhere. This page covers the essentials of shipping the FastAPI LLM service to production.

1.1 A production Dockerfile¶

# no-run — Dockerfile for a FastAPI LLM service (app/main.py exposes `app`)
FROM python:3.12-slim AS build
WORKDIR /src
COPY requirements.txt .
RUN pip wheel --no-cache-dir --wheel-dir /wheels -r requirements.txt

FROM python:3.12-slim
ENV PYTHONDONTWRITEBYTECODE=1 PYTHONUNBUFFERED=1
WORKDIR /app
COPY --from=build /wheels /wheels
RUN pip install --no-cache-dir /wheels/* && rm -rf /wheels && useradd --create-home appuser
COPY app/ app/
# Never run as root
USER appuser
EXPOSE 8000
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000", "--workers", "2"]

Choice	Why
Multi-stage build	Build tools stay out of the runtime image: smaller, less to attack
Pinned `requirements.txt`, copied before the code	Reproducible builds; code changes rebuild in seconds
Non-root user	A compromised app can't take over the container
`PYTHONUNBUFFERED=1`	Logs appear immediately when you're debugging a crash

1.2 Configuration and secrets¶

Everything that differs between environments comes from environment variables, never from code or the image. Validate them at startup, so a missing key fails the deploy instead of the first user's request:

REQUIRED = ("LLM_API_KEY", "LLM_MODEL")
DEFAULTS = {"LLM_TIMEOUT_S": "30", "MAX_OUTPUT_TOKENS": "800"}

def load_settings(env: dict) -> dict:
    missing = [k for k in REQUIRED if not env.get(k)]
    if missing:
        raise SystemExit(f"Refusing to start: missing settings {missing}")
    settings = {**DEFAULTS, **{k: v for k, v in env.items() if k in REQUIRED or k in DEFAULTS}}
    return {k: int(v) if k in DEFAULTS else v for k, v in settings.items()}

def describe(settings: dict) -> dict:
    """Safe to log: secrets masked."""
    return {k: ("***" + str(v)[-4:] if "KEY" in k else v) for k, v in settings.items()}

print(describe(load_settings({"LLM_API_KEY": "sk-live-abc123xyz9876", "LLM_MODEL": "gpt-4o-mini"})))
try:
    load_settings({"LLM_MODEL": "gpt-4o-mini"})
except SystemExit as err:
    print(err)

Output

{'LLM_TIMEOUT_S': 30, 'MAX_OUTPUT_TOKENS': 800, 'LLM_API_KEY': '***9876', 'LLM_MODEL': 'gpt-4o-mini'}
Refusing to start: missing settings ['LLM_API_KEY']

1.3 Health checks and where to deploy¶

/health (liveness) only says the process is alive. /ready (readiness) checks your own dependencies (database, Redis, vector store).
Never call the LLM provider in a health check. A short provider outage would mark every instance unhealthy, and the platform would restart your whole fleet for nothing.

Option	Good for
PaaS (Railway, Render, Fly.io)	Small teams and API-based LLM apps
Serverless containers (Cloud Run, Azure Container Apps)	Spiky traffic, scale to zero
Kubernetes	Many services, GPUs, platform teams
GPU platforms (Modal, RunPod, Baseten)	Self-hosted open models without running Kubernetes

1.4 Releasing prompt and model changes¶

In LLM apps a "release" isn't only code: prompts, models and retrieval settings change behaviour too. Version them like code, pass the eval gate in CI, then roll out gradually with a canary (5–10% of traffic) or shadow mode (the new version runs silently on real traffic), with instant rollback.

In the full LLMOps guide (PDF)

The complete production Dockerfile with health check, .dockerignore and image-size tuning
Docker Compose for the full stack (API + Redis + worker) with health-checked dependencies
Liveness vs readiness endpoints in FastAPI, with code
A deployment comparison with costs and pitfalls for each platform
The release playbook: rolling, blue/green, canary, feature flags and shadow mode, step by step
Model answers to the interview questions below

See the full guide in the Store → · Get notified when it goes on sale ↓

Interview questions¶

How would you containerise and deploy an LLM service?
Why shouldn't a health check call the LLM provider?
How do you release a new prompt or model safely?

Next: 2. Serving open models Back to: LLMOps overview · Notes overview