Setting Up an Ollama Server on a Host

Purpose

An Ollama server is a local model backend that an Animus instance (or any OpenAI-compatible consumer) calls for inference. This SOP takes a fresh Linux host to a remotely-callable, right-sized model server.

Prerequisites

• A Linux host; for Path A, systemd and sudo; for Path B, Docker
• NVIDIA GPU with a working driver (nvidia-smi succeeds) — or accept CPU/hybrid speeds knowingly (see sizing below)
• Plan disk before anything else: model libraries reach tens to hundreds of GB. Decide where models live before pulling the first one.

Memory Sizing (read before choosing models)

Reference points — total memory (weights + runtime overhead), measured on real hardware:

Model	Quant / format	Total memory
Gemma4-31b	default	~32 GB
Qwen3.6 27B	Q4KM	~18 GB
Qwen3.6 27B	Q8_0	~30 GB
Qwen3.6 27B	FP16	~61 GB
DeepSeek-R1 32B	Q4KM	~20 GB
DeepSeek-R1 32B	Q8_0	~35 GB
DeepSeek-R1 32B	FP16	~66 GB
GLM-5.3-Flash	native FP8	~306 GB
GLM-5.3-Flash	hybrid	~350 GB

Rules of thumb:

• Hybrid mode: a model can rely partially or wholly on RAM instead of VRAM. It works — output slows proportionally to the offloaded fraction. Field-verified: a 27B-class Q4KM (~18 GB) ran entirely in system RAM on a GPU-less host — functional for batch use, visibly slow for interactive output. Native VRAM residency is the performance tier; hybrid is the "it fits" tier.
• KV cache adds memory beyond weights — long contexts (≥32k) can add multiple GB on top of the table figures. Budget headroom, don't budget to the line.
• One loaded model ≈ its table figure resident. OLLAMA_MAX_LOADED_MODELS controls how many coexist.

Choosing a Path

	Path A — Host install	Path B — Docker
Best when	Dedicated model box; you want systemd to own it	Shared host; you already run everything as containers
GPU wiring	Installer handles CUDA	Needs nvidia-container-toolkit (one-time)
Configuration	systemd override file	Container env vars / compose file
Upgrade	Re-run install script	docker pull + recreate

Both paths end at the same place: an Ollama daemon on 11434 with models on your chosen disk. Configuration is env-var driven in both — only the delivery mechanism differs.

---

Path A — Host Install (systemd)

A1. Install

curl -fsSL https://ollama.com/install.sh | sh

Creates the ollama user, a systemd unit, and binds 127.0.0.1:11434.

Verify: systemctl is-active ollama → active. curl -s http://127.0.0.1:11434/api/tags → {"models":[]} (empty list, valid JSON).

A2. Relocate the models directory (if OS disk is small)

Default is /usr/share/ollama/.ollama — often on the OS disk.

sudo systemctl stop ollama
sudo mkdir -p /data/ollama
sudo rsync -a /usr/share/ollama/.ollama/ /data/ollama/
sudo systemctl edit ollama

In the override:

[Service]
Environment="OLLAMA_MODELS=/data/ollama"

sudo systemctl daemon-reload && sudo systemctl start ollama

Verify: curl -s http://127.0.0.1:11434/api/tags still responds; pull a small model and confirm files land under the new path.

A3. Configure remote access and concurrency

sudo systemctl edit ollama

[Service]
Environment="OLLAMA_HOST=0.0.0.0"
Environment="OLLAMA_KEEP_ALIVE=30m"
Environment="OLLAMA_NUM_PARALLEL=2"
Environment="OLLAMA_MAX_LOADED_MODELS=1"

Firewall — allow only the consumer host, never the world:

sudo ufw allow from <consumer-ip> to any port 11434 proto tcp

Verify (from the consumer host): curl -s http://<server>:11434/api/tags returns the model list. From any other host, the same curl must time out.

Editing to a desired state later: add/change Environment= lines in the override, then daemon-reload + restart ollama. The override file is the configuration's source of truth — keep every setting there, never in ad-hoc unit edits.

---

Path B — Docker Deployment

B1. GPU hosts: install the container toolkit (one-time)

CPU-only hosts skip this step.

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
  | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
  | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
  | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

B2. Run the container

Models live in the container's /root/.ollama. Bind-mount it to a directory on your big disk (named volumes work too, but a plain directory is inspectable and survives volume-prune):

sudo mkdir -p /opt/ollama
docker run -d --name ollama \
  --restart unless-stopped \
  -p 11434:11434 \
  -v /opt/ollama:/root/.ollama \
  --memory=24g \
  ollama/ollama

• GPU host: add --gpus all.
• Exposure control without firewall rules: publish to a specific interface — -p 10.0.0.5:11434:11434 instead of all interfaces.
• --memory is prudent on shared hosts; the model budget from the sizing table guides the value.

Verify: docker ps shows Up; curl -s http://<host>:11434/api/tags returns {"models":[]}; docker logs ollama shows the daemon listening. On GPU hosts: docker exec ollama nvidia-smi must print the GPU table.

B3. Editing the configuration to a desired state

The configuration surface is environment variables — the same OLLAMA_* set as Path A. Two disciplines:

Ad-hoc — recreate with new flags (state survives in the mount):

docker rm -f ollama
docker run -d --name ollama ... \
  -e OLLAMA_KEEP_ALIVE=30m \
  -e OLLAMA_NUM_PARALLEL=2 \
  -e OLLAMA_MAX_LOADED_MODELS=1 \
  ollama/ollama

Durable — keep a compose file as the source of truth (recommended; this file is the whole deployment, versionable and re-runnable):

services:
  ollama:
    image: ollama/ollama
    restart: unless-stopped
    ports:
      - "11434:11434"
    volumes:
      - /opt/ollama:/root/.ollama
    environment:
      - OLLAMA_KEEP_ALIVE=30m
      - OLLAMA_NUM_PARALLEL=2
      - OLLAMA_MAX_LOADED_MODELS=1
    mem_limit: 24g
    # GPU hosts: uncomment
    # deploy:
    #   resources:
    #     reservations:
    #       devices:
    #         - driver: nvidia
    #           count: all
    #           capabilities: [gpu]

docker compose up -d   # apply changes after editing

Models, the server identity keypair (id_ed25519), and CLI state persist in the mounted directory across recreation — recreating the container is cheap; nothing of value lives inside it.

Tuning notes (both paths):

• OLLAMA_KEEP_ALIVE (default 5m) — model residency after last use. Raise for steady traffic (no cold-start stall); 0 unloads immediately, -1 keeps forever.
• OLLAMA_NUM_PARALLEL — concurrent request slots per model; each slot costs KV-cache memory at long contexts. Default is conservative.
• OLLAMA_MAX_LOADED_MODELS — how many models coexist; set to what the memory budget actually fits.

---

Pull Models

Path A: sudo -u ollama ollama pull <tag> Path B: docker exec -it ollama ollama pull <tag>

Verify: ollama list (via the same wrapper) shows exact tags — read them carefully, they are the IDs every consumer must use. See Common Issues for the tag-mismatch failure mode.

Round-Trip Test (the real verification)

time curl -s http://<host>:11434/api/generate -d '{
  "model": "<tag-from-ollama-list>",
  "prompt": "Say ready.",
  "stream": false
}'

• First call: a load pause (tens of seconds for big models), then a response — note the time; that's your cold-start cost.
• Second call: near-instant by comparison — that's warm.
• ollama ps during/after: shows the model and its processor split (100% GPU vs partial CPU) — the honest view of whether you're in native or hybrid territory.

Common Issues

Consumer gets "model not found" while ollama list shows it. Tag mismatch. The tag the consumer configures must match ollama list exactly — including :quant suffixes and Ollama-Cloud :cloud suffixes. Consumers often validate model IDs against their own catalog before ever reaching Ollama, so an ID can pass the provider API but die at the consumer's allowlist. Fix discipline: copy the tag verbatim from ollama list into the consumer config; when the consumer supports an allowlist, check both gates.

First token takes forever. Cold load — the model was evicted (OLLAMA_KEEP_ALIVE too low for the traffic pattern) or another model took its slot (OLLAMA_MAX_LOADED_MODELS). Check ollama ps before and during the request.

Model loads but generation crawls. Hybrid mode — weights partially or wholly in RAM. Check the processor split in ollama ps, then pick a smaller quant from the sizing table or add VRAM.

Load fails / process OOM-killed. Model footprint exceeds VRAM + RAM (Path A) or the container memory limit (Path B — check docker inspect ollama | grep OOMKilled). Drop a quant level or raise the limit.

GPU present but unused. Host layer first: nvidia-smi must work before install; after kernel upgrades, reinstall the driver. Path B adds a container layer: nvidia-container-toolkit installed and nvidia-ctk runtime configure run, --gpus all present on the run — docker exec ollama nvidia-smi discriminates host vs container failure.

Ops Notes

• Logs: Path A journalctl -u ollama -f; Path B docker logs -f ollama. First place to look, always.
• Upgrades: Path A re-runs the install script (keeps unit, models, config). Path B: docker pull ollama/ollama + recreate — models and identity persist in the mount.
• Back up nothing except your model list — models are re-pullable; the expensive part is having recorded which tags consumers expect.