# llm-host The Ollama machine, from [Phase 3 of the project plan](../../docs/project-plan.md). A **separate physical host** from the container host, on purpose — see "Why a separate machine" below. Everything in this project that wants inference calls this one server: | Caller | What it asks for | If this host is off | |---|---|---| | Home Assistant (Assist / AI Task) | Conversation, tool calls, Phase 4's brightness/colour JSON | Assist's LLM agent is unavailable; **presence → light still works**, on plain automations | | `digest-engine` | Quarter-daily synthesis + the counter-run verification pass | The run is skipped; the timer doesn't error | | `pantry-vision` | Grocery-item recognition from one photo | The scan flow reports it can't identify; Grocy is untouched | | `chores` | Bin/dishes/litter vision checks, and reminder phrasing | Camera checks skip; nudges use the plain template | ## The guardrail this host is built around **Every consumer must degrade to "unavailable", never to "broken", when this machine is off.** The project plan's testing checklist states it directly — *"Does the reactive path (presence → light on) work with the LLM host powered off? (It must.)"* That's the whole reason this is a separate box rather than more containers on the Phase 1 host: it *can* be off — for power, for noise, because you pulled the GPU — and the house still works. The setup script's closing output walks you through actually testing that, and it's worth doing once for real rather than assuming. If something *breaks* rather than degrading when this host is down, that's a bug in the consumer, not here. ## Two tiers Auto-detected from whether `nvidia-smi` both exists **and** succeeds (a leftover driver package on a machine whose card was pulled satisfies the first but not the second). Override with `TIER` at the top of the script. | Tier | Model | Reality | |---|---|---| | `gpu` | `qwen2.5:14b-instruct` | What Phase 3 specifies | | `cpu` | `qwen2.5:7b-instruct` | Single-digit tokens/sec. Enough to validate the entire pipeline end to end before buying a card — `docs/components.md`'s deliberate "skip GPU" fallback, not a failure mode | Plus a vision model (`llava` by default) for `pantry-vision` and `chores`. Set `PULL_VISION_MODEL=false` to skip it and save several GB if you're not running those camera paths yet. **The vision model choice is not a considered one.** `llava` is simply the default those two services already ship with, and open decision #18 flags the pick as unmade and completely unbenchmarked. If grocery recognition turns out too slow or too inaccurate to be usable, this is the first knob to turn — `qwen2.5vl` and `moondream` are the obvious alternatives to measure against. ## Contention: interactive vs. batch on one GPU The real scheduling problem here (project plan open decision #4), and the script's defaults take a position on it: - **Assist is interactive** — a person is standing in the room waiting. - **`digest-engine` is batch** — every 6h, nobody watching. - **The vision callers are occasional but want a *different model resident*.** | Setting | Default | Why | |---|---|---| | `OLLAMA_KEEP_ALIVE` | `30m` | Ollama's own default of 5m means a household that talks to Assist a few times an hour pays the model-load cost nearly every time. 30m keeps it warm through normal use | | `OLLAMA_MAX_LOADED_MODELS` | `1` | **Deliberate.** A 14B text model and a vision model don't co-fit in 8–12GB; letting Ollama try produces VRAM thrash or an OOM mid-request instead of an honest swap. 1 means "swap predictably, pay the reload when vision is actually needed." Raise it only if you have the VRAM *and have checked* | | `OLLAMA_NUM_PARALLEL` | `1` | Predictable latency for whoever is speaking, over throughput nothing here needs | This is a **reasoned default, not a measured one** — none of it has been run against a real GPU under real concurrent load. The remaining half of open decision #4 (whether `DIGEST_SCHEDULE`'s `00,06,12,18` overlaps real Assist usage) needs actual usage data to settle; the settings above at least make the failure mode a predictable swap rather than an OOM. ## Security: Ollama has no authentication None. Not a token, not a password. And its API is not read-only — it can **pull and delete models**, not just generate. Anyone who can reach `:11434` can do all of that. The network is therefore the entire boundary: keep this host on the smart-home VLAN and never port-forward it, exactly as `docs/network-integration.md` §1 says for everything else. It's now in that document's port table for the same reason. ## Why a container, not the native installer Ollama's official install is `curl -fsSL https://ollama.com/install.sh | sh`, which pipes a fetched script straight into a root shell. The container path gives a pinned image, an uninstall that's `docker rm`, and no arbitrary remote code executed as root — the same reasoning behind every other component in this project running in Docker. The native install is a perfectly legitimate alternative, and on some GPU setups it's less fuss than the NVIDIA Container Toolkit. If you go that way, the one thing you **must** still do is set `OLLAMA_HOST=0.0.0.0:11434` in the systemd unit — see below. ## The one configuration mistake that looks like a dead host Ollama binds `127.0.0.1` by default. In a container, that means the published port forwards to a socket nothing is listening on, and **every caller gets a connection refused that is indistinguishable from "the LLM host is powered off"** — which, given that every consumer here is built to tolerate exactly that, degrades silently and looks like nothing is wrong. The compose file sets `OLLAMA_HOST=0.0.0.0:11434` for this reason. Check it first if inference is mysteriously "unavailable" everywhere at once. (This is the same class of bug as `chores`' env template pointing at `127.0.0.1` for a sibling container — see the project plan's open decision #38.) ## Run it ```sh sudo -E tools/setup-llm-host.sh ``` Edit the variables at the top first — `BASE_DIR` above all, since models are large (a 14B Q4 model is ~9GB, a vision model another 5–8GB) and it defaults to `/opt/llm-host`. For the GPU tier, **the NVIDIA driver must already work** (`nvidia-smi` prints your card). The script installs the Container Toolkit that lets Docker see the GPU, but deliberately does not install the driver: that's the most hardware- and kernel-specific step on this machine, and silently choosing a driver version for someone is a good way to produce a box that doesn't boot. Afterwards the script prints exactly what to paste into HA and into each service's env file on the container host. ## Manual verification still outstanding 1. **None of this has been run.** No Debian machine, no GPU, no Ollama server — the script is syntax-checked and its generated compose file is validated as YAML for both tiers, and that is the entire extent of the testing. Same honesty rule as every other unbuilt host in this repo. 2. **The model tags are library names that upstream does rename.** `qwen2.5:14b-instruct` and `llava` are written from Ollama's library as documented, not confirmed pullable today. A failed pull is deliberately non-fatal — the server stays up and you fix the tag by hand — but check if one fails. 3. **The NVIDIA Container Toolkit repo/apt steps are from NVIDIA's documented install**, not run on a real machine. `nvidia-ctk runtime configure --runtime=docker` followed by a Docker restart is the documented shape; verify against NVIDIA's current docs before trusting it on hardware you care about. 4. **The contention defaults are unmeasured** — see the table above. 5. **Whether a 14B model at Q4 actually fits your card** is not checked anywhere. On 8GB it will be tight-to-impossible; on 12GB+ it's comfortable. If it OOMs, drop to the CPU tier's 7B tag on the GPU, which is the cheap first thing to try. 6. **No Wake-on-LAN.** If you want the digest timer to wake this host rather than skip its run, that's a BIOS + `ethtool -s wol g` + a `wakeonlan` call from the container host's timer — deliberately not scripted here, since it depends on hardware that hasn't been chosen.