# llm-host

The Ollama machine, from [Phase 3 of the project plan](../../docs/project-plan.md).
A **separate physical host** from the container host, on purpose — see "Why a separate
machine" below.

Everything in this project that wants inference calls this one server:

| Caller | What it asks for | If this host is off |
|---|---|---|
| Home Assistant (Assist / AI Task) | Conversation, tool calls, Phase 4's brightness/colour JSON | Assist's LLM agent is unavailable; **presence → light still works**, on plain automations |
| `digest-engine` | Quarter-daily synthesis + the counter-run verification pass | The run is skipped; the timer doesn't error |
| `pantry-vision` | Grocery-item recognition from one photo | The scan flow reports it can't identify; Grocy is untouched |
| `chores` | Bin/dishes/litter vision checks, and reminder phrasing | Camera checks skip; nudges use the plain template |

## The guardrail this host is built around

**Every consumer must degrade to "unavailable", never to "broken", when this machine
is off.** The project plan's testing checklist states it directly — *"Does the reactive
path (presence → light on) work with the LLM host powered off? (It must.)"*

That's the whole reason this is a separate box rather than more containers on the
Phase 1 host: it *can* be off — for power, for noise, because you pulled the GPU — and
the house still works. The setup script's closing output walks you through actually
testing that, and it's worth doing once for real rather than assuming.

If something *breaks* rather than degrading when this host is down, that's a bug in
the consumer, not here.

## Two tiers

Auto-detected from whether `nvidia-smi` both exists **and** succeeds (a leftover driver
package on a machine whose card was pulled satisfies the first but not the second).
Override with `TIER` at the top of the script.

| Tier | Model | Reality |
|---|---|---|
| `gpu` | `qwen2.5:14b-instruct` | What Phase 3 specifies |
| `cpu` | `qwen2.5:7b-instruct` | Single-digit tokens/sec. Enough to validate the entire pipeline end to end before buying a card — `docs/components.md`'s deliberate "skip GPU" fallback, not a failure mode |

Plus a vision model (`llava` by default) for `pantry-vision` and `chores`. Set
`PULL_VISION_MODEL=false` to skip it and save several GB if you're not running those
camera paths yet.

**The vision model choice is not a considered one.** `llava` is simply the default
those two services already ship with, and open decision #18 flags the pick as unmade
and completely unbenchmarked. If grocery recognition turns out too slow or too
inaccurate to be usable, this is the first knob to turn — `qwen2.5vl` and `moondream`
are the obvious alternatives to measure against.

## Contention: interactive vs. batch on one GPU

The real scheduling problem here (project plan open decision #4), and the script's
defaults take a position on it:

- **Assist is interactive** — a person is standing in the room waiting.
- **`digest-engine` is batch** — every 6h, nobody watching.
- **The vision callers are occasional but want a *different model resident*.**

| Setting | Default | Why |
|---|---|---|
| `OLLAMA_KEEP_ALIVE` | `30m` | Ollama's own default of 5m means a household that talks to Assist a few times an hour pays the model-load cost nearly every time. 30m keeps it warm through normal use |
| `OLLAMA_MAX_LOADED_MODELS` | `1` | **Deliberate.** A 14B text model and a vision model don't co-fit in 8–12GB; letting Ollama try produces VRAM thrash or an OOM mid-request instead of an honest swap. 1 means "swap predictably, pay the reload when vision is actually needed." Raise it only if you have the VRAM *and have checked* |
| `OLLAMA_NUM_PARALLEL` | `1` | Predictable latency for whoever is speaking, over throughput nothing here needs |

This is a **reasoned default, not a measured one** — none of it has been run against a
real GPU under real concurrent load. The remaining half of open decision #4 (whether
`DIGEST_SCHEDULE`'s `00,06,12,18` overlaps real Assist usage) needs actual usage data
to settle; the settings above at least make the failure mode a predictable swap rather
than an OOM.

## Security: Ollama has no authentication

None. Not a token, not a password. And its API is not read-only — it can **pull and
delete models**, not just generate. Anyone who can reach `:11434` can do all of that.

The network is therefore the entire boundary: keep this host on the smart-home VLAN
and never port-forward it, exactly as `docs/network-integration.md` §1 says for
everything else. It's now in that document's port table for the same reason.

## Why a container, not the native installer

Ollama's official install is `curl -fsSL https://ollama.com/install.sh | sh`, which
pipes a fetched script straight into a root shell. The container path gives a pinned
image, an uninstall that's `docker rm`, and no arbitrary remote code executed as root
— the same reasoning behind every other component in this project running in Docker.

The native install is a perfectly legitimate alternative, and on some GPU setups it's
less fuss than the NVIDIA Container Toolkit. If you go that way, the one thing you
**must** still do is set `OLLAMA_HOST=0.0.0.0:11434` in the systemd unit — see below.

## The one configuration mistake that looks like a dead host

Ollama binds `127.0.0.1` by default. In a container, that means the published port
forwards to a socket nothing is listening on, and **every caller gets a connection
refused that is indistinguishable from "the LLM host is powered off"** — which, given
that every consumer here is built to tolerate exactly that, degrades silently and
looks like nothing is wrong.

The compose file sets `OLLAMA_HOST=0.0.0.0:11434` for this reason. Check it first if
inference is mysteriously "unavailable" everywhere at once. (This is the same class of
bug as `chores`' env template pointing at `127.0.0.1` for a sibling container — see
the project plan's open decision #38.)

## Run it

```sh
sudo -E tools/setup-llm-host.sh
```

Edit the variables at the top first — `BASE_DIR` above all, since models are large
(a 14B Q4 model is ~9GB, a vision model another 5–8GB) and it defaults to
`/opt/llm-host`.

For the GPU tier, **the NVIDIA driver must already work** (`nvidia-smi` prints your
card). The script installs the Container Toolkit that lets Docker see the GPU, but
deliberately does not install the driver: that's the most hardware- and
kernel-specific step on this machine, and silently choosing a driver version for
someone is a good way to produce a box that doesn't boot.

Afterwards the script prints exactly what to paste into HA and into each service's env
file on the container host.

## Manual verification still outstanding

1. **None of this has been run.** No Debian machine, no GPU, no Ollama server — the
   script is syntax-checked and its generated compose file is validated as YAML for
   both tiers, and that is the entire extent of the testing. Same honesty rule as
   every other unbuilt host in this repo.
2. **The model tags are library names that upstream does rename.** `qwen2.5:14b-instruct`
   and `llava` are written from Ollama's library as documented, not confirmed pullable
   today. A failed pull is deliberately non-fatal — the server stays up and you fix the
   tag by hand — but check <https://ollama.com/library> if one fails.
3. **The NVIDIA Container Toolkit repo/apt steps are from NVIDIA's documented install**,
   not run on a real machine. `nvidia-ctk runtime configure --runtime=docker` followed
   by a Docker restart is the documented shape; verify against NVIDIA's current docs
   before trusting it on hardware you care about.
4. **The contention defaults are unmeasured** — see the table above.
5. **Whether a 14B model at Q4 actually fits your card** is not checked anywhere. On
   8GB it will be tight-to-impossible; on 12GB+ it's comfortable. If it OOMs, drop to
   the CPU tier's 7B tag on the GPU, which is the cheap first thing to try.
6. **No Wake-on-LAN.** If you want the digest timer to wake this host rather than skip
   its run, that's a BIOS + `ethtool -s <iface> wol g` + a `wakeonlan` call from the
   container host's timer — deliberately not scripted here, since it depends on
   hardware that hasn't been chosen.
