147 lines
8.2 KiB
Markdown
147 lines
8.2 KiB
Markdown
# llm-host
|
||
|
||
The Ollama machine, from [Phase 3 of the project plan](../../docs/project-plan.md).
|
||
A **separate physical host** from the container host, on purpose — see "Why a separate
|
||
machine" below.
|
||
|
||
Everything in this project that wants inference calls this one server:
|
||
|
||
| Caller | What it asks for | If this host is off |
|
||
|---|---|---|
|
||
| Home Assistant (Assist / AI Task) | Conversation, tool calls, Phase 4's brightness/colour JSON | Assist's LLM agent is unavailable; **presence → light still works**, on plain automations |
|
||
| `digest-engine` | Quarter-daily synthesis + the counter-run verification pass | The run is skipped; the timer doesn't error |
|
||
| `pantry-vision` | Grocery-item recognition from one photo | The scan flow reports it can't identify; Grocy is untouched |
|
||
| `chores` | Bin/dishes/litter vision checks, and reminder phrasing | Camera checks skip; nudges use the plain template |
|
||
|
||
## The guardrail this host is built around
|
||
|
||
**Every consumer must degrade to "unavailable", never to "broken", when this machine
|
||
is off.** The project plan's testing checklist states it directly — *"Does the reactive
|
||
path (presence → light on) work with the LLM host powered off? (It must.)"*
|
||
|
||
That's the whole reason this is a separate box rather than more containers on the
|
||
Phase 1 host: it *can* be off — for power, for noise, because you pulled the GPU — and
|
||
the house still works. The setup script's closing output walks you through actually
|
||
testing that, and it's worth doing once for real rather than assuming.
|
||
|
||
If something *breaks* rather than degrading when this host is down, that's a bug in
|
||
the consumer, not here.
|
||
|
||
## Two tiers
|
||
|
||
Auto-detected from whether `nvidia-smi` both exists **and** succeeds (a leftover driver
|
||
package on a machine whose card was pulled satisfies the first but not the second).
|
||
Override with `TIER` at the top of the script.
|
||
|
||
| Tier | Model | Reality |
|
||
|---|---|---|
|
||
| `gpu` | `qwen2.5:14b-instruct` | What Phase 3 specifies |
|
||
| `cpu` | `qwen2.5:7b-instruct` | Single-digit tokens/sec. Enough to validate the entire pipeline end to end before buying a card — `docs/components.md`'s deliberate "skip GPU" fallback, not a failure mode |
|
||
|
||
Plus a vision model (`llava` by default) for `pantry-vision` and `chores`. Set
|
||
`PULL_VISION_MODEL=false` to skip it and save several GB if you're not running those
|
||
camera paths yet.
|
||
|
||
**The vision model choice is not a considered one.** `llava` is simply the default
|
||
those two services already ship with, and open decision #18 flags the pick as unmade
|
||
and completely unbenchmarked. If grocery recognition turns out too slow or too
|
||
inaccurate to be usable, this is the first knob to turn — `qwen2.5vl` and `moondream`
|
||
are the obvious alternatives to measure against.
|
||
|
||
## Contention: interactive vs. batch on one GPU
|
||
|
||
The real scheduling problem here (project plan open decision #4), and the script's
|
||
defaults take a position on it:
|
||
|
||
- **Assist is interactive** — a person is standing in the room waiting.
|
||
- **`digest-engine` is batch** — every 6h, nobody watching.
|
||
- **The vision callers are occasional but want a *different model resident*.**
|
||
|
||
| Setting | Default | Why |
|
||
|---|---|---|
|
||
| `OLLAMA_KEEP_ALIVE` | `30m` | Ollama's own default of 5m means a household that talks to Assist a few times an hour pays the model-load cost nearly every time. 30m keeps it warm through normal use |
|
||
| `OLLAMA_MAX_LOADED_MODELS` | `1` | **Deliberate.** A 14B text model and a vision model don't co-fit in 8–12GB; letting Ollama try produces VRAM thrash or an OOM mid-request instead of an honest swap. 1 means "swap predictably, pay the reload when vision is actually needed." Raise it only if you have the VRAM *and have checked* |
|
||
| `OLLAMA_NUM_PARALLEL` | `1` | Predictable latency for whoever is speaking, over throughput nothing here needs |
|
||
|
||
This is a **reasoned default, not a measured one** — none of it has been run against a
|
||
real GPU under real concurrent load. The remaining half of open decision #4 (whether
|
||
`DIGEST_SCHEDULE`'s `00,06,12,18` overlaps real Assist usage) needs actual usage data
|
||
to settle; the settings above at least make the failure mode a predictable swap rather
|
||
than an OOM.
|
||
|
||
## Security: Ollama has no authentication
|
||
|
||
None. Not a token, not a password. And its API is not read-only — it can **pull and
|
||
delete models**, not just generate. Anyone who can reach `:11434` can do all of that.
|
||
|
||
The network is therefore the entire boundary: keep this host on the smart-home VLAN
|
||
and never port-forward it, exactly as `docs/network-integration.md` §1 says for
|
||
everything else. It's now in that document's port table for the same reason.
|
||
|
||
## Why a container, not the native installer
|
||
|
||
Ollama's official install is `curl -fsSL https://ollama.com/install.sh | sh`, which
|
||
pipes a fetched script straight into a root shell. The container path gives a pinned
|
||
image, an uninstall that's `docker rm`, and no arbitrary remote code executed as root
|
||
— the same reasoning behind every other component in this project running in Docker.
|
||
|
||
The native install is a perfectly legitimate alternative, and on some GPU setups it's
|
||
less fuss than the NVIDIA Container Toolkit. If you go that way, the one thing you
|
||
**must** still do is set `OLLAMA_HOST=0.0.0.0:11434` in the systemd unit — see below.
|
||
|
||
## The one configuration mistake that looks like a dead host
|
||
|
||
Ollama binds `127.0.0.1` by default. In a container, that means the published port
|
||
forwards to a socket nothing is listening on, and **every caller gets a connection
|
||
refused that is indistinguishable from "the LLM host is powered off"** — which, given
|
||
that every consumer here is built to tolerate exactly that, degrades silently and
|
||
looks like nothing is wrong.
|
||
|
||
The compose file sets `OLLAMA_HOST=0.0.0.0:11434` for this reason. Check it first if
|
||
inference is mysteriously "unavailable" everywhere at once. (This is the same class of
|
||
bug as `chores`' env template pointing at `127.0.0.1` for a sibling container — see
|
||
the project plan's open decision #38.)
|
||
|
||
## Run it
|
||
|
||
```sh
|
||
cd hosts/llm-host/scripts
|
||
sudo ./setup-llm-host.sh
|
||
```
|
||
|
||
Edit the variables at the top first — `BASE_DIR` above all, since models are large
|
||
(a 14B Q4 model is ~9GB, a vision model another 5–8GB) and it defaults to
|
||
`/opt/llm-host`.
|
||
|
||
For the GPU tier, **the NVIDIA driver must already work** (`nvidia-smi` prints your
|
||
card). The script installs the Container Toolkit that lets Docker see the GPU, but
|
||
deliberately does not install the driver: that's the most hardware- and
|
||
kernel-specific step on this machine, and silently choosing a driver version for
|
||
someone is a good way to produce a box that doesn't boot.
|
||
|
||
Afterwards the script prints exactly what to paste into HA and into each service's env
|
||
file on the container host.
|
||
|
||
## Manual verification still outstanding
|
||
|
||
1. **None of this has been run.** No Debian machine, no GPU, no Ollama server — the
|
||
script is syntax-checked and its generated compose file is validated as YAML for
|
||
both tiers, and that is the entire extent of the testing. Same honesty rule as
|
||
every other unbuilt host in this repo.
|
||
2. **The model tags are library names that upstream does rename.** `qwen2.5:14b-instruct`
|
||
and `llava` are written from Ollama's library as documented, not confirmed pullable
|
||
today. A failed pull is deliberately non-fatal — the server stays up and you fix the
|
||
tag by hand — but check <https://ollama.com/library> if one fails.
|
||
3. **The NVIDIA Container Toolkit repo/apt steps are from NVIDIA's documented install**,
|
||
not run on a real machine. `nvidia-ctk runtime configure --runtime=docker` followed
|
||
by a Docker restart is the documented shape; verify against NVIDIA's current docs
|
||
before trusting it on hardware you care about.
|
||
4. **The contention defaults are unmeasured** — see the table above.
|
||
5. **Whether a 14B model at Q4 actually fits your card** is not checked anywhere. On
|
||
8GB it will be tight-to-impossible; on 12GB+ it's comfortable. If it OOMs, drop to
|
||
the CPU tier's 7B tag on the GPU, which is the cheap first thing to try.
|
||
6. **No Wake-on-LAN.** If you want the digest timer to wake this host rather than skip
|
||
its run, that's a BIOS + `ethtool -s <iface> wol g` + a `wakeonlan` call from the
|
||
container host's timer — deliberately not scripted here, since it depends on
|
||
hardware that hasn't been chosen.
|