SmartestHome/hosts/llm-host/README.md

llm-host

The Ollama machine, from Phase 3 of the project plan. A separate physical host from the container host, on purpose — see "Why a separate machine" below.

Everything in this project that wants inference calls this one server:

Caller	What it asks for	If this host is off
Home Assistant (Assist / AI Task)	Conversation, tool calls, Phase 4's brightness/colour JSON	Assist's LLM agent is unavailable; presence → light still works, on plain automations
digest-engine	Quarter-daily synthesis + the counter-run verification pass	The run is skipped; the timer doesn't error
pantry-vision	Grocery-item recognition from one photo	The scan flow reports it can't identify; Grocy is untouched
chores	Bin/dishes/litter vision checks, and reminder phrasing	Camera checks skip; nudges use the plain template

The guardrail this host is built around

Every consumer must degrade to "unavailable", never to "broken", when this machine is off. The project plan's testing checklist states it directly — "Does the reactive path (presence → light on) work with the LLM host powered off? (It must.)"

That's the whole reason this is a separate box rather than more containers on the Phase 1 host: it can be off — for power, for noise, because you pulled the GPU — and the house still works. The setup script's closing output walks you through actually testing that, and it's worth doing once for real rather than assuming.

If something breaks rather than degrading when this host is down, that's a bug in the consumer, not here.

Two tiers

Auto-detected from whether nvidia-smi both exists and succeeds (a leftover driver package on a machine whose card was pulled satisfies the first but not the second). Override with TIER at the top of the script.

Tier	Model	Reality
gpu	qwen2.5:14b-instruct	What Phase 3 specifies
cpu	qwen2.5:7b-instruct	Single-digit tokens/sec. Enough to validate the entire pipeline end to end before buying a card — docs/components.md's deliberate "skip GPU" fallback, not a failure mode

Plus a vision model (llava by default) for pantry-vision and chores. Set PULL_VISION_MODEL=false to skip it and save several GB if you're not running those camera paths yet.

The vision model choice is not a considered one. llava is simply the default those two services already ship with, and open decision #18 flags the pick as unmade and completely unbenchmarked. If grocery recognition turns out too slow or too inaccurate to be usable, this is the first knob to turn — qwen2.5vl and moondream are the obvious alternatives to measure against.

Contention: interactive vs. batch on one GPU

The real scheduling problem here (project plan open decision #4), and the script's defaults take a position on it:

• Assist is interactive — a person is standing in the room waiting.
• digest-engine is batch — every 6h, nobody watching.
• The vision callers are occasional but want a different model resident.

Setting	Default	Why
OLLAMA_KEEP_ALIVE	30m	Ollama's own default of 5m means a household that talks to Assist a few times an hour pays the model-load cost nearly every time. 30m keeps it warm through normal use
OLLAMA_MAX_LOADED_MODELS	1	Deliberate. A 14B text model and a vision model don't co-fit in 8–12GB; letting Ollama try produces VRAM thrash or an OOM mid-request instead of an honest swap. 1 means "swap predictably, pay the reload when vision is actually needed." Raise it only if you have the VRAM and have checked
OLLAMA_NUM_PARALLEL	1	Predictable latency for whoever is speaking, over throughput nothing here needs

This is a reasoned default, not a measured one — none of it has been run against a real GPU under real concurrent load. The remaining half of open decision #4 (whether DIGEST_SCHEDULE's 00,06,12,18 overlaps real Assist usage) needs actual usage data to settle; the settings above at least make the failure mode a predictable swap rather than an OOM.

Security: Ollama has no authentication

None. Not a token, not a password. And its API is not read-only — it can pull and delete models, not just generate. Anyone who can reach :11434 can do all of that.

The network is therefore the entire boundary: keep this host on the smart-home VLAN and never port-forward it, exactly as docs/network-integration.md §1 says for everything else. It's now in that document's port table for the same reason.

Why a container, not the native installer

Ollama's official install is curl -fsSL https://ollama.com/install.sh | sh, which pipes a fetched script straight into a root shell. The container path gives a pinned image, an uninstall that's docker rm, and no arbitrary remote code executed as root — the same reasoning behind every other component in this project running in Docker.

The native install is a perfectly legitimate alternative, and on some GPU setups it's less fuss than the NVIDIA Container Toolkit. If you go that way, the one thing you must still do is set OLLAMA_HOST=0.0.0.0:11434 in the systemd unit — see below.

The one configuration mistake that looks like a dead host

Ollama binds 127.0.0.1 by default. In a container, that means the published port forwards to a socket nothing is listening on, and every caller gets a connection refused that is indistinguishable from "the LLM host is powered off" — which, given that every consumer here is built to tolerate exactly that, degrades silently and looks like nothing is wrong.

The compose file sets OLLAMA_HOST=0.0.0.0:11434 for this reason. Check it first if inference is mysteriously "unavailable" everywhere at once. (This is the same class of bug as chores' env template pointing at 127.0.0.1 for a sibling container — see the project plan's open decision #38.)

Run it

sudo -E tools/setup-llm-host.sh

Edit the variables at the top first — BASE_DIR above all, since models are large (a 14B Q4 model is ~9GB, a vision model another 5–8GB) and it defaults to /opt/llm-host.

For the GPU tier, the NVIDIA driver must already work (nvidia-smi prints your card). The script installs the Container Toolkit that lets Docker see the GPU, but deliberately does not install the driver: that's the most hardware- and kernel-specific step on this machine, and silently choosing a driver version for someone is a good way to produce a box that doesn't boot.

Afterwards the script prints exactly what to paste into HA and into each service's env file on the container host.

Manual verification still outstanding

1. None of this has been run. No Debian machine, no GPU, no Ollama server — the script is syntax-checked and its generated compose file is validated as YAML for both tiers, and that is the entire extent of the testing. Same honesty rule as every other unbuilt host in this repo.
2. The model tags are library names that upstream does rename. qwen2.5:14b-instruct and llava are written from Ollama's library as documented, not confirmed pullable today. A failed pull is deliberately non-fatal — the server stays up and you fix the tag by hand — but check https://ollama.com/library if one fails.
3. The NVIDIA Container Toolkit repo/apt steps are from NVIDIA's documented install, not run on a real machine. nvidia-ctk runtime configure --runtime=docker followed by a Docker restart is the documented shape; verify against NVIDIA's current docs before trusting it on hardware you care about.
4. The contention defaults are unmeasured — see the table above.
5. Whether a 14B model at Q4 actually fits your card is not checked anywhere. On 8GB it will be tight-to-impossible; on 12GB+ it's comfortable. If it OOMs, drop to the CPU tier's 7B tag on the GPU, which is the cheap first thing to try.
6. No Wake-on-LAN. If you want the digest timer to wake this host rather than skip its run, that's a BIOS + ethtool -s <iface> wol g + a wakeonlan call from the container host's timer — deliberately not scripted here, since it depends on hardware that hasn't been chosen.