Amir Alexander Abdelbaki
ea82ee70ad
Installation was six scripts each carrying its own copy of the container host's IP, three that had to agree on IDENTITY_TOKEN, and every service URL typed by hand with a port in it. Any one could be wrong, and the symptom was always the same and always late: an image that boots fine and then can't reach something, found after a 40-minute build and a reboot. Two properties fix that class of bug: - Nothing is written twice. No script in tools/ contains an IP, port or token. - Anything derivable is derived. You give the subnet prefix once and one last octet per host; every address and service URL is computed from those. THE TWINNED PAIR. container_host.ip_last_octet 12 and llm_host 13 mean the container host's OLLAMA_HOST *is* http://<prefix>.13:11434 — computed in the same build, not typed into two files and kept in sync. Move the LLM host to .21 and the container host's Ollama URL follows; change the subnet and both halves move along with every kiosk's URLs. Neither image can be built pointing at an address the other isn't using. Both carry the same SMARTHOME_PAIR_ID (a hash of the config's meaning, not its bytes) so two USB sticks can be checked against each other later. validate-config.py runs before every build and refuses to start on an error, so a mistake costs seconds not an hour. It catches duplicate ports (including the music_assistant/pantry_vision 8095 clash that Compose can't see because MA runs network_mode:host — open decision #31), both hosts on one address, duplicate hostnames across kiosks and audio endpoints, placeholder tokens (checked before the length check, so padding "changeme" to 32 chars doesn't pass), a private key pasted where the public one goes, and a kiosk pointed at a disabled service. build-all.sh is the normal entry point — the images are a set that has to agree with itself, so building one is the exception. It builds the core pair, every kiosk, and every audio endpoint including both architectures (amd64 live-build ISO and arm64 rpi-image-gen img are different toolchains, not one image). The two new host ISOs install unattended with everything burnt in, including service env files generated from derived values — which permanently removes the class of bug that had chores.env shipping IDENTITY_URL=http://127.0.0.1:8097. setup-container-host.sh and setup-llm-host.sh now read every config value as ${VAR:-default} so the images configure them without editing. That also makes every ISO a credential: Wi-Fi PSK, tokens, MQTT and HA credentials are readable by anyone holding the stick. .gitignore covers the filled-in CoreSystemConfig.json and build-output/. Tested: 43 config validation/derivation checks and 44 builder checks against the real code paths with only `lb` stubbed — every generated env file, preseed, network config, first-boot unit and build stamp is verified, including that a port collision refuses the build before writing anything. No ISO has been built; `lb build` needs live-build, root and a long fetch. tools/README.md says so. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| README.md | ||
README.md
llm-host
The Ollama machine, from Phase 3 of the project plan. A separate physical host from the container host, on purpose — see "Why a separate machine" below.
Everything in this project that wants inference calls this one server:
| Caller | What it asks for | If this host is off |
|---|---|---|
| Home Assistant (Assist / AI Task) | Conversation, tool calls, Phase 4's brightness/colour JSON | Assist's LLM agent is unavailable; presence → light still works, on plain automations |
digest-engine |
Quarter-daily synthesis + the counter-run verification pass | The run is skipped; the timer doesn't error |
pantry-vision |
Grocery-item recognition from one photo | The scan flow reports it can't identify; Grocy is untouched |
chores |
Bin/dishes/litter vision checks, and reminder phrasing | Camera checks skip; nudges use the plain template |
The guardrail this host is built around
Every consumer must degrade to "unavailable", never to "broken", when this machine is off. The project plan's testing checklist states it directly — "Does the reactive path (presence → light on) work with the LLM host powered off? (It must.)"
That's the whole reason this is a separate box rather than more containers on the Phase 1 host: it can be off — for power, for noise, because you pulled the GPU — and the house still works. The setup script's closing output walks you through actually testing that, and it's worth doing once for real rather than assuming.
If something breaks rather than degrading when this host is down, that's a bug in the consumer, not here.
Two tiers
Auto-detected from whether nvidia-smi both exists and succeeds (a leftover driver
package on a machine whose card was pulled satisfies the first but not the second).
Override with TIER at the top of the script.
| Tier | Model | Reality |
|---|---|---|
gpu |
qwen2.5:14b-instruct |
What Phase 3 specifies |
cpu |
qwen2.5:7b-instruct |
Single-digit tokens/sec. Enough to validate the entire pipeline end to end before buying a card — docs/components.md's deliberate "skip GPU" fallback, not a failure mode |
Plus a vision model (llava by default) for pantry-vision and chores. Set
PULL_VISION_MODEL=false to skip it and save several GB if you're not running those
camera paths yet.
The vision model choice is not a considered one. llava is simply the default
those two services already ship with, and open decision #18 flags the pick as unmade
and completely unbenchmarked. If grocery recognition turns out too slow or too
inaccurate to be usable, this is the first knob to turn — qwen2.5vl and moondream
are the obvious alternatives to measure against.
Contention: interactive vs. batch on one GPU
The real scheduling problem here (project plan open decision #4), and the script's defaults take a position on it:
- Assist is interactive — a person is standing in the room waiting.
digest-engineis batch — every 6h, nobody watching.- The vision callers are occasional but want a different model resident.
| Setting | Default | Why |
|---|---|---|
OLLAMA_KEEP_ALIVE |
30m |
Ollama's own default of 5m means a household that talks to Assist a few times an hour pays the model-load cost nearly every time. 30m keeps it warm through normal use |
OLLAMA_MAX_LOADED_MODELS |
1 |
Deliberate. A 14B text model and a vision model don't co-fit in 8–12GB; letting Ollama try produces VRAM thrash or an OOM mid-request instead of an honest swap. 1 means "swap predictably, pay the reload when vision is actually needed." Raise it only if you have the VRAM and have checked |
OLLAMA_NUM_PARALLEL |
1 |
Predictable latency for whoever is speaking, over throughput nothing here needs |
This is a reasoned default, not a measured one — none of it has been run against a
real GPU under real concurrent load. The remaining half of open decision #4 (whether
DIGEST_SCHEDULE's 00,06,12,18 overlaps real Assist usage) needs actual usage data
to settle; the settings above at least make the failure mode a predictable swap rather
than an OOM.
Security: Ollama has no authentication
None. Not a token, not a password. And its API is not read-only — it can pull and
delete models, not just generate. Anyone who can reach :11434 can do all of that.
The network is therefore the entire boundary: keep this host on the smart-home VLAN
and never port-forward it, exactly as docs/network-integration.md §1 says for
everything else. It's now in that document's port table for the same reason.
Why a container, not the native installer
Ollama's official install is curl -fsSL https://ollama.com/install.sh | sh, which
pipes a fetched script straight into a root shell. The container path gives a pinned
image, an uninstall that's docker rm, and no arbitrary remote code executed as root
— the same reasoning behind every other component in this project running in Docker.
The native install is a perfectly legitimate alternative, and on some GPU setups it's
less fuss than the NVIDIA Container Toolkit. If you go that way, the one thing you
must still do is set OLLAMA_HOST=0.0.0.0:11434 in the systemd unit — see below.
The one configuration mistake that looks like a dead host
Ollama binds 127.0.0.1 by default. In a container, that means the published port
forwards to a socket nothing is listening on, and every caller gets a connection
refused that is indistinguishable from "the LLM host is powered off" — which, given
that every consumer here is built to tolerate exactly that, degrades silently and
looks like nothing is wrong.
The compose file sets OLLAMA_HOST=0.0.0.0:11434 for this reason. Check it first if
inference is mysteriously "unavailable" everywhere at once. (This is the same class of
bug as chores' env template pointing at 127.0.0.1 for a sibling container — see
the project plan's open decision #38.)
Run it
sudo -E tools/setup-llm-host.sh
Edit the variables at the top first — BASE_DIR above all, since models are large
(a 14B Q4 model is ~9GB, a vision model another 5–8GB) and it defaults to
/opt/llm-host.
For the GPU tier, the NVIDIA driver must already work (nvidia-smi prints your
card). The script installs the Container Toolkit that lets Docker see the GPU, but
deliberately does not install the driver: that's the most hardware- and
kernel-specific step on this machine, and silently choosing a driver version for
someone is a good way to produce a box that doesn't boot.
Afterwards the script prints exactly what to paste into HA and into each service's env file on the container host.
Manual verification still outstanding
- None of this has been run. No Debian machine, no GPU, no Ollama server — the script is syntax-checked and its generated compose file is validated as YAML for both tiers, and that is the entire extent of the testing. Same honesty rule as every other unbuilt host in this repo.
- The model tags are library names that upstream does rename.
qwen2.5:14b-instructandllavaare written from Ollama's library as documented, not confirmed pullable today. A failed pull is deliberately non-fatal — the server stays up and you fix the tag by hand — but check https://ollama.com/library if one fails. - The NVIDIA Container Toolkit repo/apt steps are from NVIDIA's documented install,
not run on a real machine.
nvidia-ctk runtime configure --runtime=dockerfollowed by a Docker restart is the documented shape; verify against NVIDIA's current docs before trusting it on hardware you care about. - The contention defaults are unmeasured — see the table above.
- Whether a 14B model at Q4 actually fits your card is not checked anywhere. On 8GB it will be tight-to-impossible; on 12GB+ it's comfortable. If it OOMs, drop to the CPU tier's 7B tag on the GPU, which is the cheap first thing to try.
- No Wake-on-LAN. If you want the digest timer to wake this host rather than skip
its run, that's a BIOS +
ethtool -s <iface> wol g+ awakeonlancall from the container host's timer — deliberately not scripted here, since it depends on hardware that hasn't been chosen.