# AI-Managed Smart Home — Full Build Plan (v2) Local-first, open-source stack: Home Assistant + RuView presence + Bermuda BLE identity + local LLM (Ollama) + Zigbee sensors/lighting + Frigate (peephole face recognition) + Grocy kitchen kiosk + Nextcloud calendar sync + Node-RED glue logic + a Sway thin-client media station + a Sway touch panel + a camera-vision kitchen/fridge display + a voice/touch identity-registration door panel + a quarter-daily LLM digest. --- ## 1. Hardware ### 1.1 Already in your possession | Item | Use in this build | |---|---| | Framework 12 (primary) | Dev machine, dotfiles/Claude Code work | | ThinkPad T440p (secondary) | Spare compute, webcam presence daemon testing | | Nextcloud instance | Calendar backend (CalDAV sync target) | | "Scraps" x86 machine | Candidate for LLM host (CPU-only tier) or Proxmox host | | Haozee Zigbee USB dongle (CC2652P + CH340C) | Zigbee coordinator — USB passthrough into container, adapter type `zstack` | ### 1.2 Core AI / HA hosts | Item | Est. Price (EUR) | Notes | |---|---|---| | Mini PC / SFF for Home Assistant + attached services (Raspberry Pi 5, 8GB) | €80 | Runs HA Core (Container), Mosquitto, Zigbee2MQTT, Frigate, Grocy, Node-RED, monitoring | | SSD for boot (USB3, 128GB+) | €20 | Avoid SD card for reliability | | Official Pi 5 PSU + case | €25 | | | Used RTX 3060 12GB | €200–250 | LLM inference host (Qwen2.5-14B Q4) | | Cheap SFF/tower host for GPU (spare PCIe x16 slot, 170W+ on 12V rail) | €40–60 | Only needs to feed the GPU | | *(Fallback: skip GPU)* Use existing "scraps" x86 box, CPU-only Ollama | €0 | Slower (5–10 tok/s), fine to validate before buying GPU | **Subtotal (with GPU tier): ~€365–435** | **Subtotal (CPU-only fallback): ~€125** ### 1.3 Zigbee backbone Already covered — using your existing Haozee CC2652P USB dongle. No coordinator purchase needed. *Contingency: if Zigbee2MQTT can't establish a network, the dongle may need CC2652P coordinator firmware reflashed (generic-branded gateways sometimes ship with non-coordinator firmware).* ### 1.4 Zigbee sensors (price per unit — multiply by room count) | Item | Est. Price/unit | Suggested qty | Notes | |---|---|---|---| | innr RB 285 C (Zigbee RGBWW bulb) | €15–20 | 1 per AI-managed room | Tunable 2200K–6500K, acts as Zigbee router | | Sonoff SNZB-02P (temp/humidity) | €10–12 | 1 per room | Avoid older Aqara — mesh drop issues | | Sonoff SNZB-06P (presence/illuminance) or Aqara Light Sensor | €12–15 | 1 per room needing lux data | Feeds LLM brightness decisions | | Sonoff SNZB-04P (door/window contact) | €13–15 | 1 per door/window of interest | Gates automations | | IKEA Vindstyrka (air quality, optional) | ~€30 | 1–2 | Requires Zigbee2MQTT | ### 1.5 BLE identity layer (Bermuda) | Item | Est. Price/unit | Suggested qty | Notes | |---|---|---|---| | ESP32 board (D1 Mini32 or similar — no CSI needed, just BT+WiFi) | €5–10 | 1 per room for identity resolution | Runs ESPHome `bluetooth_proxy`, separate hardware from RuView nodes | | Fixed-MAC BLE tag per household member (optional but recommended) | €5–8 | 1 per person | More reliable anchor than phone MAC (which randomizes on iOS/Android); Bermuda + HA's Private BLE Device can also resolve phones via IRK if you skip this | ### 1.6 RuView presence mesh | Item | Est. Price/unit | Suggested qty | Notes | |---|---|---|---| | ESP32-S3 dev board | ~€9 | 1 per room/zone needing CSI presence | RuView's primary/confirmed target board (`firmware/ruview/README.md`) — separate boards from the Bermuda BLE proxies above (one chip = one firmware). **ESP32-C3 and the original ESP32 are explicitly unsupported** ("single-core, insufficient for CSI DSP," per RuView's own README) — this is now a confirmed hardware requirement, not just a recommendation | | ESP32-C6 dev board (optional) | €6–10 | Research/WiFi-6 variant | Not required for basic per-room presence — see `firmware/ruview/README.md` §1 | ### 1.7 Door-spy / peephole camera | Item | Est. Price | Notes | |---|---|---| | Peephole IP camera (ONVIF/RTSP, color, "-IR" variant only if hallway is dark) | €50–65 | Isolate on its own VLAN, block WAN egress | | *(You're handling the physical mounting yourself)* | €0 | | ### 1.8 Frigate acceleration (optional) | Item | Est. Price | Notes | |---|---|---| | Hailo-8L (Raspberry Pi 5 AI kit) | €70 | Only if CPU/iGPU isn't keeping up with more cameras later | | *(Default: none needed for 1 camera — use CPU or Intel iGPU/OpenVINO)* | €0 | | ### 1.9 Kitchen inventory kiosk (camera + touchscreen) | Item | Est. Price | Notes | |---|---|---| | Raspberry Pi 3B+ or 4 (2–4GB) | €35–50 | Chromium kiosk + camera decode | | 7" official/compatible touchscreen | €30–40 | | | USB webcam w/ macro focus | €10–15 | Barcode scanning via Grocy's browser PWA | | Case/mount for kitchen wall or cupboard | €10–15 | | **Subtotal: ~€85–120** ### 1.10 Deferred / out of scope for now - **Projector + camera tabletop AI surface** — genuinely interesting but high-effort (projector-camera calibration, fingertip touch detection is an active CV research problem, poor legibility with ambient light). If pursued, treat as an isolated experimental side project, not wired into critical automations. Not included in this plan's budget or phases. ### 1.11 Sway thin-client hardware | Item | Est. Price (EUR) | Notes | |---|---|---| | Mini PC / SFF (Intel N100/N305-class, Quick Sync, wired GbE) | €150–220 | Wired Ethernet strongly preferred over WiFi for Steam Link latency | | Display (HDMI monitor/TV) | €0–150 | May already have one | | Keyboard + mouse or remote | €20–40 | Local fallback input — primary control is HA/MQTT + wayvnc, not this | | USB mic + speaker (mic-enabled rooms only) | €25–50/room | Only for the specific rooms chosen for voice interactivity — see §3 Phase 11 open decisions | *(No new hardware for Phase 12 or Phase 13 — `digest-engine`/`digest-web` and `admin-canvas`/`admin-web` all run as containers on the existing container-host from Phase 1.)* ### 1.12 ESP32-S3 voice/display satellite hardware (Phase 14) | Item | Est. Price (EUR) | Notes | |---|---|---| | Waveshare ESP32-S3-Touch-LCD-1.85C, **V2 revision** | €35–45 | Round 360×360 LCD + dual mic (ES7210, with echo-cancellation reference path) + speaker (ES8311) + 8Ω 2W speaker. **Must be V2** — V1 has no AEC circuit and different audio pins, see `firmware/esp32-s3-touch-lcd-1.85c/README.md`. An alternative to Home Assistant Voice PE for rooms that also want a status display, not a wholesale replacement of §1.11's mic-enabled-room hardware | ### 1.13 Headless audio endpoint hardware (Phase 15) | Item | Est. Price (EUR) | Notes | |---|---|---| | Raspberry Pi (Zero 2 W is enough for one Spotify Connect stream; any 40-pin-header Pi works) | €15–35 | arm64 variant | | HiFiBerry Amp2 HAT | ~€45 | 60W Class D, drives 4–8Ω passive speakers directly — GPIO HAT, Raspberry-Pi-only | | Passive bookshelf speakers (e.g. Micca MB42X G2) | ~€100/pair | Same pairing already recommended for the thin-client rooms | | *(amd64 alternative)* Spare x86 mini PC + USB DAC/amp (e.g. Fosi Audio V3, ~€130) | €0–130 | No GPIO HAT involved — a USB Audio Class device instead, for rooms getting a repurposed mini PC rather than a Raspberry Pi | ### 1.14 Touch panel hardware (Phase 16) | Item | Est. Price (EUR) | Notes | |---|---|---| | Mini PC or SBC with a capacitive touchscreen (e.g. 10–15" USB-touch monitor + small x86 mini PC, or an all-in-one touch panel PC) | €150–300 | **No specific unit has been chosen.** A native Wayland `wl_touch` device is preferred but no longer a hard requirement — `hosts/touch-panel/` also ships a `type:pointer` fallback (usable single-touch tap/click-drag) for touchscreens that misreport as an emulated-mouse HID device, plus a udev override template to reclassify a specific known device back to full touch semantics. See that host's "Touch input: two tiers" README section | | Speakers or a wired connection to existing room audio | €0–100 | For local Spotify GUI-client playback — same speaker guidance as the thin-client/audio-endpoint rooms | *(No container-host changes — `touchpanel-agent` talks to the existing Mosquitto/HA from Phase 1, same as every other MQTT-connected host in this plan.)* ### 1.15 Kitchen/fridge display hardware (Phase 17) | Item | Est. Price (EUR) | Notes | |---|---|---| | Mini PC or SBC with a touchscreen, mounted near the fridge/pantry | €150–300 | Prefers a native Wayland `wl_touch` device, same as §1.14 — unlike that host, `hosts/kitchen-display/` does not (yet) ship the `type:pointer`/udev-override fallback described there; a misclassified touchscreen here is still an open problem | | USB webcam, pointed at wherever items get held up for scanning | €15–40 | Either built into the panel or on a short cable/gooseneck so its angle can be set independently of the screen. Any UVC webcam works — this is the same "no depth camera, no accelerator needed" bar as the thin client's gesture-control camera (§1.11), just used for a photo instead of continuous tracking | | Zigbee door contact sensor, one per cold appliance door | €10–15 | ~3 to start (fridge, freezer compartment, second freezer). Mounted on the **outside** face of the door — a coin cell at −18 °C is a false economy and every opening condenses room air onto whatever is inside. Worth buying on their own merits ("has the freezer been open four minutes") before any camera exists; see `docs/fridge-item-location.md` | | Doorway camera, one per appliance (optional, later) | €30–40 | Outward-facing, above the door on the hinge side, ~1.8–2 m, ingested by Frigate like every other camera in this project. **Never inside the appliance.** Buy one, not two — and only after the door sensors have been lived with, per `docs/fridge-item-location.md`'s recommended order | *(No new container-host hardware — `pantry-vision` is a container on the existing Phase 1 host, calling the existing Phase 3/9 LLM host for vision inference and the already-running `grocy` container for storage. It does gain one small bind mount, `/data`, for the doorway-hint database — see Phase 17 item 4a.)* ### 1.16 Door/wardrobe panel hardware (Phase 18) | Item | Est. Price (EUR) | Notes | |---|---|---| | Mini PC or SBC with a touchscreen, mounted by the door/wardrobe | €150–300 | Same native Wayland `wl_touch` preference as §1.14/§1.15 — no specific unit chosen | | USB microphone + speaker (or a combo device) | €20–40 | This device's actual purpose is voice registration ("register me as ``"), unlike every other host's opt-in mic — budget for one from the start rather than treating it as optional | *(No camera required — unlike §1.15, identity resolution here is BLE/IRK-based, not camera-based; a registration photo is a nice-to-have via the same webcam pool as §1.15's spares if one ends up mounted, not a requirement. No new container-host hardware — `identity` is a container on the existing Phase 1 host, same as `pantry-vision`.)* ### 1.17 TP-Link Tapo pan/tilt security cameras (Phase 20) | Item | Est. Price (EUR) | Notes | |---|---|---| | TP-Link Tapo C500/C520WS (or similar pan/tilt "look around" model) | €40–70 each | Number of units and placement (trash-bin sightlines, general household areas) is your call, not this plan's — start with 1–2 and expand once Frigate integration is confirmed working | *(No new container-host hardware — these are IP cameras feeding into the existing Phase 5 Frigate NVR as additional camera sources, same as the peephole cam (§1.7). **Not RTSP-onvif-native out of the box** — Tapo cameras generally need either "Advanced Settings -> Camera Account" enabled for a direct RTSP stream, or go2rtc (bundled with recent Frigate) as a protocol bridge; VERIFY the exact stream URL and whether your specific model's firmware exposes RTSP at all before buying more than one — see `identity/README.md`'s and `chores/README.md`'s own "not verified against real hardware" callouts for everything downstream of this.)* --- ## 2. Software (all open source / self-hosted) | Layer | Software | Purpose | |---|---|---| | Home automation core | **Home Assistant** (Container install) | Central hub, automations, dashboards | | Zigbee | **Zigbee2MQTT** + **Mosquitto** | Broadest device compatibility, USB passthrough to your CC2652P dongle | | BLE identity | **Bermuda** (HACS) + **ESPHome** (`bluetooth_proxy`) + HA's **Private BLE Device** | Room-level BLE presence, resolves randomized MACs via IRK, feeds `person.*` entities | | Automation glue | **Node-RED** | Visual flows for cross-system logic — home for the face↔MAC identity-correlation logic, easy iteration/debugging | | LLM runtime | **Ollama** | Serves Qwen2.5-14B-Instruct (GPU tier) or Qwen2.5-7B/3B (CPU tier) | | Conversation agent | **HA Ollama conversation integration** | Ties LLM into Assist + AI Task | | Voice STT/TTS | **Wyoming faster-whisper** + **Piper** | Local speech pipeline | | Presence (CSI) | **RuView** ([github.com/ruvnet/ruview](https://github.com/ruvnet/ruview), Rust/ESP-IDF firmware, integrated not forked — `firmware/ruview/`) | Room-level presence via WiFi CSI, **not anonymous** — also publishes 10 inferred semantic states (sleep, vitals-adjacent, bathroom occupancy, fall risk, etc.) over MQTT auto-discovery; see `firmware/ruview/README.md` §2's privacy callout before wiring automations against anything beyond plain presence | | Presence (webcam) | Your custom daemon (dotfiles repo) | Desktop-specific, not tied to HA sensors directly | | Camera NVR / face recognition | **Frigate** (0.16+) | Native face recognition, object detection, peephole cam ingest | | Identity store | Small dict/table (SQLite or JSON) — name, face label, associated MACs, confidence | Built up conversationally via LLM tool calls (`propose_person_link`, `confirm_person`, `rename_person`); merges require confirmation, never silent | | Container runtime | **Docker + Docker Compose** | Hosts all of the above | | Inventory / shopping list | **Grocy** | Self-hosted stock + shopping list, native browser barcode scanning | | Meal planning (optional) | **Mealie** | Recipe manager + shopping list + meal-plan calendar | | Calendar backend | **Nextcloud Calendar** (already running) | Source of truth for household schedule | | Calendar bridge | **HA CalDAV integration** | Read/write bridge, `calendar.create_event` supported; keep recurring events created in Nextcloud directly | | Monitoring | **Netdata** | Consistent with your existing Ansible-fleet monitoring evaluation | | Dashboard | **Homepage** or **Homarr** | Single landing page for HA / Grocy / Frigate / Zigbee2MQTT / Node-RED UIs | | Container management | **Portainer** (optional) | GUI over the Docker host | | Notifications | **ntfy** | Self-hosted push, replaces relying on a phone OS's proprietary push channel | | Long-term stats (optional) | **InfluxDB + Grafana** | Only if you want history beyond HA's default recorder retention | | Backup | **restic** | Scheduled encrypted backups of all stateful volumes | | Kiosk browser | **Chromium (kiosk mode)** | Displays Grocy PWA on the kitchen touchscreen | | Thin-client OS build | **live-build** | Builds the thin-client ISO from `hosts/thin-client/live-build/` | | Thin-client compositor | **Sway** | Kiosk Wayland compositor | | Thin-client autologin | **greetd** | Autologin straight into `sway`, no separate greeter UI | | Thin-client remote view/control | **wayvnc** | VNC for wlroots compositors — the chosen remote-control channel (Sway/wlroots has no maintained RDP path; wayvnc replaces RDP for this project) | | Thin-client scripted control | **thinclient-agent** (custom) | HA MQTT-discovery entity + `swaymsg`/app-process control; the only surface the LLM can reach, always mediated through HA | | Thin-client media | **mpv** + **mpv-mpris** | Local playback with MPRIS2 D-Bus control, bridged into `thinclient-agent`'s HA `media_player` entity | | Thin-client music | **spotifyd** / **librespot** | Headless Spotify Connect receiver (Premium required, unofficial protocol) | | Thin-client game streaming | **Steam Link** (Flatpak/Flathub) + **Xwayland** | Remote Steam play; Xwayland avoids native-Wayland black-screen/flicker bugs on wlroots | | Thin-client browser | **Firefox (kiosk)** | General browsing + the rendering surface for the LLM-generated digest canvas | | Thin-client voice | **wyoming-satellite** + **openWakeWord** | Local wake-word spotting, streams to the existing Phase 3 Wyoming faster-whisper/Piper Assist pipeline — no new STT/TTS infrastructure | | Digest scheduling | **systemd timer** | 4x/day cadence, same `OnCalendar` pattern as the existing restic backup timer | | Digest engine | **digest-engine** (custom Python) | Ingests mail/messages/news/financial data, calls the Phase 3 Ollama host, renders the personal/political/household/network digest sections, generating only the ones the household ticked in `identity` | | Digest static serving | **digest-web** (nginx:alpine/Caddy) | Serves the rendered digest artifact read-only to both the thin client and an HA iframe card | | Digest ingestion — Signal | **signal-cli** | Linked-device (JSON-RPC) read access to Signal messages | | Digest ingestion — Telegram | **Telethon** | MTProto client logged in as the real account — the Bot API can't read personal DMs | | Digest ingestion — Discord | **discord.py** | Bot with Message Content intent, scoped to servers you own/admin only — no personal-DM access | | Digest ingestion — WhatsApp | **whatsapp-bridge** (custom, Node.js + `whatsapp-web.js`/Puppeteer + Xvfb, opt-in) | Real WhatsApp Web session in a virtual display, headful Chromium to avoid headless-detection; still opt-in via `ENABLE_WHATSAPP_INGEST`, off by default | | Digest ingestion — mail | **imapclient** | IMAP fetch (App Password or OAuth2/XOAUTH2) | | Digest ingestion — news | **feedparser** | Curated OPML feed list, including `https://www.marxist.com/feed/rss` | | Digest ingestion — financial | **FRED API** + **Stooq** | Macro/unemployment indicators + stock/oil/commodity data | | Digest rendering | **digest-canvas SDK** (custom, vendored) | Offline globe (`addMarker(lat, lon, {icon, color, glow})`), window/panel chrome, and glow/holo CSS primitives the LLM composes against each run | | Admin canvas write API | **admin-canvas** (custom Python, stdlib `http.server`) | Small always-on service, `POST /show` + `POST /media/`, bearer-token gated, no published port — reachable only from Home Assistant on the compose network. The sys-admin-llm's on-demand counterpart to the scheduled digest-engine | | Admin canvas static serving | **admin-web** (nginx:alpine) | Serves the admin canvas's rendered JSON + uploaded media read-only to the thin client — same role as digest-web, separate instance | | Admin canvas rendering | **canvas-sdk** (custom, vendored, duplicated from digest-canvas SDK) | Same window/panel chrome and glow theme, minus the globe, plus `stat`/`image`/`video`/`chart` window kinds (the `chart` kind is a dependency-free inline-SVG bar/sparkline) | | Voice/display satellite firmware | **ESPHome** (custom config, `firmware/esp32-s3-touch-lcd-1.85c/`) | Display (LVGL), on-device wake word (`micro_wake_word`, `okay_nabu`), `voice_assistant` streaming into the existing Phase 3 Assist pipeline, and a media/cover-art-over-idle-weather-time-date priority display | | Headless audio endpoint (arm64) | **rpi-image-gen** (custom config, `hosts/audio-endpoint/rpi-image-gen/`) | Raspberry Pi Foundation's current custom-image tool; builds a flashable `.img` with a per-room Spotify Connect receiver + HiFiBerry Amp2 audio | | Headless audio endpoint (amd64) | **live-build** (custom config, `hosts/audio-endpoint/live-build-amd64/`) | Reuses the thin client's own build tool, stripped of the whole graphical/kiosk stack — headless boot straight to the same per-room Spotify Connect receiver | | Headless audio endpoint software | **librespot** or **spotifyd** (apt-first, documented fallback) | Independent, per-room Spotify Connect — no Snapcast, no shared/synced stream; each endpoint is its own device in Spotify's picker, same model as the thin client's own per-room spotifyd/librespot (Phase 11.6) | | Touch-panel OS build | **live-build** (custom config, `hosts/touch-panel/live-build/`) | Reuses the thin client's own build tool and directory-split convention, not its live-build tree — a different, smaller image for a different device | | Touch-panel compositor | **Sway** | Same choice as the thin client, configured for direct touch instead of remote control: an always-on dock instead of no bars, no gesture-camera/wayvnc machinery | | Touch-panel scripted control | **touchpanel-agent** (custom) | HA MQTT-discovery entity + `swaymsg`/app-process control, identical security shape to `thinclient-agent` — LLM tool call → HA service call → MQTT → this agent, never a direct path | | Touch-panel music | **Spotify (official Linux client, Flathub `com.spotify.Client`)** | Full GUI client, not a headless Connect receiver — the point of a touch panel is a screen you interact with directly (Premium required, same as the headless receivers) | | Touch-panel home dashboard | **Chromium (kiosk mode, `--app=`)** | Dedicated, always-open window pointed at Home Assistant, auto-restarted if it crashes | | Touch-panel browser | **Firefox** | General browsing, minimal (not kiosk-locked) chrome — back/forward/reload/address bar | | Touch-panel on-screen keyboard | **wvkbd** (apt-first, documented fallback) | Manually toggled from the touch dock — no shell-level input-method integration on this image, so no automatic show-on-focus | | Touch-panel navigation | **eww** (apt-first, documented fallback) | An always-on, layer-shell-reserved touch dock (Spotify / Home / Web / Keyboard) — the touch-first equivalent of the thin client's HA-only **Screen** select | | Kitchen-display OS build | **live-build** (custom config, `hosts/kitchen-display/live-build/`) | Same toolchain/convention reuse as the touch panel, a third and simpler sibling image — one workspace, one app | | Kitchen-display camera capture | Browser-native `getUserMedia()` (Chromium) | No native capture app on-device at all — the kiosk page itself talks to the webcam and hands a frame straight to `pantry-vision` | | Kitchen-display scripted control | **kitchen-display-agent** (custom) | HA MQTT-discovery entity for **Show scan/inventory/recipes** only — the actual camera/Grocy read-write path is `pantry-vision`'s own published API, called directly by the kiosk browser, not mediated through this agent | | Grocery vision recognition | **Ollama** (a vision-capable model, e.g. `llava`/`qwen2.5vl` — TBD, not yet pulled or benchmarked) | Identifies a grocery item from one photo and estimates its shelf life; the proposal is always human-reviewed before anything is written (see `pantry-vision/README.md`) | | Grocery inventory backend | **pantry-vision** (custom Python, stdlib `http.server`) | `POST /identify` (photo → proposal), `POST /confirm` (human-reviewed proposal → Grocy stock write), `GET /inventory`/`GET /recipes` (proxy Grocy, reshaped). Bearer-token gated, **and — unlike admin-canvas — LAN-published**, since the kitchen display's kiosk browser calls it directly rather than through Home Assistant | | Grocery inventory storage | **Grocy** (already running, Phase 1/7) | The single source of truth for stock/best-before-dates/recipes; `pantry-vision` is a client of Grocy's own REST API, not a replacement for it | | Kitchen-display static serving | **pantry-web** (nginx:alpine) | Serves `pantry-vision/frontend/`'s single-page app (unload / consume / expired / edit, plus Inventory and Recipes) read-only to the kitchen display — same role `digest-web`/`admin-web` play for their own hosts | | Identity registry backend | **identity** (custom Python, stdlib `http.server` + `sqlite3`) | Person <-> BLE-identifier registry: registration (voice or touchscreen), presence resolution, visit history, per-device rights, a weather proxy. **Published**, unlike admin-canvas — `hosts/kitchen-display/`'s and `hosts/door-panel/`'s kiosk browsers call it directly, bearer-token gated | | Identity static serving | **identity-web** (nginx:alpine) | Serves `identity/frontend/`'s `register.html`/`dashboard.html`/`admin.html` read-only — same role as `pantry-web`/`digest-web`/`admin-web`. The two kiosk pages are what the wall panels load; `admin.html` is deliberately not linked from either (see Phase 6b) | | Door-panel OS build | **live-build** (custom config, `hosts/door-panel/live-build/`) | Reuses the thin client's build tool/convention, structurally `hosts/kitchen-display/`'s twin — see Phase 18 | | Door-panel scripted control | **door-panel-agent** (custom) | HA MQTT-discovery entity for **Show home/registration** only — identical security shape to every other host's agent | | Door-panel voice | **wyoming-satellite** + **openWakeWord** | Same components as the thin client's Phase 11.8 rooms and `hosts/kitchen-display/`'s opt-in mic, but **on by default** here — voice registration is this device's actual purpose | | Trash-day sync | **trash-calendar** (custom Python, stdlib) | Reads a personal collection-date ICS feed (Kennelbach, AT), writes matching events onto the shared household CalDAV calendar. Read-only against the source feed, write-only (create/update, never delete anyone else's events) against Nextcloud — see `trash-calendar/README.md` | | Public transit — schedules | **GTFS** (static feed, Vorarlberg/VAO) + `transit/sync_gtfs.py` | Parsed with stdlib `csv`/`zipfile`, no external deps; `GET /departures` answers "when's the next bus/train from X" for voice | | Public transit — trip planning | **OpenTripPlanner (OTP)** (self-hosted, official image) | Multi-modal journey planning ("get me from A to B"), proxied via GraphQL through `transit/server.py`'s `/plan` — this repo does not build or manage the OTP graph itself, see `transit/README.md`'s "Route planning scope" (Austria-wide is a moderate commitment, global is a real infrastructure decision) | | Public transit backend | **transit** (custom Python, stdlib `http.server`) | `GET /departures` + `GET /plan`, published (unlike admin-canvas) since HA's `rest_command` needs to reach it and the `homeassistant` container's `network_mode: host` means it can't resolve container DNS names | | Household chore distribution | **chores** (custom Python, stdlib) | Presence/calendar-driven nudge-and-redirect system (see Phase 20) — reads `identity`'s `/presence`, the same trash-day ICS feed as `trash-calendar`, and (optionally) a CalDAV busy-check and Frigate camera checks. Runs as a systemd-timed oneshot, not a long-lived service | | Music library/multi-room audio | **Music Assistant** (own container, official image) | Unifies Spotify Connect + any local/streaming sources behind one HA-native multi-room player abstraction. **Not an HA add-on** — this project runs HA as a plain Container install (§2, above), which has no add-on store, so Music Assistant runs as its own `music-assistant` compose service (`ENABLE_MUSIC_ASSISTANT`, `setup-container-host.sh`, `network_mode: host` for player-discovery mDNS) with HA's own Music Assistant integration pointed at it. **Optional and additive** — does not replace any host's existing per-room spotifyd/librespot/Spotify-client setup (Phase 11.6/15/16), those keep working standalone either way. VERIFY its default port (assumed 8095) against `PANTRY_VISION_PORT` before enabling both — see the setup script's own callout | ### HA integrations catalog — nothing under this repo builds these; they're HA-side installs/configs against systems this repo doesn't touch | Integration | Purpose | Notes | |---|---|---| | **UniFi Network** (HA core) | Presence/device tracking off your UniFi access points, network health sensors | A second, corroborating presence signal alongside Bermuda/RuView/Frigate face recognition — same "OR-ed in, never authoritative for registration" principle as every other presence source in this plan | | **CalDAV** (HA core) | Read/write bridge to Nextcloud Calendar | Already load-bearing in this plan (Phase 8, `trash-calendar`, `chores`'s busy-check) — listed here for completeness as an explicit HA-side integration too, not just this repo's own direct CalDAV clients | | **Matter** (HA core, via the Thread/Matter add-on) | Native support for Matter-certified smart-home devices, if any get added later | Not currently required by anything in this plan's hardware list (§1) — listed as available groundwork, not a current dependency | | **1-Wire** (HA core) | Temperature/humidity sensors on a 1-Wire bus (e.g. DS18B20), if wired in later | Same "available groundwork, not a current dependency" status as Matter above | | **Proxmox VE** (HA core) | VM/container/node status sensors, if any of this stack ends up virtualized on a Proxmox host | Purely a monitoring integration — does not change where any container in this plan actually runs | | **Steam** (HA core) | Friend/game-status sensors from a Steam account | Unrelated to Phase 16's Steam Link game-streaming client — this is presence/activity data, not the streaming path itself | | **Discord** (HA core, notify platform) | Send HA notifications to a Discord channel/webhook | A second notification channel alongside `ntfy` — not a replacement, `chores`/`identity`/everything else keeps using ntfy by default | | **HP iLO** (HA core, via `hpilo` sensor or IPMI) | Server health/power sensors, if any host in this stack is HP server hardware with iLO | Purely a monitoring integration, same class as Proxmox above | | **GTFS** (HA core `gtfs` sensor) | A second, simpler departure-board sensor directly in HA, alongside this repo's own `transit`/`GET /departures` | Redundant with `transit` by design, not a replacement for it — HA's own sensor is single-stop/single-route per entity, `transit`'s voice path is the more general "ask about any stop" interface | ### Self-check / hardware monitoring integrations catalog — same "nothing under this repo builds these" status | Integration | Purpose | Notes | |---|---|---| | **System Monitor** (HA core) | CPU/RAM/disk/network sensors for the machine HA itself runs on | The baseline "is the container host healthy" check, zero extra software | | **SNMP** (HA core) | Polls SNMP-capable network gear (managed switches, the OPNsense box itself, NAS units) | Complements UniFi Network above for anything not UniFi-branded | | **Network UPS Tools (NUT)** (HA core) | Battery/load/runtime sensors from a UPS, and a clean-shutdown trigger on low battery | Recommended if a UPS protects the container host — an unplanned power loss is a real risk to `identity`/`chores`/every other SQLite-backed service's on-disk state | | **Glances** (HA core) | A richer alternative/companion to System Monitor — per-process detail, more sensor granularity | Optional; System Monitor alone covers the basics with no extra service to run | | **Uptime Kuma** (self-hosted, HACS integration or its own HA `webhook`) | External uptime/latency checks for this repo's own published services (`identity`, `pantry-vision`, `transit`, `digest-web`/`admin-web`) | The one entry here this repo's *services* are actually the target of, not the network they run on — worth pointing at every bearer-token-gated published port in this plan once deployed | | **Netdata** | Already in this plan (§2, Phase 9) — per-container/per-host real-time resource monitoring | Listed again here only to make clear it's the same "self-check" category as the rest of this table, not a separate concern | --- ## 3. Implementation Plan ### Phase 0 — Infrastructure prep 1. Decide: Proxmox host (if virtualizing) vs. bare-metal. Given your homelab habits, Proxmox with HAOS/container-host as VMs/LXCs is reasonable. 2. Set up a dedicated **camera VLAN** (peephole cam + future cameras), no WAN egress, firewalled from the main LAN. 3. Provision the container host (Debian 12 or Raspberry Pi OS Lite). ### Phase 1 — Core containers + Zigbee 1. Docker + Compose stack: Home Assistant, Mosquitto, Zigbee2MQTT (USB passthrough to the Haozee dongle), Node-RED. 2. Pair Zigbee sensors room by room: bulb → temp/humidity → illuminance → door/window contact. 3. Confirm entities populate correctly in HA. ### Phase 2 — Presence: RuView + Bermuda, in parallel 1. Deploy RuView ESP32-S3 nodes per room (dedicated CSI firmware) — `firmware/ruview/` now documents the real upstream project ([github.com/ruvnet/ruview](https://github.com/ruvnet/ruview)), a per-room provisioning wrapper (`provision-room.sh`), and a real privacy callout: RuView publishes 10 inferred semantic states (sleep, vitals-adjacent, bathroom occupancy, fall risk, etc.) over MQTT, not just anonymous occupancy — read that README's §2 before wiring automations beyond plain presence. 2. Separately, flash plain ESP32 boards with ESPHome (`bluetooth_proxy` — `firmware/esphome-ble-proxy/`, a stock ESPHome component, safe to build directly unlike RuView's own DSP firmware), install Bermuda via HACS, configure Private BLE Device for phone IRK resolution, and/or distribute fixed-MAC BLE tags per person. 3. Build plain HA automations: presence (RuView) on → light on at neutral default; off (with delay) → light off. **Validate this works with the LLM host powered off — this is your safety-net baseline.** ### Phase 3 — LLM host + conversation agent 1. Stand up the GPU host (or CPU fallback) with Ollama, pull Qwen2.5-14B-Instruct (or 7B/3B for CPU). **Scripted**: `tools/setup-llm-host.sh` auto-detects the tier (`nvidia-smi` must both exist *and* succeed), installs Docker + the NVIDIA Container Toolkit, and runs Ollama as a pinned container rather than `curl | sh` into a root shell. It deliberately does **not** install the GPU driver — the most hardware/kernel-specific step on that box, and silently picking a version is how you get a machine that doesn't boot. 2. Add the Ollama integration in HA, point it at the LLM host over LAN. 3. Set up Wyoming faster-whisper + Piper, configure an Assist pipeline. 4. Test open-ended conversation and basic tool-calling before wiring into presence logic. 5. **Then power the LLM host off and confirm the house still works** — presence → light, the door panel, chore nudges (plain template instead of LLM-phrased), digest skipping rather than erroring. This is the testing-checklist guardrail below, and it's the entire reason this is a separate machine: it has to be able to be off. Anything that *breaks* rather than degrading is a bug in the consumer, not in the LLM host. 6. **Ollama has no authentication of any kind**, and its API can pull and delete models, not only generate — so the network is the whole boundary. Smart-home VLAN, never port-forwarded; it's in `docs/network-integration.md`'s port table for that reason. ### Phase 4 — AI-managed brightness/color 1. Script triggered after the presence-automation light-on event, calling `ai_task.generate_data` with room/time/occupancy context, requesting structured JSON (`brightness`, `rgb_color`). 2. Apply via `light.turn_on`; fallback to neutral default on timeout/error. 3. Feed illuminance sensor data into the LLM's exposed-entity list. ### Phase 5 — Peephole camera + Frigate 1. Mount your color camera (your own work), isolate on the camera VLAN. 2. Deploy Frigate, add the RTSP stream, enable native face recognition. 3. Enroll known faces; automation for known vs. unknown at the door. 4. Set conservative confidence thresholds given the narrow FOV/low light. ### Phase 6 — Identity registration (multi-device, anti-spoofing, voice) New top-level `identity/` directory (container-host service + frontend). Backs `hosts/door-panel/`'s and `hosts/kitchen-display/`'s registration and presence surfaces (Phases 17/18). **Built directly as a Python service, not a Node-RED flow first** — a deliberate deviation from this phase's original sketch (passive Frigate-face + Bermuda-BLE co-occurrence tallying in Node-RED). Once "usable via voice, defends against MAC spoofing, and handles multiple phones per person" became the actual bar, the shape that falls out of it — real request/response semantics for a multi-step registration attempt, a relational multi-identifier-per-person data model, unit-testable candidate-resolution logic — fits a small Python HTTP service (the same `admin-canvas`/`pantry-vision` shape) far better than a visual flow tool. Passive co-occurrence tallying itself was **dropped, not deferred**: registration is now always an explicit, human-initiated act (a spoken command or a touchscreen tap), not something inferred from ambient signal over time — a stronger, simpler invariant than the original confidence-threshold design, and the reason `propose_person_link`/ `confirm_person`/`rename_person` from the original sketch don't appear in the shipped API at all. 1. **The model**: a person has zero or more identifiers, each a Home Assistant `entity_id`. Multi-phone support isn't a special case — register twice under the same spoken name (private phone, then work phone) and the second identifier just joins the same person record. See `identity/README.md`'s "The model". 2. **Anti-spoofing is an allowlist, not a filter**: a raw Bluetooth MAC — especially a randomized one, the iOS/Android default — is never accepted as a candidate identifier at all. Only `entity_id`s matching `TRUSTED_ENTITY_PREFIXES` are eligible, meant to contain exclusively HA's Private BLE Device (IRK-resolved) entities and manually provisioned fixed-MAC BLE tag entities (§1.5). This is a defense against passive/opportunistic spoofing, explicitly **not** a claim of cryptographic non-repudiation — see `identity/README.md`'s full threat-model section for the honest boundary (a compromised phone/IRK is out of scope). 3. **Voice is single-utterance, not multi-turn**: "register me as ``," one sentence, no follow-up question. HA Assist's multi-turn/continue-conversation support is newer and more version-sensitive than a single custom-sentence intent with a captured slot, and this is a materially more robust thing to build against. **Nothing under this repo builds the HA-side custom-sentence/intent- script/`rest_command` wiring** — same convention as every other HA integration point in this project (digest-engine, admin-canvas) — `identity/README.md` has a worked, unverified-against-a-real-instance example. 4. **Never auto-commit on ambiguity**: zero candidates, more than one, or an already-claimed one — nothing gets written, and a human disambiguates on the touchscreen. The one case that *does* commit within a single call is the clean one (exactly one trusted, unclaimed candidate), because the spoken command itself is the human confirmation — this mirrors, not weakens, the original phase's "never auto-commit a merge silently" rule. 5. **People with no device at all are a first-class case, not an edge case** — a household member without a smartphone (the concrete example that drove this: a grandmother). `POST /register` with `no_device: true` creates a person with zero identifiers; `POST /presence/manual` gives them a hand-operated Home/Away toggle on the door panel's dashboard, since there's nothing to resolve automatically. Reporting them as "away" by default (rather than "unknown") would be actively wrong the moment they're actually home, not just uninformative — see `identity/server.py`'s `presence()` docstring. 6. **A separate "doesn't need to know who it is" path**: `POST /register/guest`, no name, no device, always a new "Guest N" record (never deduped the way named people are). `DELETE /people/` cleans up a stale one afterwards. 7. **Every person gets a profile picture, automatically** — whichever registration photo was captured most recently (`identity/README.md`'s "Every person gets a profile picture" section), fetched via a bearer-token-gated endpoint + blob URL, not a bare ``. The photo is **never run through face-matching** — it's an audit/reference artifact only; BLE/IRK resolution is what actually decides who registered. Camera-based identity, if ever wanted, is a Frigate face-recognition integration (Phase 5), not a new pipeline here. 8. **Floor-plan groundwork, not the floor plan**: `/presence` reports a best-effort `room` per person, read from whichever area/room attribute your BLE presence integration (Bermuda) attaches to a trusted entity's state (`AREA_ATTRIBUTE`, unverified default). The actual floor-plan UI — an image, a room↔coordinate mapping, any rendering — is deliberately **not** built: there's no floor plan or fixed room list to design a format against yet, and building one now would be guessing, not engineering. This is groundwork specifically so that future UI doesn't require `identity`'s data model to change again. 9. Nothing here has been run against a real HA instance, real Private BLE Device entities, or a real voice pipeline — see the itemized list in `identity/README.md`, `TRUSTED_ENTITY_PREFIXES`' defaults above all. ### Phase 6b — Household admin panel (people, history, rights) Extends Phase 6's `identity` rather than adding a service: all of this is facts about who someone is, which this project already made `identity`'s job. 1. **`identity/frontend/admin.html` — the one page here that isn't kiosk-shaped.** `register.html`/`dashboard.html` are wall panels read from across a room; the admin page is dense, has destructive actions, and is meant for a phone or laptop. A wall panel anyone can walk up to must not carry a "prune these six people" button, which is why it's a separate URL and nothing in `hosts/door-panel/` links to it. 2. **Nicknames are an input alias, never an output one.** A person can have a `nickname` that `/resolve` accepts ("is Bibi home?"), but every payload also carries `speak_name`, always the real name, and voice/TTS consumers read that. The asymmetry *is* the feature: a nickname is something people grant each other, not something a machine should presume back. A nickname colliding with anyone else's name or nickname is refused; a spoken string that somehow matches two people refuses with `ambiguous_name` rather than picking, extending Phase 6's own never-auto-commit rule to a second kind of ambiguity. 3. **Visit history is sampled by `identity` itself**, not pushed at it — a poller writes arrival/departure rows off its own `/presence`. `home: null` (unknown) writes **nothing**, ever: an HA outage must not record a departure that didn't happen. A departure needs `DEPARTURE_GRACE_SECONDS` of sustained absence (BLE flaps) and is recorded as of the last moment the person was actually seen. 4. **"Who was home with whom" is a query, not a table** — overlapping visit intervals, computed on read. No second copy of the same truth to drift; O(visits²) in the window, fine for a household, stated plainly rather than hidden. 5. **Pruning splits selecting from deleting.** `GET /prune/candidates` fills the checkboxes; `POST /people/prune` deletes the explicit ids that came back. The filter is never re-run at delete time, so nobody who walks in the door mid-review gets swept up by a filter that quietly re-evaluated. 6. **Per-device rights are an answer, never an action** (the "let my cousin unlock the front door herself" case). `identity` stores grants and answers `GET /device-access`; **HA asks and HA acts**, same "HA mediates, nothing auto-acts" rule as every other control path in this plan. Deny is the default and the only fallback — the one place in this service that fails closed rather than degrading, because a lock has no useful "unknown". Grants can expire; every check is logged, allowed and denied alike. 7. **Chore assignment is a preference, not a lock** — an assignee who's home gets nudged first, an assignee who's away doesn't block the chore ("I don't care who does it, as long as it gets done"), and `CHORE_ASSIGNMENT_STRICT` flips that for households that meant the stronger thing. **Litter still can't be assigned**, for the same reason it ignores exemptions. 8. Admin-added identifiers **stay behind the anti-spoofing boundary** — `TRUSTED_ENTITY_PREFIXES` is still enforced. What the admin path relaxes is only "must be in range right now" (so a fixed BLE tag can be provisioned before it's near the door), never the allowlist itself. An admin panel is not a reason to hand out an exception to the one security property this service has. 9. **Arrival notifications ride the same transition as the visit log** — opt-in per person (`notify_on_arrival`), with a separate per-person opt-*out* of being announced (`announce_arrivals`, default on) for anyone who doesn't want their comings and goings broadcast, the same concern open decision #32 raises about RuView. Delivered via the self-hosted ntfy this stack already runs for `chores` — **`identity` itself never touches the WAN**. The first sample after startup notifies nobody, establishing a baseline instead of firing a burst of false arrivals for everyone already home. 10. **Whether the push reaches a phone that's away from home is a network decision, not a feature of this service** — `docs/network-integration.md`'s existing answer (WireGuard in, never a port-forward) applies unchanged. Android + WireGuard keeps this fully local; **iOS can't**, since ntfy's iOS app needs APNs and therefore an upstream relay through a third party. That's an Apple constraint, but it makes "everyone gets arrival notifications" a household-platform question worth deciding consciously — see `identity/README.md`'s table. 11. Covered by API-level tests, not browser ones — see `identity/README.md`'s verification list, `DEPARTURE_GRACE_SECONDS`' untuned default above all. ### Phase 7 — Kitchen inventory kiosk 1. Deploy Grocy via Compose. 2. Pi + touchscreen running Chromium kiosk mode pointed at Grocy's PWA. 3. Attach USB webcam, test in-browser barcode scanning (Open Food Facts lookup). 4. Use Grocy's own add/consume UI — no extra hardware/logic needed for the "in vs. out" problem. ### Phase 8 — Nextcloud calendar sync Written up in full in **`docs/caldav-integration.md`** — the four independent clients and their directions, the shared-app-password decision and what it costs, and the recurrence/TLS traps. Nextcloud itself is pre-existing; nothing in this repo deploys it. 1. Add HA's CalDAV integration, point at Nextcloud's CalDAV URL. 2. Confirm read + write (`calendar.create_event`) both work. 3. Expose the calendar entity to the LLM's tool list. 4. Gate delete/move actions behind a confirmation step; create recurring events directly in Nextcloud, not via HA. 5. **One Nextcloud app password, shared by all four clients** (HA plus this repo's three), never the account password — revocable on its own, and mandatory anyway once 2FA is on, since DAV endpoints can't prompt for a second factor. Note the two consequences: rotating it means editing three env files *and* HA, and a stale credential fails **quietly** because every client here degrades rather than crashes. 6. **The read-only invariant on `digest-engine`/`chores` is a code property, not a permission boundary** — a Nextcloud app password cannot be scoped read-only or to a single calendar, so the credential those services hold could delete every event you own. What prevents it is that those modules only ever issue reads. Review changes to them with that in mind; the server will not catch a regression. ### Phase 9 — Observability, dashboard, backup 1. Deploy Netdata (or your team's eventual choice from the Netdata/Checkmk evaluation) for container/host monitoring. 2. Deploy Homepage/Homarr as a single landing page across all service UIs. 3. Deploy ntfy for local push notifications (door alerts, automation failures). 4. Set up restic scheduled backups of all stateful volumes (HA config, Zigbee2MQTT device DB, Grocy data, Frigate face embeddings, Node-RED flows). ### Phase 10 — Optional expansion 1. Mealie for meal-plan-aware shopping lists. 2. InfluxDB + Grafana if you want longer-term historical dashboards than HA's recorder retains. 3. Additional RuView/Bermuda nodes as more rooms are covered. 4. Dedicated Frigate accelerator (Hailo-8L) only if you expand beyond 1–2 cameras. ### Phase 11 — Sway thin-client ISO 1. Scaffold a `live-build` tree at `hosts/thin-client/live-build/` (Debian 12, matching container-host's OS). `config/package-lists/thin-client.list.chroot` pulls `sway`, `greetd`, `wayvnc`, `xwayland`, `firefox-esr`, `mpv`, `mpv-mpris`, `spotifyd` (or `librespot`), `flatpak` (Steam Link), `wyoming-satellite` + `openwakeword` deps, plus `pipewire`/`wireplumber`. `tools/build-thin-client-iso.sh` drives `lb config && lb build`. 2. Autologin straight into a kiosk Sway session via **greetd** (`initial_session` block runs `sway` directly, no greeter UI) — not the older getty+`.bash_profile` hack. 3. Remote control: **wayvnc** for interactive screen view/control. **Sway/wlroots has no maintained RDP path** (wlroots dropped its RDP backend; xrdp is X11-only) — wayvnc is the deliberate, confirmed replacement for "RDP" in this project, not a stopgap. 4. Build `thinclient-agent` (Python, `hosts/thin-client/agent/`) as a systemd service baked into the image: - Connects to Mosquitto, does HA MQTT-discovery: a `media_player` entity (driven by mpv's MPRIS2 D-Bus state via `mpv-mpris`, bridged in-process), plus `button`/`select` entities for launching apps, switching Sway workspaces, and opening/expanding the digest canvas. - On MQTT command, shells out to `swaymsg` (`$SWAYSOCK`) and manages app processes (Firefox, Steam Link, mpv). - **Security principle**: the LLM never gets a raw network path to the thin client. Every control path is LLM tool call → HA service call → MQTT → `thinclient-agent`, mirroring the Phase 6/8 "HA mediates, nothing auto-acts" precedent. 5. Dedicate one Sway workspace to a kiosk Firefox window pointed at `digest-engine`'s local HTTP endpoint (Phase 12) — this doubles as the LLM's rendering surface for "free windows/graphics." 6. Spotify via `spotifyd`/`librespot` (Connect receiver, no GUI login, Premium required, unofficial protocol — minor ongoing-maintenance risk, not a blocker). 7. Steam Link via Flathub Flatpak, run under **Xwayland** (documented workaround for native-Wayland black-screen/flicker bugs on wlroots). 8. **Voice interactivity**: - Only the rooms with a chosen thin client *and* an attached mic run `wyoming-satellite` (openWakeWord), streaming to the existing Phase 3 Wyoming faster-whisper/Piper pipeline — no new STT/TTS infrastructure. - `thinclient-agent` accepts an Assist-resolved "play my digest" intent: switches the dedicated Firefox workspace into the "full/thorough" canvas view (vs. the HA dashboard's "compact" view) and narrates via the existing Piper TTS output. - **Room/person routing — voice-activated, automatically recognized, never asked**: the digest canvas is shown only when somebody asks for it (a spoken "play my digest", or the HA button). **Nothing displays it because a person walked past a screen**, and nothing here polls presence to decide to show something. When the wake word fires, HA calls `identity`'s `GET /speaker?area=`, which resolves who is asking from the two signals it already fuses: an IRK-resolved BLE identifier in that area, and a Frigate face sighting inside `FACE_PRESENCE_WINDOW_SECONDS`. One person in the room is them; several, and the most recent camera sighting decides. That answer becomes the `person` parameter the thin client passes to the canvas, which then shows exactly that person's chosen sections (Phase 12 step 5). **An unresolved answer means show less, not ask**: the canvas renders every section except the personal one, which is the same rule as before — automating the recognition is only acceptable because the unresolved case still fails closed. This replaces the earlier "Assist asks *whose digest?*" disambiguation: the household wanted recognition, not an interrogation. There is no speaker identification in this stack and `/speaker` does not pretend otherwise — it identifies who is in the room, not whose voice it was. 9. Network placement: plain trusted LAN for now (no VLAN precedent exists for a general client device class yet — only the unbuilt camera-VLAN concept). Revisitable later as a Phase-10-style expansion item, not a blocker now. 10. **Validate the image boots to a working kiosk session (Sway, local mpv/Spotify playback) with Mosquitto/HA/container-host powered off** — must not hang waiting on the network, same "reactive path never depends on a remote service" philosophy applied to the thin client's own boot path. ### Phase 12 — Quarter-daily LLM digest 1. Add `digest-engine` to `hosts/container-host`'s compose stack via the existing `ENABLE_X`/`X_BLOCK` pattern (`ENABLE_DIGEST_ENGINE="false"`, off by default until credentials are provisioned). Its build context points at the new top-level `digest-engine/` directory — **the first locally-built image in the stack** (everything else pulls prebuilt registry images). 2. Add a companion `digest-web` static-file service (nginx:alpine/Caddy) in the same block, serving the shared output volume read-only to **both** display surfaces: the thin client's Firefox workspace (compact-vs-full toggle) and an HA Lovelace HTML/iframe card. 3. Schedule via a **systemd timer**, mirroring the existing restic-backup convention: `smart-home-digest.service` (oneshot, `docker compose run --rm digest-engine`) + `smart-home-digest.timer` (`OnCalendar=*-*-* 00,06,12,18:00:00`, adjust once Ollama contention is settled — see open decisions). 4. Ingestion modules, each independently toggleable, each reading credentials from a not-committed `.env`: - **Email** — IMAP via `imapclient`; Gmail needs an App Password (2FA-gated) or OAuth2/XOAUTH2 — App Password recommended for this personal-use case. - **Signal** — `signal-cli` linked as a secondary device (JSON-RPC daemon mode); lowest risk of the four message platforms. - **Telegram** — Telethon (MTProto, logs in as the real account) since the Bot API can't read personal DMs; this is a userbot, ToS-grey but lower enforcement risk than WhatsApp/Discord-selfbot. - **Discord** — `discord.py` bot with Message Content intent, scoped to **servers you own/admin only**; cannot read personal DMs or others' servers without a selfbot (not built — real ban risk, explicit ToS violation). - **WhatsApp** — no officially-sanctioned API option exists. Rather than a protocol-reimplementation library (Baileys), run a small **`whatsapp-bridge`** sidecar (Node.js, `digest-engine/whatsapp-bridge/`): a real Chromium logged into the actual web.whatsapp.com client via **whatsapp-web.js** (Puppeteer), inside its own container running **Xvfb** so Chromium executes **headful** (not `headless: true`) — WhatsApp's automation detection specifically fingerprints headless Chrome, so a virtual-display "real browser" session is meaningfully lower-risk than either Baileys or true-headless whatsapp-web.js, though not zero-risk (it's still automated use of a personal account). One-time interactive QR-code login persists a session directory (mounted volume) so subsequent runs don't need re-scanning. The bridge exposes incoming messages over a local-only channel (e.g. a Unix socket or a small internal HTTP endpoint on the compose network, never published to the LAN) that `digest-engine/ingest/whatsapp_ingest.py` reads each run. Still gate behind `ENABLE_WHATSAPP_INGEST="false"`, off by default, with a warning in script output + `digest-engine/README.md`; recommend a secondary/non-critical number if enabled. Build this one last. - **News** — `feedparser` over a curated OPML list (`digest-engine/feeds/curated-feeds.opml`), seeded with the confirmed `https://www.marxist.com/feed/rss` plus a mainstream-outlet list (exact outlets: see open decisions). - **Financial** — FRED API (macro/unemployment, e.g. `UNRATE`) + Stooq keyless CSV (stocks/oil/commodities, preferred over Alpha Vantage's tight free-tier cap). 5. LLM synthesis: assemble the run's ingested content into context, call the existing Phase 3 Ollama host with one prompt template per section (`digest-engine/synth/prompts/{personal,political,household,network}.md`): - **Personal** — from personal-flagged mail/messages. - **Political** — Marxist/working-class analytical framing (marxist.com feed as theoretical basis) synthesizing mainstream news + financial indicators + politically-flagged mail, laid out on the "holo globe" with colored/glowing markers (e.g. revolutionary-situation markers in red with a hammer-and-sickle/star motif). - **Household/calendar** — from the existing Nextcloud CalDAV integration (Phase 8) and Grocy state (Phase 7). - **Network** — the OPNsense/Suricata intrusion-detection summary, split out of the household section (implementation note, later than the original plan) so the two can be wanted separately. - **The archive** (implementation note): `digest-engine/archive.py` keeps every ingested item and every measured number in SQLite across runs, so a run can say "up from 5.1% in June", "this IDS signature has fired every night this week", or "this story first appeared on Monday". Items are deduplicated on a fingerprint so `first_seen_at` means something; numbers are stored one row per measurement so a trend is a query. History enters each prompt as its own timestamped block, and it is what lifts the "one snapshot is not a trend" prohibition — but only with the figures and dates attached. It also persists mail and messages for its retention window, which `DIGEST_ARCHIVE_EXCLUDE_SOURCES` exists to bound. - **Meeting agendas** (implementation note): `digest-engine/agenda.py` matches a Tagesordnung PDF arriving by mail or WhatsApp to the calendar event it belongs to (date in the filename/subject/heading, then wording in common, each labelled with its confidence), reads it with pypdf, and extracts its numbered points mechanically. The household section lists the points and derives a todo window from what the document actually asks for; the political section gets the points as a relevance filter — a story touching Thursday's agenda outranks a bigger one that doesn't. A scanned agenda extracts nothing and is named but never characterised; there is no OCR here. - **Per-person section toggles** (implementation note): each person picks their own sections in `identity`'s admin panel (`people.digest_sections`, `GET /digest-preferences`). A run generates the **union** of what the household asked for — a section nobody wants costs neither an LLM call nor its sources' ingestion — and each surface filters to the person Home Assistant resolved. That last half is a display filter, not an access control: `digest-web` serves the whole artifact read-only to the LAN. An unreachable `identity` means "generate everything", never "generate nothing". - A **detail-level** parameter (`compact` for the HA iframe, `full` for the thin-client fullscreen view) makes the thin-client rendering genuinely more thorough without needing two independent generation passes. 6. Rendering: vendor the offline **digest-canvas SDK** under `digest-engine/render/digest-canvas-sdk/` (globe + `addMarker()`, window/panel chrome, glow/holo CSS utility, no CDN dependency). Each run's LLM job is to call into this SDK with structured content, not hand-roll projection math. Use a custom inline SVG or Unicode ☭ (U+262D, explicit font-fallback + CSS glow) for hammer-and-sickle iconography since Nerd Fonts has no such glyph. 7. **Live follow-up voice Q&A**: persist each run's actually-used ingested-context bundle (not full raw content) as `digest-engine/output//context.json`. Expose a small HA tool (`digest_followup_query`) so a spoken follow-up ("tell me more about the unemployment numbers") feeds the cached context + question back into Ollama for a grounded, low-latency answer — no fresh ingestion pass. The answer can push a new small window/card onto the already-open thin-client canvas via a websocket, keeping the "flexible windows" idea alive live, not just at generation time. 8. **Security/scope principle**: `digest-engine` is the first component in this project with routine WAN egress (mail, message platforms, news, financial APIs). Run it as its own compose service, no inbound port exposure beyond `digest-web`'s read-only LAN serving; credentials in a git-ignored `.env`. **Everything here is read-only summarization — it must never perform a write action anywhere** (no auto-reply, no mail archive/delete beyond what IMAP fetch requires, no CalDAV/Grocy writes, no message-platform writes), extending the Phase 6/8 "no silent mutation" precedent to its logical extreme: no mutation path exists at all. ### Phase 13 — On-demand admin canvas (sys-admin-llm display surface) No new hardware — this reuses the Phase 11 thin client and Phase 12's container host. 1. New top-level `admin-canvas/` directory (`admin-canvas/README.md`): a small stdlib-only Python HTTP service (`server.py`), the write-side counterpart to `digest-engine` — except long-running (`restart: unless-stopped`) rather than a oneshot, since content here arrives whenever Home Assistant's tool-calling LLM (the household's "sys-admin-llm", in its admin/ops-facing role — distinct from digest-engine's own synthesis LLM) decides to push something, not on a schedule. Two bearer-token-gated endpoints: `POST /show` (JSON `{"windows": [...]}`, overwrites `output/latest.json`, no run history — this is "what's on screen right now", not a scheduled artifact) and `POST /media/` (raw image/video bytes, filename allowlist-validated by both pattern and extension before being written under `output/media/`). 2. **No published port on `admin-canvas` itself.** It is reachable only from other containers on the container host's compose network — i.e. Home Assistant — the same trust boundary `mosquitto`/`homeassistant` already share. A companion `admin-web` (nginx:alpine, LAN-published, read-only, wired into `setup-container-host.sh` behind `ENABLE_ADMIN_CANVAS`) serves the shared `output/` volume plus the vendored SDK, exactly mirroring `digest-web`. 3. Rendering: `admin-canvas/render/canvas-sdk/` is a **duplicated**, not shared, copy of the digest-canvas SDK's window chrome and glow theme (renamed `digest-` → `admin-` throughout) — a deliberate choice to keep the two canvases fully decoupled rather than extracting a shared library out of a working, already-documented Phase 12 component. The globe kind is dropped (nothing here is a lat/lon marker); four kinds are added instead: `stat` (a big number/label/unit/trend), `image` and `video` (same "no scheme, no leading `/`, no `..`" src validation as the thin-client's own MQTT-payload invariant below, just relocated to this write API's trust boundary), and `chart` (dependency-free inline SVG — bar or sparkline, no charting library, consistent with this project's existing no-unnecessary-deps calls). Same degrade-instead-of-throw philosophy as the digest's renderer: a malformed window falls back to a `
` dump, never a blank page.
4. Thin client: a fourth Sway workspace `4:admin` (`hosts/thin-client/configs/sway/config`), a new `thinclient_agent/admin_canvas.py` module (sibling of `digest_canvas.py`, but simpler — it never inspects its own MQTT payload at all, since there is nothing content-specific for it to decide), and a single new HA button entity **"Show admin canvas"** (`mqtt_discovery.py`'s `register_admin_canvas`). Unlike the digest workspace, `4:admin` is **not** auto-launched at session start — this surface is on-demand by nature, so it starts empty until the first "Show admin canvas" command. Its own Firefox profile and launcher script (`admin-browser`), on the same kill-and-relaunch approach as `digest-browser`, with both scripts' pkill/pgrep patterns scoped to their own `--profile` path specifically so the two can never kill or race against each other.
5. **Security principle, unchanged from Phase 11.4**: this does not add a network path to the thin client. "Show admin canvas" only ever switches workspace and opens a fixed, locally-configured URL (`ADMIN_WEB_URL/canvas.html`) — identical in shape to "Show digest canvas". All actual *content* takes a completely separate path that never touches the thin client's MQTT surface: sys-admin-llm → HA tool call → HA `rest_command` → `admin-canvas`'s write API → `admin-web` → the browser's own poll (every 15s, shorter than the digest's 5 minutes since this is meant to feel closer to live).
6. **Nothing under this repo builds the HA side** — same convention as every other HA integration point in this project (Node-RED flows, CalDAV/Grocy wiring, the Lovelace card mentioned in `hosts/thin-client/README.md`). `admin-canvas/README.md` documents the expected `rest_command:` shape and worked example JSON for each window kind, but the actual HA config, the specific tool/intent definition, and which entities/history it reads to answer something like "show me the kitchen outlet's power draw" are the household's own to build.

### Phase 14 — ESP32-S3 voice/display satellite

New hardware: §1.12, **V2 revision specifically** — V1 has no echo-cancellation circuit and different audio pins, and this phase's firmware will not work on it.

1. New top-level `firmware/esp32-s3-touch-lcd-1.85c/` directory — the first ESPHome firmware this repo actually ships (`firmware/ruview/` and `firmware/esphome-ble-proxy/` remain unbuilt placeholders from Phase 2). `voice-display.yaml`'s display/touch/audio hardware bring-up (the ST77916 QSPI init sequence, CST816 touch, PCA9554 reset-pin wiring, I2S mic/speaker pins) is adapted from a community-verified config for this exact board rather than re-derived, since a wrong register sequence just shows static; see the file's own header comment and `firmware/esp32-s3-touch-lcd-1.85c/README.md` for the source and the cross-checked V2-specific quirks (EXIO2 reset, 80MHz data rate).
2. **Wake word stays on-device**, consistent with Phase 11.8's principle: ESPHome's `micro_wake_word` component (`okay_nabu` model — the same phrase as the thin client's `VOICE_WAKE_WORD`, so the household has one wake phrase regardless of which kind of satellite answers) runs TensorFlow Lite wake-word detection on the ESP32-S3 itself and explicitly starts a `voice_assistant` session on detection, rather than streaming continuously to Home Assistant for server-side spotting. Everything after the wake word streams into the **existing** Phase 3 Wyoming faster-whisper/Piper Assist pipeline — no new STT/TTS infrastructure, same as Phase 11.8's `wyoming-satellite` rooms.
3. **Screen priority, in order**: (1) a media/cover-art page, shown the instant the configured `media_player` entity's state becomes `playing` — cover art fetched via ESPHome's `online_image` platform (`entity_picture` resolved against a configured `ha_base_url`, since the device fetches it directly over HTTP rather than through the HA connection); (2) failing that, an idle page cycling every 8 seconds between a clock and the weather; (3) a voice-state visualizer (a colour-coded ring, listening/thinking/replying/error) as an LVGL `top_layer` overlay, shown regardless of which of the above is underneath — "a visualizer when speaking" is an overlay concern, not a fourth competing page.
4. **Both HA entities this device mirrors are placeholders** (`media_player_entity_id`, `weather_entity_id` in `secrets.yaml`, see `secrets.yaml.example`) — no real data source is picked yet, same "don't build against a guess" rule as Phase 12's calendar/Grocy sourcing and Phase 13's power-monitoring entity.
5. **This is a per-room device — media status must be that specific room's, never any other room's.** A `room` substitution (used in the device's hostname/AP name/friendly name) and a per-unit `secrets.yaml` (never shared across units) are both required, mirroring the thin client's per-image `THINCLIENT_NAME` convention. For rooms with more than one real audio source (thin client + a Spotify Connect speaker + a cast device, say), `media_player_entity_id` should point at a Home Assistant [Universal Media Player](https://www.home-assistant.io/integrations/universal/) that aggregates that room's real entities, rather than any single hardcoded device — see `firmware/esp32-s3-touch-lcd-1.85c/README.md`'s "Multiple rooms" section. This firmware has no way to detect a misconfigured `media_player_entity_id` pointed at the wrong room; it is a configuration invariant, not something the code can verify at runtime.
6. **Nothing under this repo builds the HA side** here either — the device is a standard ESPHome device once flashed and adopted (Settings → Devices → Add Device → ESPHome), and plugs into whichever Assist pipeline Phase 3 already has configured. No new add-on, no new container.
7. Validated so far with `esphome config voice-display.yaml` (ESPHome's own schema validator — passes cleanly), **not flashed to real hardware**. See the itemized unverified list in `firmware/esp32-s3-touch-lcd-1.85c/README.md`.

### Phase 15 — Headless audio endpoint (rooms without a thin client)

New hardware: §1.13. Two separate build pipelines, arm64 and amd64 — see below for why.

1. New top-level `hosts/audio-endpoint/` directory. **Per-room independent Spotify Connect, not a synced whole-house stream**: each endpoint is its own Spotify Connect device with its own name, exactly mirroring the thin client's own per-room spotifyd/librespot receiver (Phase 11.6). No Snapcast, no shared source, no container-host changes — a self-contained appliance per room, by explicit choice over the alternative (a single synced multi-room source) considered and rejected during planning.
2. **Two build pipelines, not one image with a flag**: the HiFiBerry Amp2 (arm64's assumed audio hardware) is a Raspberry Pi GPIO HAT with no amd64 equivalent — an x86 mini PC instead uses a USB DAC/amp (e.g. Fosi Audio V3, USB Audio Class, no driver needed). arm64 uses **rpi-image-gen** (the Raspberry Pi Foundation's current officially-recommended tool for custom images, superseding pi-gen for this use case) to produce a real flashable `.img`. amd64 reuses **the thin client's own live-build toolchain** rather than a third, unfamiliar one — a new, much smaller live-build tree with the entire graphical/kiosk stack stripped out, headless boot straight to `multi-user.target`.
3. **Spotify Connect install is apt-first with a documented fallback, reused identically on both variants** from `hosts/thin-client/live-build/config/hooks/normal/0500-spotify-connect.hook.chroot`'s own already-established logic (neither `spotifyd` nor `librespot` is in Debian bookworm main) — not two independent judgment calls about the same package.
4. **Audio output is direct ALSA, not PipeWire**, on both variants — deliberately different from the thin client, since this is a single-purpose headless appliance with no multi-app mixing need and no desktop session to run a per-user PipeWire daemon in the first place.
5. ~~Per-room identity differs by variant~~ — **converged** (`tools/`): **both** variants now bake the hostname in at build time from `CoreSystemConfig.json`'s `audio_endpoints` array, one image per room, so no device in the household gets its identity typed in after the build. arm64 was the last exception (a generic `.img` named in Raspberry Pi Imager's OS Customisation dialog at flash time), and it was worth converging because it was the only remaining hand-entry point *and* it failed silently when mistyped — `spotify-connect-start` reads `$(hostname)`, so a typo yields a Spotify Connect device under the wrong room name rather than an error. **The tradeoff is explicit**: one full `rpi-image-gen` run per arm64 room (qemu-emulated on an x86 builder) instead of one for all; `build.arm64_prebake: false` restores the single-generic-image behaviour. Wi-Fi and SSH keys are still set in Imager on arm64 — only the hostname moved into the build. Both converge on the same mechanism at the software layer: `spotify-connect-start` reads `$(hostname)` fresh at every service start, regardless of which point in the pipeline set it.
6. **rpi-image-gen's exact config/layer YAML schema is the single highest-risk unverified part of this phase** — written from the tool's documented structure, not hands-on validated (no ARM build environment available). See `hosts/audio-endpoint/README.md`'s prominent callout before a real build.
7. Neither image gives Home Assistant any control or visibility over these endpoints — no MQTT, no HA entities. Deliberate scope decision matching the "per-room independent" choice, not an oversight; revisit as a separate addition if remote control/monitoring from HA is wanted later.
8. Nothing built or flashed on real hardware — no Raspberry Pi, no HiFiBerry Amp2, no x86 test box available. See the itemized unverified list in `hosts/audio-endpoint/README.md`.

### Phase 16 — Sway touch panel

New hardware: §1.14. No container-host changes.

1. New top-level `hosts/touch-panel/` directory — **a different device, not a variant
   of the Phase 11 thin client**: that image is a couch-distance media station
   deliberately built around *remote* control (HA/MQTT, wayvnc) with no on-screen
   bars; this one is touched directly, so it inverts that — an always-on touch dock,
   native Wayland touch input, no gesture-camera/remote-view machinery. It reuses the
   thin client's live-build toolchain and `configs/`+`agent/` split, not its
   live-build tree, exactly the relationship `hosts/audio-endpoint/`'s amd64 build
   already has to the thin client's.
2. **Three fixed workspaces, no on-demand ones**: `1:spotify` (the real, official
   Spotify Linux GUI client via Flathub `com.spotify.Client` — not the thin client's
   headless `spotifyd`/`librespot` Connect receiver, since the whole point of a touch
   panel is a screen you interact with directly), `2:home` (a persistent Chromium
   `--kiosk --app=$HA_URL` window, auto-restarted on crash by its own supervising
   loop rather than by sway or systemd), and `3:web` (Firefox, minimal chrome —
   general browsing, not kiosk-locked, mirroring the thin client's own `web_browser`
   app in shape but on a separate, duplicated profile).
3. **Touch navigation is a persistent on-screen dock (`eww`, layer-shell
   `:exclusive true`), not a keyboard/mouse-driven `bar {}`**: reserving a strip at
   the bottom means the dock can never be covered by, or lose screen area behind,
   whatever app is on screen — the opposite trade-off from the thin client's
   deliberately bar-less, overlay-only widgets. Same session-scoped-not-agent
   ownership rule as the thin client's now-playing widget: this is compositor UI, not
   device control, and dies with sway rather than opening a second inbound channel
   into the security-sensitive agent.
4. **`touchpanel-agent` mirrors `thinclient-agent`'s exact security boundary**: LLM
   tool call → HA service call → MQTT → this agent, never a direct network path. It
   publishes a **Screen** select (the touch dock's HA-reachable equivalent) and
   **Show Spotify** / **Show Home** / **Show web browser** buttons — the latter two
   focus the already-running app rather than relaunching it (both are meant to stay
   open and stateful), unlike the thin client's stateless kill-and-relaunch digest/
   admin kiosk pages. Also publishes a `media_player`-shaped entity set bridged from
   Spotify's own MPRIS interface, identical in shape to the thin client's MPRIS
   bridge, pointed at the `spotify` player name instead of `mpv`/`spotifyd`.
5. **An on-screen keyboard (`wvkbd`) with no auto-show, by explicit choice**: proper
   show-on-text-field-focus needs the `text-input-v3`/`virtual-keyboard-v1` protocols
   wired through a shell component (the way Phosh does it for squeekboard), which this
   image doesn't run. Rather than build that integration now, the dock's Keyboard
   button is a manual toggle — a documented, accepted limitation, not a gap hidden
   from the README.
6. **No wayvnc on this image**, unlike the thin client — a deliberate scope decision,
   not an oversight: the primary interaction model here is a finger at the panel, not
   a remote viewer, so SSH alone covers the "something's broken, fix it from a shell"
   case. Revisit only if a real need for remote *view* shows up once this runs on real
   hardware; adding wayvnc back would be a small, isolated change (see
   `hosts/touch-panel/README.md`).
7. **Both apt-first/documented-fallback installs already established elsewhere in this
   repo are reused, not reinvented**: `eww` (same as the thin client's now-playing
   widget) and `wvkbd` — neither is in Debian bookworm main, and both hooks stop at a
   documented placeholder rather than a hardcoded release URL if apt doesn't have them.
8. **Touch input degrades instead of hard-requiring `wl_touch`**: `configs/sway/config`
   ships both `input type:touch` (full multi-touch, if the hardware reports
   correctly) and an `input type:pointer` fallback (flat accel profile, no pointer
   acceleration, cursor hidden on idle) for touchscreens that misreport as an
   emulated-mouse HID device — a real, documented failure mode for cheap touch
   controllers, not a hypothetical. `configs/udev/99-touchscreen-override.rules` is
   the actual fix once a specific unit's USB vendor/product ID is known: it re-tags
   the device so udev/libinput treat it as a genuine touchscreen instead of settling
   for the pointer fallback's single-touch ceiling. Ships as an inert template
   (`0000:0000`, matches nothing) until then.
9. Nothing built or flashed on real hardware — no touch-panel unit has been chosen
   (§1.14). Both the `wl_touch` path and the `type:pointer` fallback above are
   unverified against real touch hardware either way — see the itemized list in
   `hosts/touch-panel/README.md`.

### Phase 17 — Kitchen/fridge display (camera-vision grocery cataloguing)

New hardware: §1.15. New top-level `pantry-vision/` directory (container-host
service) plus `hosts/kitchen-display/` (a third, simpler kiosk image).

1. **The workflow this phase is built around, stated once because everything below
   serves it**: come home, put down the shopping bag, hold one item up to the
   kitchen display's camera, the system proposes what it is and roughly how long it
   keeps, the person confirms (editing anything wrong) before anything is written
   anywhere, put the item away. The same display then shows inventory ordered by
   what expires soonest, and Grocy's recipes, on request.
2. **`pantry-vision` never writes to Grocy from `/identify` alone — only `/confirm`,
   a separate human-reviewed call, ever does.** This is the single guardrail the
   whole phase is designed around, the same "propose, never auto-commit" rule this
   project already applies to identity-merge confirmation (the *Identity store* row
   in §2). A wrong camera guess costs one tap to fix on the kiosk screen, not a wrong
   fact silently written into the household's inventory. The book-out direction obeys
   it too, and it costs a tap there as well: recognition never consumes anything by
   itself, an ambiguous brand is asked about instead of picked, and a throw-away
   states its amount first.
2a. **Stock only moves in four ways, so the display has four screens** — *Unload
   groceries* (the camera runs in a loop, one confirm per item, straight on to the
   next), *Consume article* (hold up what you are eating; multi-unit rows ask how
   many, multi-brand kinds ask which), *List expired foods* (cleared by scanning what
   you are binning, booked out as **spoiled** so Grocy keeps waste apart from eating),
   and *Edit inventory* (`−`/`+` and a freeform amount — the one screen that
   deliberately does not use the camera, because it is where you go when the camera
   got something wrong). Corrections use Grocy's inventory-correction endpoint, not
   consume/add, so the stock journal doesn't fill up with corrections dressed as
   meals.
2b. **Two invariants the whole feature rests on.** (i) *Stock is counted in individual
   units, never packages* — a twelve-pack of eggs is twelve, because that is the
   question people ask and it makes consuming three of them arithmetic instead of
   fractions of a pack; the model's `units_per_package` is a multiplier applied once
   on the confirm screen, where it is editable and where the result is stated before
   anything is written. (ii) *The fold key is the brand-free product kind* — twelve
   eggs of brand X plus ten of brand Y are twenty-two eggs, recorded as a Grocy
   **product group** so the grouping lives in Grocy's own data model rather than in a
   second classification scheme here, with the per-brand rows kept underneath because
   "take ten off brand Y" has to stay possible on the edit screen. The folding is
   deliberately dumb (casefold, strip the brand suffix); the clever version is a
   synonym problem nobody wants adjudicated by a kitchen display at 19:00, and its
   failure mode — two lines that should be one — is at least visible, which a wrongly
   merged line is not.
3. **A real, LAN-published network listener — the one deliberate exception to this
   project's usual "the LLM only ever reaches a device through HA→MQTT" shape.**
   `admin-canvas` (Phase 13) has no published port because only Home Assistant calls
   it; `pantry-vision` is different on purpose because the kitchen display is a
   separate physical device that has to get an identification back synchronously,
   with no HA round-trip in the loop for that specific call. The bearer token — not
   network placement — is the actual boundary here; see
   `pantry-vision/README.md`'s "A real network listener, unlike admin-canvas"
   section. Controlling *which screen is showing* is a separate, narrower path that
   does stay HA/MQTT-mediated: `kitchen-display-agent`, identical in shape to every
   other host's agent in this project. Its "Show scan" button still publishes the
   fragment `scan`, which `frontend/app.js` treats as an alias for the unload screen —
   an old name living on costs nothing next to an HA button that silently stops
   working.
4. **Grocy is the system of record, not a new inventory store** — `pantry-vision` is
   a client of Grocy's own REST API (already running unconditionally since Phase 1/7)
   for the reads and for every stock write (`/confirm`, `/consume`, `/adjust`,
   `/transfer`). Nothing here duplicates or replaces Grocy's own data model. The one
   thing `pantry-vision` keeps in a file of its own is the doorway hints below, and
   that is precisely because they are *not* inventory.
4a. **Multiple cold appliances, answered twice over — and only one of the answers is
   authoritative.** Grocy **locations** are the record: the confirm screen's placement
   resolves to one, `/inventory` reports it, and `POST /transfer` moves an amount
   between appliances. That transfer action is the whole point — without it a location
   decays into "where it was when it was bought", which is worse than no answer.
   Alongside it, **doorway hints** (`pantry-vision/doorway.py`): a Zigbee contact
   sensor on each appliance door drives an HA automation that POSTs `/doorway-event`,
   which answers 202 immediately and pulls a short camera burst off the request
   thread. Whatever the vision model recognises is written to its own SQLite file with
   a timestamp and a confidence, **never to stock** — a camera at a door cannot tell
   in from out, misses two-items-at-once, and sees nothing behind an arm, so it
   produces "last seen at Freezer (loggia), 20 min ago" for a person to evaluate.
   `docs/fridge-item-location.md` is the full argument, including why nothing goes
   *inside* an appliance (no interior power, condensation on every opening, −18 °C
   outside every consumer camera module's rating) and why the door sensors are worth
   buying even if no camera ever is. **An appliance configured with a sensor and no
   camera is a supported, recommended starting point.**
5. **The vision-identification prompt asks for one photo, not a live video stream**,
   and degrades to a low-confidence placeholder proposal (never an error page) if the
   model call fails or its response isn't parseable JSON — same "degrade, don't
   blank" rule as the digest/admin canvas renderers and `llm_client.py`'s own
   `_fallback_document`. The unload screen's "camera searching for products" is still
   built out of single photos: `frontend/app.js` samples a 32×24 greyscale thumbnail
   every 700 ms and spends an `/identify` call only when the frame has **settled** and
   **changed** since the last identification. Both gates are latency economics — see
   open decision #18 — and a degraded answer stops the loop rather than
   re-photographing the counter at a model that is down.
5a. **The model is asked to read a printed best-before date off the packaging**, and
   that is the one place the "propose, never auto-commit" rule is deliberately looser
   than a pure estimate would be. The screen always labels which of the two it is
   showing ("read off the packaging — check it" vs "estimated from the category"), and
   the server discards any date more than a year past or ten years out, because a
   misread label is this feature's most likely failure and that is its shape.
6. **The kitchen display has no native camera-capture app at all** — Chromium's own
   `getUserMedia()`, called from `pantry-vision/frontend/app.js`, talks to the webcam
   directly inside the kiosk page and hands a captured frame straight to
   `POST /identify` as a blob. `--use-fake-ui-for-media-stream` on the kiosk launch
   auto-accepts the permission prompt that would otherwise sit unanswered on an
   unattended screen.
7. **`hosts/kitchen-display/` is one workspace, one app — no touch dock, no
   multi-app switching** the way `hosts/touch-panel` has: Home/Inventory/Recipes is
   in-page tab navigation inside `pantry-vision/frontend/`'s single-page app, with the
   four stock-movement screens opening from Home, since there's only one thing this
   device does. `kitchen-display-agent`'s MQTT surface is correspondingly narrow:
   three "Show X" buttons and nothing else — no media_player, no
   capture/audio/remote-desktop entities.
8. **The Grocy API integration is written from documentation, not verified against a
   live instance** — the exact `GET /api/stock` response shape (whether product
   names arrive nested by default), the minimum required fields for
   `POST /api/objects/products`, and whether the Recipes/fulfillment endpoints need
   explicit setup before they return anything meaningful are all flagged in
   `pantry-vision/README.md` alongside the live OpenAPI spec URL every real Grocy
   instance exposes (`/api/openapi/specification`) as the way to actually check.
9. **The vision model choice is unpicked and unbenchmarked** (`OLLAMA_VISION_MODEL`
   defaults to `llava`, not confirmed pulled or even correct for a given LLM host's
   GPU/CPU tier) — same "don't build against a guess" flag as every other
   LLM-model/hardware pairing in this plan. Latency in particular is unmeasured and
   is the number that decides whether "hold it up, wait, confirm" feels usable at
   the counter or like standing there for 30 seconds.
10. Nothing built or run against real hardware, a real camera, a real vision model,
    or a real Grocy instance. See the itemized lists in `pantry-vision/README.md`
    and `hosts/kitchen-display/README.md`.

### Phase 18 — Door/wardrobe panel (identity + ambient dashboard)

New hardware: §1.16. New host `hosts/door-panel/` — no new backend directory; it's
entirely a client of Phase 6's `identity` (and, for one dashboard section, Phase
17's `pantry-vision`).

1. **Structurally `hosts/kitchen-display/`'s twin, not a new device shape**: one
   Sway workspace, one Chromium kiosk window, a thin agent that only ever switches
   which page is showing (`Show home` / `Show registration`, mirroring "Show scan/
   inventory/recipes"). The differences are what it shows by default
   (`identity`'s `dashboard.html` instead of `pantry-vision`'s scan/inventory/
   recipes) and that its microphone is the device's actual purpose, not an
   edge-case opt-in — `ENABLE_VOICE_SATELLITE` defaults to `true` in this host's own
   build script, the only host in this project where that's the case.
2. **The dashboard**: weather + a deterministic clothing suggestion (a plain
   temperature/condition lookup table, `identity/frontend/dashboard.js`'s
   `clothingSuggestion()` — not an LLM call; this is a solved-enough problem that a
   round trip to Ollama would only add latency and a failure mode for no real gain),
   who's home (`identity`'s `/presence`, three-state: home / away / unknown, plus a
   manual toggle for anyone with no device), and groceries running low
   (`pantry-vision`'s `/shopping-list`, a thin reshape of Grocy's own
   `/api/stock/volatile` `missing_products` — distinct from Phase 17's "soonest to
   expire" sort, this is "below minimum stock" instead).
3. **No camera is required on this device**, unlike `hosts/kitchen-display/` —
   identity resolution here is BLE/IRK-based, not camera-based; the registration
   photo `identity` captures is a nice-to-have audit artifact/profile picture, never
   load-bearing for who gets registered.
4. Nothing built or run against real hardware — no door-panel unit has been chosen,
   and the full voice-registration chain (wyoming-satellite → HA Assist → the
   custom intent script → `identity`'s `/register`) has the most untested moving
   pieces of any single interaction in this project. See the itemized list in
   `hosts/door-panel/README.md`.

### Phase 19 — Trash-day calendar sync + public transit (Kennelbach, AT)

New backend directories `trash-calendar/` and `transit/`. No new hardware, no new
host — both are container-host services with no kiosk-facing UI of their own;
`transit`'s voice path goes through HA Assist like every other voice interaction
in this plan.

1. **Trash-day sync (`trash-calendar/`)**: a scheduled (systemd timer, daily 06:15)
   oneshot that reads Kennelbach's personal collection-date ICS feed and writes
   matching "Trash: " events onto the shared household CalDAV calendar —
   read-only against the source feed, and write-only (create/update its own
   events, never touch anyone else's) against Nextcloud. `chores/check.py`'s own
   trash-day-eve check (Phase 20) reads this exact same feed independently, rather
   than depending on this service's calendar writes — see "duplicated, not
   shared" in `chores/README.md`.
2. **Public transit — departures + trip planning (`transit/`)**: `GET /departures`
   answers "when's the next bus/train from X" from a static GTFS feed (Vorarlberg
   VAO), refreshed weekly (systemd timer, Monday 04:00 — schedules are published a
   season at a time, not daily). `GET /plan` answers "get me from A to B" by
   proxying OpenTripPlanner's GraphQL API — **this repo does not build or manage
   the OTP graph itself**, that's a manual, one-time-per-OSM/GTFS-update step; see
   `transit/README.md`'s "Route planning scope" section for why "Austria, possibly
   global" is a real infrastructure sizing decision (Geofabrik extract size, OTP
   memory requirements), not a config flag. Both endpoints are voice-usable
   through HA Assist, same "custom sentence → intent script → this service's API"
   shape as `identity`'s voice registration.
3. **Slow walking speed assumed throughout** — `WALK_SPEED_MPS=0.9` (about half of
   OTP's ~1.4 m/s default), configurable, applied to every `/plan` call.
4. Nothing here has been run against a live GTFS feed or a real OTP instance —
   both `trash-calendar`'s ICS parsing and `transit`'s GTFS/OTP integration are
   written from documented formats/APIs, not a verified live source; see each
   service's own README for the exact assumptions flagged.

### Phase 20 — Tapo pan/tilt cameras (presence + trash-bin/litter checks) + household chore distribution

New hardware: §1.17. New backend directory `chores/`. No new host — this phase is
entirely container-host services plus additional Frigate camera sources.

1. **Tapo cameras feed into Frigate as additional camera sources** (§1.17), same
   integration point as the existing peephole cam (Phase 5) — this repo does not
   add a separate camera-control layer, Frigate (and, for pan/tilt aiming, its PTZ
   preset API) is the one interface everything downstream talks to.
2. **Presence integration is `identity`'s job, not a new one** — Phase 6's
   `identity` already gained Frigate face-recognition as a second, corroborating
   presence signal (never a registration signal, see `identity/README.md`'s
   "Camera face recognition" section) specifically so that Tapo cameras plug into
   the exact same anti-spoofing-respecting presence pipeline as everything else,
   rather than this phase inventing its own.
3. **Trash-bin fullness / dishes / litter / groceries-left-out checks are `chores/`'s
   job**: every ~2 hours (systemd timer, `RandomizedDelaySec=1800` — the "+/-30 min in
   case something else is running" jitter, systemd's own built-in feature, not custom
   code), `chores/check.py` grabs a Frigate snapshot per configured watch point
   (optionally moving a PTZ camera to a preset first) and asks an Ollama vision
   model a one-word question. A "needs attention" result opens a chore; a "clear"
   result auto-closes one.
3a. **`groceries_out_of_place` is `litter`'s twin, with a shorter fuse.** Same
   culprit-attribution (whoever the camera last recognised nearby), same
   exemption-proofing, same unassignability — because it is the same situation,
   somebody carried something in and put it down. The one deliberate difference is
   `GROCERIES_NEGLECT_THRESHOLD_HOURS` (30 minutes against the usual four), and it is
   the only place in `chores/` where one chore type is treated as more urgent than
   another: litter left for an afternoon is untidiness, a tub of ice cream left for an
   afternoon is a bin bag. Its prompt names what to **ignore** (fruit bowls, bread
   bins, coffee, spices, cookware) as well as what to look for, because the general
   question — "is anything out of place?" — is one a vision model answers YES to for a
   kitchen that is simply a kitchen, every two hours, until the household mutes the
   topic. It knows food is out; it never knows *what* the food is and never writes to
   Grocy. That is `pantry-vision`'s job, at a display someone is standing at — two
   systems looking at the same counter for two different reasons, which is the
   intended shape rather than a duplication to collapse.
4. **Household chore distribution — presence/calendar-driven nudging, not LLM
   assignment.** The governing principle, stated in `chores/check.py`'s own module
   docstring: *"I don't care who does it, as long as it gets done."* This is a
   nudge-and-redirect system, not a fair-assignment algorithm — the first run
   after a chore opens nudges whoever `identity` reports home right now; if it's
   still open after `NEGLECT_THRESHOLD_HOURS` (and the household calendar isn't
   showing a busy window, via a household-wide — not per-person — CalDAV
   busy-check that fails open), the nudge redirects to someone different who's
   available, "the next person that walks by." A rolling 30-day tally of who got
   nudged about what is kept and logged **purely for fairness comparison** — it
   never feeds back into who gets nudged next, a hard rule carried over from an
   explicit correction during this phase's design (an earlier LLM-picks-a-fair-
   assignee design was scrapped in favor of this one).
5. **Anyone leaving trash out gets told to bin it — this is the one case that
   ignores chore-exemption status.** `identity`'s `chore_exempt` flag (§ below)
   takes a household member out of the general nudge rotation (e.g. a frequent
   guest who isn't a household member and doesn't owe chores), but litter is
   special-cased in both directions: it's attributed to whoever the camera most
   recently recognized nearby (a best-effort "who left this" guess, not a
   certainty) rather than just whoever's home in general, and chore-exempt status
   is deliberately ignored for it — putting away trash you personally left out
   isn't "doing a chore," it's cleaning up after yourself.
6. **Per-person chore settings live in `identity`, not `chores`** — `chore_exempt`
   (bool) and `chore_reminder_style` (free text, e.g. "be assertive" / "be gentle,
   give me a few minutes of grace") are set via `identity`'s
   `POST /people//chore-settings` and read by `chores` off `GET /presence`,
   the same "identity owns who someone is" principle as the profile-photo and
   floor-plan groundwork already living there. `chore_reminder_style` is passed to
   an optional LLM call that **phrases** the reminder message in that tone — it
   never decides who or when, only how the words come out, keeping the same
   deterministic-fallback discipline as every other LLM-in-the-loop feature in
   this project (an un-styled plain template if no model is configured, the style
   is empty, or the call fails).
7. Nothing here has been run against real hardware or a real Frigate PTZ
   integration — no Tapo camera model has been chosen, the PTZ move-to-preset API
   shape is assumed from Frigate's general feature set, and vision-model accuracy
   for "is this bin full" / "is this counter dirty" / "is there litter out" is
   completely unmeasured. See `chores/README.md`'s "Manual verification still
   outstanding" for the full list.

### Testing checklist before calling any phase "done"
- Does the reactive path (presence → light on) work with the LLM host powered off? (It must.)
- Does a bad/slow LLM response ever block a light switch? (It must not.)
- Are cameras verified to have zero WAN egress?
- Does the calendar integration survive a Nextcloud restart without orphaning entities?
- Does an identity merge ever get auto-committed without confirmation? (It must not.)
- Does the thin client boot to a usable kiosk session with Mosquitto/HA/container-host powered off? (It must not hang.)
- Can the LLM reach the thin client through any path other than HA service call → MQTT → `thinclient-agent`? (It must not.)
- Does `digest-engine` ever perform a write action anywhere? (It must not — read-only only.)
- Is WhatsApp ingestion off by default, requiring the explicit opt-in + warning? (It must be.)
- Does voice digest playback ever read the wrong person's personal section when multiple people are present? (It must not — must ask, never guess.)
- Does a follow-up voice question ever trigger a write, or only grounded read/synthesis against cached context? (It must stay read-only.)
- If a run's LLM output produces malformed canvas-SDK calls, does the digest fall back to plain text instead of a broken/blank page?
- Are all ingestion platform credentials kept out of git (`.env`, gitignored), matching the restic-password handling convention?
- Does a stale/unreachable "was the digest viewed" signal ever cause runs to merge forever, instead of degrading to "assume viewed" after one missed check? (It must degrade, not compound.)
- Does the compact HA-dashboard iframe view ever mark a digest as viewed? (It must not — only an actual thin-client canvas display or voice playback counts.)
- Does the counter run actually drop a fabricated quote/figure/theoretical connection, rather than waving it through? (It must drop it.)
- Does the counter run ever flag a correctly-grounded piece of Marxist analysis as "unverifiable" for being theoretical rather than a bare fact? (It must not — see synth/prompts/counter_run.md.)
- If the counter run itself fails to reach the LLM host, is the original document kept and marked unverified, rather than either passed through silently or blanked? (It must be marked, not silently either extreme.)
- Does `admin-canvas` ever accept a request without a valid bearer token? (It must not — an unset `ADMIN_CANVAS_TOKEN` must fail closed, reject everything, not "auth optional".)
- Is `admin-canvas`'s write port ever published to the LAN in the generated compose file? (It must not be — reachable only from other containers on the compose network.)
- Does `admin-canvas` ever accept an `image`/`video` window whose `src` isn't a bare `media/` path (no scheme, no leading `/`, no `..`)? (It must not — same invariant as `mqtt_discovery.py`'s "a payload never becomes a URL host", enforced server-side and again client-side.)
- Can the sys-admin-llm reach the thin client through any path other than "Show admin canvas" → MQTT → `thinclient-agent`, with all actual *content* arriving via the separate `admin-canvas` write API instead? (It must not — same boundary as the digest canvas.)
- Does opening the admin canvas ever kill the digest canvas's kiosk Firefox window, or vice versa? (It must not — both `digest-browser` and `admin-browser`'s pkill/pgrep patterns are scoped to their own `--profile` path.)
- If `admin-canvas`'s `output/latest.json` holds a malformed or unrecognized window, does the admin canvas fall back to plain text instead of a broken/blank page? (Same rule as the digest's canvas-SDK renderer.)
- Does the voice/display satellite's wake-word spotting ever stream continuously to Home Assistant instead of triggering locally? (It must not — on-device `micro_wake_word`, same Phase 11.8 principle as the thin client's `wyoming-satellite` rooms.)
- Does the idle weather/time/date cycle ever show *over* an active media page, or does media ever fail to take priority the instant playback starts? (It must not — media priority is the one hard behavioral requirement of Phase 14.)
- Does the voice-state visualizer ever replace or hide the underlying page's content instead of overlaying it? (It must not — it's a `top_layer` overlay by design, never a page swap.)
- Is the ESP32-S3-Touch-LCD-1.85C firmware ever flashed onto a V1 board? (It must not be — V1 has no AEC circuit and different audio pins; this phase's config assumes V2 throughout.)
- Does a headless audio endpoint's Spotify Connect device name ever fail to match the room it's actually in? (It must not — `spotify-connect-start` always reads `$(hostname)`, which is set per-room by Raspberry Pi Imager on arm64 or `IMAGE_HOSTNAME` at build time on amd64; a copy-pasted/reused image for two rooms is a configuration mistake this software has no way to detect, same class of risk already called out for the ESP32 firmware's `media_player_entity_id`.)
- Does the headless audio endpoint ever gain a shared/synced audio path, an MQTT connection, or an HA entity without a deliberate new decision to add one? (It must not — Phase 15 is scoped to per-room independent Spotify Connect only, on purpose.)
- Can the LLM reach the touch panel through any path other than HA service call → MQTT → `touchpanel-agent`? (It must not — identical boundary to the thin client's.)
- Does the touch panel boot to a usable session (Spotify + the browser) with the container host powered off? (It must not hang — Home showing a connection error is the expected/acceptable degraded state, since unlike the thin client this device has no local media source to fall back to.)
- Does tapping the touch dock's Home/Spotify buttons ever relaunch (rather than just focus) an already-running window? (It must not — both are stateful apps; relaunching would throw away login/scroll/playback state for no reason.)
- Does `pantry-vision`'s `/identify` endpoint ever write to Grocy by itself? (It must not — only a separate, human-reviewed `/confirm` call ever writes anything.)
- Does `pantry-vision` ever accept a request without a valid bearer token, on any of its four endpoints — including the two GETs? (It must not — unlike admin-canvas, this service is LAN-published, so the token is the actual boundary, not network placement.)
- Can the LLM reach the kitchen display through any path other than HA service call → MQTT → `kitchen-display-agent`, for *which screen is showing*? (It must not — reading/writing the actual inventory is a separate, intentionally-published path through `pantry-vision` itself, not a violation of this rule.)
- If the vision model's response is unparseable or the call fails outright, does `/identify` ever return a broken/blank result instead of a flagged, low-confidence placeholder proposal? (It must not.)
- Does `identity` ever accept a raw/unresolved Bluetooth MAC (as opposed to an entity_id matching `TRUSTED_ENTITY_PREFIXES`) as a registration candidate? (It must not — that allowlist is the entire anti-spoofing boundary.)
- Does `POST /register` ever commit a registration when zero, more than one, or an already-claimed-by-someone-else candidate was found? (It must not — only the single-unambiguous-candidate case commits within one call; every other case requires a human to disambiguate.)
- Does `/presence` ever report a device-less person (no identifiers, no manual override set) as `home: false`? (It must report `null`/unknown — defaulting to "away" would be actively wrong the moment they're actually home, not just imprecise.)
- Can the LLM reach the door panel through any path other than HA service call → MQTT → `door-panel-agent`, for *which screen is showing*? (It must not — registration and presence/weather/groceries reads are separate, intentionally-published paths through `identity`/`pantry-vision` themselves, not a violation of this rule.)
- Does the door panel's voice registration path ever bypass HA's Assist pipeline (i.e. the kiosk device talking to `identity` on its own initiative from a wake word, with no HA intent script in between)? (It must not — voice is HA Assist → a custom intent script → `identity`'s API, same "HA mediates" shape as every other voice/tool-call path in this project.)
- Does `trash-calendar` ever delete or modify an event on the shared calendar that it didn't itself create? (It must not — write-only in the sense of create/update its own events, never touch anyone else's.)
- Does `transit`'s `/plan` ever silently fall back to guessed directions if `OTP_URL` is unset or OTP is unreachable? (It must not — it returns an explicit "not configured"/error, never a fabricated route.)
- Does Frigate face recognition (the Tapo-camera presence signal) ever create or claim a new `identity` person on its own? (It must not — it is only ever an OR-ed-in corroborating signal for an already-registered person, same hard rule as every other camera-adjacent presence source in this plan.)
- Does `chores`' nudge logic ever read its own fairness tally to decide who gets nudged next? (It must not — the tally is passive reporting only, per this phase's explicit design correction.)
- Does a `chore_exempt` person ever get nudged about a non-litter chore? (It must not.) Does a `chore_exempt` person ever get skipped for a litter chore? (It must not — litter ignores exemption status in both directions.)
- Does `chores`' optional LLM message-phrasing (`chore_reminder_style`) ever change *who* gets nudged or *when*, rather than only the wording of the notification? (It must not — and a failed/empty LLM call must fall back to the plain template, never block the nudge from going out.)
- Does `chores`' household calendar busy-check, if unreachable or misconfigured, ever become the reason nudges stop going out entirely? (It must not — it fails open, treating an error as "not busy.")

---

## 4. Open decisions (Phases 6, 11–20)

These need a decision before their respective implementation steps can be built — everything above is written to accommodate any answer, but nothing should be built against an unresolved item.

1. ~~RDP vs. VNC vs. desktop-environment swap~~ — **resolved**: wayvnc (VNC) replaces RDP for this project; Sway is kept.
2. ~~WhatsApp ingestion approach~~ — **resolved**: `whatsapp-bridge` (headful Chromium + `whatsapp-web.js` in an Xvfb virtual display), not Baileys. Still opt-in (`ENABLE_WHATSAPP_INGEST`); still recommend a secondary/non-critical number, since automating a personal account carries some risk even via the real web client.
3. **Mainstream news source list** — `digest-engine/feeds/curated-feeds.opml` ships with `marxist.com/feed/rss` (real) plus BBC World/Al Jazeera/Guardian World/DW as clearly-marked placeholders (Reuters/AP were skipped — both have restricted their public RSS and guessing a live URL seemed worse than an honest placeholder). Still needs the user's actual sign-off/edit.
4. **Ollama contention** — **half resolved** (Phase 3, `hosts/llm-host/`). The *resource* half now has a defensible default: `OLLAMA_MAX_LOADED_MODELS=1` (a 14B text model and a vision model don't co-fit in 8–12GB, so force a predictable swap rather than VRAM thrash or an OOM mid-request), `OLLAMA_NUM_PARALLEL=1` (predictable latency for whoever is speaking to Assist over throughput nothing here needs), and `OLLAMA_KEEP_ALIVE=30m` (Ollama's 5m default makes a household that talks to Assist a few times an hour pay the model-load cost nearly every time). **Still open**: these are reasoned, not measured — nothing has run against a real GPU under concurrent load — and the *scheduling* half is untouched, `DIGEST_SCHEDULE` still defaults to `00,06,12,18` with no adjustment for real Assist-traffic overlap, which needs actual usage data to settle.
5. **Credential storage** — implemented as a git-ignored `.env` seeded from `digest-engine.env.example` (matches the restic-password precedent); `age`/`sops` was not built, considered adequate for now.
6. **Exact mic-enabled room list** — still needed; `ENABLE_VOICE_SATELLITE` in `build-thin-client-iso.sh` defaults to `false` per-image until rooms are chosen.
7. **Exact thin-client hardware target** — still needed; nothing in the built image assumes specific hardware, but Steam Link/Xwayland decode performance can't be validated without it.
8. **Personal-digest visibility on shared displays** — not yet resolved; not blocking, since the thin-client rendering built so far doesn't yet distinguish "shared screen" from "private."
9. **Household/calendar ingestion has no real data source yet** (new, found during Phase 12 implementation) — the plan named CalDAV (Phase 8) and Grocy (Phase 7) as the source but no ingest module was written for either; `digest-engine/run.py` currently passes empty calendar/Grocy context and the household prompt is told to say "nothing scheduled" rather than hallucinate. Needs either a new ingest module or a decision to pull this from HA directly.
10. **HA has no core MQTT `media_player` platform** (new, found during Phase 11 implementation) — `thinclient-agent` publishes the `media_player` discovery payload as specified, but stock Home Assistant ignores it without the HACS "MQTT Media Player" custom integration installed. `button`/`sensor`/`number` entities are also published as a fallback that works on a plain HA install; decide whether to install the HACS integration or keep relying on the fallback entities.
11. **Several package-availability items still need verification on real hardware** before first boot, all flagged in-code rather than guessed: the Steam Link Flatpak app ID (`com.valvesoftware.SteamLink`), spotifyd/librespot packaging on Debian bookworm (not in main — three fallback install routes documented, none wired to a hardcoded download URL), and `mpv-mpris` packaging.
12. **wayvnc ships with no password** — `start-wayvnc` fails closed on a sentinel value (`CHANGEME-SET-ON-FIRST-BOOT`) rather than serving unauthenticated VNC; a real password must be generated on the booted machine before wayvnc will start (see `hosts/thin-client/README.md`).
13. **The sys-admin-llm's HA-side wiring has no real data source or tool definition yet** (new, Phase 13) — same shape as open decision #9 above: `admin-canvas/README.md` documents the expected `rest_command:`/tool-call contract and worked example JSON, but which HA entities/history back something like "the kitchen outlet's power draw" is unresolved, and no metering-capable Zigbee smart plug is in this plan's hardware list (§1.4) yet. Needs a hardware decision (a power-monitoring outlet) and an actual HA tool/script, neither of which exists in this repo by design — see the Phase 13 "nothing under this repo builds the HA side" note.
14. **The voice/display satellite has no real hardware verification, and two of its data sources are unpicked** (new, Phase 14) — `firmware/esp32-s3-touch-lcd-1.85c/voice-display.yaml` passes ESPHome's own config validator but has never been flashed to a physical unit; the display init sequence, the AEC audio path, and wake-word sensitivity are all adapted/assumed, not measured (see the itemized list in that directory's README). Separately, `media_player_entity_id` and `weather_entity_id` are placeholders — which media player this unit should mirror needs a decision if the household ends up with more than one active at a time.
15. **The headless audio endpoint has no real hardware verification on either architecture** (new, Phase 15) — neither the arm64 (`rpi-image-gen`) nor amd64 (`live-build`) pipeline has been run to completion, let alone flashed and booted. rpi-image-gen's exact config/layer YAML schema in particular was written from documentation only, with no ARM build environment available to validate it against the real tool — see `hosts/audio-endpoint/README.md`'s prominent risk callout. Also still open: whether `spotifyd` or `librespot` actually installs cleanly from current bookworm apt sources at all (the fallback list is documented, not exercised), and how stable USB DAC/amp ALSA enumeration turns out to be on a real amd64 mini PC.
16. **No touch-panel hardware has been chosen** (new, Phase 16) — nothing in `hosts/touch-panel/` has been booted on real metal or a real touchscreen. Which of the two shipped input tiers a real device lands in (native `wl_touch`, or the `type:pointer` fallback for hardware that misreports as an emulated-mouse HID device — see `hosts/touch-panel/README.md`'s "Touch input: two tiers") is unverified either way, as is whether `configs/udev/99-touchscreen-override.rules`' `ATTRS{idVendor}`/`ATTRS{idProduct}` match actually walks up to the right USB parent for a given controller — see that README's verification list, item 3.
17. **The touch panel's Spotify/Chromium integration details are assumed, not confirmed** (new, Phase 16) — the Flathub app ID `com.spotify.Client`, the Spotify Flatpak's real MPRIS bus name (assumed `spotify`), and Chromium's Wayland `app_id` for a `--app=` kiosk window (assumed to start with `chromium`) are all flagged for on-hardware verification in `hosts/touch-panel/README.md`.
18. **No vision-capable Ollama model has been picked, pulled, or benchmarked** (new, Phase 17) — `pantry-vision`'s `OLLAMA_VISION_MODEL` defaults to `llava` with no confirmation it's the right choice for any given LLM host's hardware tier, and grocery-item identification latency/accuracy is completely unmeasured. This is the single highest-risk unknown in Phase 17: if it's too slow or too inaccurate, "hold item up to camera" stops being a usable interaction and the phase needs a different approach (a smaller/faster model, a barcode-scan fallback via Grocy's existing scanning support, or both).
19. **`pantry-vision`'s Grocy API integration is written from documentation, not a live instance** (new, Phase 17) — `GET /api/stock`'s response shape, `POST /api/objects/products`'s minimum required fields, and the Recipes/fulfillment endpoints' setup requirements are all assumed; `pantry-vision/README.md` points at each real Grocy instance's own live OpenAPI spec (`/api/openapi/specification`) as the way to check before trusting any of it. `GROCY_DEFAULT_LOCATION_ID`/`GROCY_DEFAULT_QU_ID` are fresh-install-default guesses that need confirming against Settings → Locations/Quantity units on the real instance too.
20. **`identity`'s `TRUSTED_ENTITY_PREFIXES` default is a guess, and it's the single highest-risk unknown in Phase 6** (new) — the whole anti-spoofing design rests on this allowlist actually matching real Private BLE Device / fixed-tag entity IDs; until it's checked against Developer Tools -> States on a real HA instance, registration will most likely just report "no candidate" for everything. Same open dependency as §1.5's original Bermuda/Private BLE Device setup, which itself has never been built (see the top-level README status checklist).
21. **Identity's HA-side voice wiring (custom sentence + intent script + `rest_command`) is written from HA's documented shape, not tested** (new, Phase 6) — `identity/README.md` has the worked example; nothing under this repo builds or verifies it, same convention as admin-canvas's/digest-engine's own HA-side integration points.
22. ~~The floor-plan UI itself doesn't exist~~ — **built** (Phase 6b): `identity`'s admin panel has a Floorplan tab where levels and rooms are **drawn by hand** (polygons, optional background image, normalised 0–1 coordinates so the plan renders at any size), each room mapped to the HA area string `/presence` already reported. Occupied rooms light up live. Nothing is auto-placed — nothing here knows the shape of this flat, and drawing it is what removed the need to invent a coordinate format against a guess. **Still genuinely open**: `AREA_ATTRIBUTE`'s default (`area_id`) is still an unconfirmed guess at what Bermuda publishes, so until that's verified every room will simply never light up — the editor now surfaces this directly by showing which areas HA is actually reporting, and listing people who are home but unplaceable. The editor has also never been opened in a real browser.
23. **No Tapo camera model or count has been chosen, and Frigate's PTZ move-to-preset API shape is assumed** (new, Phase 20) — §1.17 lists a placeholder model/price only; whether a specific Tapo model even exposes RTSP without go2rtc as a bridge is unverified, and `chores/check.py`'s `_frigate_snapshot()`'s `POST /api//ptz/move/` is assumed from Frigate's general PTZ feature set, not a real deployment. This is the single highest-risk unknown in Phase 20, same class of risk as open decision #18's vision-model pick for Phase 17.
24. **Vision-model accuracy for bin-fullness/dishes/litter checks is completely unmeasured** (new, Phase 20) — same caveat as open decision #18, applied to a different prompt; a wrong FULL/DIRTY/YES answer just means a chore opens or stays open incorrectly, never a hard failure, but nobody has checked how often that actually happens.
25. **`chores`' household calendar busy-check is household-wide, not per-person** (new, Phase 20) — `_household_currently_busy()` can't tell that only one person is in a flagged-busy calendar event and nudge someone else who's free; everyone's nudges pause together. A real per-person availability model would need per-person calendars, which this project doesn't have. Documented as a known limitation, not a bug, in `chores/README.md`.
26. ~~No frontend exists yet for setting `chore_exempt`/`chore_reminder_style`~~ — **resolved** (Phase 6b): `identity/frontend/admin.html` edits both, plus chore assignment, on the People tab. **Still open in a smaller way**: `admin.js`'s `CHORE_TYPES` list is kept in sync with `chores/check.py`'s `_CHORE_PROMPTS` **by hand**, because `chores` is a oneshot timer job with no HTTP surface to query for it — adding a fourth chore type means editing both files, and nothing catches it if you don't.
27. **No way to manually mark a chore done exists yet** (new, Phase 20) — the only way a `chores` chore currently closes is a camera re-check finding it clear, or direct SQLite surgery; a real deployment probably wants an HA button or voice phrase for "mark the trash as done," deliberately left out of this pass rather than guessed at.
28. **Which UniFi/CalDAV/Matter/1-Wire/Proxmox/Steam/Discord/HP-iLO/GTFS HA integrations actually get installed is unresolved** (new) — all nine are catalogued in §2's "HA integrations catalog" as available options with their purpose/notes, but none has been installed, configured, or verified against real hardware/accounts; several (Matter, 1-Wire, Proxmox, HP iLO) also depend on hardware/infrastructure decisions this plan hasn't made yet (whether anything in the household actually uses those platforms at all).
29. **Music Assistant has not been installed or configured** (new) — catalogued in §2 as an optional, additive HA add-on; whether it's worth adding on top of the existing per-room spotifyd/librespot/Spotify-client setup (which keeps working standalone regardless) is a real usage-pattern question, not answerable until the existing per-room setups (Phase 11.6/15/16) are actually running.
30. **The self-check/hardware-monitoring integrations catalog (§2) is a menu, not a deployment plan** (new) — System Monitor, SNMP, NUT, Glances, and Uptime Kuma are all listed with their purpose, but which ones are actually worth installing depends on hardware decisions not yet made (is there a UPS? managed switches? which of this repo's published services matter enough to alert on?).
31. ~~Music Assistant's default port is a guess, and it collides with `PANTRY_VISION_PORT` in this exact stack~~ — **mechanically resolved** by `tools/`: both ports are now declared in `CoreSystemConfig.json`, and `validate-config.py` **fails the build** on any duplicate, so the clash cannot reach a deployment. The template assigns Music Assistant 8101. **Still genuinely open**: its real default (8095) is still an unverified guess, and because it runs `network_mode: host`, moving it requires configuring Music Assistant itself — the config file can declare the port but cannot make the service bind to it.
32. ~~RuView's semantic-state MQTT entities have no opt-out or visibility restriction beyond this network's normal trust boundary~~ — **household decision made**: real automations are now built on this data (sleep → dim lights, possible-distress → whole-household alert, concurrent elevated heart rate → colored lighting, bathroom occupancy → an external door indicator — see `firmware/ruview/README.md` §5 and `firmware/ruview/automations.yaml.example`). **Still genuinely open**: there is no technical opt-out for a specific person/room and no access restriction on these MQTT topics beyond this network's normal trust boundary — worth revisiting if anyone not on board with being sensed this way ever stays over. Every automation's `entity_id` is also still an unconfirmed placeholder (see #33), and rule 3 (concurrent two-person heart rate) rests on an unconfirmed assumption that a single RuView node can report two people's heart rates at once — multi-target vital-sign separation from WiFi CSI is a genuinely hard, unconfirmed capability, not something to trust until checked against real entities.
33. **RuView's build/flash commands and `provision.py`'s exact flags beyond `--port`/`--ssid`/`--password`/`--mqtt` are transcribed from its README, not independently run** (new, Phase 2) — see `firmware/ruview/README.md`'s own "Manual verification still outstanding," same category of risk as every other "written from documentation, not a live instance" open decision in this list (#19, #21).
34. **`identity`'s `DEPARTURE_GRACE_SECONDS` default (15 min) is an untuned guess at how much a real Private BLE Device setup flaps** (new, Phase 6b) — too low and one evening at home is recorded as several separate "visits," too high and a quick trip out never registers. The entire usefulness of the visit history and the co-presence view rests on this number, and nobody has watched a real BLE presence entity over a day to pick it. First thing to check once `GET /visits` has real data in it.
35. **The device-rights HA wiring is a worked example, not a tested one, and this is the one place where getting it wrong opens a door** (new, Phase 6b) — `identity` only ever *answers* `GET /device-access`; an HA automation that calls `lock.unlock` outside the `condition: template` guard in `identity/README.md`'s example would unlock regardless of that answer, and nothing on this side can detect or prevent that. Same "documented shape, not verified" caveat as #21, with a materially worse failure mode.
36. **The admin panel has never been opened in a real browser** (new, Phase 6b) — every endpoint it calls is covered by API-level tests, but the page itself (the ``-based person editor, the prune checkbox flow) has only been checked for syntax. `showModal()` needs a reasonably current browser; the door panel's Chromium qualifies, something older might not.
37. ~~iOS cannot receive `identity`'s arrival notifications without WAN egress through a third party~~ — **household decision made** (Phase 6b): **no Apple devices are used here**, which removes the only forcing function for exposing ntfy at all. ntfy therefore **stays LAN-only** — no DMZ, no port forward, no NAT-reflection hairpin; `identity → ntfy` is a container-to-container call that never reaches OPNsense, and remote delivery rides a **WireGuard split tunnel** routing only the smart-home VLAN. The full comparison (including why the hairpin variant, which avoids an inter-VLAN rule, was rejected for making an internal path depend on the ISP) is recorded in `docs/network-integration.md` §2.2 so it doesn't get re-litigated. **Still genuinely open**: nothing has been delivered to a real phone yet, and the split tunnel's `AllowedIPs` is the fiddly part — routing all of `192.168.0.0/16` would collide with typical café/hotel LANs and break the phone's connectivity there. Revisit §2.2 only if an Apple device ever joins the household.
38. **Both of `chores`' shipped container URLs pointed at `127.0.0.1` and could never have worked** (found while wiring Phase 6b's arrival notifications) — inside the `chores` container that address is the chores container itself, not the host. `NTFY_URL=http://127.0.0.1:8090` meant every nudge failed to send, and `IDENTITY_URL=http://127.0.0.1:8097` meant `_presence()` never reached identity at all, so **`chores` would have done nothing whatsoever** — and silently, since both failure paths log and continue by design (a deliberate never-block-on-a-dependency choice that here hid a total outage). Corrected in the template to `http://ntfy` and `http://identity:8097` (compose-bridge DNS; ntfy on its internal port 80, not the 8090 published to the LAN). **An already-deployed `/opt/smart-home/chores/chores.env` still has the old values** — templates are copied once at setup and never re-synced, so existing installs need this edited by hand. The broader lesson worth acting on: several services fail soft on an unreachable dependency, which means a misconfigured address produces silence rather than an error — nothing in this repo currently distinguishes "nothing to do" from "never reached the thing that would have told me".
39. **Nothing enforces that voice/TTS consumers actually read `speak_name` rather than `nickname`** (new, Phase 6b) — the field exists, is documented, and `chores/` uses it, but a future HA intent script that reaches for the friendlier-looking `nickname` would break the "the assistant uses real names" rule silently. Worth checking whenever a new consumer of `/presence` or `/resolve` is written.
40. **Whether a vision model can identify a grocery item in a moving hand at doorway distance** (new, Phase 17) — the assumption the entire `/doorway-event` path rests on, and a materially harder task than the kitchen display's held-still-against-a-plain-background one. It is also the cheapest open decision on this list to close: point the kitchen's existing webcam at somebody walking past with a carton and run `/identify` on the frames. Until then the door sensors are the part worth deploying, and `PANTRY_DOOR_APPLIANCES` entries can be configured without a camera — see `docs/fridge-item-location.md`, whose recommended order (sensors, then a month of living with them, *then* one camera) the code deliberately does not shortcut.
41. **Whether `groceries_out_of_place` can tell a carton of milk from a fruit bowl** (new, Phase 20) — its prompt lists what to ignore precisely because the general question gets a YES for any normal kitchen, but that is a mitigation written blind. The failure mode is not a missed chore, it is a false one every two hours, which is how a household learns to ignore the notification channel entirely. Point it at a *clean* counter for a day before trusting a YES, and note this is the first chore type whose false positives cost more than its false negatives.
42. **Contact-sensor battery life on an appliance door is unverified** (new, Phase 17) — a fridge door opens far more often than the window these sensors are sold for, and the sensor sits in a cold, humid draught even when mounted outside the compartment (which `docs/components.md` says to do, for the separate reason that a coin cell at −18 °C is a false economy). Check one after a month before buying more.