# AI-Managed Smart Home Local-first, open-source smart home: Home Assistant + Zigbee + RuView (CSI presence) + Bermuda (BLE identity) + local LLM (Ollama) + Frigate (peephole face recognition) + Grocy (kitchen kiosk) + Nextcloud calendar sync + a Sway thin-client media station + a Sway touch panel + a camera-vision kitchen/fridge display + a voice/touch identity-registration door panel + a quarter-daily LLM-generated digest. See [`docs/project-plan.md`](docs/project-plan.md) for the full hardware list, software stack, and phased implementation plan. ## Repo layout ``` docs/ Project plan and design notes hosts/ container-host/ Docker Compose stack: HA, Mosquitto, Zigbee2MQTT, Frigate, Grocy, Node-RED, monitoring, etc. configs/ Per-service config files (mosquitto.conf, etc.) scripts/ Host setup / bootstrap scripts llm-host/ Ollama + GPU host setup (separate physical machine) thin-client/ Sway kiosk/media-station ISO (live-build) + thinclient-agent audio-endpoint/ Headless Spotify Connect appliance for rooms with no thin client — arm64 (rpi-image-gen) + amd64 (live-build) images touch-panel/ Touch-driven Sway panel: full Spotify GUI, a dedicated Home Assistant Chromium kiosk window, and a general web browser, switched via an always-on touch dock or by HA/ the local LLM over MQTT kitchen-display/ Single-purpose Sway kiosk for the fridge/pantry: one Chromium window showing pantry-vision's Scan/Inventory/ Recipes frontend, camera capture via the browser itself door-panel/ Single-purpose Sway kiosk by the door/wardrobe: identity's weather+clothing/who's-home/groceries-low dashboard by default, registration on demand — this host's twin relationship to kitchen-display, but the mic is the point here, not an opt-in edge case firmware/ ruview/ RuView ESP32-S3 CSI presence node configs esphome-ble-proxy/ ESPHome configs for Bermuda BLE proxy nodes esp32-s3-touch-lcd-1.85c/ ESPHome voice satellite + status display (round LCD, media/cover-art priority over an idle weather/time/ date cycle, voice-state visualizer) identity/ Person <-> BLE-identifier registry: multi-phone support, anti-spoofing (allowlisted IRK-resolved/fixed-tag entities only, never a raw MAC), voice/touch registration, presence resolution, visit history ("who was home when, with whom"), nicknames, per-device rights, and an admin panel (write API + frontend/ static serving, consumed by kitchen-display and door-panel) digest-engine/ Quarter-daily LLM digest: mail/message/news/financial ingestion, LLM synthesis, digest-canvas SDK rendering admin-canvas/ On-demand sys-admin-llm display surface for the thin clients: stats/graphics/media, pushed on demand rather than on a schedule (write API + admin-web static serving) pantry-vision/ Kitchen-display backend: a photo held up to the camera -> an Ollama vision-model proposal -> human-confirmed write into Grocy stock; also proxies Grocy's inventory (soonest-expiring first) and recipes to the kiosk frontend (write API + frontend/ static serving) trash-calendar/ Reads Kennelbach's personal trash-collection ICS feed, writes matching events onto the shared CalDAV calendar (daily systemd timer, read-only against the feed) transit/ Public transit: GET /departures (static GTFS lookup) and GET /plan (OpenTripPlanner proxy, voice-usable via HA Assist) — weekly GTFS refresh via systemd timer chores/ Presence/calendar-driven household chore nudging + a passive fairness tally + camera-verified trash-bin/ dishes/litter checks (systemd-timed oneshot, no long-lived service, no LLM-picked assignment) ``` ## Status - [x] Project plan drafted - [x] Container host setup script v1 (HA, Mosquitto, Zigbee2MQTT USB, Frigate, Grocy) - [x] Node-RED + monitoring (Netdata) + dashboard (Homepage) + ntfy + Portainer added to compose stack - [x] Backup (restic) setup — scripted, off by default until a backup target is picked (`ENABLE_BACKUPS`) - [ ] Bermuda / ESPHome BLE proxy configs — `firmware/esphome-ble-proxy/` built (stock ESPHome `bluetooth_proxy` component), not yet flashed to real hardware, see that directory's README - [ ] RuView node configs — `firmware/ruview/` documents the real upstream project ([github.com/ruvnet/ruview](https://github.com/ruvnet/ruview), integrated not forked) + a per-room provisioning wrapper + `automations.yaml.example` (sleep → dim lights, possible-distress → whole-household alert, concurrent elevated heart rate → colored lighting, bathroom occupancy → an external door indicator). **Every automation's entity_id is an unconfirmed placeholder**, and the concurrent-two-person-heart-rate rule rests on an unconfirmed assumption about RuView's multi-target vital-sign capability — see `firmware/ruview/README.md` §5–6 and `docs/project-plan.md` open decisions #32–33 before relying on any of it - [ ] Frigate peephole camera config (real RTSP details) - [ ] Grocy kiosk (Pi + touchscreen) setup - [ ] LLM host (Ollama) setup script - [ ] CalDAV / Nextcloud calendar integration notes - [x] Sway thin-client ISO (live-build) + thinclient-agent — built, not yet boot-tested on real hardware; RDP replaced by wayvnc (resolved), remaining open items (mic-enabled rooms, exact hardware target, wayvnc password provisioning) in `docs/project-plan.md` §4 - [ ] Thin-client follow-ups in progress: fullscreen-aware now-playing widget (cover art + controls), minimal Firefox chrome + uBlock Origin/SponsorBlock, persistent audio-output selection, outbound RDP/VNC client (`rdp-vnc.json`), HA mobile-app browser remote control (text input + mouse buttons), capture-card ("receiver box") video source selection on a new `5:capture` workspace, idle-gallery weather/clock overlay (clock always, weather via a new `smarthome/weather/current` MQTT topic an HA automation has to publish) — built, not yet tried against real capture-card hardware or a real weather automation, see `hosts/thin-client/README.md` - [x] Quarter-daily digest engine (mail/Signal/Telegram/Discord/WhatsApp, news, financial ingestion; LLM synthesis; digest-canvas SDK) — built and wired into `setup-container-host.sh` (`ENABLE_DIGEST_ENGINE`, off by default), not yet run against real credentials; household/calendar ingest (CalDAV/Grocy) still needs a real data source wired in, see `docs/project-plan.md` §4 - [x] admin-canvas + admin-web (sys-admin-llm on-demand display surface for the thin clients) — built and wired into `setup-container-host.sh` (`ENABLE_ADMIN_CANVAS`, off by default); the HA-side tool/rest_command wiring and the specific entities it surfaces (e.g. power-monitoring) are still undecided, see `docs/project-plan.md` §4 - [ ] ESP32-S3-Touch-LCD-1.85C-V2 voice satellite + status display (`firmware/esp32-s3-touch-lcd-1.85c/`) — ESPHome config written and passes `esphome config`, not yet flashed to real hardware; `media_player`/`weather` entity IDs still need to be chosen, see `docs/project-plan.md` §4 - [ ] Headless audio endpoint (`hosts/audio-endpoint/`) — per-room independent Spotify Connect appliance for rooms without a thin client, arm64 (Raspberry Pi + HiFiBerry Amp2, rpi-image-gen) and amd64 (mini PC + USB DAC/amp, live-build) build pipelines written, **neither built/flashed/booted on real hardware** — rpi-image-gen's exact config schema in particular is unverified, see `hosts/audio-endpoint/README.md` - [ ] Sway touch panel (`hosts/touch-panel/`) — touch-driven Sway image: full Spotify GUI (Flathub), a dedicated Home Assistant Chromium kiosk window, a general web browser, an always-on touch dock for app switching, an on-screen keyboard (toggled manually, no auto-show), and `touchpanel-agent` (HA MQTT control, same LLM-mediated-through-HA security model as the thin client) — built, **no touch-panel hardware chosen and nothing booted on real metal**, see `hosts/touch-panel/README.md` - [ ] Kitchen/fridge display + `pantry-vision` (`hosts/kitchen-display/`, `pantry-vision/`) — hold a grocery item up to the camera, an Ollama vision model proposes what it is and roughly how long it keeps, a human confirms (never auto-committed) before it's written into Grocy stock; the display then shows inventory sorted by soonest-to-expire, groceries running low, and Grocy's recipes — built and wired into `setup-container-host.sh` (`ENABLE_PANTRY_VISION`, off by default), **nothing run against a real camera, vision model, or Grocy instance** — the Grocy API call shapes in particular are written from documentation only, see `pantry-vision/README.md` and `hosts/kitchen-display/README.md` - [ ] `identity` + door panel (`identity/`, `hosts/door-panel/`) — the person <-> BLE-identifier registry: "register me as ``" by voice or touchscreen, multi-phone support (multiple identifiers per person), anti-spoofing (only allowlisted IRK-resolved/fixed-tag entities are ever accepted as candidates, never a raw MAC), device-less people (a "no device" flag plus a hand-operated Home/Away toggle — the concrete case: a grandmother without a smartphone), and an anonymous "Guest" path. Backs `hosts/door-panel/`'s weather+clothing/who's-home/groceries-running-low dashboard and `hosts/kitchen-display/`'s "Show registration" screen — built and wired into `setup-container-host.sh` (`ENABLE_IDENTITY`, off by default), **nothing run against a real HA instance, real Private BLE Device entities, or a real voice pipeline** — `TRUSTED_ENTITY_PREFIXES` above all needs checking against Developer Tools -> States, see `identity/README.md` and `hosts/door-panel/README.md` - [ ] `identity` also corroborates presence from Frigate face recognition (Phase 20, Tapo pan/tilt cameras) — an OR-ed-in second signal only, **never** a registration signal; and owns the per-person chore-system settings (`chore_exempt`, `chore_reminder_style`, plus chore assignment) consumed by `chores/`, see `identity/README.md` - [ ] `identity`'s admin panel (`identity/frontend/admin.html`, Phase 6b) — managing people/guests: edit every field, **nicknames** (an input alias only — `/resolve` accepts them, but the assistant always speaks the real `speak_name`), **visit history** sampled from `/presence` plus a derived "who was home with whom" view, **"select all that last visited before ``"** pruning (the filter selects, a human confirms the exact list, the filter is never re-run at delete time), **per-device rights** for self-entry via a smart lock (`identity` only ever *answers* `GET /device-access` — HA asks and HA acts, deny is the default), chore assignment, and **opt-in arrival push notifications** ("tell me when someone gets home", via the self-hosted ntfy this stack already runs — `identity` itself never touches the WAN; ntfy stays LAN-only and remote delivery rides a WireGuard split tunnel — see `docs/network-integration.md` §2.2 for why a DMZ/port-forward was weighed and rejected). Deliberately **not** a kiosk page and not linked from any wall panel. Covered by API-level tests; **never opened in a real browser**, and `DEPARTURE_GRACE_SECONDS` is an untuned guess — see `identity/README.md` - [ ] `trash-calendar` + `transit` (Phase 19, Kennelbach AT trash pickup + Vorarlberg public transit) — built and wired into `setup-container-host.sh` (`ENABLE_TRASH_CALENDAR`/`ENABLE_TRANSIT`/`ENABLE_TRIP_PLANNING`, all off by default), **nothing run against a live ICS feed, a live GTFS feed, or a real OpenTripPlanner instance** — trip planning also needs a manually-built OTP graph this repo does not build for you, see `trash-calendar/README.md` and `transit/README.md`'s "Route planning scope" - [ ] `chores` (Phase 20) — presence/calendar-driven household chore nudging: "I don't care who does it, as long as it gets done" — prefers whoever's been assigned a chore in `identity`'s admin panel but falls through to whoever's actually home rather than waiting (`CHORE_ASSIGNMENT_STRICT` flips that), redirects to someone else if a chore goes neglected, keeps a passive fairness tally that never feeds back into who gets nudged, and camera-checks trash bins/dishes/litter via Frigate + an Ollama vision model. **Litter remains the exception to everything** — it ignores both chore-exemption and assignment, because cleaning up what you left out was never a task anyone could be assigned. Built and wired into `setup-container-host.sh` (`ENABLE_CHORES`, off by default, every-2-hours systemd timer), **no Tapo camera hardware chosen and nothing run against real hardware**, see `chores/README.md` - [ ] Music Assistant (optional, additive multi-room audio) — wired into `setup-container-host.sh` (`ENABLE_MUSIC_ASSISTANT`, off by default), **its default port is an unverified guess that collides with `PANTRY_VISION_PORT`** if both are enabled together, see `docs/project-plan.md` open decision #31 - [ ] `docs/network-integration.md` (OPNsense VLAN segmentation, the WireGuard split tunnel that carries arrival notifications, and why nothing here — ntfy included — gets port-forwarded to the WAN) — written, not run against a real OPNsense instance ## Quick start ```bash cd hosts/container-host/scripts sudo ./setup-container-host.sh ``` Edit the variables at the top of the script first (timezone, Zigbee USB device path, Mealie/hardware-accel toggles, and whether to enable Node-RED/Netdata/ Homepage/ntfy/Portainer and restic backups — all but backups are on by default). See `docs/project-plan.md` for the full phased rollout order — don't skip straight to Phase 4+ automations before the Phase 2 reactive baseline (presence -> light, no LLM in the loop) is working. The script brings up everything that runs on this one Debian host: Home Assistant, Mosquitto, Zigbee2MQTT, Node-RED, Frigate, Grocy, Netdata, a Homepage dashboard, ntfy, and Portainer, plus an optional scheduled restic backup timer and an optional quarter-daily digest engine (`ENABLE_DIGEST_ENGINE`, off by default — needs `digest-engine/` checked out on the host and its `.env` filled in first, see `digest-engine/README.md`). What it can't do for you, because they need separate hardware, credentials, or physical setup: pairing Zigbee sensors, flashing RuView/ESPHome/Bermuda BLE proxy boards, pointing Frigate at a real camera RTSP URL, the Grocy kiosk touchscreen, the separate LLM/GPU host, wiring up the Nextcloud CalDAV integration, building/flashing the thin-client ISO (`hosts/thin-client/`), and provisioning real credentials for the digest engine's mail/message/news/financial sources — see the Status checklist above and `docs/project-plan.md` for those.