AI-Managed Smart Home

Local-first, open-source smart home: Home Assistant + Zigbee + RuView (CSI presence)

• Bermuda (BLE identity) + local LLM (Ollama) + Frigate (peephole face recognition)
• Grocy (kitchen kiosk) + Nextcloud calendar sync + a Sway thin-client media station
• a Sway touch panel + a camera-vision kitchen/fridge display + a voice/touch identity-registration door panel + a quarter-daily LLM-generated digest.

See docs/project-plan.md for the full hardware list, software stack, and phased implementation plan.

Repo layout

docs/                       Project plan and design notes
hosts/
  container-host/           Docker Compose stack: HA, Mosquitto, Zigbee2MQTT,
                             Frigate, Grocy, Node-RED, monitoring, etc.
    configs/                Per-service config files (mosquitto.conf, etc.)
  llm-host/                 Ollama + GPU host (separate physical machine)
  thin-client/              Sway kiosk/media-station ISO (live-build) + thinclient-agent
  audio-endpoint/           Headless Spotify Connect appliance for rooms with no thin
                             client — arm64 (rpi-image-gen) + amd64 (live-build) images
  touch-panel/              Touch-driven Sway panel: full Spotify GUI, a dedicated
                             Home Assistant Chromium kiosk window, and a general web
                             browser, switched via an always-on touch dock or by HA/
                             the local LLM over MQTT
  kitchen-display/           Single-purpose Sway kiosk for the fridge/pantry: one
                             Chromium window showing pantry-vision's Scan/Inventory/
                             Recipes frontend, camera capture via the browser itself
  door-panel/                 Single-purpose Sway kiosk by the door/wardrobe:
                             identity's weather+clothing/who's-home/groceries-low
                             dashboard by default, registration on demand — this
                             host's twin relationship to kitchen-display, but the
                             mic is the point here, not an opt-in edge case
firmware/
  ruview/                   RuView ESP32-S3 CSI presence node configs
  esphome-ble-proxy/        ESPHome configs for Bermuda BLE proxy nodes
  esp32-s3-touch-lcd-1.85c/ ESPHome voice satellite + status display (round LCD,
                            media/cover-art priority over an idle weather/time/
                            date cycle, voice-state visualizer)
tools/                      All build + setup scripts, driven by one
                             CoreSystemConfig.json: twinned container-host/LLM-host
                             ISO pair, every kiosk ISO, and a validator that refuses
                             a build on port collisions, placeholder tokens, or a
                             kiosk pointed at a disabled service
identity/                   Person <-> BLE-identifier registry: multi-phone support,
                             anti-spoofing (allowlisted IRK-resolved/fixed-tag
                             entities only, never a raw MAC), voice/touch
                             registration, presence resolution, visit history
                             ("who was home when, with whom"), nicknames, per-device
                             rights, and an admin panel (write API + frontend/
                             static serving, consumed by kitchen-display and
                             door-panel)
digest-engine/              Quarter-daily LLM digest: mail/message/news/financial
                            ingestion, LLM synthesis, digest-canvas SDK rendering
admin-canvas/               On-demand sys-admin-llm display surface for the thin
                             clients: stats/graphics/media, pushed on demand
                             rather than on a schedule (write API + admin-web
                             static serving)
pantry-vision/               Kitchen-display backend: a photo held up to the camera
                             -> an Ollama vision-model proposal -> human-confirmed
                             write into Grocy stock; also proxies Grocy's inventory
                             (soonest-expiring first) and recipes to the kiosk
                             frontend (write API + frontend/ static serving)
trash-calendar/             Reads Kennelbach's personal trash-collection ICS feed,
                             writes matching events onto the shared CalDAV calendar
                             (daily systemd timer, read-only against the feed)
transit/                    Public transit: GET /departures (static GTFS lookup)
                             and GET /plan (OpenTripPlanner proxy, voice-usable via
                             HA Assist) — weekly GTFS refresh via systemd timer
chores/                      Presence/calendar-driven household chore nudging + a
                             passive fairness tally + camera-verified trash-bin/
                             dishes/litter checks (systemd-timed oneshot, no
                             long-lived service, no LLM-picked assignment)

Status

• Project plan drafted
• Container host setup script v1 (HA, Mosquitto, Zigbee2MQTT USB, Frigate, Grocy)
• Node-RED + monitoring (Netdata) + dashboard (Homepage) + ntfy + Portainer added to compose stack
• Backup (restic) setup — scripted, off by default until a backup target is picked (ENABLE_BACKUPS)
• Bermuda / ESPHome BLE proxy configs — firmware/esphome-ble-proxy/ built (stock ESPHome bluetooth_proxy component), not yet flashed to real hardware, see that directory's README
• RuView node configs — firmware/ruview/ documents the real upstream project (github.com/ruvnet/ruview, integrated not forked) + a per-room provisioning wrapper + automations.yaml.example (sleep → dim lights, possible-distress → whole-household alert, concurrent elevated heart rate → colored lighting, bathroom occupancy → an external door indicator). Every automation's entity_id is an unconfirmed placeholder, and the concurrent-two-person-heart-rate rule rests on an unconfirmed assumption about RuView's multi-target vital-sign capability — see firmware/ruview/README.md §5–6 and docs/project-plan.md open decisions #32–33 before relying on any of it
• Frigate peephole camera config (real RTSP details)
• Grocy kiosk (Pi + touchscreen) setup
• LLM host (Ollama) setup script — hosts/llm-host/ written: auto-detects a GPU vs CPU-only tier, runs Ollama as a pinned container (no curl | sh into root), pulls the Phase 3 text model + a vision model for pantry-vision/chores, and takes a reasoned position on Assist-vs-digest GPU contention (MAX_LOADED_MODELS=1 — swap predictably rather than OOM mid-request). Never run — no Debian machine, no GPU, no Ollama server; the script is syntax-checked and its generated compose validated as YAML for both tiers, and that's the whole of the testing. The vision-model pick is still unmade (open decision #18), and the contention defaults are reasoned, not measured (#4) — see hosts/llm-host/README.md
• CalDAV / Nextcloud calendar integration notes — docs/caldav-integration.md: the four independent clients (digest-engine read-only, chores' busy-check read-only, trash-calendar create-only, HA's own bridge), why they share one Nextcloud app password and what that costs (rotation touches three env files; the read-only invariant is a code property, not a permission boundary — an app password can't be scoped read-only or to one calendar), trash-calendar's UID-prefix ownership invariant, and the two traps that cost real debugging time (unexpanded recurrence reporting a meeting on the day it was created; CALDAV_VERIFY_TLS=false). Nextcloud itself is pre-existing — nothing here deploys it
• Sway thin-client ISO (live-build) + thinclient-agent — built, not yet boot-tested on real hardware; RDP replaced by wayvnc (resolved), remaining open items (mic-enabled rooms, exact hardware target, wayvnc password provisioning) in docs/project-plan.md §4
• Thin-client follow-ups in progress: fullscreen-aware now-playing widget (cover art + controls), minimal Firefox chrome + uBlock Origin/SponsorBlock, persistent audio-output selection, outbound RDP/VNC client (rdp-vnc.json), HA mobile-app browser remote control (text input + mouse buttons), capture-card ("receiver box") video source selection on a new 5:capture workspace, idle-gallery weather/clock overlay (clock always, weather via a new smarthome/weather/current MQTT topic an HA automation has to publish) — built, not yet tried against real capture-card hardware or a real weather automation, see hosts/thin-client/README.md
• Quarter-daily digest engine (mail/Signal/Telegram/Discord/WhatsApp, news, financial ingestion; LLM synthesis; digest-canvas SDK) — built and wired into setup-container-host.sh (ENABLE_DIGEST_ENGINE, off by default), not yet run against real credentials; household/calendar ingest (CalDAV/Grocy) still needs a real data source wired in, see docs/project-plan.md §4
• admin-canvas + admin-web (sys-admin-llm on-demand display surface for the thin clients) — built and wired into setup-container-host.sh (ENABLE_ADMIN_CANVAS, off by default); the HA-side tool/rest_command wiring and the specific entities it surfaces (e.g. power-monitoring) are still undecided, see docs/project-plan.md §4
• ESP32-S3-Touch-LCD-1.85C-V2 voice satellite + status display (firmware/esp32-s3-touch-lcd-1.85c/) — ESPHome config written and passes esphome config, not yet flashed to real hardware; media_player/weather entity IDs still need to be chosen, see docs/project-plan.md §4
• Headless audio endpoint (hosts/audio-endpoint/) — per-room independent Spotify Connect appliance for rooms without a thin client, arm64 (Raspberry Pi + HiFiBerry Amp2, rpi-image-gen) and amd64 (mini PC + USB DAC/amp, live-build) build pipelines written, neither built/flashed/booted on real hardware — rpi-image-gen's exact config schema in particular is unverified, see hosts/audio-endpoint/README.md
• Sway touch panel (hosts/touch-panel/) — touch-driven Sway image: full Spotify GUI (Flathub), a dedicated Home Assistant Chromium kiosk window, a general web browser, an always-on touch dock for app switching, an on-screen keyboard (toggled manually, no auto-show), and touchpanel-agent (HA MQTT control, same LLM-mediated-through-HA security model as the thin client) — built, no touch-panel hardware chosen and nothing booted on real metal, see hosts/touch-panel/README.md
• Kitchen/fridge display + pantry-vision (hosts/kitchen-display/, pantry-vision/) — hold a grocery item up to the camera, an Ollama vision model proposes what it is and roughly how long it keeps, a human confirms (never auto-committed) before it's written into Grocy stock; the display then shows inventory sorted by soonest-to-expire, groceries running low, and Grocy's recipes — built and wired into setup-container-host.sh (ENABLE_PANTRY_VISION, off by default), nothing run against a real camera, vision model, or Grocy instance — the Grocy API call shapes in particular are written from documentation only, see pantry-vision/README.md and hosts/kitchen-display/README.md
• identity + door panel (identity/, hosts/door-panel/) — the person <-> BLE-identifier registry: "register me as <name>" by voice or touchscreen, multi-phone support (multiple identifiers per person), anti-spoofing (only allowlisted IRK-resolved/fixed-tag entities are ever accepted as candidates, never a raw MAC), device-less people (a "no device" flag plus a hand-operated Home/Away toggle — the concrete case: a grandmother without a smartphone), and an anonymous "Guest" path. Backs hosts/door-panel/'s weather+clothing/who's-home/groceries-running-low dashboard and hosts/kitchen-display/'s "Show registration" screen — built and wired into setup-container-host.sh (ENABLE_IDENTITY, off by default), nothing run against a real HA instance, real Private BLE Device entities, or a real voice pipeline — TRUSTED_ENTITY_PREFIXES above all needs checking against Developer Tools -> States, see identity/README.md and hosts/door-panel/README.md
• identity also corroborates presence from Frigate face recognition (Phase 20, Tapo pan/tilt cameras) — an OR-ed-in second signal only, never a registration signal; and owns the per-person chore-system settings (chore_exempt, chore_reminder_style, plus chore assignment) consumed by chores/, see identity/README.md
• identity's admin panel (identity/frontend/admin.html, Phase 6b) — managing people/guests: edit every field, nicknames (an input alias only — /resolve accepts them, but the assistant always speaks the real speak_name), visit history sampled from /presence plus a derived "who was home with whom" view, "select all that last visited before <date>" pruning (the filter selects, a human confirms the exact list, the filter is never re-run at delete time), per-device rights for self-entry via a smart lock (identity only ever answers GET /device-access — HA asks and HA acts, deny is the default), chore assignment, a floorplan editor (draw levels and rooms as polygons, map each to an HA area, and watch occupied rooms light up — resolves the long-deferred open decision #22), and opt-in arrival push notifications ("tell me when someone gets home", via the self-hosted ntfy this stack already runs — identity itself never touches the WAN; ntfy stays LAN-only and remote delivery rides a WireGuard split tunnel — see docs/network-integration.md §2.2 for why a DMZ/port-forward was weighed and rejected). Deliberately not a kiosk page and not linked from any wall panel. Covered by API-level tests; never opened in a real browser, and DEPARTURE_GRACE_SECONDS is an untuned guess — see identity/README.md
• trash-calendar + transit (Phase 19, Kennelbach AT trash pickup + Vorarlberg public transit) — built and wired into setup-container-host.sh (ENABLE_TRASH_CALENDAR/ENABLE_TRANSIT/ENABLE_TRIP_PLANNING, all off by default), nothing run against a live ICS feed, a live GTFS feed, or a real OpenTripPlanner instance — trip planning also needs a manually-built OTP graph this repo does not build for you, see trash-calendar/README.md and transit/README.md's "Route planning scope"
• chores (Phase 20) — presence/calendar-driven household chore nudging: "I don't care who does it, as long as it gets done" — prefers whoever's been assigned a chore in identity's admin panel but falls through to whoever's actually home rather than waiting (CHORE_ASSIGNMENT_STRICT flips that), redirects to someone else if a chore goes neglected, keeps a passive fairness tally that never feeds back into who gets nudged, and camera-checks trash bins/dishes/litter via Frigate + an Ollama vision model. Litter remains the exception to everything — it ignores both chore-exemption and assignment, because cleaning up what you left out was never a task anyone could be assigned. Built and wired into setup-container-host.sh (ENABLE_CHORES, off by default, every-2-hours systemd timer), no Tapo camera hardware chosen and nothing run against real hardware, see chores/README.md
• Music Assistant (optional, additive multi-room audio) — wired into setup-container-host.sh (ENABLE_MUSIC_ASSISTANT, off by default), its default port is an unverified guess that collides with PANTRY_VISION_PORT if both are enabled together, see docs/project-plan.md open decision #31
• docs/network-integration.md (OPNsense VLAN segmentation, the WireGuard split tunnel that carries arrival notifications, and why nothing here — ntfy included — gets port-forwarded to the WAN) — written, not run against a real OPNsense instance
• tools/ + CoreSystemConfig.json — every build and setup script in one place, reading one config. The container host and LLM host build as a twinned pair: you set two last octets and the container host's OLLAMA_HOST is derived from the LLM host's, so the two ISOs cannot be built disagreeing about where the other one is; every kiosk's service URLs derive from the container host's address the same way. build-all.sh builds the set, validate-config.py refuses a build on duplicate ports (the music_assistant/pantry_vision 8095 clash, open decision #31), placeholder or padded tokens, duplicate hostnames, or a kiosk pointed at a disabled service. All secrets are burnt into the images so installs are unattended — which makes every ISO a credential; the filled-in config and iso-out/ are gitignored. No ISO has ever been built with this (lb build needs live-build, root and a long fetch) — what is tested is config validation/derivation and every generated artifact, with lb stubbed. See tools/README.md

Quick start

Everything is built from one config file — CoreSystemConfig.json — so an address or token can only ever be wrong in one place:

cp CoreSystemConfig.json.template CoreSystemConfig.json
$EDITOR CoreSystemConfig.json          # subnet, two host octets, tokens, your kiosks
tools/validate-config.py               # catches typos in seconds, not after a 40-min build
sudo -E tools/build-all.sh             # every ISO, all agreeing with each other

The container host and LLM host come out as a twinned pair: you give each a last octet, and the container host's OLLAMA_HOST is derived from the LLM host's — change one and the other follows on the next build, with nothing to keep in sync by hand. Every kiosk's service URLs derive from the container host's address the same way. See tools/README.md.

Built ISOs contain every secret in the config, by design (nothing to configure post-install) — which makes each one a credential. .gitignore covers both the filled-in config and iso-out/.

To set up a host by hand instead of from an ISO:

sudo -E tools/setup-container-host.sh

Edit the variables at the top of the script first (timezone, Zigbee USB device path, Mealie/hardware-accel toggles, and whether to enable Node-RED/Netdata/ Homepage/ntfy/Portainer and restic backups — all but backups are on by default). See docs/project-plan.md for the full phased rollout order — don't skip straight to Phase 4+ automations before the Phase 2 reactive baseline (presence -> light, no LLM in the loop) is working.

The script brings up everything that runs on this one Debian host: Home Assistant, Mosquitto, Zigbee2MQTT, Node-RED, Frigate, Grocy, Netdata, a Homepage dashboard, ntfy, and Portainer, plus an optional scheduled restic backup timer and an optional quarter-daily digest engine (ENABLE_DIGEST_ENGINE, off by default — needs digest-engine/ checked out on the host and its .env filled in first, see digest-engine/README.md). What it can't do for you, because they need separate hardware, credentials, or physical setup: pairing Zigbee sensors, flashing RuView/ESPHome/Bermuda BLE proxy boards, pointing Frigate at a real camera RTSP URL, the Grocy kiosk touchscreen, the separate LLM/GPU host, wiring up the Nextcloud CalDAV integration, building/flashing the thin-client ISO (hosts/thin-client/), and provisioning real credentials for the digest engine's mail/message/news/financial sources — see the Status checklist above and docs/project-plan.md for those.