102 lines
8.6 KiB
Markdown
102 lines
8.6 KiB
Markdown
# AI-Managed Smart Home
|
|
|
|
Local-first, open-source smart home: Home Assistant + Zigbee + RuView (CSI presence)
|
|
+ Bermuda (BLE identity) + local LLM (Ollama) + Frigate (peephole face recognition)
|
|
+ Grocy (kitchen kiosk) + Nextcloud calendar sync + a Sway thin-client media station
|
|
+ a Sway touch panel + a camera-vision kitchen/fridge display + a quarter-daily
|
|
LLM-generated digest.
|
|
|
|
See [`docs/project-plan.md`](docs/project-plan.md) for the full hardware list,
|
|
software stack, and phased implementation plan.
|
|
|
|
## Repo layout
|
|
|
|
```
|
|
docs/ Project plan and design notes
|
|
hosts/
|
|
container-host/ Docker Compose stack: HA, Mosquitto, Zigbee2MQTT,
|
|
Frigate, Grocy, Node-RED, monitoring, etc.
|
|
configs/ Per-service config files (mosquitto.conf, etc.)
|
|
scripts/ Host setup / bootstrap scripts
|
|
llm-host/ Ollama + GPU host setup (separate physical machine)
|
|
thin-client/ Sway kiosk/media-station ISO (live-build) + thinclient-agent
|
|
audio-endpoint/ Headless Spotify Connect appliance for rooms with no thin
|
|
client — arm64 (rpi-image-gen) + amd64 (live-build) images
|
|
touch-panel/ Touch-driven Sway panel: full Spotify GUI, a dedicated
|
|
Home Assistant Chromium kiosk window, and a general web
|
|
browser, switched via an always-on touch dock or by HA/
|
|
the local LLM over MQTT
|
|
kitchen-display/ Single-purpose Sway kiosk for the fridge/pantry: one
|
|
Chromium window showing pantry-vision's Scan/Inventory/
|
|
Recipes frontend, camera capture via the browser itself
|
|
firmware/
|
|
ruview/ RuView ESP32-S3 CSI presence node configs
|
|
esphome-ble-proxy/ ESPHome configs for Bermuda BLE proxy nodes
|
|
esp32-s3-touch-lcd-1.85c/ ESPHome voice satellite + status display (round LCD,
|
|
media/cover-art priority over an idle weather/time/
|
|
date cycle, voice-state visualizer)
|
|
identity/ Face<->MAC<->name correlation logic (Node-RED flow
|
|
export once stabilized, or a Python service)
|
|
digest-engine/ Quarter-daily LLM digest: mail/message/news/financial
|
|
ingestion, LLM synthesis, digest-canvas SDK rendering
|
|
admin-canvas/ On-demand sys-admin-llm display surface for the thin
|
|
clients: stats/graphics/media, pushed on demand
|
|
rather than on a schedule (write API + admin-web
|
|
static serving)
|
|
pantry-vision/ Kitchen-display backend: a photo held up to the camera
|
|
-> an Ollama vision-model proposal -> human-confirmed
|
|
write into Grocy stock; also proxies Grocy's inventory
|
|
(soonest-expiring first) and recipes to the kiosk
|
|
frontend (write API + frontend/ static serving)
|
|
```
|
|
|
|
## Status
|
|
|
|
- [x] Project plan drafted
|
|
- [x] Container host setup script v1 (HA, Mosquitto, Zigbee2MQTT USB, Frigate, Grocy)
|
|
- [x] Node-RED + monitoring (Netdata) + dashboard (Homepage) + ntfy + Portainer added to compose stack
|
|
- [x] Backup (restic) setup — scripted, off by default until a backup target is picked (`ENABLE_BACKUPS`)
|
|
- [ ] Bermuda / ESPHome BLE proxy configs
|
|
- [ ] RuView node configs
|
|
- [ ] Frigate peephole camera config (real RTSP details)
|
|
- [ ] Grocy kiosk (Pi + touchscreen) setup
|
|
- [ ] LLM host (Ollama) setup script
|
|
- [ ] CalDAV / Nextcloud calendar integration notes
|
|
- [ ] Identity correlation flow (Node-RED)
|
|
- [x] Sway thin-client ISO (live-build) + thinclient-agent — built, not yet boot-tested on real hardware; RDP replaced by wayvnc (resolved), remaining open items (mic-enabled rooms, exact hardware target, wayvnc password provisioning) in `docs/project-plan.md` §4
|
|
- [ ] Thin-client follow-ups in progress: fullscreen-aware now-playing widget (cover art + controls), minimal Firefox chrome + uBlock Origin/SponsorBlock, persistent audio-output selection, outbound RDP/VNC client (`rdp-vnc.json`), HA mobile-app browser remote control (text input + mouse buttons), capture-card ("receiver box") video source selection on a new `5:capture` workspace, idle-gallery weather/clock overlay (clock always, weather via a new `smarthome/weather/current` MQTT topic an HA automation has to publish) — built, not yet tried against real capture-card hardware or a real weather automation, see `hosts/thin-client/README.md`
|
|
- [x] Quarter-daily digest engine (mail/Signal/Telegram/Discord/WhatsApp, news, financial ingestion; LLM synthesis; digest-canvas SDK) — built and wired into `setup-container-host.sh` (`ENABLE_DIGEST_ENGINE`, off by default), not yet run against real credentials; household/calendar ingest (CalDAV/Grocy) still needs a real data source wired in, see `docs/project-plan.md` §4
|
|
- [x] admin-canvas + admin-web (sys-admin-llm on-demand display surface for the thin clients) — built and wired into `setup-container-host.sh` (`ENABLE_ADMIN_CANVAS`, off by default); the HA-side tool/rest_command wiring and the specific entities it surfaces (e.g. power-monitoring) are still undecided, see `docs/project-plan.md` §4
|
|
- [ ] ESP32-S3-Touch-LCD-1.85C-V2 voice satellite + status display (`firmware/esp32-s3-touch-lcd-1.85c/`) — ESPHome config written and passes `esphome config`, not yet flashed to real hardware; `media_player`/`weather` entity IDs still need to be chosen, see `docs/project-plan.md` §4
|
|
- [ ] Headless audio endpoint (`hosts/audio-endpoint/`) — per-room independent Spotify Connect appliance for rooms without a thin client, arm64 (Raspberry Pi + HiFiBerry Amp2, rpi-image-gen) and amd64 (mini PC + USB DAC/amp, live-build) build pipelines written, **neither built/flashed/booted on real hardware** — rpi-image-gen's exact config schema in particular is unverified, see `hosts/audio-endpoint/README.md`
|
|
- [ ] Sway touch panel (`hosts/touch-panel/`) — touch-driven Sway image: full Spotify GUI (Flathub), a dedicated Home Assistant Chromium kiosk window, a general web browser, an always-on touch dock for app switching, an on-screen keyboard (toggled manually, no auto-show), and `touchpanel-agent` (HA MQTT control, same LLM-mediated-through-HA security model as the thin client) — built, **no touch-panel hardware chosen and nothing booted on real metal**, see `hosts/touch-panel/README.md`
|
|
- [ ] Kitchen/fridge display + `pantry-vision` (`hosts/kitchen-display/`, `pantry-vision/`) — hold a grocery item up to the camera, an Ollama vision model proposes what it is and roughly how long it keeps, a human confirms (never auto-committed) before it's written into Grocy stock; the display then shows inventory sorted by soonest-to-expire and Grocy's recipes — built and wired into `setup-container-host.sh` (`ENABLE_PANTRY_VISION`, off by default), **nothing run against a real camera, vision model, or Grocy instance** — the Grocy API call shapes in particular are written from documentation only, see `pantry-vision/README.md` and `hosts/kitchen-display/README.md`
|
|
|
|
## Quick start
|
|
|
|
```bash
|
|
cd hosts/container-host/scripts
|
|
sudo ./setup-container-host.sh
|
|
```
|
|
|
|
Edit the variables at the top of the script first (timezone, Zigbee USB device
|
|
path, Mealie/hardware-accel toggles, and whether to enable Node-RED/Netdata/
|
|
Homepage/ntfy/Portainer and restic backups — all but backups are on by
|
|
default). See `docs/project-plan.md` for the full phased rollout order — don't
|
|
skip straight to Phase 4+ automations before the Phase 2 reactive baseline
|
|
(presence -> light, no LLM in the loop) is working.
|
|
|
|
The script brings up everything that runs on this one Debian host: Home
|
|
Assistant, Mosquitto, Zigbee2MQTT, Node-RED, Frigate, Grocy, Netdata, a
|
|
Homepage dashboard, ntfy, and Portainer, plus an optional scheduled restic
|
|
backup timer and an optional quarter-daily digest engine (`ENABLE_DIGEST_ENGINE`,
|
|
off by default — needs `digest-engine/` checked out on the host and its `.env`
|
|
filled in first, see `digest-engine/README.md`). What it can't do for you,
|
|
because they need separate hardware, credentials, or physical setup: pairing
|
|
Zigbee sensors, flashing RuView/ESPHome/Bermuda BLE proxy boards, pointing
|
|
Frigate at a real camera RTSP URL, the Grocy kiosk touchscreen, the separate
|
|
LLM/GPU host, wiring up the Nextcloud CalDAV integration, building/flashing
|
|
the thin-client ISO (`hosts/thin-client/`), and provisioning real credentials
|
|
for the digest engine's mail/message/news/financial sources — see the Status
|
|
checklist above and `docs/project-plan.md` for those.
|