12 KiB
12 KiB
AI-Managed Smart Home — Full Build Plan (v2)
Local-first, open-source stack: Home Assistant + RuView presence + Bermuda BLE identity + local LLM (Ollama) + Zigbee sensors/lighting + Frigate (peephole face recognition) + Grocy kitchen kiosk + Nextcloud calendar sync + Node-RED glue logic.
1. Hardware
1.1 Already in your possession
| Item | Use in this build |
|---|---|
| Framework 12 (primary) | Dev machine, dotfiles/Claude Code work |
| ThinkPad T440p (secondary) | Spare compute, webcam presence daemon testing |
| Nextcloud instance | Calendar backend (CalDAV sync target) |
| "Scraps" x86 machine | Candidate for LLM host (CPU-only tier) or Proxmox host |
| Haozee Zigbee USB dongle (CC2652P + CH340C) | Zigbee coordinator — USB passthrough into container, adapter type zstack |
1.2 Core AI / HA hosts
| Item | Est. Price (EUR) | Notes |
|---|---|---|
| Mini PC / SFF for Home Assistant + attached services (Raspberry Pi 5, 8GB) | €80 | Runs HA Core (Container), Mosquitto, Zigbee2MQTT, Frigate, Grocy, Node-RED, monitoring |
| SSD for boot (USB3, 128GB+) | €20 | Avoid SD card for reliability |
| Official Pi 5 PSU + case | €25 | |
| Used RTX 3060 12GB | €200–250 | LLM inference host (Qwen2.5-14B Q4) |
| Cheap SFF/tower host for GPU (spare PCIe x16 slot, 170W+ on 12V rail) | €40–60 | Only needs to feed the GPU |
| (Fallback: skip GPU) Use existing "scraps" x86 box, CPU-only Ollama | €0 | Slower (5–10 tok/s), fine to validate before buying GPU |
Subtotal (with GPU tier): ~€365–435 | Subtotal (CPU-only fallback): ~€125
1.3 Zigbee backbone
Already covered — using your existing Haozee CC2652P USB dongle. No coordinator purchase needed. Contingency: if Zigbee2MQTT can't establish a network, the dongle may need CC2652P coordinator firmware reflashed (generic-branded gateways sometimes ship with non-coordinator firmware).
1.4 Zigbee sensors (price per unit — multiply by room count)
| Item | Est. Price/unit | Suggested qty | Notes |
|---|---|---|---|
| innr RB 285 C (Zigbee RGBWW bulb) | €15–20 | 1 per AI-managed room | Tunable 2200K–6500K, acts as Zigbee router |
| Sonoff SNZB-02P (temp/humidity) | €10–12 | 1 per room | Avoid older Aqara — mesh drop issues |
| Sonoff SNZB-06P (presence/illuminance) or Aqara Light Sensor | €12–15 | 1 per room needing lux data | Feeds LLM brightness decisions |
| Sonoff SNZB-04P (door/window contact) | €13–15 | 1 per door/window of interest | Gates automations |
| IKEA Vindstyrka (air quality, optional) | ~€30 | 1–2 | Requires Zigbee2MQTT |
1.5 BLE identity layer (Bermuda)
| Item | Est. Price/unit | Suggested qty | Notes |
|---|---|---|---|
| ESP32 board (D1 Mini32 or similar — no CSI needed, just BT+WiFi) | €5–10 | 1 per room for identity resolution | Runs ESPHome bluetooth_proxy, separate hardware from RuView nodes |
| Fixed-MAC BLE tag per household member (optional but recommended) | €5–8 | 1 per person | More reliable anchor than phone MAC (which randomizes on iOS/Android); Bermuda + HA's Private BLE Device can also resolve phones via IRK if you skip this |
1.6 RuView presence mesh
| Item | Est. Price/unit | Suggested qty | Notes |
|---|---|---|---|
| ESP32-S3 dev board | €8–12 | 1 per room/zone needing CSI presence | Runs RuView firmware — separate boards from the Bermuda BLE proxies above (one chip = one firmware) |
1.7 Door-spy / peephole camera
| Item | Est. Price | Notes |
|---|---|---|
| Peephole IP camera (ONVIF/RTSP, color, "-IR" variant only if hallway is dark) | €50–65 | Isolate on its own VLAN, block WAN egress |
| (You're handling the physical mounting yourself) | €0 |
1.8 Frigate acceleration (optional)
| Item | Est. Price | Notes |
|---|---|---|
| Hailo-8L (Raspberry Pi 5 AI kit) | €70 | Only if CPU/iGPU isn't keeping up with more cameras later |
| (Default: none needed for 1 camera — use CPU or Intel iGPU/OpenVINO) | €0 |
1.9 Kitchen inventory kiosk (camera + touchscreen)
| Item | Est. Price | Notes |
|---|---|---|
| Raspberry Pi 3B+ or 4 (2–4GB) | €35–50 | Chromium kiosk + camera decode |
| 7" official/compatible touchscreen | €30–40 | |
| USB webcam w/ macro focus | €10–15 | Barcode scanning via Grocy's browser PWA |
| Case/mount for kitchen wall or cupboard | €10–15 |
Subtotal: ~€85–120
1.10 Deferred / out of scope for now
- Projector + camera tabletop AI surface — genuinely interesting but high-effort (projector-camera calibration, fingertip touch detection is an active CV research problem, poor legibility with ambient light). If pursued, treat as an isolated experimental side project, not wired into critical automations. Not included in this plan's budget or phases.
2. Software (all open source / self-hosted)
| Layer | Software | Purpose |
|---|---|---|
| Home automation core | Home Assistant (Container install) | Central hub, automations, dashboards |
| Zigbee | Zigbee2MQTT + Mosquitto | Broadest device compatibility, USB passthrough to your CC2652P dongle |
| BLE identity | Bermuda (HACS) + ESPHome (bluetooth_proxy) + HA's Private BLE Device |
Room-level BLE presence, resolves randomized MACs via IRK, feeds person.* entities |
| Automation glue | Node-RED | Visual flows for cross-system logic — home for the face↔MAC identity-correlation logic, easy iteration/debugging |
| LLM runtime | Ollama | Serves Qwen2.5-14B-Instruct (GPU tier) or Qwen2.5-7B/3B (CPU tier) |
| Conversation agent | HA Ollama conversation integration | Ties LLM into Assist + AI Task |
| Voice STT/TTS | Wyoming faster-whisper + Piper | Local speech pipeline |
| Presence (CSI) | RuView (ESP32-S3 firmware + server) | Anonymous room-level presence, feeds gating automations |
| Presence (webcam) | Your custom daemon (dotfiles repo) | Desktop-specific, not tied to HA sensors directly |
| Camera NVR / face recognition | Frigate (0.16+) | Native face recognition, object detection, peephole cam ingest |
| Identity store | Small dict/table (SQLite or JSON) — name, face label, associated MACs, confidence | Built up conversationally via LLM tool calls (propose_person_link, confirm_person, rename_person); merges require confirmation, never silent |
| Container runtime | Docker + Docker Compose | Hosts all of the above |
| Inventory / shopping list | Grocy | Self-hosted stock + shopping list, native browser barcode scanning |
| Meal planning (optional) | Mealie | Recipe manager + shopping list + meal-plan calendar |
| Calendar backend | Nextcloud Calendar (already running) | Source of truth for household schedule |
| Calendar bridge | HA CalDAV integration | Read/write bridge, calendar.create_event supported; keep recurring events created in Nextcloud directly |
| Monitoring | Netdata | Consistent with your existing Ansible-fleet monitoring evaluation |
| Dashboard | Homepage or Homarr | Single landing page for HA / Grocy / Frigate / Zigbee2MQTT / Node-RED UIs |
| Container management | Portainer (optional) | GUI over the Docker host |
| Notifications | ntfy | Self-hosted push, replaces relying on a phone OS's proprietary push channel |
| Long-term stats (optional) | InfluxDB + Grafana | Only if you want history beyond HA's default recorder retention |
| Backup | restic | Scheduled encrypted backups of all stateful volumes |
| Kiosk browser | Chromium (kiosk mode) | Displays Grocy PWA on the kitchen touchscreen |
3. Implementation Plan
Phase 0 — Infrastructure prep
- Decide: Proxmox host (if virtualizing) vs. bare-metal. Given your homelab habits, Proxmox with HAOS/container-host as VMs/LXCs is reasonable.
- Set up a dedicated camera VLAN (peephole cam + future cameras), no WAN egress, firewalled from the main LAN.
- Provision the container host (Debian 12 or Raspberry Pi OS Lite).
Phase 1 — Core containers + Zigbee
- Docker + Compose stack: Home Assistant, Mosquitto, Zigbee2MQTT (USB passthrough to the Haozee dongle), Node-RED.
- Pair Zigbee sensors room by room: bulb → temp/humidity → illuminance → door/window contact.
- Confirm entities populate correctly in HA.
Phase 2 — Presence: RuView + Bermuda, in parallel
- Deploy RuView ESP32-S3 nodes per room (dedicated CSI firmware).
- Separately, flash plain ESP32 boards with ESPHome (
bluetooth_proxy), install Bermuda via HACS, configure Private BLE Device for phone IRK resolution, and/or distribute fixed-MAC BLE tags per person. - Build plain HA automations: presence (RuView) on → light on at neutral default; off (with delay) → light off. Validate this works with the LLM host powered off — this is your safety-net baseline.
Phase 3 — LLM host + conversation agent
- Stand up the GPU host (or CPU fallback) with Ollama, pull Qwen2.5-14B-Instruct (or 7B/3B for CPU).
- Add the Ollama integration in HA, point it at the LLM host over LAN.
- Set up Wyoming faster-whisper + Piper, configure an Assist pipeline.
- Test open-ended conversation and basic tool-calling before wiring into presence logic.
Phase 4 — AI-managed brightness/color
- Script triggered after the presence-automation light-on event, calling
ai_task.generate_datawith room/time/occupancy context, requesting structured JSON (brightness,rgb_color). - Apply via
light.turn_on; fallback to neutral default on timeout/error. - Feed illuminance sensor data into the LLM's exposed-entity list.
Phase 5 — Peephole camera + Frigate
- Mount your color camera (your own work), isolate on the camera VLAN.
- Deploy Frigate, add the RTSP stream, enable native face recognition.
- Enroll known faces; automation for known vs. unknown at the door.
- Set conservative confidence thresholds given the narrow FOV/low light.
Phase 6 — Identity correlation (face ↔ MAC ↔ name)
- Build the correlation logic as a Node-RED flow first: listen to Frigate face-recognition MQTT events + Bermuda/BLE MQTT topics, tally co-occurrence, apply a confidence threshold.
- Expose LLM tool calls (
propose_person_link,confirm_person,rename_person) so the LLM can name people conversationally and propose merges — never auto-commit a merge silently, require confirmation. - Once the logic stabilizes, consider porting to a small Python service if it outgrows Node-RED's comfort zone (git/test-ability).
Phase 7 — Kitchen inventory kiosk
- Deploy Grocy via Compose.
- Pi + touchscreen running Chromium kiosk mode pointed at Grocy's PWA.
- Attach USB webcam, test in-browser barcode scanning (Open Food Facts lookup).
- Use Grocy's own add/consume UI — no extra hardware/logic needed for the "in vs. out" problem.
Phase 8 — Nextcloud calendar sync
- Add HA's CalDAV integration, point at Nextcloud's CalDAV URL.
- Confirm read + write (
calendar.create_event) both work. - Expose the calendar entity to the LLM's tool list.
- Gate delete/move actions behind a confirmation step; create recurring events directly in Nextcloud, not via HA.
Phase 9 — Observability, dashboard, backup
- Deploy Netdata (or your team's eventual choice from the Netdata/Checkmk evaluation) for container/host monitoring.
- Deploy Homepage/Homarr as a single landing page across all service UIs.
- Deploy ntfy for local push notifications (door alerts, automation failures).
- Set up restic scheduled backups of all stateful volumes (HA config, Zigbee2MQTT device DB, Grocy data, Frigate face embeddings, Node-RED flows).
Phase 10 — Optional expansion
- Mealie for meal-plan-aware shopping lists.
- InfluxDB + Grafana if you want longer-term historical dashboards than HA's recorder retains.
- Additional RuView/Bermuda nodes as more rooms are covered.
- Dedicated Frigate accelerator (Hailo-8L) only if you expand beyond 1–2 cameras.
Testing checklist before calling any phase "done"
- Does the reactive path (presence → light on) work with the LLM host powered off? (It must.)
- Does a bad/slow LLM response ever block a light switch? (It must not.)
- Are cameras verified to have zero WAN egress?
- Does the calendar integration survive a Nextcloud restart without orphaning entities?
- Does an identity merge ever get auto-committed without confirmation? (It must not.)