diff --git a/README.md b/README.md index 607c035..3676a03 100644 --- a/README.md +++ b/README.md @@ -3,7 +3,8 @@ Local-first, open-source smart home: Home Assistant + Zigbee + RuView (CSI presence) + Bermuda (BLE identity) + local LLM (Ollama) + Frigate (peephole face recognition) + Grocy (kitchen kiosk) + Nextcloud calendar sync + a Sway thin-client media station -+ a quarter-daily LLM-generated digest. ++ a Sway touch panel + a camera-vision kitchen/fridge display + a quarter-daily +LLM-generated digest. See [`docs/project-plan.md`](docs/project-plan.md) for the full hardware list, software stack, and phased implementation plan. @@ -21,6 +22,13 @@ hosts/ thin-client/ Sway kiosk/media-station ISO (live-build) + thinclient-agent audio-endpoint/ Headless Spotify Connect appliance for rooms with no thin client — arm64 (rpi-image-gen) + amd64 (live-build) images + touch-panel/ Touch-driven Sway panel: full Spotify GUI, a dedicated + Home Assistant Chromium kiosk window, and a general web + browser, switched via an always-on touch dock or by HA/ + the local LLM over MQTT + kitchen-display/ Single-purpose Sway kiosk for the fridge/pantry: one + Chromium window showing pantry-vision's Scan/Inventory/ + Recipes frontend, camera capture via the browser itself firmware/ ruview/ RuView ESP32-S3 CSI presence node configs esphome-ble-proxy/ ESPHome configs for Bermuda BLE proxy nodes @@ -35,6 +43,11 @@ admin-canvas/ On-demand sys-admin-llm display surface for the thin clients: stats/graphics/media, pushed on demand rather than on a schedule (write API + admin-web static serving) +pantry-vision/ Kitchen-display backend: a photo held up to the camera + -> an Ollama vision-model proposal -> human-confirmed + write into Grocy stock; also proxies Grocy's inventory + (soonest-expiring first) and recipes to the kiosk + frontend (write API + frontend/ static serving) ``` ## Status @@ -56,6 +69,8 @@ admin-canvas/ On-demand sys-admin-llm display surface for the thin - [x] admin-canvas + admin-web (sys-admin-llm on-demand display surface for the thin clients) — built and wired into `setup-container-host.sh` (`ENABLE_ADMIN_CANVAS`, off by default); the HA-side tool/rest_command wiring and the specific entities it surfaces (e.g. power-monitoring) are still undecided, see `docs/project-plan.md` §4 - [ ] ESP32-S3-Touch-LCD-1.85C-V2 voice satellite + status display (`firmware/esp32-s3-touch-lcd-1.85c/`) — ESPHome config written and passes `esphome config`, not yet flashed to real hardware; `media_player`/`weather` entity IDs still need to be chosen, see `docs/project-plan.md` §4 - [ ] Headless audio endpoint (`hosts/audio-endpoint/`) — per-room independent Spotify Connect appliance for rooms without a thin client, arm64 (Raspberry Pi + HiFiBerry Amp2, rpi-image-gen) and amd64 (mini PC + USB DAC/amp, live-build) build pipelines written, **neither built/flashed/booted on real hardware** — rpi-image-gen's exact config schema in particular is unverified, see `hosts/audio-endpoint/README.md` +- [ ] Sway touch panel (`hosts/touch-panel/`) — touch-driven Sway image: full Spotify GUI (Flathub), a dedicated Home Assistant Chromium kiosk window, a general web browser, an always-on touch dock for app switching, an on-screen keyboard (toggled manually, no auto-show), and `touchpanel-agent` (HA MQTT control, same LLM-mediated-through-HA security model as the thin client) — built, **no touch-panel hardware chosen and nothing booted on real metal**, see `hosts/touch-panel/README.md` +- [ ] Kitchen/fridge display + `pantry-vision` (`hosts/kitchen-display/`, `pantry-vision/`) — hold a grocery item up to the camera, an Ollama vision model proposes what it is and roughly how long it keeps, a human confirms (never auto-committed) before it's written into Grocy stock; the display then shows inventory sorted by soonest-to-expire and Grocy's recipes — built and wired into `setup-container-host.sh` (`ENABLE_PANTRY_VISION`, off by default), **nothing run against a real camera, vision model, or Grocy instance** — the Grocy API call shapes in particular are written from documentation only, see `pantry-vision/README.md` and `hosts/kitchen-display/README.md` ## Quick start diff --git a/docs/project-plan.md b/docs/project-plan.md index d1ec2c9..e5e886f 100644 --- a/docs/project-plan.md +++ b/docs/project-plan.md @@ -1,6 +1,6 @@ # AI-Managed Smart Home — Full Build Plan (v2) -Local-first, open-source stack: Home Assistant + RuView presence + Bermuda BLE identity + local LLM (Ollama) + Zigbee sensors/lighting + Frigate (peephole face recognition) + Grocy kitchen kiosk + Nextcloud calendar sync + Node-RED glue logic + a Sway thin-client media station + a quarter-daily LLM digest. +Local-first, open-source stack: Home Assistant + RuView presence + Bermuda BLE identity + local LLM (Ollama) + Zigbee sensors/lighting + Frigate (peephole face recognition) + Grocy kitchen kiosk + Nextcloud calendar sync + Node-RED glue logic + a Sway thin-client media station + a Sway touch panel + a camera-vision kitchen/fridge display + a quarter-daily LLM digest. --- @@ -98,6 +98,25 @@ Already covered — using your existing Haozee CC2652P USB dongle. No coordinato | Passive bookshelf speakers (e.g. Micca MB42X G2) | ~€100/pair | Same pairing already recommended for the thin-client rooms | | *(amd64 alternative)* Spare x86 mini PC + USB DAC/amp (e.g. Fosi Audio V3, ~€130) | €0–130 | No GPIO HAT involved — a USB Audio Class device instead, for rooms getting a repurposed mini PC rather than a Raspberry Pi | +### 1.14 Touch panel hardware (Phase 16) +| Item | Est. Price (EUR) | Notes | +|---|---|---| +| Mini PC or SBC with a capacitive touchscreen (e.g. 10–15" USB-touch monitor + small x86 mini PC, or an all-in-one touch panel PC) | €150–300 | **No specific unit has been chosen** — the one hard requirement is that the touchscreen exposes itself as a native Wayland `wl_touch` device under libinput, not merely an emulated-mouse HID interface, since `hosts/touch-panel/` has no gesture/pointer-translation fallback the way the thin client does | +| Speakers or a wired connection to existing room audio | €0–100 | For local Spotify GUI-client playback — same speaker guidance as the thin-client/audio-endpoint rooms | + +*(No container-host changes — `touchpanel-agent` talks to the existing Mosquitto/HA +from Phase 1, same as every other MQTT-connected host in this plan.)* + +### 1.15 Kitchen/fridge display hardware (Phase 17) +| Item | Est. Price (EUR) | Notes | +|---|---|---| +| Mini PC or SBC with a touchscreen, mounted near the fridge/pantry | €150–300 | Same "must be a native Wayland `wl_touch` device" requirement as §1.14 — `hosts/kitchen-display/` has no pointer-emulation fallback either | +| USB webcam, pointed at wherever items get held up for scanning | €15–40 | Either built into the panel or on a short cable/gooseneck so its angle can be set independently of the screen. Any UVC webcam works — this is the same "no depth camera, no accelerator needed" bar as the thin client's gesture-control camera (§1.11), just used for a photo instead of continuous tracking | + +*(No new container-host hardware — `pantry-vision` is a container on the existing +Phase 1 host, calling the existing Phase 3/9 LLM host for vision inference and the +already-running `grocy` container for storage.)* + --- ## 2. Software (all open source / self-hosted) @@ -155,6 +174,21 @@ Already covered — using your existing Haozee CC2652P USB dongle. No coordinato | Headless audio endpoint (arm64) | **rpi-image-gen** (custom config, `hosts/audio-endpoint/rpi-image-gen/`) | Raspberry Pi Foundation's current custom-image tool; builds a flashable `.img` with a per-room Spotify Connect receiver + HiFiBerry Amp2 audio | | Headless audio endpoint (amd64) | **live-build** (custom config, `hosts/audio-endpoint/live-build-amd64/`) | Reuses the thin client's own build tool, stripped of the whole graphical/kiosk stack — headless boot straight to the same per-room Spotify Connect receiver | | Headless audio endpoint software | **librespot** or **spotifyd** (apt-first, documented fallback) | Independent, per-room Spotify Connect — no Snapcast, no shared/synced stream; each endpoint is its own device in Spotify's picker, same model as the thin client's own per-room spotifyd/librespot (Phase 11.6) | +| Touch-panel OS build | **live-build** (custom config, `hosts/touch-panel/live-build/`) | Reuses the thin client's own build tool and directory-split convention, not its live-build tree — a different, smaller image for a different device | +| Touch-panel compositor | **Sway** | Same choice as the thin client, configured for direct touch instead of remote control: an always-on dock instead of no bars, no gesture-camera/wayvnc machinery | +| Touch-panel scripted control | **touchpanel-agent** (custom) | HA MQTT-discovery entity + `swaymsg`/app-process control, identical security shape to `thinclient-agent` — LLM tool call → HA service call → MQTT → this agent, never a direct path | +| Touch-panel music | **Spotify (official Linux client, Flathub `com.spotify.Client`)** | Full GUI client, not a headless Connect receiver — the point of a touch panel is a screen you interact with directly (Premium required, same as the headless receivers) | +| Touch-panel home dashboard | **Chromium (kiosk mode, `--app=`)** | Dedicated, always-open window pointed at Home Assistant, auto-restarted if it crashes | +| Touch-panel browser | **Firefox** | General browsing, minimal (not kiosk-locked) chrome — back/forward/reload/address bar | +| Touch-panel on-screen keyboard | **wvkbd** (apt-first, documented fallback) | Manually toggled from the touch dock — no shell-level input-method integration on this image, so no automatic show-on-focus | +| Touch-panel navigation | **eww** (apt-first, documented fallback) | An always-on, layer-shell-reserved touch dock (Spotify / Home / Web / Keyboard) — the touch-first equivalent of the thin client's HA-only **Screen** select | +| Kitchen-display OS build | **live-build** (custom config, `hosts/kitchen-display/live-build/`) | Same toolchain/convention reuse as the touch panel, a third and simpler sibling image — one workspace, one app | +| Kitchen-display camera capture | Browser-native `getUserMedia()` (Chromium) | No native capture app on-device at all — the kiosk page itself talks to the webcam and hands a frame straight to `pantry-vision` | +| Kitchen-display scripted control | **kitchen-display-agent** (custom) | HA MQTT-discovery entity for **Show scan/inventory/recipes** only — the actual camera/Grocy read-write path is `pantry-vision`'s own published API, called directly by the kiosk browser, not mediated through this agent | +| Grocery vision recognition | **Ollama** (a vision-capable model, e.g. `llava`/`qwen2.5vl` — TBD, not yet pulled or benchmarked) | Identifies a grocery item from one photo and estimates its shelf life; the proposal is always human-reviewed before anything is written (see `pantry-vision/README.md`) | +| Grocery inventory backend | **pantry-vision** (custom Python, stdlib `http.server`) | `POST /identify` (photo → proposal), `POST /confirm` (human-reviewed proposal → Grocy stock write), `GET /inventory`/`GET /recipes` (proxy Grocy, reshaped). Bearer-token gated, **and — unlike admin-canvas — LAN-published**, since the kitchen display's kiosk browser calls it directly rather than through Home Assistant | +| Grocery inventory storage | **Grocy** (already running, Phase 1/7) | The single source of truth for stock/best-before-dates/recipes; `pantry-vision` is a client of Grocy's own REST API, not a replacement for it | +| Kitchen-display static serving | **pantry-web** (nginx:alpine) | Serves `pantry-vision/frontend/`'s Scan/Inventory/Recipes single-page app read-only to the kitchen display — same role `digest-web`/`admin-web` play for their own hosts | --- @@ -296,6 +330,131 @@ New hardware: §1.13. Two separate build pipelines, arm64 and amd64 — see belo 7. Neither image gives Home Assistant any control or visibility over these endpoints — no MQTT, no HA entities. Deliberate scope decision matching the "per-room independent" choice, not an oversight; revisit as a separate addition if remote control/monitoring from HA is wanted later. 8. Nothing built or flashed on real hardware — no Raspberry Pi, no HiFiBerry Amp2, no x86 test box available. See the itemized unverified list in `hosts/audio-endpoint/README.md`. +### Phase 16 — Sway touch panel + +New hardware: §1.14. No container-host changes. + +1. New top-level `hosts/touch-panel/` directory — **a different device, not a variant + of the Phase 11 thin client**: that image is a couch-distance media station + deliberately built around *remote* control (HA/MQTT, wayvnc) with no on-screen + bars; this one is touched directly, so it inverts that — an always-on touch dock, + native Wayland touch input, no gesture-camera/remote-view machinery. It reuses the + thin client's live-build toolchain and `configs/`+`agent/` split, not its + live-build tree, exactly the relationship `hosts/audio-endpoint/`'s amd64 build + already has to the thin client's. +2. **Three fixed workspaces, no on-demand ones**: `1:spotify` (the real, official + Spotify Linux GUI client via Flathub `com.spotify.Client` — not the thin client's + headless `spotifyd`/`librespot` Connect receiver, since the whole point of a touch + panel is a screen you interact with directly), `2:home` (a persistent Chromium + `--kiosk --app=$HA_URL` window, auto-restarted on crash by its own supervising + loop rather than by sway or systemd), and `3:web` (Firefox, minimal chrome — + general browsing, not kiosk-locked, mirroring the thin client's own `web_browser` + app in shape but on a separate, duplicated profile). +3. **Touch navigation is a persistent on-screen dock (`eww`, layer-shell + `:exclusive true`), not a keyboard/mouse-driven `bar {}`**: reserving a strip at + the bottom means the dock can never be covered by, or lose screen area behind, + whatever app is on screen — the opposite trade-off from the thin client's + deliberately bar-less, overlay-only widgets. Same session-scoped-not-agent + ownership rule as the thin client's now-playing widget: this is compositor UI, not + device control, and dies with sway rather than opening a second inbound channel + into the security-sensitive agent. +4. **`touchpanel-agent` mirrors `thinclient-agent`'s exact security boundary**: LLM + tool call → HA service call → MQTT → this agent, never a direct network path. It + publishes a **Screen** select (the touch dock's HA-reachable equivalent) and + **Show Spotify** / **Show Home** / **Show web browser** buttons — the latter two + focus the already-running app rather than relaunching it (both are meant to stay + open and stateful), unlike the thin client's stateless kill-and-relaunch digest/ + admin kiosk pages. Also publishes a `media_player`-shaped entity set bridged from + Spotify's own MPRIS interface, identical in shape to the thin client's MPRIS + bridge, pointed at the `spotify` player name instead of `mpv`/`spotifyd`. +5. **An on-screen keyboard (`wvkbd`) with no auto-show, by explicit choice**: proper + show-on-text-field-focus needs the `text-input-v3`/`virtual-keyboard-v1` protocols + wired through a shell component (the way Phosh does it for squeekboard), which this + image doesn't run. Rather than build that integration now, the dock's Keyboard + button is a manual toggle — a documented, accepted limitation, not a gap hidden + from the README. +6. **No wayvnc on this image**, unlike the thin client — a deliberate scope decision, + not an oversight: the primary interaction model here is a finger at the panel, not + a remote viewer, so SSH alone covers the "something's broken, fix it from a shell" + case. Revisit only if a real need for remote *view* shows up once this runs on real + hardware; adding wayvnc back would be a small, isolated change (see + `hosts/touch-panel/README.md`). +7. **Both apt-first/documented-fallback installs already established elsewhere in this + repo are reused, not reinvented**: `eww` (same as the thin client's now-playing + widget) and `wvkbd` — neither is in Debian bookworm main, and both hooks stop at a + documented placeholder rather than a hardcoded release URL if apt doesn't have them. +8. Nothing built or flashed on real hardware — no touch-panel unit has been chosen + (§1.14), and in particular whether the touchscreen exposes itself as a native + Wayland `wl_touch` device (this image's one hard assumption, since there is no + pointer-emulation fallback the way the thin client's gesture control has one) is + unverified. See the itemized list in `hosts/touch-panel/README.md`. + +### Phase 17 — Kitchen/fridge display (camera-vision grocery cataloguing) + +New hardware: §1.15. New top-level `pantry-vision/` directory (container-host +service) plus `hosts/kitchen-display/` (a third, simpler kiosk image). + +1. **The workflow this phase is built around, stated once because everything below + serves it**: come home, put down the shopping bag, hold one item up to the + kitchen display's camera, the system proposes what it is and roughly how long it + keeps, the person confirms (editing anything wrong) before anything is written + anywhere, put the item away. The same display then shows inventory ordered by + what expires soonest, and Grocy's recipes, on request. +2. **`pantry-vision` never writes to Grocy from `/identify` alone — only `/confirm`, + a separate human-reviewed call, ever does.** This is the single guardrail the + whole phase is designed around, the same "propose, never auto-commit" rule this + project already applies to identity-merge confirmation (the *Identity store* row + in §2). A wrong camera guess costs one tap to fix on the kiosk screen, not a wrong + fact silently written into the household's inventory. +3. **A real, LAN-published network listener — the one deliberate exception to this + project's usual "the LLM only ever reaches a device through HA→MQTT" shape.** + `admin-canvas` (Phase 13) has no published port because only Home Assistant calls + it; `pantry-vision` is different on purpose because the kitchen display is a + separate physical device that has to get an identification back synchronously, + with no HA round-trip in the loop for that specific call. The bearer token — not + network placement — is the actual boundary here; see + `pantry-vision/README.md`'s "A real network listener, unlike admin-canvas" + section. Controlling *which screen is showing* (Scan/Inventory/Recipes) is a + separate, narrower path that does stay HA/MQTT-mediated: `kitchen-display-agent`, + identical in shape to every other host's agent in this project. +4. **Grocy is the system of record, not a new inventory store** — `pantry-vision` is + a client of Grocy's own REST API (already running unconditionally since Phase 1/7) + for both reads (`/inventory`, `/recipes`) and the one write (`/confirm` → Grocy + stock). Nothing here duplicates or replaces Grocy's own data model. +5. **The vision-identification prompt asks for one photo, not a live video stream**, + and degrades to a low-confidence placeholder proposal (never an error page) if the + model call fails or its response isn't parseable JSON — same "degrade, don't + blank" rule as the digest/admin canvas renderers and `llm_client.py`'s own + `_fallback_document`. +6. **The kitchen display has no native camera-capture app at all** — Chromium's own + `getUserMedia()`, called from `pantry-vision/frontend/app.js`, talks to the webcam + directly inside the kiosk page and hands a captured frame straight to + `POST /identify` as a blob. `--use-fake-ui-for-media-stream` on the kiosk launch + auto-accepts the permission prompt that would otherwise sit unanswered on an + unattended screen. +7. **`hosts/kitchen-display/` is one workspace, one app — no touch dock, no + multi-app switching** the way `hosts/touch-panel` has: Scan/Inventory/Recipes is + in-page tab navigation inside `pantry-vision/frontend/`'s single-page app, since + there's only one thing this device does. `kitchen-display-agent`'s MQTT surface + is correspondingly narrow: three "Show X" buttons and nothing else — no + media_player, no capture/audio/remote-desktop entities. +8. **The Grocy API integration is written from documentation, not verified against a + live instance** — the exact `GET /api/stock` response shape (whether product + names arrive nested by default), the minimum required fields for + `POST /api/objects/products`, and whether the Recipes/fulfillment endpoints need + explicit setup before they return anything meaningful are all flagged in + `pantry-vision/README.md` alongside the live OpenAPI spec URL every real Grocy + instance exposes (`/api/openapi/specification`) as the way to actually check. +9. **The vision model choice is unpicked and unbenchmarked** (`OLLAMA_VISION_MODEL` + defaults to `llava`, not confirmed pulled or even correct for a given LLM host's + GPU/CPU tier) — same "don't build against a guess" flag as every other + LLM-model/hardware pairing in this plan. Latency in particular is unmeasured and + is the number that decides whether "hold it up, wait, confirm" feels usable at + the counter or like standing there for 30 seconds. +10. Nothing built or run against real hardware, a real camera, a real vision model, + or a real Grocy instance. See the itemized lists in `pantry-vision/README.md` + and `hosts/kitchen-display/README.md`. + ### Testing checklist before calling any phase "done" - Does the reactive path (presence → light on) work with the LLM host powered off? (It must.) - Does a bad/slow LLM response ever block a light switch? (It must not.) @@ -327,10 +486,17 @@ New hardware: §1.13. Two separate build pipelines, arm64 and amd64 — see belo - Is the ESP32-S3-Touch-LCD-1.85C firmware ever flashed onto a V1 board? (It must not be — V1 has no AEC circuit and different audio pins; this phase's config assumes V2 throughout.) - Does a headless audio endpoint's Spotify Connect device name ever fail to match the room it's actually in? (It must not — `spotify-connect-start` always reads `$(hostname)`, which is set per-room by Raspberry Pi Imager on arm64 or `IMAGE_HOSTNAME` at build time on amd64; a copy-pasted/reused image for two rooms is a configuration mistake this software has no way to detect, same class of risk already called out for the ESP32 firmware's `media_player_entity_id`.) - Does the headless audio endpoint ever gain a shared/synced audio path, an MQTT connection, or an HA entity without a deliberate new decision to add one? (It must not — Phase 15 is scoped to per-room independent Spotify Connect only, on purpose.) +- Can the LLM reach the touch panel through any path other than HA service call → MQTT → `touchpanel-agent`? (It must not — identical boundary to the thin client's.) +- Does the touch panel boot to a usable session (Spotify + the browser) with the container host powered off? (It must not hang — Home showing a connection error is the expected/acceptable degraded state, since unlike the thin client this device has no local media source to fall back to.) +- Does tapping the touch dock's Home/Spotify buttons ever relaunch (rather than just focus) an already-running window? (It must not — both are stateful apps; relaunching would throw away login/scroll/playback state for no reason.) +- Does `pantry-vision`'s `/identify` endpoint ever write to Grocy by itself? (It must not — only a separate, human-reviewed `/confirm` call ever writes anything.) +- Does `pantry-vision` ever accept a request without a valid bearer token, on any of its four endpoints — including the two GETs? (It must not — unlike admin-canvas, this service is LAN-published, so the token is the actual boundary, not network placement.) +- Can the LLM reach the kitchen display through any path other than HA service call → MQTT → `kitchen-display-agent`, for *which screen is showing*? (It must not — reading/writing the actual inventory is a separate, intentionally-published path through `pantry-vision` itself, not a violation of this rule.) +- If the vision model's response is unparseable or the call fails outright, does `/identify` ever return a broken/blank result instead of a flagged, low-confidence placeholder proposal? (It must not.) --- -## 4. Open decisions (Phases 11–15) +## 4. Open decisions (Phases 11–17) These need a decision before their respective implementation steps can be built — everything above is written to accommodate any answer, but nothing should be built against an unresolved item. @@ -349,3 +515,7 @@ These need a decision before their respective implementation steps can be built 13. **The sys-admin-llm's HA-side wiring has no real data source or tool definition yet** (new, Phase 13) — same shape as open decision #9 above: `admin-canvas/README.md` documents the expected `rest_command:`/tool-call contract and worked example JSON, but which HA entities/history back something like "the kitchen outlet's power draw" is unresolved, and no metering-capable Zigbee smart plug is in this plan's hardware list (§1.4) yet. Needs a hardware decision (a power-monitoring outlet) and an actual HA tool/script, neither of which exists in this repo by design — see the Phase 13 "nothing under this repo builds the HA side" note. 14. **The voice/display satellite has no real hardware verification, and two of its data sources are unpicked** (new, Phase 14) — `firmware/esp32-s3-touch-lcd-1.85c/voice-display.yaml` passes ESPHome's own config validator but has never been flashed to a physical unit; the display init sequence, the AEC audio path, and wake-word sensitivity are all adapted/assumed, not measured (see the itemized list in that directory's README). Separately, `media_player_entity_id` and `weather_entity_id` are placeholders — which media player this unit should mirror needs a decision if the household ends up with more than one active at a time. 15. **The headless audio endpoint has no real hardware verification on either architecture** (new, Phase 15) — neither the arm64 (`rpi-image-gen`) nor amd64 (`live-build`) pipeline has been run to completion, let alone flashed and booted. rpi-image-gen's exact config/layer YAML schema in particular was written from documentation only, with no ARM build environment available to validate it against the real tool — see `hosts/audio-endpoint/README.md`'s prominent risk callout. Also still open: whether `spotifyd` or `librespot` actually installs cleanly from current bookworm apt sources at all (the fallback list is documented, not exercised), and how stable USB DAC/amp ALSA enumeration turns out to be on a real amd64 mini PC. +16. **No touch-panel hardware has been chosen** (new, Phase 16) — nothing in `hosts/touch-panel/` has been booted on real metal or a real touchscreen. In particular, whether the panel's touch input actually surfaces as a native Wayland `wl_touch` device (this image's one hard assumption) is unverified — see `hosts/touch-panel/README.md`'s verification list, item 3. +17. **The touch panel's Spotify/Chromium integration details are assumed, not confirmed** (new, Phase 16) — the Flathub app ID `com.spotify.Client`, the Spotify Flatpak's real MPRIS bus name (assumed `spotify`), and Chromium's Wayland `app_id` for a `--app=` kiosk window (assumed to start with `chromium`) are all flagged for on-hardware verification in `hosts/touch-panel/README.md`. +18. **No vision-capable Ollama model has been picked, pulled, or benchmarked** (new, Phase 17) — `pantry-vision`'s `OLLAMA_VISION_MODEL` defaults to `llava` with no confirmation it's the right choice for any given LLM host's hardware tier, and grocery-item identification latency/accuracy is completely unmeasured. This is the single highest-risk unknown in Phase 17: if it's too slow or too inaccurate, "hold item up to camera" stops being a usable interaction and the phase needs a different approach (a smaller/faster model, a barcode-scan fallback via Grocy's existing scanning support, or both). +19. **`pantry-vision`'s Grocy API integration is written from documentation, not a live instance** (new, Phase 17) — `GET /api/stock`'s response shape, `POST /api/objects/products`'s minimum required fields, and the Recipes/fulfillment endpoints' setup requirements are all assumed; `pantry-vision/README.md` points at each real Grocy instance's own live OpenAPI spec (`/api/openapi/specification`) as the way to check before trusting any of it. `GROCY_DEFAULT_LOCATION_ID`/`GROCY_DEFAULT_QU_ID` are fresh-install-default guesses that need confirming against Settings → Locations/Quantity units on the real instance too. diff --git a/hosts/container-host/scripts/setup-container-host.sh b/hosts/container-host/scripts/setup-container-host.sh index e3d5728..a33a8c1 100755 --- a/hosts/container-host/scripts/setup-container-host.sh +++ b/hosts/container-host/scripts/setup-container-host.sh @@ -28,6 +28,10 @@ # - admin-canvas + admin-web (Phase 13 sys-admin-llm display surface, optional, # off by default — needs admin-canvas/ from this repo checked out on this # host, see ADMIN_CANVAS_SRC below and admin-canvas/README.md) +# - pantry-vision + pantry-web (Phase 17 kitchen-display camera cataloguing +# backend, optional, off by default — needs pantry-vision/ from this repo +# checked out on this host, see PANTRY_VISION_SRC below and +# pantry-vision/README.md) # # Run as: sudo ./setup-container-host.sh # @@ -94,6 +98,19 @@ ADMIN_CANVAS_SRC="/opt/smart-home/src/admin-canvas" ADMIN_CANVAS_PORT="8092" # internal only — no `ports:` mapping, HA-reachable only ADMIN_WEB_PORT="8094" # LAN-facing read-only static serving +# --- Kitchen-display camera cataloguing backend (Phase 17) — off by default until +# --- PANTRY_VISION_TOKEN and GROCY_API_KEY are provisioned. See pantry-vision/README.md. +# Unlike admin-canvas, this one IS meant to be reached directly by a separate physical +# device (hosts/kitchen-display's kiosk browser), not only by Home Assistant — so, +# deliberately unlike ADMIN_CANVAS_PORT above, it DOES get a `ports:` mapping. The +# bearer token is the actual boundary here, not network placement — see +# pantry-vision/README.md's "A real network listener, unlike admin-canvas" section. +ENABLE_PANTRY_VISION="false" +# Where this repo's pantry-vision/ directory lives on THIS host (build context). +PANTRY_VISION_SRC="/opt/smart-home/src/pantry-vision" +PANTRY_VISION_PORT="8095" # LAN-facing — the kitchen display's kiosk browser calls this directly +PANTRY_WEB_PORT="8096" # LAN-facing read-only static serving (the kiosk's frontend) + # --------------------------------------------------------------------------- # Sanity checks # --------------------------------------------------------------------------- @@ -130,6 +147,12 @@ if [[ "$ENABLE_ADMIN_CANVAS" == "true" && ! -d "$ADMIN_CANVAS_SRC" ]]; then echo " build context for the admin-canvas image — then re-run." fi +if [[ "$ENABLE_PANTRY_VISION" == "true" && ! -d "$PANTRY_VISION_SRC" ]]; then + echo "Warning: ENABLE_PANTRY_VISION=true but $PANTRY_VISION_SRC does not exist." + echo " Copy or clone this repo's pantry-vision/ directory there — it is the" + echo " build context for the pantry-vision image — then re-run." +fi + if [[ "$ENABLE_WHATSAPP_INGEST" == "true" ]]; then echo "WARNING: WhatsApp ingestion is enabled." echo " There is no officially sanctioned way to read WhatsApp programmatically." @@ -252,6 +275,17 @@ if [[ "$ENABLE_ADMIN_CANVAS" == "true" ]]; then echo " Seeded $BASE_DIR/admin-canvas/admin-canvas.env from the template — fill in a real ADMIN_CANVAS_TOKEN." fi fi +if [[ "$ENABLE_PANTRY_VISION" == "true" ]]; then + mkdir -p "$BASE_DIR"/pantry-vision + # Same handling as admin-canvas.env above: seed from the committed template on + # first run, 600, never committed (repo .gitignore covers *.env). + if [[ ! -f "$BASE_DIR/pantry-vision/pantry-vision.env" ]]; then + cp "$PANTRY_VISION_SRC/pantry-vision.env.example" "$BASE_DIR/pantry-vision/pantry-vision.env" + chmod 600 "$BASE_DIR/pantry-vision/pantry-vision.env" + echo " Seeded $BASE_DIR/pantry-vision/pantry-vision.env from the template — fill in a real" + echo " PANTRY_VISION_TOKEN and GROCY_API_KEY (Grocy's own UI: Settings -> Manage API keys)." + fi +fi # --------------------------------------------------------------------------- # 4. Mosquitto config @@ -625,6 +659,46 @@ if [[ "$ENABLE_ADMIN_CANVAS" == "true" ]]; then " fi +# pantry-vision (Phase 17) — the kitchen display's camera-cataloguing backend. Same +# build/write-service shape as admin-canvas above, but see this block's own comment +# where it differs (a published port, since a separate physical device calls it +# directly) and pantry-vision/README.md for the full reasoning. `depends_on: grocy` +# is a startup-order hint only — pantry-vision's own outbound calls already retry per +# request, so a slow-starting Grocy delays the first real call, not the container. +PANTRY_VISION_BLOCK="" +PANTRY_WEB_BLOCK="" +if [[ "$ENABLE_PANTRY_VISION" == "true" ]]; then + PANTRY_VISION_BLOCK=" + pantry-vision: + build: ${PANTRY_VISION_SRC} + image: smart-home/pantry-vision:local + container_name: pantry-vision + restart: unless-stopped + depends_on: + - grocy + ports: + - \"${PANTRY_VISION_PORT}:${PANTRY_VISION_PORT}\" + env_file: + - ${BASE_DIR}/pantry-vision/pantry-vision.env + environment: + - PANTRY_VISION_PORT=${PANTRY_VISION_PORT} + - TZ=${TIMEZONE} +" + + PANTRY_WEB_BLOCK=" + pantry-web: + image: nginx:alpine + container_name: pantry-web + restart: unless-stopped + ports: + - \"${PANTRY_WEB_PORT}:80\" + volumes: + - ${PANTRY_VISION_SRC}/frontend:/usr/share/nginx/html:ro + environment: + - TZ=${TIMEZONE} +" +fi + if [[ "$ENABLE_DIGEST_ENGINE" == "true" && "$ENABLE_WHATSAPP_INGEST" == "true" ]]; then # Long-lived, unlike digest-engine: holds the logged-in WhatsApp Web session # open and appends to /data/messages.jsonl, which digest-engine drains each @@ -721,7 +795,7 @@ ${FRIGATE_DEVICES} - PUID=1000 - PGID=1000 - TZ=${TIMEZONE} -${MEALIE_BLOCK}${NODERED_BLOCK}${NETDATA_BLOCK}${HOMEPAGE_BLOCK}${NTFY_BLOCK}${PORTAINER_BLOCK}${GALLERY_SMB_BLOCK}${DIGEST_ENGINE_BLOCK}${DIGEST_WEB_BLOCK}${WHATSAPP_BRIDGE_BLOCK}${ADMIN_CANVAS_BLOCK}${ADMIN_WEB_BLOCK} +${MEALIE_BLOCK}${NODERED_BLOCK}${NETDATA_BLOCK}${HOMEPAGE_BLOCK}${NTFY_BLOCK}${PORTAINER_BLOCK}${GALLERY_SMB_BLOCK}${DIGEST_ENGINE_BLOCK}${DIGEST_WEB_BLOCK}${WHATSAPP_BRIDGE_BLOCK}${ADMIN_CANVAS_BLOCK}${ADMIN_WEB_BLOCK}${PANTRY_VISION_BLOCK}${PANTRY_WEB_BLOCK} EOF # --------------------------------------------------------------------------- @@ -880,6 +954,10 @@ fi if [[ "$ENABLE_ADMIN_CANVAS" == "true" ]]; then echo " Admin canvas : http://${HOST_IP}:${ADMIN_WEB_PORT}/canvas.html" fi +if [[ "$ENABLE_PANTRY_VISION" == "true" ]]; then + echo " Pantry API : http://${HOST_IP}:${PANTRY_VISION_PORT} (bearer-token gated)" + echo " Pantry display : http://${HOST_IP}:${PANTRY_WEB_PORT}/index.html?api=http://${HOST_IP}:${PANTRY_VISION_PORT}&token=" +fi if [[ "$ENABLE_GALLERY_SMB" == "true" ]]; then echo " Gallery SMB : \\\\${HOST_IP}\\gallery (user: ${GALLERY_SMB_USERNAME})" fi @@ -927,6 +1005,15 @@ if [[ "$ENABLE_ADMIN_CANVAS" == "true" ]]; then echo " admin-canvas/README.md for the worked example and the 'Show admin canvas'" echo " entity documented in hosts/thin-client/README.md." fi +if [[ "$ENABLE_PANTRY_VISION" == "true" ]]; then + echo " 15. Fill in $BASE_DIR/pantry-vision/pantry-vision.env before the kitchen display can" + echo " identify anything: PANTRY_VISION_TOKEN (also goes into" + echo " hosts/kitchen-display/scripts/build-kitchen-display-iso.sh — both sides need the" + echo " SAME value) and GROCY_API_KEY (Grocy's own UI: Settings -> Manage API keys, at" + echo " http://${HOST_IP}:9283). Also pull a vision-capable Ollama model on the LLM host" + echo " (e.g. 'ollama pull llava') — OLLAMA_VISION_MODEL defaults to one that is NOT" + echo " confirmed pulled or even correct for your setup. See pantry-vision/README.md." +fi echo echo "Updating later: cd $BASE_DIR && docker compose pull && docker compose up -d" echo "Backing up manually: sudo $BASE_DIR/backup.sh (requires ENABLE_BACKUPS=true was run once)" diff --git a/hosts/kitchen-display/README.md b/hosts/kitchen-display/README.md new file mode 100644 index 0000000..21514d6 --- /dev/null +++ b/hosts/kitchen-display/README.md @@ -0,0 +1,125 @@ +# Sway kitchen display + +Phase 17 of `docs/project-plan.md`. Builds a Debian 12 live ISO for a single-purpose +fridge/pantry display: hold a grocery item up to its camera, confirm what the vision +model thinks it is and when it'll go bad, put it away — then browse the resulting +inventory (soonest-to-expire first) and Grocy's recipes, all touch-driven. + +**The camera-vision and Grocy read/write logic does not live in this directory** — +that's [`../../pantry-vision/`](../../pantry-vision/README.md), a container-host +service this device's kiosk browser calls directly. This host is deliberately thin: +a Sway session, one Chromium kiosk window, and `kitchen-display-agent`, which only +ever controls *which screen is showing* (Scan / Inventory / Recipes) over MQTT — the +same "the agent controls the surface, a separate write API owns the content" split +`hosts/thin-client`'s admin canvas already established. + +**A third, simpler sibling of `../thin-client/` and `../touch-panel/`**, not a variant +of either — one workspace, one app, no dock, no music, no general browsing. It reuses +the thin client's live-build toolchain and `configs/`+`agent/` split. + +| | | +|---|---| +| Compositor | Sway, one workspace, one Chromium kiosk window | +| Autologin | greetd, straight into `/usr/local/bin/kiosk-session` | +| Touch input | Native Wayland `wl_touch`, same assumption as `../touch-panel/` | +| Camera | Browser-native `getUserMedia()` inside the Chromium kiosk page — no separate native capture app on this device at all | +| Content | `pantry-vision`'s frontend (`../../pantry-vision/frontend/`), served by `pantry-web`; in-page tabs, not separate Sway workspaces | +| Remote control (HA/LLM) | `kitchen-display-agent` — **Show scan / Show inventory / Show recipes** buttons only | +| Remote control (human) | SSH only — no wayvnc, same scope decision as `../touch-panel/` | + +## Hardware + +**No specific unit has been chosen.** What's assumed: a touchscreen (native +`wl_touch`, same caveat as `../touch-panel/README.md`) mounted somewhere near the +fridge/pantry, with a USB webcam pointed at wherever items get held up — either +built into the panel or a separate camera on a short cable/gooseneck mount so the +angle can be adjusted independently of the screen. See §1.15 of +`docs/project-plan.md` for the (unverified, non-specific) hardware line item. + +## Before you build + +Deploy `pantry-vision`/`pantry-web` first (`ENABLE_PANTRY_VISION` in +`hosts/container-host/scripts/setup-container-host.sh`) — this image builds and boots +fine without it, but every tab will show a connection error until it exists. Then edit +the `# CONFIGURATION` block at the top of +[`scripts/build-kitchen-display-iso.sh`](scripts/build-kitchen-display-iso.sh): + +| Variable | What to put in it | +|---|---| +| `MQTT_BROKER_HOST` | LAN IP of the container host running Mosquitto | +| `PANTRY_WEB_URL` | The `pantry-web` service's URL, e.g. `http://192.168.1.10:8096` | +| `PANTRY_VISION_URL` | The `pantry-vision` API's URL, e.g. `http://192.168.1.10:8095` — read by `app.js` from the kiosk window's own URL query params (see `pantry-kiosk`'s comment), never baked into the served frontend files | +| `PANTRY_VISION_TOKEN` | **Must exactly match** `PANTRY_VISION_TOKEN` in `pantry-vision/pantry-vision.env` on the container host — there's no way for this repo to push that value between the two hosts for you | +| `KIOSK_USERNAME` / `IMAGE_HOSTNAME` / `KITCHEN_DISPLAY_NAME` | Per-device identity | +| `SSH_AUTHORIZED_KEY` | Optional — password auth is disabled and there is no wayvnc | + +## Build + +```sh +sudo ./scripts/build-kitchen-display-iso.sh +``` + +Same directory-split convention as every other host in this project: `configs/` and +`agent/` are human-edited and git-tracked; `live-build/config/includes.chroot/` is +generated, gitignored, never hand-edited. + +## Why the camera has no native capture app + +`../touch-panel/` and `../thin-client/` both shell out to native processes for their +media (mpv, the Spotify Flatpak). This device doesn't need that: Chromium's own +`getUserMedia()` API, called from `pantry-vision/frontend/app.js`, talks to the +webcam directly and hands a captured frame straight to `POST /identify` as a blob — +no intermediate script, no temp file, no separate process to supervise. The one thing +this needs from the kiosk launch itself is `--use-fake-ui-for-media-stream` +(`configs/sway/pantry-kiosk`), which auto-accepts the camera permission prompt that +would otherwise sit unanswered on a screen nobody is there to click "Allow" on. + +## Home Assistant entities + +`kitchen-display-agent` publishes three buttons on connect: **Show scan**, **Show +inventory**, **Show recipes** — each kills and relaunches the kiosk Chromium window +at `PANTRY_WEB_URL/index.html?...#`. That's the entire MQTT surface; there is no +media_player, no workspace select (one workspace), no capture/audio/remote-desktop +entities the way the thin client has. Reading or writing the actual inventory (from +voice, e.g. "what's about to expire?" or "add three eggs") is a **separate** path — +Home Assistant/the LLM calling `pantry-vision`'s own published API directly, not +through this agent, exactly mirroring the split `hosts/thin-client`'s admin canvas +already established between "which screen is showing" (MQTT, this agent) and "what's +actually on it" (a dedicated write API). + +### Security boundary + +Same shape and reasoning as every other host's agent in this project: +`kitchen_display_agent/mqtt_discovery.py` is the entire inbound MQTT control surface +of this machine — LLM tool call → HA service call → MQTT → this agent, no HTTP +listener, no websocket, no exposed Sway IPC socket, no VNC. `pantry-vision` is a +**separate, deliberately published** service with its own bearer-token boundary — see +`../../pantry-vision/README.md`'s "A real network listener, unlike admin-canvas" +section for why that one's trust model is different on purpose. + +## Manual verification still outstanding + +None of this has been run on hardware. In rough order — on top of everything already +flagged as unverified in `../../pantry-vision/README.md` (the Grocy API assumptions, +the vision model's accuracy/latency, which this device's whole usefulness rides on): + +1. The ISO builds at all. +2. greetd lands in Sway with no login prompt. +3. **Touch input as native `wl_touch`** — same open item as `../touch-panel/`. +4. **Camera enumeration and `--use-fake-ui-for-media-stream`** — whether Chromium + picks the right `/dev/videoN` when more than one video device is present (a + built-in laptop-style webcam plus a dedicated one, say), and whether that flag + actually suppresses the permission prompt on this Chromium/Wayland combination + specifically (it's a well-known flag for automated testing, not confirmed here + against a real kiosk session). +5. `kitchen-display-agent` connects to Mosquitto and the device appears in HA. +6. End-to-end scan latency: camera capture → `/identify` → a usable proposal on + screen, with a real vision model on real LLM-host hardware. This is the number + that decides whether the whole "hold it up, wait, confirm" workflow feels usable + or like standing at the fridge for 30 seconds waiting on an LLM. +7. Whether `?api=...&token=...` survives being passed through `--app=` URL escaping + correctly for every character that can appear in `PANTRY_VISION_URL` — the + `sed`-based percent-encoding in `configs/sway/pantry-kiosk` handles `/` and `:` + only, which is enough for a bare `http://host:port` but not for a URL with a path + or query string of its own. +8. Idle-blank timeout (20 minutes, `configs/sway/config`) and touch-resume — untested. diff --git a/hosts/kitchen-display/agent/kitchen-display-agent.service b/hosts/kitchen-display/agent/kitchen-display-agent.service new file mode 100644 index 0000000..e8f39dd --- /dev/null +++ b/hosts/kitchen-display/agent/kitchen-display-agent.service @@ -0,0 +1,20 @@ +[Unit] +Description=Kitchen display agent (Home Assistant MQTT control surface) +Documentation=file:///opt/kitchen-display-agent +After=network.target + +[Service] +Type=simple +User=@KIOSK_USERNAME@ +Group=@KIOSK_USERNAME@ +WorkingDirectory=/opt/kitchen-display-agent +Environment=PYTHONPATH=/opt/kitchen-display-agent +Environment=PYTHONUNBUFFERED=1 +EnvironmentFile=-/etc/kitchen-display-agent/config.env +ExecStart=/usr/bin/python3 -m kitchen_display_agent.main +Restart=always +RestartSec=5 +NoNewPrivileges=true + +[Install] +WantedBy=multi-user.target diff --git a/hosts/kitchen-display/agent/kitchen_display_agent/__init__.py b/hosts/kitchen-display/agent/kitchen_display_agent/__init__.py new file mode 100644 index 0000000..1c83f10 --- /dev/null +++ b/hosts/kitchen-display/agent/kitchen_display_agent/__init__.py @@ -0,0 +1,3 @@ +"""kitchen-display-agent — Home Assistant MQTT control surface for the kitchen display.""" + +__version__ = "0.1.0" diff --git a/hosts/kitchen-display/agent/kitchen_display_agent/main.py b/hosts/kitchen-display/agent/kitchen_display_agent/main.py new file mode 100644 index 0000000..a2ac355 --- /dev/null +++ b/hosts/kitchen-display/agent/kitchen_display_agent/main.py @@ -0,0 +1,137 @@ +"""kitchen-display-agent entrypoint.""" + +from __future__ import annotations + +import logging +import os +import signal +import socket +import sys +import threading + +import paho.mqtt.client as mqtt + +from .mqtt_discovery import Discovery +from .session import launch_pantry_kiosk + +CONFIG_PATH = os.environ.get("KITCHEN_DISPLAY_AGENT_CONFIG", "/etc/kitchen-display-agent/config.env") + +CONFIG_KEYS = ( + "MQTT_BROKER_HOST", + "MQTT_BROKER_PORT", + "MQTT_USERNAME", + "MQTT_PASSWORD", + "PANTRY_WEB_URL", + "PANTRY_VISION_URL", + "PANTRY_VISION_TOKEN", + "KIOSK_USERNAME", + "KITCHEN_DISPLAY_NAME", +) + +log = logging.getLogger("kitchen-display-agent") + + +def load_config(path: str = CONFIG_PATH) -> dict[str, str]: + values: dict[str, str] = {} + try: + with open(path, encoding="utf-8") as handle: + for line in handle: + line = line.strip() + if not line or line.startswith("#") or "=" not in line: + continue + key, _, value = line.partition("=") + values[key.strip()] = value.strip().strip('"').strip("'") + except OSError as exc: + log.warning("could not read %s (%s); falling back to the environment", path, exc) + + for key in CONFIG_KEYS: + if key in os.environ: + values[key] = os.environ[key] + + return values + + +def make_client(client_id: str) -> mqtt.Client: + callback_api = getattr(mqtt, "CallbackAPIVersion", None) + if callback_api is not None: + return mqtt.Client(callback_api.VERSION1, client_id=client_id) + return mqtt.Client(client_id=client_id) + + +def main() -> int: + logging.basicConfig( + level=logging.INFO, + format="%(asctime)s %(levelname)s %(name)s: %(message)s", + stream=sys.stdout, + ) + + config = load_config() + hostname = socket.gethostname() + node_id = "".join(c if c.isalnum() else "_" for c in hostname).strip("_") or "kitchendisplay" + friendly_name = config.get("KITCHEN_DISPLAY_NAME") or f"Kitchen display ({hostname})" + + broker_host = config.get("MQTT_BROKER_HOST", "") + broker_port = int(config.get("MQTT_BROKER_PORT") or 1883) + + client = make_client(f"kitchen-display-agent-{node_id}") + if config.get("MQTT_USERNAME"): + client.username_pw_set(config["MQTT_USERNAME"], config.get("MQTT_PASSWORD") or None) + + discovery = Discovery(client, node_id, friendly_name) + + def on_show(fragment: str) -> None: + launch_pantry_kiosk(fragment) + + def on_connect(_client, _userdata, _flags, rc): + if rc != 0: + log.error("MQTT connection refused (rc=%s)", rc) + return + log.info("connected to MQTT broker %s:%s", broker_host, broker_port) + discovery.register_screens(on_show) + discovery.subscribe_all() + discovery.publish_available(True) + + def on_disconnect(_client, _userdata, rc): + log.warning("disconnected from MQTT broker (rc=%s); paho will retry", rc) + + def on_message(_client, _userdata, message): + discovery.dispatch(message.topic, message.payload.decode("utf-8", "replace")) + + client.on_connect = on_connect + client.on_disconnect = on_disconnect + client.on_message = on_message + client.will_set(discovery.availability_topic, "offline", qos=1, retain=True) + + stop_event = threading.Event() + + def handle_signal(_signum, _frame): + stop_event.set() + + signal.signal(signal.SIGTERM, handle_signal) + signal.signal(signal.SIGINT, handle_signal) + + if not broker_host: + log.error("MQTT_BROKER_HOST is not set in %s — running without HA control", CONFIG_PATH) + else: + # connect_async + loop_start, never a blocking connect(): same "reactive path + # never depends on a remote service" rule as every other host's agent — the + # kitchen display's local scan/inventory/recipes flow must work with the + # container host powered off, only the HA "Show X" buttons need it. + client.connect_async(broker_host, broker_port, keepalive=60) + client.loop_start() + + log.info("kitchen-display-agent %s started (node_id=%s)", node_id, node_id) + try: + stop_event.wait() + finally: + log.info("shutting down") + if broker_host: + discovery.publish_available(False) + client.loop_stop() + client.disconnect() + + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/hosts/kitchen-display/agent/kitchen_display_agent/mqtt_discovery.py b/hosts/kitchen-display/agent/kitchen_display_agent/mqtt_discovery.py new file mode 100644 index 0000000..f23de99 --- /dev/null +++ b/hosts/kitchen-display/agent/kitchen_display_agent/mqtt_discovery.py @@ -0,0 +1,105 @@ +"""Home Assistant MQTT Discovery payloads and command dispatch. + +SECURITY BOUNDARY — this module is the entire remote-control API of the kitchen +display. Same principle as every other host's agent in this project (Phase 11.4): +the local LLM never gets a network path to this machine directly. The only chain is +LLM tool call -> Home Assistant service call -> MQTT -> this dispatcher. There is no +HTTP listener, no websocket server, no exposed Sway IPC socket here. + +This is deliberately NOT how the LLM reaches pantry-vision's actual inventory data — +that's a separate, already-published API (pantry-vision/server.py) that Home +Assistant or the LLM can call directly for reads/writes to Grocy. This agent only +ever controls what the physical screen is showing, exactly the same division of +labour as hosts/thin-client's "Show admin canvas" button vs. admin-canvas's own +write API. +""" + +from __future__ import annotations + +import json +import logging +from typing import Callable + +from . import __version__ + +log = logging.getLogger(__name__) + +DISCOVERY_PREFIX = "homeassistant" + + +class Discovery: + def __init__(self, client, node_id: str, friendly_name: str): + self.client = client + self.node_id = node_id + self.friendly_name = friendly_name + self.base = f"kitchendisplay/{node_id}" + self.availability_topic = f"{self.base}/availability" + self._handlers: dict[str, Callable[[str], None]] = {} + + self.device = { + "identifiers": [f"kitchendisplay_{node_id}"], + "name": friendly_name, + "manufacturer": "SmartestHome", + "model": "Sway kitchen display", + "sw_version": __version__, + } + + def _publish_config(self, component: str, object_id: str, payload: dict) -> None: + payload = { + "availability_topic": self.availability_topic, + "device": self.device, + "unique_id": f"{self.node_id}_{object_id}", + **payload, + } + topic = f"{DISCOVERY_PREFIX}/{component}/{self.node_id}/{object_id}/config" + self.client.publish(topic, json.dumps(payload), qos=1, retain=True) + + def _command_topic(self, suffix: str, handler) -> str: + topic = f"{self.base}/{suffix}" + self._handlers[topic] = handler + return topic + + def subscribe_all(self) -> None: + for topic in self._handlers: + self.client.subscribe(topic, qos=1) + + def dispatch(self, topic: str, payload: str) -> None: + handler = self._handlers.get(topic) + if handler is None: + log.warning("no handler for %s", topic) + return + try: + handler(payload) + except Exception: + log.exception("handler for %s failed", topic) + + def publish_available(self, available: bool = True) -> None: + self.client.publish( + self.availability_topic, + "online" if available else "offline", + qos=1, + retain=True, + ) + + def register_screens(self, on_show) -> None: + """One button per screen — payload is the fixed page fragment (scan/ + inventory/recipes), an enumerated constant handled in main.py, never passed + through as a URL/argv element unchanged (same rule as every other agent's + mqtt_discovery.py in this project). + """ + for object_id, name, fragment, icon in ( + ("show_scan", "Show scan", "scan", "mdi:camera"), + ("show_inventory", "Show inventory", "inventory", "mdi:fridge-outline"), + ("show_recipes", "Show recipes", "recipes", "mdi:chef-hat"), + ): + self._publish_config( + "button", + object_id, + { + "name": name, + "command_topic": self._command_topic( + f"show/{fragment}", lambda _payload, fragment=fragment: on_show(fragment) + ), + "icon": icon, + }, + ) diff --git a/hosts/kitchen-display/agent/kitchen_display_agent/session.py b/hosts/kitchen-display/agent/kitchen_display_agent/session.py new file mode 100644 index 0000000..d0197f0 --- /dev/null +++ b/hosts/kitchen-display/agent/kitchen_display_agent/session.py @@ -0,0 +1,47 @@ +"""Session-environment resolution and process launching. + +A trimmed version of hosts/thin-client's/hosts/touch-panel's sway_control.py: this +device has only one Sway workspace and one app (the pantry-kiosk Chromium window), so +there is no workspace-switching or window-focus surface to wrap — just enough to run +`/usr/local/bin/pantry-kiosk [fragment]` with the graphical session's environment. +""" + +from __future__ import annotations + +import glob +import logging +import os +import subprocess + +log = logging.getLogger(__name__) + + +def runtime_dir() -> str: + return os.environ.get("XDG_RUNTIME_DIR") or f"/run/user/{os.getuid()}" + + +def session_env() -> dict[str, str]: + env = dict(os.environ) + env["XDG_RUNTIME_DIR"] = runtime_dir() + env.setdefault("DBUS_SESSION_BUS_ADDRESS", f"unix:path={runtime_dir()}/bus") + env.setdefault("WAYLAND_DISPLAY", "wayland-1") + matches = sorted(glob.glob(os.path.join(runtime_dir(), "sway-ipc.*.sock"))) + if matches: + env["SWAYSOCK"] = matches[-1] + return env + + +def launch_pantry_kiosk(fragment: str) -> None: + command = ["/usr/local/bin/pantry-kiosk", fragment] if fragment else ["/usr/local/bin/pantry-kiosk"] + log.info("launching %s", " ".join(command)) + try: + subprocess.Popen( + command, + env=session_env(), + stdin=subprocess.DEVNULL, + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + start_new_session=True, + ) + except OSError as exc: + log.error("could not launch pantry-kiosk: %s", exc) diff --git a/hosts/kitchen-display/agent/requirements.txt b/hosts/kitchen-display/agent/requirements.txt new file mode 100644 index 0000000..b92e61e --- /dev/null +++ b/hosts/kitchen-display/agent/requirements.txt @@ -0,0 +1,4 @@ +# The ISO installs this from apt (python3-paho-mqtt) rather than pip — see +# live-build/config/package-lists/kitchen-display.list.chroot. This file is for +# running the agent outside the image (development, a venv on a test box). +paho-mqtt>=1.6 diff --git a/hosts/kitchen-display/configs/greetd/config.toml b/hosts/kitchen-display/configs/greetd/config.toml new file mode 100644 index 0000000..0bb52e4 --- /dev/null +++ b/hosts/kitchen-display/configs/greetd/config.toml @@ -0,0 +1,9 @@ +# greetd — autologin straight into the kiosk Sway session, no greeter UI. Same +# shape as hosts/touch-panel's and hosts/thin-client's own config.toml. + +[terminal] +vt = 1 + +[default_session] +command = "/usr/local/bin/kiosk-session" +user = "@KIOSK_USERNAME@" diff --git a/hosts/kitchen-display/configs/greetd/kiosk-session b/hosts/kitchen-display/configs/greetd/kiosk-session new file mode 100755 index 0000000..d662bfa --- /dev/null +++ b/hosts/kitchen-display/configs/greetd/kiosk-session @@ -0,0 +1,17 @@ +#!/bin/sh +# greetd's default_session command. Installed to /usr/local/bin/kiosk-session. +set -eu + +[ -r /etc/kitchen-display-agent/config.env ] && . /etc/kitchen-display-agent/config.env +export MQTT_BROKER_HOST PANTRY_WEB_URL PANTRY_VISION_URL PANTRY_VISION_TOKEN + +export XDG_CURRENT_DESKTOP=sway +export XDG_SESSION_TYPE=wayland +export XDG_SESSION_DESKTOP=sway +export MOZ_ENABLE_WAYLAND=1 +export OZONE_PLATFORM=wayland + +: "${XDG_RUNTIME_DIR:=/run/user/$(id -u)}" +export XDG_RUNTIME_DIR + +exec sway diff --git a/hosts/kitchen-display/configs/sway/config b/hosts/kitchen-display/configs/sway/config new file mode 100644 index 0000000..b188de4 --- /dev/null +++ b/hosts/kitchen-display/configs/sway/config @@ -0,0 +1,56 @@ +# Sway kiosk session for the kitchen display (docs/project-plan.md Phase 17). +# +# Installed to /home//.config/sway/config by +# build-kitchen-display-iso.sh. A single-purpose device — one workspace, one +# Chromium kiosk window pointed at pantry-web's frontend (Scan / Inventory / Recipes +# is in-page tab navigation there, not separate Sway workspaces the way +# hosts/touch-panel switches between three different apps). + +set $mod Mod4 +set $ws_pantry 1:pantry + +input type:keyboard { + xkb_layout @KEYBOARD_LAYOUT@ +} + +# Real touchscreen hardware — native Wayland wl_touch, same assumption and same +# caveat as hosts/touch-panel/configs/sway/config (see that host's README +# verification list item 3). +input type:touch { + map_to_output "*" +} + +output * bg #101014 solid_color + +default_border none +default_floating_border none +hide_edge_borders both +gaps inner 0 +gaps outer 0 + +# kitchen-display-agent is NOT started here — systemd owns it, same reasoning as +# every other agent in this project (reachable over MQTT whether or not a graphical +# session is up, survives a sway restart). + +for_window [app_id="chromium.*"] fullscreen enable +for_window [app_id="chromium.*"] inhibit_idle fullscreen + +exec sh -c '[ -n "$PANTRY_WEB_URL" ] && { swaymsg workspace $ws_pantry; /usr/local/bin/pantry-kiosk; }' + +# Blank after 20 minutes; any touch resumes it. No lock screen — same reasoning as +# every other kiosk image in this project: a shared kitchen display has nobody to +# unlock it. +exec swayidle -w \ + timeout 1200 'swaymsg "output * power off"' \ + resume 'swaymsg "output * power on"' + +bindsym $mod+Shift+Ctrl+m exec foot --title maintenance-shell +for_window [title="maintenance-shell"] floating enable, resize set width 800 height 500, move position center + +bindsym $mod+Return exec foot +bindsym $mod+q kill +bindsym $mod+Shift+c reload + +# Deliberately no exit binding — same reasoning as the thin client/touch panel. + +workspace $ws_pantry diff --git a/hosts/kitchen-display/configs/sway/pantry-kiosk b/hosts/kitchen-display/configs/sway/pantry-kiosk new file mode 100755 index 0000000..596199e --- /dev/null +++ b/hosts/kitchen-display/configs/sway/pantry-kiosk @@ -0,0 +1,58 @@ +#!/bin/sh +# Chromium kiosk window pointed at pantry-vision's frontend (pantry-web). Installed +# to /usr/local/bin/pantry-kiosk. Called both at session start and by +# kitchen-display-agent when a "Show scan/inventory/recipes" MQTT command arrives. +# +# Kill-and-relaunch, not focus-and-navigate — same reasoning as hosts/thin-client's +# digest-browser: Chromium's --kiosk window has no address bar to drive remotely, and +# the page underneath (pantry-vision/frontend/) is cheap to reload. Camera capture is +# momentary (hold item up, tap Capture) so losing an in-progress scan to a "Show +# inventory" command is an acceptable, rare edge case, not a design flaw to solve here. +set -eu + +PROFILE_DIR="${HOME:-/home/$(id -un)}/.config/kitchen-display-chromium" + +BASE_URL="${PANTRY_WEB_URL:-}" +[ -n "$BASE_URL" ] || { echo "pantry-kiosk: PANTRY_WEB_URL is unset" >&2; exit 1; } +[ -n "${PANTRY_VISION_URL:-}" ] || { echo "pantry-kiosk: PANTRY_VISION_URL is unset" >&2; exit 1; } +[ -n "${PANTRY_VISION_TOKEN:-}" ] || { echo "pantry-kiosk: PANTRY_VISION_TOKEN is unset" >&2; exit 1; } + +# api=/token= are read by frontend/app.js — see that file's top comment for why +# config travels this way instead of being baked into the served static files. +FRAGMENT="${1:-}" +URL="${BASE_URL%/}/index.html?api=$(printf '%s' "$PANTRY_VISION_URL" | sed 's/\//%2F/g; s/:/%3A/g')&token=${PANTRY_VISION_TOKEN}" +[ -n "$FRAGMENT" ] && URL="${URL}#${FRAGMENT}" + +if command -v chromium >/dev/null 2>&1; then + CHROMIUM=chromium +elif command -v chromium-browser >/dev/null 2>&1; then + CHROMIUM=chromium-browser +else + echo "pantry-kiosk: no chromium/chromium-browser binary found" >&2 + exit 1 +fi + +mkdir -p "$PROFILE_DIR" + +pkill -u "$(id -u)" -f "$CHROMIUM .*--user-data-dir=$PROFILE_DIR" 2>/dev/null || true +i=0 +while pgrep -u "$(id -u)" -f "$CHROMIUM .*--user-data-dir=$PROFILE_DIR" >/dev/null 2>&1 && [ "$i" -lt 20 ]; do + sleep 0.25 + i=$((i + 1)) +done + +# --use-fake-ui-for-media-stream auto-accepts the getUserMedia camera prompt instead +# of leaving it sitting unanswered on a screen nobody is there to click "Allow" on — +# VERIFY: the simplest documented way to do this: the alternative, +# VideoCaptureAllowedUrls enterprise policy, needs a policies.json this image doesn't +# currently ship, so start with the flag and only add that if it proves insufficient. +exec "$CHROMIUM" \ + --user-data-dir="$PROFILE_DIR" \ + --ozone-platform=wayland \ + --kiosk --app="$URL" \ + --use-fake-ui-for-media-stream \ + --start-fullscreen \ + --noerrdialogs --disable-infobars --disable-session-crashed-bubble \ + --overscroll-history-navigation=0 \ + --touch-events=enabled \ + --check-for-update-interval=31536000 diff --git a/hosts/kitchen-display/live-build/config/hooks/normal/0100-user-setup.hook.chroot b/hosts/kitchen-display/live-build/config/hooks/normal/0100-user-setup.hook.chroot new file mode 100755 index 0000000..9ade97e --- /dev/null +++ b/hosts/kitchen-display/live-build/config/hooks/normal/0100-user-setup.hook.chroot @@ -0,0 +1,42 @@ +#!/bin/sh +# Creates the kiosk account. Identical logic to hosts/touch-panel's/hosts/thin-client's +# own 0100-user-setup.hook.chroot. +set -eu + +. /etc/kitchen-display-agent/config.env + +if ! id "$KIOSK_USERNAME" >/dev/null 2>&1; then + useradd --create-home --shell /bin/bash --comment "Kitchen display kiosk session" "$KIOSK_USERNAME" +fi + +for grp in audio video input render dialout netdev plugdev seat _seatd; do + if getent group "$grp" >/dev/null 2>&1; then + adduser "$KIOSK_USERNAME" "$grp" >/dev/null + fi +done + +passwd --lock "$KIOSK_USERNAME" >/dev/null +adduser "$KIOSK_USERNAME" sudo >/dev/null + +cat > "/etc/sudoers.d/010-${KIOSK_USERNAME}" < /etc/ssh/sshd_config.d/10-kitchen-display.conf <<'EOF' +PermitRootLogin no +PasswordAuthentication no +KbdInteractiveAuthentication no +PubkeyAuthentication yes +EOF + +if [ -d "/home/${KIOSK_USERNAME}/.ssh" ]; then + chmod 700 "/home/${KIOSK_USERNAME}/.ssh" + [ -f "/home/${KIOSK_USERNAME}/.ssh/authorized_keys" ] && \ + chmod 600 "/home/${KIOSK_USERNAME}/.ssh/authorized_keys" +fi + +chown -R "${KIOSK_USERNAME}:${KIOSK_USERNAME}" "/home/${KIOSK_USERNAME}" + +systemctl enable ssh >/dev/null 2>&1 || true diff --git a/hosts/kitchen-display/live-build/config/hooks/normal/0200-greetd.hook.chroot b/hosts/kitchen-display/live-build/config/hooks/normal/0200-greetd.hook.chroot new file mode 100755 index 0000000..c7724cc --- /dev/null +++ b/hosts/kitchen-display/live-build/config/hooks/normal/0200-greetd.hook.chroot @@ -0,0 +1,19 @@ +#!/bin/sh +# Makes greetd the boot target. Identical logic to hosts/touch-panel's own +# 0200-greetd.hook.chroot. +set -eu + +. /etc/kitchen-display-agent/config.env + +chmod 0755 /usr/local/bin/kiosk-session + +if ! id greeter >/dev/null 2>&1; then + useradd --system --create-home --home-dir /var/lib/greetd --shell /usr/sbin/nologin greeter +fi + +systemctl enable greetd +systemctl set-default graphical.target +systemctl mask getty@tty1.service + +mkdir -p "/home/${KIOSK_USERNAME}/.config/sway" +chown -R "${KIOSK_USERNAME}:${KIOSK_USERNAME}" "/home/${KIOSK_USERNAME}/.config" diff --git a/hosts/kitchen-display/live-build/config/hooks/normal/0300-kitchen-display-agent.hook.chroot b/hosts/kitchen-display/live-build/config/hooks/normal/0300-kitchen-display-agent.hook.chroot new file mode 100755 index 0000000..ef750ea --- /dev/null +++ b/hosts/kitchen-display/live-build/config/hooks/normal/0300-kitchen-display-agent.hook.chroot @@ -0,0 +1,12 @@ +#!/bin/sh +# Installs the kitchen-display-agent systemd unit. Identical logic to +# hosts/touch-panel's own 0400-touchpanel-agent.hook.chroot. +set -eu + +install -m 0644 /opt/kitchen-display-agent/kitchen-display-agent.service \ + /etc/systemd/system/kitchen-display-agent.service + +chmod 0644 /etc/kitchen-display-agent/config.env +chown -R root:root /opt/kitchen-display-agent + +systemctl enable kitchen-display-agent diff --git a/hosts/kitchen-display/live-build/config/package-lists/kitchen-display.list.chroot b/hosts/kitchen-display/live-build/config/package-lists/kitchen-display.list.chroot new file mode 100644 index 0000000..2b6b158 --- /dev/null +++ b/hosts/kitchen-display/live-build/config/package-lists/kitchen-display.list.chroot @@ -0,0 +1,46 @@ +# Kitchen-display package list (live-build .list.chroot format: one package per +# line). Phase 17, docs/project-plan.md. + +# --- Wayland compositor / session --- +sway +swayidle +swaybg +greetd +foot + +# --- Browser (the only app on this device — a Chromium kiosk window pointed at +# pantry-web's frontend, which uses getUserMedia for camera capture directly in +# the browser rather than a native capture app) --- +chromium + +# --- Agent runtime --- +python3 +python3-venv +python3-paho-mqtt +procps + +# --- Networking --- +network-manager + +# --- Out-of-band admin — no wayvnc on this image either, same scope decision as +# hosts/touch-panel (see that host's README) --- +openssh-server +sudo + +# --- Camera troubleshooting (`v4l2-ctl --list-devices` to confirm the webcam +# enumerates correctly) — Chromium/getUserMedia talks to it directly, nothing +# here shells out to v4l2 itself --- +v4l-utils + +# --- Firmware for real hardware --- +firmware-linux +firmware-iwlwifi +firmware-realtek +firmware-misc-nonfree + +# --- Misc --- +ca-certificates +curl +openssl +less +vim-tiny diff --git a/hosts/kitchen-display/live-build/config/preseed.cfg b/hosts/kitchen-display/live-build/config/preseed.cfg new file mode 100644 index 0000000..bffd46a --- /dev/null +++ b/hosts/kitchen-display/live-build/config/preseed.cfg @@ -0,0 +1,3 @@ +# Only relevant when ENABLE_INSTALLER="true" in build-kitchen-display-iso.sh (off by +# default). Identical mechanism/caveat as the other hosts' preseed.cfg. +d-i keyboard-configuration/xkb-keymap select de diff --git a/hosts/kitchen-display/scripts/build-kitchen-display-iso.sh b/hosts/kitchen-display/scripts/build-kitchen-display-iso.sh new file mode 100755 index 0000000..667b7c1 --- /dev/null +++ b/hosts/kitchen-display/scripts/build-kitchen-display-iso.sh @@ -0,0 +1,265 @@ +#!/usr/bin/env bash +# +# Smart Home Kitchen-Display ISO Builder +# Target: builds a Debian 12 (Bookworm) live ISO on a Debian/Ubuntu build machine +# +# Drives `lb config && lb build` over hosts/kitchen-display/live-build/ to produce +# the fridge/pantry camera-cataloguing display described in docs/project-plan.md +# Phase 17: +# - greetd autologin straight into a kiosk Sway session (no greeter UI) +# - ONE Chromium kiosk window pointed at pantry-web's frontend (Scan / Inventory / +# Recipes is in-page tab navigation, not separate Sway workspaces) +# - kitchen-display-agent (Python, systemd) — HA MQTT "Show scan/inventory/recipes" +# buttons only; the actual camera capture and Grocy read/write path is +# pantry-vision's own published API, called directly by the browser +# +# Reuses hosts/thin-client's live-build toolchain and directory-split convention, not +# its live-build tree — same relationship hosts/touch-panel and hosts/audio-endpoint +# already have to the thin client's. +# +# Run as: sudo ./build-kitchen-display-iso.sh +# +# EDIT THE VARIABLES BELOW BEFORE RUNNING. + +set -euo pipefail + +# --------------------------------------------------------------------------- +# CONFIGURATION — edit these before running +# --------------------------------------------------------------------------- +DEBIAN_RELEASE="bookworm" +KIOSK_USERNAME="kiosk" +IMAGE_HOSTNAME="kitchen-display" +KITCHEN_DISPLAY_NAME="Kitchen fridge display" + +KEYBOARD_LAYOUT="de" + +ENABLE_INSTALLER="false" + +# --- Where the kitchen display talks to ------------------------------------- +MQTT_BROKER_HOST="192.168.1.10" # <-- EDIT: container-host IP running Mosquitto +MQTT_BROKER_PORT="1883" +MQTT_USERNAME="" +MQTT_PASSWORD="" + +# pantry-web (nginx, serves pantry-vision/frontend/) and pantry-vision (the API the +# frontend calls directly from the browser) — both from ENABLE_PANTRY_VISION in +# setup-container-host.sh. Placeholders until that's deployed; the image builds and +# boots fine without it, the kiosk window just shows a connection error. +PANTRY_WEB_URL="http://192.168.1.10:8096" # <-- EDIT once pantry-web is deployed +PANTRY_VISION_URL="http://192.168.1.10:8095" # <-- EDIT once pantry-vision is deployed +# Same value as PANTRY_VISION_TOKEN in pantry-vision/pantry-vision.env — there is no +# way for this repo to push it between the two hosts for you, same as every other +# credential pair that spans two machines in this project. +PANTRY_VISION_TOKEN="" # <-- EDIT: must match pantry-vision's own token + +SSH_AUTHORIZED_KEY="" + +# --------------------------------------------------------------------------- +# Paths +# --------------------------------------------------------------------------- +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +KITCHEN_DISPLAY_DIR="$(dirname "$SCRIPT_DIR")" +CONFIGS_DIR="${KITCHEN_DISPLAY_DIR}/configs" +AGENT_DIR="${KITCHEN_DISPLAY_DIR}/agent" +LIVE_BUILD_DIR="${KITCHEN_DISPLAY_DIR}/live-build" +INCLUDES="${LIVE_BUILD_DIR}/config/includes.chroot" +PACKAGE_LIST="${LIVE_BUILD_DIR}/config/package-lists/kitchen-display.list.chroot" + +# --------------------------------------------------------------------------- +# Sanity checks +# --------------------------------------------------------------------------- +if [[ $EUID -ne 0 ]]; then + echo "Warning: not running as root. 'lb build' needs root; re-run with: sudo $0" + echo " Continuing anyway so you can at least regenerate includes.chroot..." +fi + +if ! command -v lb &> /dev/null; then + if [[ $EUID -eq 0 ]]; then + echo "--- Installing live-build ---" + apt-get update + apt-get install -y live-build + else + echo "live-build is not installed and this script is not running as root." >&2 + echo " Install it first: sudo apt-get install live-build" >&2 + exit 1 + fi +else + echo "--- live-build already installed, skipping ---" +fi + +if [[ ! -f "$PACKAGE_LIST" ]]; then + echo "Missing package list: $PACKAGE_LIST" >&2 + exit 1 +fi + +if [[ "$MQTT_BROKER_HOST" == "192.168.1.10" ]]; then + echo "Warning: MQTT_BROKER_HOST is still the placeholder IP — edit it before building." +fi + +if [[ -z "$PANTRY_VISION_TOKEN" ]]; then + echo "Warning: PANTRY_VISION_TOKEN is empty. The kiosk window will load but every" + echo " call to pantry-vision will fail (401) until this matches the token in" + echo " pantry-vision/pantry-vision.env on the container host." +fi + +echo +echo "=== Smart Home Kitchen-Display ISO Builder ===" +echo "Debian release : $DEBIAN_RELEASE" +echo "Kiosk user : $KIOSK_USERNAME" +echo "Image hostname : $IMAGE_HOSTNAME" +echo "MQTT broker : ${MQTT_BROKER_HOST}:${MQTT_BROKER_PORT}" +echo "pantry-web : $PANTRY_WEB_URL" +echo "pantry-vision : $PANTRY_VISION_URL" +echo "Keyboard layout : $KEYBOARD_LAYOUT" +echo + +# --------------------------------------------------------------------------- +# 1. Regenerate includes.chroot from configs/ and agent/ +# --------------------------------------------------------------------------- +echo "--- Regenerating $INCLUDES ---" +rm -rf "$INCLUDES" +mkdir -p \ + "$INCLUDES/etc/greetd" \ + "$INCLUDES/etc/kitchen-display-agent" \ + "$INCLUDES/usr/local/bin" \ + "$INCLUDES/opt/kitchen-display-agent" \ + "$INCLUDES/home/${KIOSK_USERNAME}/.config/sway" \ + "$INCLUDES/home/${KIOSK_USERNAME}/.ssh" + +subst() { + sed -e "s/@KIOSK_USERNAME@/${KIOSK_USERNAME}/g" \ + -e "s/@KEYBOARD_LAYOUT@/${KEYBOARD_LAYOUT}/g" "$1" > "$2" +} + +subst "${CONFIGS_DIR}/greetd/config.toml" "$INCLUDES/etc/greetd/config.toml" +subst "${CONFIGS_DIR}/sway/config" "$INCLUDES/home/${KIOSK_USERNAME}/.config/sway/config" +subst "${AGENT_DIR}/kitchen-display-agent.service" "$INCLUDES/opt/kitchen-display-agent/kitchen-display-agent.service" + +install -m 0755 "${CONFIGS_DIR}/greetd/kiosk-session" "$INCLUDES/usr/local/bin/kiosk-session" +install -m 0755 "${CONFIGS_DIR}/sway/pantry-kiosk" "$INCLUDES/usr/local/bin/pantry-kiosk" + +mkdir -p "$INCLUDES/etc/default" +cat > "$INCLUDES/etc/default/keyboard" < "$INCLUDES/home/${KIOSK_USERNAME}/.ssh/authorized_keys" + chmod 600 "$INCLUDES/home/${KIOSK_USERNAME}/.ssh/authorized_keys" + echo " Baked an SSH authorized_keys entry for ${KIOSK_USERNAME}." +else + echo " No SSH_AUTHORIZED_KEY set — SSH admin access will not be possible on this image." +fi + +# --------------------------------------------------------------------------- +# 2. Runtime config +# --------------------------------------------------------------------------- +echo "--- Writing /etc/kitchen-display-agent/config.env into includes.chroot ---" +cat > "$INCLUDES/etc/kitchen-display-agent/config.env" < **No touch-panel hardware has been chosen.** Nothing here has been booted on real +> metal or a real touchscreen. Touch-as-`wl_touch` (as opposed to some panels that only +> expose an emulated-mouse HID interface) is assumed, not confirmed. + +Edit the `# CONFIGURATION` block at the top of +[`scripts/build-touch-panel-iso.sh`](scripts/build-touch-panel-iso.sh): + +| Variable | What to put in it | +|---|---| +| `MQTT_BROKER_HOST` | LAN IP of the container host running Mosquitto (Phase 1) | +| `HA_URL` | Home Assistant URL, e.g. `http://192.168.1.10:8123` — this is what loads in the Home workspace | +| `KIOSK_USERNAME` | Autologin account name (`kiosk`) | +| `TOUCHPANEL_NAME` / `IMAGE_HOSTNAME` | Per-panel identity; each touch panel needs its own | +| `SSH_AUTHORIZED_KEY` | Optional. Password auth is disabled and there is no wayvnc, so without a key the only admin path is the local console. | + +## Build + +```sh +sudo ./scripts/build-touch-panel-iso.sh +``` + +Same directory-split convention as the thin client: `configs/` and `agent/` are the +human-edited, git-tracked source of truth; `live-build/config/includes.chroot/` is +**generated** — wiped and rebuilt on every run, gitignored, never hand-edited. The only +templated tokens are `@KIOSK_USERNAME@`/`@KEYBOARD_LAYOUT@`; everything else the +in-chroot hooks need comes from `/etc/touchpanel-agent/config.env`. + +## Why there's no wayvnc on this image + +The thin client's primary control surface is deliberately remote (HA/MQTT + wayvnc) +because nobody is standing at a couch-distance media station. This panel is the +opposite: it's mounted somewhere and touched directly, so remote *view/control* isn't +the primary need the way it is for the thin client — SSH already covers "something's +broken, let me fix it from a shell." If a real need for a remote-view channel shows up +once this is on real hardware, adding wayvnc back is a small, isolated change (copy +`../thin-client/configs/wayvnc/` and its two hooks) — deliberately left out for now +rather than built against a guess. + +## Home Assistant login + +The Chromium kiosk window's profile (`~/.config/touchpanel-chromium-ha`) is persistent +across restarts specifically so logging into Home Assistant once survives a reboot — +unlike the thin client's digest/admin kiosk Firefox profiles, which are stateless +pages re-copied on every launch. Log in on first boot (tap the dock's Home button if +the window doesn't already have focus) and it should stay logged in after that, +**unverified** — depends on HA's own session cookie lifetime and whether Chromium's +Wayland profile persistence behaves the way assumed. + +## Spotify: full GUI, not a headless Connect receiver + +`../thin-client/` and `../audio-endpoint/` both run `spotifyd`/`librespot` — a +Spotify Connect *receiver* with no interface of its own, controlled from the phone +app or HA. This panel runs the real, official Spotify Linux client instead +(`configs/spotify/spotify-launch`, Flathub `com.spotify.Client`), because the whole +point of a touch panel is a screen you interact with directly: browsing your library, +searching, picking a playlist with a finger. **Spotify Premium is required for +playback either way** — same requirement as the headless receivers. + +Login is interactive, on the device, on first launch, and persists in the Flatpak's +own data directory (`~/.var/app/com.spotify.Client`) — nothing here wipes or reseeds +it. + +`touchpanel_agent/mpris_bridge.py` bridges Spotify's MPRIS interface (via +`playerctl`) into an HA `media_player`-shaped set of entities, identical in shape to +the thin client's own bridge, just pointed at `spotify` instead of `mpv`/`spotifyd` as +the player name — **unverified that the Spotify Flatpak's MPRIS bus name is actually +`spotify`** (some Flatpak-sandboxed apps register under a `flatpak.`-shaped +name instead). If Playback state never updates, check `playerctl -l` on the booted +panel and adjust `PLAYER_PRIORITY` in `mpris_bridge.py`. + +## Touch dock + +An always-visible bar reserved at the bottom of the screen (`configs/eww/`, layer-shell +`:exclusive true` so nothing ever tiles under or draws over it) with four buttons: +**Spotify**, **Home**, **Web**, and **Keyboard**. The first three call `swaymsg +workspace` directly — fixed constants in `eww.yuck`, nothing from MQTT/HA is ever +interpolated into them, same rule as the thin client's now-playing widget. This is the +touch-first equivalent of the thin client's HA **Screen** select entity — both exist, +one for a finger, one for the LLM. + +Session-scoped (`exec_always` in `configs/sway/config`), not part of +`touchpanel-agent`, for the identical reason the thin client's now-playing widget +isn't: compositor UI should die with the compositor, not need a restart-independent +lifecycle or open a second inbound control channel into the security-sensitive agent. + +**If `eww` isn't installed** (it's not in Debian bookworm main — +`0500-eww-dock.hook.chroot` tries apt and otherwise leaves a documented placeholder), +there is no on-screen app switcher at all. The panel still boots and shows Home; HA/ +LLM-driven app switching via `touchpanel-agent`'s **Screen** select and **Show +\** buttons still works either way, since that's a separate control path. + +## On-screen keyboard — no auto-show, by design + +`configs/keyboard/toggle-keyboard` toggles `wvkbd` on and off. There is deliberately +**no automatic show-on-text-field-focus** — that needs the `text-input-v3`/ +`virtual-keyboard-v1` Wayland protocols wired all the way through each app via a shell +component (the way Phosh does it for squeekboard), which this image doesn't run. +Instead: tap the dock's Keyboard button before typing, tap it again to dismiss. This +is a known, accepted limitation, not a bug to chase before the panel is otherwise +useful — see the verification list below if you want to pursue proper auto-show later. + +`wvkbd`, not `squeekboard`, was picked specifically because it doesn't assume that +shell integration exists; see `0700-onscreen-keyboard.hook.chroot`'s comment. It is +also not in Debian bookworm main, so that hook has the same apt-first/documented- +placeholder shape as `eww` and the thin client's `spotifyd`/`librespot` install. + +## Home Assistant entities + +`touchpanel-agent` publishes MQTT-discovery configs on connect. Under the MQTT +integration you should get one device per touch panel with: + +- **Show Spotify**, **Show Home**, **Show web browser** (buttons) — switch workspace + and, for Spotify/Home, focus the already-running app rather than relaunching it + (both are meant to stay open and stateful, unlike the thin client's stateless + digest/admin kiosk pages) +- **Screen** (select) — `1:spotify` / `2:home` / `3:web` +- **Playback state** (sensor, with track metadata as attributes), **Volume** + (number), and play/pause / next / previous / stop (buttons), bridged from + Spotify's own MPRIS interface +- A **media_player** discovery payload — see the thin client README's identical + caveat: stock Home Assistant's MQTT integration has no `media_player` platform: the + button/number/sensor entities above are what give playback control on a plain + install; the HACS *MQTT Media Player* integration is what picks up this one. + +### Security boundary + +Identical shape and reasoning to `../thin-client/README.md`'s section of the same +name: `touchpanel_agent/mqtt_discovery.py` is the **entire** inbound control surface +of this machine. The LLM never gets a direct network path here — the only chain is +*LLM tool call → HA service call → MQTT → touchpanel-agent*. No HTTP listener, no +websocket server, no exposed Sway IPC socket, no VNC (this image has none at all, see +above). New control features belong as additional MQTT entities, not as a second +listener. + +## Manual verification still outstanding + +None of this has been run on hardware. In rough order: + +1. The ISO builds at all (`lb build` is network-heavy and can fail on mirror hiccups). +2. greetd lands in Sway with no login prompt, on vt1, with getty@tty1 masked. +3. **Touch input actually arrives as `wl_touch`**, not an emulated pointer — the whole + design (native touch in Chromium/Firefox/Spotify, no gesture-translation layer) + assumes a real Wayland touchscreen driver. If it only shows up as a mouse, tapping + will still mostly work by accident (a single point maps to a click) but pinch/ + multi-touch in Chromium or Spotify won't. +4. `touchpanel-agent` connects to Mosquitto and the device appears in HA. +5. **Flathub app ID `com.spotify.Client` is correct** — flagged for verification in + `0300-flatpak-spotify.hook.chroot`, same as the thin client's Steam Link ID. +6. Spotify login persists in `~/.var/app/com.spotify.Client` across a reboot. +7. Spotify's real MPRIS bus name — assumed `spotify`, not confirmed (see the Spotify + section above); `playerctl -l` on the booted panel is the way to check. +8. **Chromium's Wayland `app_id` for a `--app=` kiosk window** — assumed to start with + `chromium` (`configs/sway/config`'s `for_window` rule and + `touchpanel_agent/main.py`'s `focus_criteria` both match on it). If focusing the + Home workspace's window from the dock/HA stops working, check the real app_id with + `swaymsg -t get_tree` and correct both places. +9. HA login persistence in the Chromium kiosk profile across a reboot (see above). +10. `eww` and `wvkbd` are not in Debian bookworm main — both hooks try apt and + otherwise leave a documented placeholder (dock absent / keyboard absent + respectively); neither has been checked against the real bookworm archive. +11. **The Firefox extension IDs in `configs/firefox/policies.json`** — same unverified + caveat as the thin client's copy; see that README's identical item. +12. **Phase 16's own "reactive path" check**: power off the container host and confirm + the panel still boots to Spotify + the browser (Home will show a connection + error, which is the expected/acceptable failure mode — unlike the thin client, + there's no local-media fallback to fall back to here since this device has no + local media source at all). +13. Keyboard layout defaults to German (`KEYBOARD_LAYOUT="de"`) — same per-image, + no-runtime-override handling as the thin client's identical setting. +14. Dock touch-target sizing (`configs/eww/eww.scss`, ~84px buttons) — a starting + guess, not measured against a real finger on real glass at the real screen DPI + chosen once hardware exists. diff --git a/hosts/touch-panel/agent/requirements.txt b/hosts/touch-panel/agent/requirements.txt new file mode 100644 index 0000000..c28136a --- /dev/null +++ b/hosts/touch-panel/agent/requirements.txt @@ -0,0 +1,8 @@ +# The ISO installs this from apt (python3-paho-mqtt, 1.6.x on bookworm) rather than +# pip — see live-build/config/package-lists/touch-panel.list.chroot. This file is for +# running the agent outside the image (development, a venv on a test box). The code +# works against both the 1.x and 2.x callback APIs. +paho-mqtt>=1.6 + +# Config is read from /etc/touchpanel-agent/config.env by a small parser in main.py, +# so there is no python-dotenv dependency. diff --git a/hosts/touch-panel/agent/touchpanel-agent.service b/hosts/touch-panel/agent/touchpanel-agent.service new file mode 100644 index 0000000..e9d2ceb --- /dev/null +++ b/hosts/touch-panel/agent/touchpanel-agent.service @@ -0,0 +1,31 @@ +[Unit] +Description=Touch panel agent (Home Assistant MQTT control surface for Sway) +Documentation=file:///opt/touchpanel-agent +# A system unit, not a `systemctl --user` unit — see thinclient-agent.service's +# identical reasoning: this must be reachable over MQTT whether or not a graphical +# session is up, and must survive sway restarting. Session-scoped bits (SWAYSOCK, +# XDG_RUNTIME_DIR, DBUS_SESSION_BUS_ADDRESS) are derived at call time in +# sway_control.SwayControl.session_env(). +# +# Deliberately NOT After=network-online.target: the panel must boot to a usable +# session (Spotify/HA/browser) with the container host powered off. +After=network.target + +[Service] +Type=simple +User=@KIOSK_USERNAME@ +Group=@KIOSK_USERNAME@ +WorkingDirectory=/opt/touchpanel-agent +Environment=PYTHONPATH=/opt/touchpanel-agent +Environment=PYTHONUNBUFFERED=1 +EnvironmentFile=-/etc/touchpanel-agent/config.env +ExecStart=/usr/bin/python3 -m touchpanel_agent.main +Restart=always +RestartSec=5 +# Sandboxing stops here on purpose: this unit's whole job is to spawn GUI child +# processes (Chromium, the Spotify flatpak, Firefox) that need the real /tmp, the +# user's home, and the session bus. +NoNewPrivileges=true + +[Install] +WantedBy=multi-user.target diff --git a/hosts/touch-panel/agent/touchpanel_agent/__init__.py b/hosts/touch-panel/agent/touchpanel_agent/__init__.py new file mode 100644 index 0000000..2c5ddc4 --- /dev/null +++ b/hosts/touch-panel/agent/touchpanel_agent/__init__.py @@ -0,0 +1,3 @@ +"""touchpanel-agent — Home Assistant MQTT control surface for a Sway touch panel.""" + +__version__ = "0.1.0" diff --git a/hosts/touch-panel/agent/touchpanel_agent/main.py b/hosts/touch-panel/agent/touchpanel_agent/main.py new file mode 100644 index 0000000..55456e9 --- /dev/null +++ b/hosts/touch-panel/agent/touchpanel_agent/main.py @@ -0,0 +1,211 @@ +"""touchpanel-agent entrypoint.""" + +from __future__ import annotations + +import logging +import os +import signal +import socket +import sys +import threading +from dataclasses import dataclass + +import paho.mqtt.client as mqtt + +from .mpris_bridge import MprisBridge +from .mqtt_discovery import Discovery +from .sway_control import WS_HOME, WS_SPOTIFY, WS_WEB, SwayControl + +CONFIG_PATH = os.environ.get("TOUCHPANEL_AGENT_CONFIG", "/etc/touchpanel-agent/config.env") + +CONFIG_KEYS = ( + "MQTT_BROKER_HOST", + "MQTT_BROKER_PORT", + "MQTT_USERNAME", + "MQTT_PASSWORD", + "HA_URL", + "KIOSK_USERNAME", + "TOUCHPANEL_NAME", +) + +WORKSPACES = (WS_SPOTIFY, WS_HOME, WS_WEB) + +log = logging.getLogger("touchpanel-agent") + + +@dataclass(frozen=True) +class App: + name: str + command: list[str] + process_pattern: str | None = None + workspace: str | None = None + focus_criteria: str | None = None + icon: str = "mdi:application" + + +def load_config(path: str = CONFIG_PATH) -> dict[str, str]: + values: dict[str, str] = {} + try: + with open(path, encoding="utf-8") as handle: + for line in handle: + line = line.strip() + if not line or line.startswith("#") or "=" not in line: + continue + key, _, value = line.partition("=") + values[key.strip()] = value.strip().strip('"').strip("'") + except OSError as exc: + log.warning("could not read %s (%s); falling back to the environment", path, exc) + + for key in CONFIG_KEYS: + if key in os.environ: + values[key] = os.environ[key] + + return values + + +def build_apps(_config: dict[str, str]) -> dict[str, App]: + return { + "spotify": App( + name="Spotify", + command=["/usr/local/bin/spotify-launch"], + process_pattern="com.spotify.Client", + workspace=WS_SPOTIFY, + focus_criteria='app_id="spotify"', + icon="mdi:spotify", + ), + "home": App( + name="Home", + # ha-kiosk is already running (started at session boot, see + # configs/sway/config) and supervises its own restart loop — this just + # switches workspace and focuses the existing window rather than + # relaunching it, since the HA dashboard is stateful. See ha-kiosk's own + # comment for why this is deliberately not a kill-and-relaunch like the + # thin client's digest canvas. + command=["/usr/local/bin/ha-kiosk"], + process_pattern="ha-kiosk", + workspace=WS_HOME, + focus_criteria='app_id="chromium.*"', + icon="mdi:home-assistant", + ), + "web_browser": App( + name="web browser", + command=["/usr/local/bin/web-browser"], + workspace=WS_WEB, + icon="mdi:web", + ), + } + + +def make_client(client_id: str) -> mqtt.Client: + # paho-mqtt 2.x requires an explicit callback API version; bookworm's + # python3-paho-mqtt is 1.6.x and has no such argument. VERSION1 is requested when + # available so the callback signatures below are identical under both. + callback_api = getattr(mqtt, "CallbackAPIVersion", None) + if callback_api is not None: + return mqtt.Client(callback_api.VERSION1, client_id=client_id) + return mqtt.Client(client_id=client_id) + + +def main() -> int: + logging.basicConfig( + level=logging.INFO, + format="%(asctime)s %(levelname)s %(name)s: %(message)s", + stream=sys.stdout, + ) + + config = load_config() + hostname = socket.gethostname() + node_id = "".join(c if c.isalnum() else "_" for c in hostname).strip("_") or "touchpanel" + friendly_name = config.get("TOUCHPANEL_NAME") or f"Touch panel ({hostname})" + + broker_host = config.get("MQTT_BROKER_HOST", "") + broker_port = int(config.get("MQTT_BROKER_PORT") or 1883) + + sway = SwayControl() + apps = build_apps(config) + + client = make_client(f"touchpanel-agent-{node_id}") + if config.get("MQTT_USERNAME"): + client.username_pw_set(config["MQTT_USERNAME"], config.get("MQTT_PASSWORD") or None) + + discovery = Discovery(client, node_id, friendly_name) + mpris = MprisBridge(discovery.publish_media_state, sway.session_env) + + def on_launch(key: str) -> None: + app = apps[key] + sway.launch_app( + app.command, + process_pattern=app.process_pattern, + workspace=app.workspace, + focus_criteria=app.focus_criteria, + ) + if app.workspace: + discovery.publish_workspace(app.workspace) + + def on_workspace(payload: str) -> None: + name = payload.strip() + # Enumerated, never passed through: see the security note in mqtt_discovery.py. + if name not in WORKSPACES: + log.warning("ignoring unknown workspace %r", name) + return + sway.switch_workspace(name) + discovery.publish_workspace(name) + + def on_connect(_client, _userdata, _flags, rc): + if rc != 0: + log.error("MQTT connection refused (rc=%s)", rc) + return + log.info("connected to MQTT broker %s:%s", broker_host, broker_port) + discovery.register_media_player(mpris.handle_command, mpris.set_volume) + discovery.register_app_launchers(apps, on_launch) + discovery.register_workspace_select(WORKSPACES, on_workspace, WS_HOME) + discovery.subscribe_all() + discovery.publish_available(True) + + def on_disconnect(_client, _userdata, rc): + log.warning("disconnected from MQTT broker (rc=%s); paho will retry", rc) + + def on_message(_client, _userdata, message): + discovery.dispatch(message.topic, message.payload.decode("utf-8", "replace")) + + client.on_connect = on_connect + client.on_disconnect = on_disconnect + client.on_message = on_message + client.will_set(discovery.availability_topic, "offline", qos=1, retain=True) + + stop_event = threading.Event() + + def handle_signal(_signum, _frame): + stop_event.set() + + signal.signal(signal.SIGTERM, handle_signal) + signal.signal(signal.SIGINT, handle_signal) + + if not broker_host: + log.error("MQTT_BROKER_HOST is not set in %s — running without HA control", CONFIG_PATH) + else: + # connect_async + loop_start, never a blocking connect(): the panel must come + # up and show Spotify/HA/the browser with the container host powered off, + # same "reactive path never depends on a remote service" rule as the thin + # client's Phase 11.10. + client.connect_async(broker_host, broker_port, keepalive=60) + client.loop_start() + + log.info("touchpanel-agent %s started (node_id=%s)", node_id, node_id) + if not config.get("HA_URL"): + log.warning("HA_URL is not set — the Home workspace will have nothing to show") + + try: + mpris.run_forever(stop_event) + finally: + log.info("shutting down") + if broker_host: + discovery.publish_available(False) + client.loop_stop() + client.disconnect() + + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/hosts/touch-panel/agent/touchpanel_agent/mpris_bridge.py b/hosts/touch-panel/agent/touchpanel_agent/mpris_bridge.py new file mode 100644 index 0000000..0a913f9 --- /dev/null +++ b/hosts/touch-panel/agent/touchpanel_agent/mpris_bridge.py @@ -0,0 +1,127 @@ +"""Bridges the Spotify client's MPRIS state to the HA media_player entity via playerctl. + +Identical in shape and reasoning to +hosts/thin-client/agent/thinclient_agent/mpris_bridge.py, with PLAYER_PRIORITY pointed +at the Spotify GUI client's own MPRIS bus name instead of mpv/spotifyd — see that +file's docstring for why this polls via subprocess rather than holding a dbus +connection open. +""" + +from __future__ import annotations + +import logging +import subprocess +import threading + +log = logging.getLogger(__name__) + +PLAYER_PRIORITY = "spotify,%any" + +_STATUS_TO_HA = { + "Playing": "playing", + "Paused": "paused", + "Stopped": "idle", +} + +_METADATA_FORMAT = "{{title}}\x1f{{artist}}\x1f{{album}}\x1f{{mpris:length}}\x1f{{mpris:artUrl}}" + + +class MprisBridge: + def __init__(self, publish_state, env_provider, poll_interval: float = 2.0): + self._publish_state = publish_state + # playerctl needs DBUS_SESSION_BUS_ADDRESS, which this system service does not + # inherit; SwayControl.session_env() derives it from the kiosk user's runtime dir. + self._env_provider = env_provider + self._poll_interval = poll_interval + self._last_state: dict | None = None + + def _playerctl(self, *args: str) -> str | None: + try: + result = subprocess.run( + ["playerctl", "-p", PLAYER_PRIORITY, *args], + env=self._env_provider(), + capture_output=True, + text=True, + timeout=5, + check=False, + ) + except (OSError, subprocess.TimeoutExpired) as exc: + log.debug("playerctl %s failed: %s", " ".join(args), exc) + return None + if result.returncode != 0: + return None + return result.stdout.strip() + + def read_state(self) -> dict: + status = self._playerctl("status") + if status is None: + return {"state": "off"} + + state = {"state": _STATUS_TO_HA.get(status, "idle")} + + metadata = self._playerctl("metadata", "--format", _METADATA_FORMAT) + if metadata: + title, artist, album, length, art_url = (metadata.split("\x1f") + [""] * 5)[:5] + state["title"] = title + state["artist"] = artist + state["album"] = album + state["art_url"] = art_url + if length.isdigit(): + state["duration"] = int(length) // 1_000_000 + + position = self._playerctl("position") + if position: + try: + state["position"] = int(float(position)) + except ValueError: + pass + + volume = self._playerctl("volume") + if volume: + try: + state["volume"] = round(float(volume), 3) + except ValueError: + pass + + return state + + def poll_once(self) -> None: + state = self.read_state() + if state != self._last_state: + self._last_state = state + self._publish_state(state) + + def run_forever(self, stop_event: threading.Event) -> None: + while not stop_event.is_set(): + try: + self.poll_once() + except Exception: + log.exception("MPRIS poll failed") + stop_event.wait(self._poll_interval) + + # --- command side ------------------------------------------------------- + def handle_command(self, command: str) -> None: + command = command.strip().upper() + action = { + "PLAY": ("play",), + "PAUSE": ("pause",), + "PLAY_PAUSE": ("play-pause",), + "TOGGLE": ("play-pause",), + "STOP": ("stop",), + "NEXT": ("next",), + "PREVIOUS": ("previous",), + "PREV": ("previous",), + }.get(command) + + if action is None: + log.warning("ignoring unknown media command %r", command) + return + + log.info("media command %s", command) + self._playerctl(*action) + self.poll_once() + + def set_volume(self, level: float) -> None: + level = max(0.0, min(1.0, level)) + self._playerctl("volume", f"{level:.3f}") + self.poll_once() diff --git a/hosts/touch-panel/agent/touchpanel_agent/mqtt_discovery.py b/hosts/touch-panel/agent/touchpanel_agent/mqtt_discovery.py new file mode 100644 index 0000000..0bfac9c --- /dev/null +++ b/hosts/touch-panel/agent/touchpanel_agent/mqtt_discovery.py @@ -0,0 +1,193 @@ +"""Home Assistant MQTT Discovery payloads and command dispatch. + +SECURITY BOUNDARY — this module is the entire remote-control API of the touch panel. + +Same principle as hosts/thin-client/agent/thinclient_agent/mqtt_discovery.py (Phase +11.4), applied to this device: the local LLM never gets a network path to this +machine. The only chain is: LLM tool call -> Home Assistant service call -> MQTT -> +this dispatcher. That property holds only as long as this stays the sole inbound +control surface: no HTTP listener, no websocket server, no exposed Sway IPC socket, no +shell endpoint. A new feature belongs as another entity below, not as another +listener. + +Every command handler here is a fixed, enumerated action. A payload never becomes an +argv element, a shell string, or a URL host — see the launch table in main.py, which +builds every command from local constants and uses the payload only to pick between +known values. +""" + +from __future__ import annotations + +import json +import logging +from typing import Callable + +from . import __version__ + +log = logging.getLogger(__name__) + +DISCOVERY_PREFIX = "homeassistant" + + +class Discovery: + def __init__(self, client, node_id: str, friendly_name: str): + self.client = client + self.node_id = node_id + self.friendly_name = friendly_name + self.base = f"touchpanel/{node_id}" + self.availability_topic = f"{self.base}/availability" + self.media_state_topic = f"{self.base}/media/state" + self._handlers: dict[str, Callable[[str], None]] = {} + + self.device = { + "identifiers": [f"touchpanel_{node_id}"], + "name": friendly_name, + "manufacturer": "SmartestHome", + "model": "Sway touch panel", + "sw_version": __version__, + } + + # --- plumbing ----------------------------------------------------------- + def _publish_config(self, component: str, object_id: str, payload: dict) -> None: + payload = { + "availability_topic": self.availability_topic, + "device": self.device, + "unique_id": f"{self.node_id}_{object_id}", + **payload, + } + topic = f"{DISCOVERY_PREFIX}/{component}/{self.node_id}/{object_id}/config" + self.client.publish(topic, json.dumps(payload), qos=1, retain=True) + + def _command_topic(self, suffix: str, handler) -> str: + topic = f"{self.base}/{suffix}" + self._handlers[topic] = handler + return topic + + def subscribe_all(self) -> None: + for topic in self._handlers: + self.client.subscribe(topic, qos=1) + + def dispatch(self, topic: str, payload: str) -> None: + handler = self._handlers.get(topic) + if handler is None: + log.warning("no handler for %s", topic) + return + try: + handler(payload) + except Exception: + log.exception("handler for %s failed", topic) + + def publish_available(self, available: bool = True) -> None: + self.client.publish( + self.availability_topic, + "online" if available else "offline", + qos=1, + retain=True, + ) + + def publish_media_state(self, state: dict) -> None: + self.client.publish(self.media_state_topic, json.dumps(state), qos=0, retain=True) + + # --- entities ----------------------------------------------------------- + def register_media_player(self, on_command, on_volume) -> None: + command_topic = self._command_topic("media/command", on_command) + volume_topic = self._command_topic("media/volume/set", lambda p: on_volume(float(p))) + + # Core Home Assistant's MQTT integration has NO media_player platform — see + # hosts/thin-client/README.md's identical caveat. The button/number entities + # below give the same transport control with stock HA. + self._publish_config( + "media_player", + "media", + { + "name": "Media", + "state_topic": self.media_state_topic, + "state_template": "{{ value_json.state }}", + "command_topic": command_topic, + "volume_command_topic": volume_topic, + "volume_state_topic": self.media_state_topic, + "volume_template": "{{ value_json.volume }}", + "title_template": "{{ value_json.title }}", + "artist_template": "{{ value_json.artist }}", + "album_template": "{{ value_json.album }}", + }, + ) + + for object_id, name, payload, icon in ( + ("media_play_pause", "Play/pause", "PLAY_PAUSE", "mdi:play-pause"), + ("media_next", "Next track", "NEXT", "mdi:skip-next"), + ("media_previous", "Previous track", "PREVIOUS", "mdi:skip-previous"), + ("media_stop", "Stop", "STOP", "mdi:stop"), + ): + self._publish_config( + "button", + object_id, + { + "name": name, + "command_topic": command_topic, + "payload_press": payload, + "icon": icon, + }, + ) + + self._publish_config( + "sensor", + "media_state", + { + "name": "Playback state", + "state_topic": self.media_state_topic, + "value_template": "{{ value_json.state }}", + "json_attributes_topic": self.media_state_topic, + "icon": "mdi:spotify", + }, + ) + + self._publish_config( + "number", + "media_volume", + { + "name": "Volume", + "command_topic": volume_topic, + "state_topic": self.media_state_topic, + "value_template": "{{ value_json.volume }}", + "min": 0, + "max": 1, + "step": 0.05, + "mode": "slider", + "icon": "mdi:volume-high", + }, + ) + + def register_app_launchers(self, apps, on_launch) -> None: + for key, app in apps.items(): + self._publish_config( + "button", + f"launch_{key}", + { + "name": f"Show {app.name}", + "command_topic": self._command_topic( + f"app/{key}/launch", + lambda _payload, key=key: on_launch(key), + ), + "icon": app.icon, + }, + ) + + def register_workspace_select(self, workspaces, on_workspace, state_topic_value) -> None: + self._publish_config( + "select", + "workspace", + { + "name": "Screen", + "command_topic": self._command_topic("workspace/set", on_workspace), + "state_topic": f"{self.base}/workspace/state", + "options": list(workspaces), + "icon": "mdi:view-dashboard", + }, + ) + self.client.publish( + f"{self.base}/workspace/state", state_topic_value, qos=1, retain=True + ) + + def publish_workspace(self, name: str) -> None: + self.client.publish(f"{self.base}/workspace/state", name, qos=1, retain=True) diff --git a/hosts/touch-panel/agent/touchpanel_agent/sway_control.py b/hosts/touch-panel/agent/touchpanel_agent/sway_control.py new file mode 100644 index 0000000..fd9a2f1 --- /dev/null +++ b/hosts/touch-panel/agent/touchpanel_agent/sway_control.py @@ -0,0 +1,115 @@ +"""Thin wrapper around swaymsg and local process launching. + +Identical in shape to hosts/thin-client/agent/thinclient_agent/sway_control.py — +duplicated rather than imported across hosts, same convention as the digest/admin +canvas SDK split in Phase 13. +""" + +from __future__ import annotations + +import glob +import logging +import os +import subprocess + +log = logging.getLogger(__name__) + +# Contract with configs/sway/config — these strings must match the `set $ws_*` lines. +WS_SPOTIFY = "1:spotify" +WS_HOME = "2:home" +WS_WEB = "3:web" + + +def runtime_dir() -> str: + return os.environ.get("XDG_RUNTIME_DIR") or f"/run/user/{os.getuid()}" + + +class SwayControl: + def _socket_path(self) -> str | None: + path = os.environ.get("SWAYSOCK") + if path and os.path.exists(path): + return path + # sway names the socket sway-ipc...sock, so the path changes every + # time sway restarts. touchpanel-agent is a system service that outlives the + # session, so the socket is re-resolved per call instead of cached at startup. + matches = sorted(glob.glob(os.path.join(runtime_dir(), "sway-ipc.*.sock"))) + return matches[-1] if matches else None + + def session_env(self) -> dict[str, str]: + env = dict(os.environ) + env["XDG_RUNTIME_DIR"] = runtime_dir() + env.setdefault("DBUS_SESSION_BUS_ADDRESS", f"unix:path={runtime_dir()}/bus") + env.setdefault("WAYLAND_DISPLAY", "wayland-1") + sock = self._socket_path() + if sock: + env["SWAYSOCK"] = sock + return env + + def swaymsg(self, *args: str) -> str | None: + if self._socket_path() is None: + log.warning("no sway IPC socket found; dropping command %s", " ".join(args)) + return None + try: + result = subprocess.run( + ["swaymsg", *args], + env=self.session_env(), + capture_output=True, + text=True, + timeout=10, + check=False, + ) + except (OSError, subprocess.TimeoutExpired) as exc: + log.warning("swaymsg %s failed: %s", " ".join(args), exc) + return None + if result.returncode != 0: + log.warning("swaymsg %s: %s", " ".join(args), result.stderr.strip()) + return None + return result.stdout + + def switch_workspace(self, name: str) -> None: + log.info("switching to workspace %s", name) + self.swaymsg("workspace", name) + + def focus_window(self, criteria: str) -> None: + self.swaymsg(f"[{criteria}] focus") + + def is_running(self, pattern: str) -> bool: + try: + result = subprocess.run( + ["pgrep", "-u", str(os.getuid()), "-f", pattern], + capture_output=True, + timeout=5, + check=False, + ) + except (OSError, subprocess.TimeoutExpired): + return False + return result.returncode == 0 + + def launch_app( + self, + command: list[str], + process_pattern: str | None = None, + workspace: str | None = None, + focus_criteria: str | None = None, + ) -> None: + if workspace: + self.switch_workspace(workspace) + + if process_pattern and self.is_running(process_pattern): + log.info("%s already running; focusing instead of launching", command[0]) + if focus_criteria: + self.focus_window(focus_criteria) + return + + log.info("launching %s", " ".join(command)) + try: + subprocess.Popen( + command, + env=self.session_env(), + stdin=subprocess.DEVNULL, + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + start_new_session=True, + ) + except OSError as exc: + log.error("could not launch %s: %s", " ".join(command), exc) diff --git a/hosts/touch-panel/configs/eww/eww.scss b/hosts/touch-panel/configs/eww/eww.scss new file mode 100644 index 0000000..7ba8dfa --- /dev/null +++ b/hosts/touch-panel/configs/eww/eww.scss @@ -0,0 +1,50 @@ +// Styling for the touch dock. Installed to +// /home//.config/eww/eww.scss by build-touch-panel-iso.sh. +// +// Touch targets are deliberately large (min ~72px tap area) — this is a finger +// interface, not a mouse one. + +$bg: rgba(16, 16, 20, 0.96); +$fg: #e8e8ec; +$muted: #9a9aa6; +$accent: #6ea8fe; + +.dock-root { + background-color: $bg; + padding: 6px 12px; +} + +.dock-btn { + min-width: 84px; + min-height: 68px; + background-color: transparent; + border: none; + border-radius: 12px; + padding: 4px 10px; + color: $muted; + + &:active { + background-color: rgba(255, 255, 255, 0.14); + } +} + +.dock-btn-active { + background-color: rgba(110, 168, 254, 0.16); + color: $fg; +} + +.dock-icon { + font-size: 26px; +} + +.dock-label { + font-size: 13px; + font-weight: 500; + color: inherit; +} + +.dock-sep { + min-width: 1px; + background-color: rgba(255, 255, 255, 0.12); + margin: 8px 4px; +} diff --git a/hosts/touch-panel/configs/eww/eww.yuck b/hosts/touch-panel/configs/eww/eww.yuck new file mode 100644 index 0000000..275c68f --- /dev/null +++ b/hosts/touch-panel/configs/eww/eww.yuck @@ -0,0 +1,44 @@ +;; The touch panel's on-screen dock — the primary local navigation surface (this +;; device is touched directly, unlike hosts/thin-client, which deliberately has no +;; bars because its primary control is HA/MQTT and wayvnc). Installed to +;; /home//.config/eww/eww.yuck by build-touch-panel-iso.sh. +;; +;; Session-scoped, not part of touchpanel-agent, same reasoning as the thin client's +;; now-playing widget: this is compositor UI, not device control, and dies with sway +;; rather than needing a restart-independent lifecycle. The :onclick strings below are +;; fixed constants in this file — nothing from MQTT, HA, or player metadata is ever +;; interpolated into them. + +(defpoll active_workspace :interval "1s" :initial "2:home" + "swaymsg -t get_workspaces | jq -r '.[] | select(.focused) | .name' 2>/dev/null || echo 2:home") + +(defwidget dock-btn [ws icon label] + (button :class {active_workspace == ws ? "dock-btn dock-btn-active" : "dock-btn"} + :onclick "swaymsg workspace ${ws}" + (box :orientation "vertical" :space-evenly false :spacing 2 + (label :class "dock-icon" :text icon) + (label :class "dock-label" :text label)))) + +(defwidget dock-root [] + (box :class "dock-root" :orientation "horizontal" :space-evenly false :spacing 8 + :halign "center" :valign "center" + (dock-btn :ws "1:spotify" :icon "🎵" :label "Spotify") + (dock-btn :ws "2:home" :icon "🏠" :label "Home") + (dock-btn :ws "3:web" :icon "🌐" :label "Web") + (box :class "dock-sep") + (button :class "dock-btn" :onclick "toggle-keyboard" + (box :orientation "vertical" :space-evenly false :spacing 2 + (label :class "dock-icon" :text "⌨") + (label :class "dock-label" :text "Keyboard"))))) + +;; :exclusive true reserves this strip so it is never overlapped by, and never steals +;; area from behind, the app on screen — unlike the thin client's overlay widgets, +;; which deliberately float over fullscreen video. A dock a finger can miss because +;; something else drew over it is worse than a smaller app area. +(defwindow touch-dock + :monitor 0 + :geometry (geometry :width "100%" :height "84px" :anchor "bottom center") + :stacking "overlay" + :exclusive true + :focusable false + (dock-root)) diff --git a/hosts/touch-panel/configs/firefox/policies.json b/hosts/touch-panel/configs/firefox/policies.json new file mode 100644 index 0000000..4cd0281 --- /dev/null +++ b/hosts/touch-panel/configs/firefox/policies.json @@ -0,0 +1,63 @@ +{ + "_comment_path": "Installed to /etc/firefox/policies/policies.json, the documented Linux location for Firefox enterprise policy. 0600-firefox.hook.chroot also links it into firefox-esr's distribution/ directory, an older location some builds read instead; whichever the installed build honours wins and the other is ignored. Every key below lives OUTSIDE the 'policies' object on purpose — Firefox flags unrecognised keys *inside* it as invalid policies in about:policies.", + + "_comment_extensions": "Same two extensions and the same unverified-ID caveat as hosts/thin-client/configs/firefox/policies.json — see that file's comment and the touch-panel README's verification list. Duplicated, not shared, per this project's established cross-host convention (Phase 13's admin-canvas/digest-canvas SDK split).", + + "_comment_kiosk_ui": "The Disable*/UserMessaging/FirefoxHome blocks all remove a prompt, tour, or nag that would otherwise sit on an unattended panel with nobody there to dismiss it.", + + "_comment_updates": "The browser is part of the image and is replaced by rebuilding and reflashing it, so in-browser updates would only produce version drift between panels plus a restart banner nobody is there to click.", + + "policies": { + "ExtensionSettings": { + "uBlock0@raymondhill.net": { + "installation_mode": "force_installed", + "install_url": "https://addons.mozilla.org/firefox/downloads/latest/ublock-origin/latest.xpi", + "default_area": "menupanel" + }, + "sponsorBlocker@ajay.app": { + "installation_mode": "force_installed", + "install_url": "https://addons.mozilla.org/firefox/downloads/latest/sponsorblock/latest.xpi", + "default_area": "menupanel" + } + }, + + "DisableProfileImport": true, + "DisableProfileRefresh": true, + "DisableFirefoxAccounts": true, + "DisableFirefoxStudies": true, + "DisableTelemetry": true, + "DisablePocket": true, + "DisableFeedbackCommands": true, + "DisableSetDesktopBackground": true, + "DontCheckDefaultBrowser": true, + "NoDefaultBookmarks": true, + "OfferToSaveLogins": false, + "PasswordManagerEnabled": false, + "PromptForDownloadLocation": false, + + "AppAutoUpdate": false, + "DisableAppUpdate": true, + + "UserMessaging": { + "WhatsNew": false, + "ExtensionRecommendations": false, + "FeatureRecommendations": false, + "UrlbarInterventions": false, + "SkipOnboarding": true, + "MoreFromMozilla": false + }, + + "FirefoxHome": { + "Search": true, + "TopSites": true, + "SponsoredTopSites": false, + "Highlights": false, + "Pocket": false, + "SponsoredPocket": false, + "Snippets": false + }, + + "OverrideFirstRunPage": "", + "OverridePostUpdatePage": "" + } +} diff --git a/hosts/touch-panel/configs/firefox/user.js b/hosts/touch-panel/configs/firefox/user.js new file mode 100644 index 0000000..d787693 --- /dev/null +++ b/hosts/touch-panel/configs/firefox/user.js @@ -0,0 +1,45 @@ +// Prefs for the touch panel's general-browsing Firefox profile. Installed by +// build-touch-panel-iso.sh into /home//.mozilla/firefox/web. +// +// Same shape as hosts/thin-client/configs/firefox/user.js — see that file's comment +// for why user.js (re-applied on every startup) rather than a one-shot prefs.js seed, +// and why anything expressible as enterprise policy lives in policies.json instead. +// Duplicated rather than shared between the two hosts on purpose — same call as +// admin-canvas/digest-canvas's SDK split in Phase 13: this profile is free to drift +// (touch-target padding, no ydotool-specific concerns) without touching the thin +// client's. + +user_pref("toolkit.legacyUserProfileCustomizations.stylesheets", true); + +// First-run / post-update interstitials. An unattended panel has nobody to close them. +user_pref("browser.startup.homepage_override.mstone", "ignore"); +user_pref("browser.aboutwelcome.enabled", false); +user_pref("browser.messaging-system.whatsNewPanel.enabled", false); +user_pref("trailhead.firstrun.didSeeAboutWelcome", true); +user_pref("datareporting.policy.firstRunURL", ""); +user_pref("datareporting.policy.dataSubmissionPolicyBypassNotification", true); + +// Session restore prompts after a power cut would leave the panel stuck on a +// "restore your session?" page nobody is there to click through. +user_pref("browser.sessionstore.resume_from_crash", false); +user_pref("browser.tabs.warnOnClose", false); +user_pref("browser.tabs.warnOnCloseOtherTabs", false); +user_pref("browser.warnOnQuit", false); + +// No permission doorhangers on a shared touch panel. Autoplay is left blocked +// (unlike the thin client's digest canvas) — general browsing has no reason to want +// arbitrary pages making sound on a household wall panel. +user_pref("permissions.default.desktop-notification", 2); +user_pref("permissions.default.geo", 2); + +user_pref("gfx.webrender.all", true); + +// Touch-scrolling tuning: Firefox's Wayland/GTK touch handling is on by default, but +// the overlay/kinetic scrollbar reads better at panel viewing distance than the +// default desktop one. +user_pref("widget.gtk.overlay-scrollbars.enabled", true); + +user_pref("browser.urlbar.suggest.topsites", false); +user_pref("browser.urlbar.suggest.quicksuggest.sponsored", false); +user_pref("browser.newtabpage.activity-stream.showSponsored", false); +user_pref("browser.newtabpage.activity-stream.showSponsoredTopSites", false); diff --git a/hosts/touch-panel/configs/firefox/userChrome.css b/hosts/touch-panel/configs/firefox/userChrome.css new file mode 100644 index 0000000..e45669c --- /dev/null +++ b/hosts/touch-panel/configs/firefox/userChrome.css @@ -0,0 +1,64 @@ +/* Minimal browser chrome for the touch panel's general-browsing Firefox window. + * Installed to /chrome/userChrome.css by build-touch-panel-iso.sh, and only + * read because user.js sets toolkit.legacyUserProfileCustomizations.stylesheets=true. + * + * What survives: back, forward, reload, and the address bar — the four things a + * finger actually needs to drive this window, same reasoning as + * hosts/thin-client/configs/firefox/userChrome.css (adapted here for a touchscreen + * instead of ydotool/wayvnc remote input). Every extra toolbar button is one more + * small target to mis-tap. + */ + +/* Tab strip. One page at a time; a tab bar on a shared panel just accumulates + * whatever the last person left open. */ +#TabsToolbar, +#tabbrowser-tabs, +#alltabs-button { + visibility: collapse !important; +} + +/* Menu bar and bookmarks bar. */ +#toolbar-menubar, +#PersonalToolbar, +#PlacesToolbar { + visibility: collapse !important; +} + +/* Everything on the nav bar except back / forward / reload / the address bar. */ +#home-button, +#library-button, +#sidebar-button, +#fxa-toolbar-menu-button, +#unified-extensions-button, +#PanelUI-button, +#downloads-button, +#save-to-pocket-button, +#pageActionButton, +#star-button-box, +#reader-mode-button, +#customizableui-special-spring1, +#customizableui-special-spring2 { + display: none !important; +} + +#urlbar-container { + min-width: 0 !important; +} + +#nav-bar { + border: none !important; + box-shadow: none !important; +} + +.titlebar-buttonbox-container, +.titlebar-spacer { + display: none !important; +} + +/* Touch targets: the remaining nav-bar buttons default to a mouse-sized hit area. + * Bumping padding here is cheaper than a full touch-optimised theme. */ +#nav-bar toolbarbutton { + padding: 8px !important; +} + +/* Findbar and notification popups stay: they are transient and user-initiated. */ diff --git a/hosts/touch-panel/configs/firefox/web-browser b/hosts/touch-panel/configs/firefox/web-browser new file mode 100755 index 0000000..1d3a87a --- /dev/null +++ b/hosts/touch-panel/configs/firefox/web-browser @@ -0,0 +1,34 @@ +#!/bin/sh +# Opens the general-browsing Firefox window — minimal chrome, not kiosk mode. +# Installed to /usr/local/bin/web-browser. Its own profile, separate from any other +# Firefox instance on this machine (there is only this one on the touch panel, but the +# profile is still named explicitly for the same reason hosts/thin-client keeps three +# separate ones: a Firefox profile can only be open in one process). +set -eu + +PROFILE_DIR="${HOME:-/home/$(id -un)}/.mozilla/firefox/web" + +if command -v firefox-esr >/dev/null 2>&1; then + FIREFOX=firefox-esr +else + FIREFOX=firefox +fi + +mkdir -p "$PROFILE_DIR/chrome" + +# Re-copied on every launch rather than once at build time: this is a locked-down +# profile with no interactive customisation expected, so keeping it in lockstep with +# /etc/touchpanel-firefox/ (edited by rebuilding the image) is simpler than a one-shot +# seed that could drift after a userChrome.css update. +cp /etc/touchpanel-firefox/userChrome.css "$PROFILE_DIR/chrome/userChrome.css" 2>/dev/null || true +cp /etc/touchpanel-firefox/user.js "$PROFILE_DIR/user.js" 2>/dev/null || true + +# Focus an already-open window rather than stacking a second one: this window holds +# state (history, a half-typed URL, a logged-in page) that a kill-and-relaunch would +# throw away. +if pgrep -u "$(id -u)" -f "$FIREFOX .*--profile $PROFILE_DIR" >/dev/null 2>&1; then + [ -n "${1:-}" ] && exec "$FIREFOX" --profile "$PROFILE_DIR" --new-tab "$1" + exit 0 +fi + +exec "$FIREFOX" --profile "$PROFILE_DIR" --new-instance --new-window "${1:-about:blank}" diff --git a/hosts/touch-panel/configs/greetd/config.toml b/hosts/touch-panel/configs/greetd/config.toml new file mode 100644 index 0000000..2025b49 --- /dev/null +++ b/hosts/touch-panel/configs/greetd/config.toml @@ -0,0 +1,14 @@ +# greetd — autologin straight into the kiosk Sway session, no greeter UI. +# Installed to /etc/greetd/config.toml by build-touch-panel-iso.sh, which substitutes +# @KIOSK_USERNAME@ with its own KIOSK_USERNAME variable on the way in. +# +# Same shape as hosts/thin-client/configs/greetd/config.toml — see that file's comment +# for why [default_session] (not [initial_session]) is what makes autologin permanent +# and self-healing after a crash or a `swaymsg exit`. + +[terminal] +vt = 1 + +[default_session] +command = "/usr/local/bin/kiosk-session" +user = "@KIOSK_USERNAME@" diff --git a/hosts/touch-panel/configs/greetd/kiosk-session b/hosts/touch-panel/configs/greetd/kiosk-session new file mode 100755 index 0000000..257283b --- /dev/null +++ b/hosts/touch-panel/configs/greetd/kiosk-session @@ -0,0 +1,25 @@ +#!/bin/sh +# greetd's default_session command. Installed to /usr/local/bin/kiosk-session. +# +# Same purpose as the thin client's own kiosk-session: gets MQTT_BROKER_HOST / HA_URL / +# SPOTIFY_APP_ID / etc into Sway's environment, since sway's config file has no way to +# read an env file itself but every `exec` line it runs inherits this process's +# environment. +set -eu + +[ -r /etc/touchpanel-agent/config.env ] && . /etc/touchpanel-agent/config.env +export MQTT_BROKER_HOST HA_URL + +export XDG_CURRENT_DESKTOP=sway +export XDG_SESSION_TYPE=wayland +export XDG_SESSION_DESKTOP=sway +export MOZ_ENABLE_WAYLAND=1 +# Chromium's Wayland/Ozone backend, needed for native touch events in the HA kiosk +# window (see configs/sway/ha-kiosk) — without it Chromium falls back to Xwayland, +# where touch input is translated through XTest and loses multi-touch gestures. +export OZONE_PLATFORM=wayland + +: "${XDG_RUNTIME_DIR:=/run/user/$(id -u)}" +export XDG_RUNTIME_DIR + +exec sway diff --git a/hosts/touch-panel/configs/keyboard/toggle-keyboard b/hosts/touch-panel/configs/keyboard/toggle-keyboard new file mode 100755 index 0000000..b24a67c --- /dev/null +++ b/hosts/touch-panel/configs/keyboard/toggle-keyboard @@ -0,0 +1,25 @@ +#!/bin/sh +# Toggles the on-screen keyboard. Installed to /usr/local/bin/toggle-keyboard, run +# from the dock's Keyboard button (configs/eww/eww.yuck) and bindable to any other +# tap target later. +# +# wvkbd, not squeekboard: squeekboard's auto-show behaviour is wired through Phosh's +# session/DBus setup, which this image doesn't run, so under plain Sway it would need +# the same manual-toggle treatment anyway while pulling in more of the GNOME-mobile +# stack. wvkbd is a small layer-shell surface built for exactly this — a wlroots +# kiosk with no shell-level input-method integration. +# +# KNOWN LIMITATION, documented rather than hidden: there is no automatic show-on- +# text-field-focus here (that needs the text-input-v3/virtual-keyboard-v1 protocols +# wired all the way through each app, which is inconsistent across Chromium/Firefox/ +# Spotify under plain wlroots). The dock's Keyboard button is therefore the only way +# to open it — tap before typing, tap again to dismiss. +set -eu + +if pgrep -u "$(id -u)" -x wvkbd-mobintl >/dev/null 2>&1; then + pkill -u "$(id -u)" -x wvkbd-mobintl +else + # -L: keyboard layer height in px. 280 is a starting point for a touch-panel-sized + # screen, not measured against real hardware — see the README's verification list. + wvkbd-mobintl -L 280 & +fi diff --git a/hosts/touch-panel/configs/spotify/spotify-launch b/hosts/touch-panel/configs/spotify/spotify-launch new file mode 100755 index 0000000..4a9366a --- /dev/null +++ b/hosts/touch-panel/configs/spotify/spotify-launch @@ -0,0 +1,20 @@ +#!/bin/sh +# Full Spotify GUI client (docs/project-plan.md Phase 16). Installed to +# /usr/local/bin/spotify-launch. +# +# Deliberately NOT spotifyd/librespot — those are the thin client's and audio-endpoint's +# headless Spotify Connect *receivers*, with no GUI at all. This device is touched +# directly, so it runs the real, official Spotify Linux client: its own library +# browser, search, and playlists. Spotify ships no apt package for bookworm, so this +# comes from Flathub (0300-flatpak.hook.chroot installs it) — same distribution +# channel already used for the thin client's Steam Link, and the only maintained one +# for Spotify on Linux. +# +# Login is interactive, on-device, on first launch (Spotify account/Premium required +# for playback — Premium is also what the thin client's headless receivers need). The +# Flatpak's own persistent data dir (~/.var/app/com.spotify.Client) keeps that session +# across restarts; nothing here wipes or reseeds it the way the kiosk Firefox profiles +# are re-copied on every launch, since staying logged in is the whole point here. +set -eu + +exec flatpak run com.spotify.Client "$@" diff --git a/hosts/touch-panel/configs/sway/config b/hosts/touch-panel/configs/sway/config new file mode 100644 index 0000000..6814dd2 --- /dev/null +++ b/hosts/touch-panel/configs/sway/config @@ -0,0 +1,126 @@ +# Sway kiosk session for the touch panel (docs/project-plan.md Phase 16). +# +# Installed to /home//.config/sway/config by build-touch-panel-iso.sh. +# Every `exec` below inherits the environment set by /usr/local/bin/kiosk-session, +# which sources /etc/touchpanel-agent/config.env — that is how $HA_URL gets here +# without this file being templated. +# +# This is a DIFFERENT device from hosts/thin-client: that one is a couch-distance +# media station whose primary control surface is HA/MQTT and wayvnc, deliberately with +# "no bars" (see its config's own comment). This one is touched directly, so the whole +# point is an always-visible on-screen dock a finger can hit — see the eww bar below. + +set $mod Mod4 +set $ws_spotify 1:spotify +set $ws_home 2:home +set $ws_web 3:web + +# Workspace names are a contract with touchpanel_agent/sway_control.py — changing one +# side means changing the other. + +# --------------------------------------------------------------------------- +# Output / input +# --------------------------------------------------------------------------- +output * bg #101014 solid_color + +input type:keyboard { + xkb_layout @KEYBOARD_LAYOUT@ +} + +# Real touchscreen hardware, not the thin client's couch-distance camera-gesture +# trick: libinput/Wayland surfaces (GTK, Chromium, Firefox) all speak wl_touch +# natively, so taps/scrolls/pinches just work without a pointer-emulation layer. +input type:touch { + map_to_output "*" +} + +input type:tablet_tool { + map_to_output "*" +} + +# --------------------------------------------------------------------------- +# Look +# --------------------------------------------------------------------------- +default_border none +default_floating_border none +hide_edge_borders both +gaps inner 0 +gaps outer 0 + +# Reserve a strip at the bottom for the touch dock (bar/app-switcher), so it is +# never covered by, and never steals area from behind, whatever app is on screen. +# :exclusive is the eww window's own setting (configs/eww/eww.yuck) — sway just +# needs to know nothing else should tile into that space, which layer-shell +# exclusivity already guarantees without a `bar {}` block here. + +# --------------------------------------------------------------------------- +# Session-scoped helpers +# --------------------------------------------------------------------------- +# touchpanel-agent is NOT started here — see thin-client's identical note. systemd +# owns it so it is reachable over MQTT whether or not a graphical session is up, and +# survives a sway restart. Starting it from sway too would give two competing MQTT +# clients. + +# The touch dock (eww bar) and the on-screen keyboard toggle both live for the +# session, not the system, for the same reason the thin client's now-playing widget +# does: they are compositor UI, not device control, and should die with sway rather +# than need their own restart-independent lifecycle. +exec_always eww daemon +exec_always eww open touch-dock + +# --------------------------------------------------------------------------- +# Applications +# --------------------------------------------------------------------------- +# No `assign` rules, same reasoning as the thin client: every launch path is explicit +# about its own workspace via touchpanel_agent.sway_control.launch_app(), so a +# separately-matched `assign` rule can't race it. +for_window [app_id="chromium.*"] fullscreen enable +for_window [app_id="spotify"] fullscreen enable + +# HA kiosk window and Spotify both auto-launch at session start — this is a fixed +# 3-app panel, not an on-demand surface like the thin client's digest/admin +# workspaces, so there is no "nothing to show yet" state to guard against beyond an +# unset HA_URL. +exec sh -c '[ -n "$HA_URL" ] && { swaymsg workspace $ws_home; /usr/local/bin/ha-kiosk; }' +exec sh -c 'swaymsg workspace $ws_spotify; /usr/local/bin/spotify-launch' +exec swaymsg workspace $ws_home + +# --------------------------------------------------------------------------- +# Idle +# --------------------------------------------------------------------------- +# Blank after 10 minutes to save the panel; any touch (or local input) resumes it. +# No lock screen, same reasoning as the thin client: a lock screen would make a wall +# panel unusable to anyone in the room who isn't holding a passcode. +exec swayidle -w \ + timeout 600 'swaymsg "output * power off"' \ + resume 'swaymsg "output * power on"' + +for_window [app_id="chromium.*"] inhibit_idle fullscreen +for_window [app_id="spotify"] inhibit_idle fullscreen + +# --------------------------------------------------------------------------- +# Local override keys — a fallback for standing in front of the machine (or a +# connected keyboard), not the primary control surface, which is touch + HA/MQTT. +# --------------------------------------------------------------------------- +bindsym $mod+Shift+Ctrl+m exec foot --title maintenance-shell +for_window [title="maintenance-shell"] floating enable, resize set width 800 height 500, move position center + +bindsym $mod+Return exec foot +bindsym $mod+q kill +bindsym $mod+f fullscreen toggle +bindsym $mod+Shift+c reload +bindsym $mod+1 workspace $ws_spotify +bindsym $mod+2 workspace $ws_home +bindsym $mod+3 workspace $ws_web + +bindsym XF86AudioPlay exec playerctl -p spotify play-pause +bindsym XF86AudioNext exec playerctl -p spotify next +bindsym XF86AudioPrev exec playerctl -p spotify previous +bindsym XF86AudioRaiseVolume exec wpctl set-volume @DEFAULT_AUDIO_SINK@ 5%+ +bindsym XF86AudioLowerVolume exec wpctl set-volume @DEFAULT_AUDIO_SINK@ 5%- +bindsym XF86AudioMute exec wpctl set-mute @DEFAULT_AUDIO_SINK@ toggle + +# Deliberately no exit binding — see the thin client's identical note: `swaymsg exit` +# would just drop to a black VT and greetd would autologin straight back in. + +workspace $ws_home diff --git a/hosts/touch-panel/configs/sway/ha-kiosk b/hosts/touch-panel/configs/sway/ha-kiosk new file mode 100755 index 0000000..2ef6363 --- /dev/null +++ b/hosts/touch-panel/configs/sway/ha-kiosk @@ -0,0 +1,47 @@ +#!/bin/sh +# Persistent Chromium kiosk window pointed at Home Assistant. Installed to +# /usr/local/bin/ha-kiosk. Launched once at session start (configs/sway/config) and +# supervised right here with a restart loop, not by sway (its `exec` has no +# restart-on-crash) or systemd (this needs the graphical session's env, and a system +# unit chasing a per-session Wayland socket is more trouble than a two-line loop). +# +# Unlike hosts/thin-client's digest-browser, this is never killed-and-relaunched on +# an MQTT command — touchpanel_agent.sway_control.launch_app()'s is_running/ +# focus_criteria path just switches workspace and focuses the existing window, since +# the HA dashboard is stateful (login session, scroll position, which view you left +# it on) and reloading it on every "show home" tap would throw that away for nothing. +set -eu + +PROFILE_DIR="${HOME:-/home/$(id -un)}/.config/touchpanel-chromium-ha" +URL="${HA_URL:-}" +[ -n "$URL" ] || { echo "ha-kiosk: HA_URL is unset" >&2; exit 1; } + +if command -v chromium >/dev/null 2>&1; then + CHROMIUM=chromium +elif command -v chromium-browser >/dev/null 2>&1; then + CHROMIUM=chromium-browser +else + echo "ha-kiosk: no chromium/chromium-browser binary found" >&2 + exit 1 +fi + +mkdir -p "$PROFILE_DIR" + +# VERIFY: Chromium's Wayland app_id for a --app= kiosk window is assumed to start +# with "chromium" (matched by configs/sway/config's [app_id="chromium.*"] and by +# touchpanel_agent's focus_criteria) — not confirmed against a real build. If the +# fullscreen/for_window rule or the "Home" dock button's focus-instead-of-relaunch +# stops working, check the real app_id with `swaymsg -t get_tree`. +while true; do + "$CHROMIUM" \ + --user-data-dir="$PROFILE_DIR" \ + --ozone-platform=wayland \ + --kiosk --app="$URL" \ + --start-fullscreen \ + --noerrdialogs --disable-infobars --disable-session-crashed-bubble \ + --overscroll-history-navigation=0 \ + --touch-events=enabled \ + --check-for-update-interval=31536000 \ + >/tmp/ha-kiosk.log 2>&1 || true + sleep 2 +done diff --git a/hosts/touch-panel/live-build/config/hooks/normal/0100-user-setup.hook.chroot b/hosts/touch-panel/live-build/config/hooks/normal/0100-user-setup.hook.chroot new file mode 100755 index 0000000..d80133f --- /dev/null +++ b/hosts/touch-panel/live-build/config/hooks/normal/0100-user-setup.hook.chroot @@ -0,0 +1,50 @@ +#!/bin/sh +# Creates the kiosk account the whole image is built around. Identical logic to +# hosts/thin-client's 0100-user-setup.hook.chroot — see that file's comments for the +# reasoning behind the locked password, passwordless sudo, and key-only SSH. +set -eu + +. /etc/touchpanel-agent/config.env + +if ! id "$KIOSK_USERNAME" >/dev/null 2>&1; then + useradd --create-home --shell /bin/bash --comment "Touch panel kiosk session" "$KIOSK_USERNAME" +fi + +for grp in audio video input render dialout netdev plugdev seat _seatd; do + if getent group "$grp" >/dev/null 2>&1; then + adduser "$KIOSK_USERNAME" "$grp" >/dev/null + fi +done + +# No password is baked in: the account is locked so it can never be used to log in +# remotely, while the physical console still autologins via greetd. +passwd --lock "$KIOSK_USERNAME" >/dev/null + +adduser "$KIOSK_USERNAME" sudo >/dev/null + +# Passwordless sudo: this image autologins to an unattended interactive Sway session +# at the physical console, so anyone standing in front of the panel already has the +# equivalent of a root shell. The boundary that actually matters is key-only SSH below +# — there is no wayvnc on this image (see the README for that scope decision). +cat > "/etc/sudoers.d/010-${KIOSK_USERNAME}" < /etc/ssh/sshd_config.d/10-touch-panel.conf <<'EOF' +PermitRootLogin no +PasswordAuthentication no +KbdInteractiveAuthentication no +PubkeyAuthentication yes +EOF + +if [ -d "/home/${KIOSK_USERNAME}/.ssh" ]; then + chmod 700 "/home/${KIOSK_USERNAME}/.ssh" + [ -f "/home/${KIOSK_USERNAME}/.ssh/authorized_keys" ] && \ + chmod 600 "/home/${KIOSK_USERNAME}/.ssh/authorized_keys" +fi + +chown -R "${KIOSK_USERNAME}:${KIOSK_USERNAME}" "/home/${KIOSK_USERNAME}" + +systemctl enable ssh >/dev/null 2>&1 || true diff --git a/hosts/touch-panel/live-build/config/hooks/normal/0200-greetd.hook.chroot b/hosts/touch-panel/live-build/config/hooks/normal/0200-greetd.hook.chroot new file mode 100755 index 0000000..485b86e --- /dev/null +++ b/hosts/touch-panel/live-build/config/hooks/normal/0200-greetd.hook.chroot @@ -0,0 +1,23 @@ +#!/bin/sh +# Makes greetd the boot target so the machine comes up straight in the kiosk Sway +# session. Identical logic to hosts/thin-client's 0200-greetd.hook.chroot. +set -eu + +. /etc/touchpanel-agent/config.env + +chmod 0755 /usr/local/bin/kiosk-session + +if ! id greeter >/dev/null 2>&1; then + useradd --system --create-home --home-dir /var/lib/greetd --shell /usr/sbin/nologin greeter +fi + +systemctl enable greetd +systemctl set-default graphical.target + +# live-config would otherwise autologin its own account on tty1 and fight greetd for +# the VT. build-touch-panel-iso.sh passes `noautologin` on the kernel command line; +# masking the getty on vt1 makes that robust even if someone edits the boot args. +systemctl mask getty@tty1.service + +mkdir -p "/home/${KIOSK_USERNAME}/.config/sway" +chown -R "${KIOSK_USERNAME}:${KIOSK_USERNAME}" "/home/${KIOSK_USERNAME}/.config" diff --git a/hosts/touch-panel/live-build/config/hooks/normal/0300-flatpak-spotify.hook.chroot b/hosts/touch-panel/live-build/config/hooks/normal/0300-flatpak-spotify.hook.chroot new file mode 100755 index 0000000..e799d9b --- /dev/null +++ b/hosts/touch-panel/live-build/config/hooks/normal/0300-flatpak-spotify.hook.chroot @@ -0,0 +1,24 @@ +#!/bin/sh +# Flathub remote + the full Spotify GUI client (docs/project-plan.md Phase 16). +# +# Spotify ships no apt package for bookworm, so this comes from Flathub — the same +# distribution channel already used for hosts/thin-client's Steam Link. No release +# version is pinned, same "don't hardcode a URL that will just rot" reasoning as that +# hook and as 0500-spotify-connect.hook.chroot's headless-receiver install. +set -eu + +flatpak remote-add --if-not-exists flathub https://dl.flathub.org/repo/flathub.flatpakrepo + +# VERIFY BEFORE THE FIRST REAL BUILD: confirm this application ID against the live +# Flathub listing (`flatpak search Spotify`). Believed correct (it's Spotify's own +# published app on Flathub) but not checked from this environment. +SPOTIFY_APP_ID="com.spotify.Client" + +if flatpak install -y --noninteractive flathub "$SPOTIFY_APP_ID"; then + echo "0300-flatpak-spotify: installed ${SPOTIFY_APP_ID}." +else + echo "0300-flatpak-spotify: WARNING — could not install ${SPOTIFY_APP_ID} during the" + echo " build (no network in the chroot, or the app ID is wrong). Run this on the" + echo " booted image instead: flatpak install -y flathub ${SPOTIFY_APP_ID}" + echo " The Spotify workspace will show a blank window until this is done." +fi diff --git a/hosts/touch-panel/live-build/config/hooks/normal/0400-touchpanel-agent.hook.chroot b/hosts/touch-panel/live-build/config/hooks/normal/0400-touchpanel-agent.hook.chroot new file mode 100755 index 0000000..aff69a0 --- /dev/null +++ b/hosts/touch-panel/live-build/config/hooks/normal/0400-touchpanel-agent.hook.chroot @@ -0,0 +1,14 @@ +#!/bin/sh +# Installs the touchpanel-agent systemd unit. Identical logic to hosts/thin-client's +# 0700-thinclient-agent.hook.chroot — the package arrives under /opt/touchpanel-agent +# via includes.chroot and runs on the system interpreter against apt's +# python3-paho-mqtt, so there is nothing to pip-install here. +set -eu + +install -m 0644 /opt/touchpanel-agent/touchpanel-agent.service \ + /etc/systemd/system/touchpanel-agent.service + +chmod 0644 /etc/touchpanel-agent/config.env +chown -R root:root /opt/touchpanel-agent + +systemctl enable touchpanel-agent diff --git a/hosts/touch-panel/live-build/config/hooks/normal/0500-eww-dock.hook.chroot b/hosts/touch-panel/live-build/config/hooks/normal/0500-eww-dock.hook.chroot new file mode 100755 index 0000000..2c98bbc --- /dev/null +++ b/hosts/touch-panel/live-build/config/hooks/normal/0500-eww-dock.hook.chroot @@ -0,0 +1,20 @@ +#!/bin/sh +# The touch dock widget (docs/project-plan.md Phase 16). Identical apt-first/ +# documented-placeholder pattern as hosts/thin-client's 0800-eww-widget.hook.chroot — +# eww is not in Debian bookworm main (a Rust binary, normally cargo-built or fetched +# as a GitHub release). +set -eu + +if apt-get install -y --no-install-recommends eww 2>/dev/null; then + echo "0500-eww-dock: installed eww from apt." +else + echo "0500-eww-dock: eww is not available from the configured apt sources." + echo " The touch dock (the on-screen Spotify/Home/Web/Keyboard bar) will not appear" + echo " on this image until eww is installed:" + echo " a) cargo install eww (needs a Rust toolchain in a build hook), or" + echo " b) fetch a release binary from https://github.com/elkowar/eww/releases" + echo " (check the current tag yourself) into /usr/local/bin/eww." + echo " Without it there is NO on-screen app switcher — the panel still boots and" + echo " shows Home, but touch navigation between apps needs the dock. Home Assistant/" + echo " MQTT app switching (touchpanel_agent) still works either way." +fi diff --git a/hosts/touch-panel/live-build/config/hooks/normal/0600-firefox.hook.chroot b/hosts/touch-panel/live-build/config/hooks/normal/0600-firefox.hook.chroot new file mode 100755 index 0000000..e3a7d16 --- /dev/null +++ b/hosts/touch-panel/live-build/config/hooks/normal/0600-firefox.hook.chroot @@ -0,0 +1,17 @@ +#!/bin/sh +# Firefox enterprise policy plumbing for the general-browsing window. Identical logic +# to hosts/thin-client's 0900-firefox.hook.chroot. +set -eu + +FIREFOX_LIB_DIR="/usr/lib/firefox-esr" + +if [ -d "$FIREFOX_LIB_DIR" ]; then + mkdir -p "$FIREFOX_LIB_DIR/distribution" + ln -sf /etc/firefox/policies/policies.json "$FIREFOX_LIB_DIR/distribution/policies.json" + echo "0600-firefox: linked policies.json into $FIREFOX_LIB_DIR/distribution/" +else + echo "0600-firefox: $FIREFOX_LIB_DIR not found (firefox-esr not installed yet, or a" + echo " different path in this Debian release) — /etc/firefox/policies/policies.json" + echo " still applies if the package's install order runs after this hook; verify" + echo " about:policies shows uBlock Origin/SponsorBlock as force-installed on first boot." +fi diff --git a/hosts/touch-panel/live-build/config/hooks/normal/0700-onscreen-keyboard.hook.chroot b/hosts/touch-panel/live-build/config/hooks/normal/0700-onscreen-keyboard.hook.chroot new file mode 100755 index 0000000..7f4f8ed --- /dev/null +++ b/hosts/touch-panel/live-build/config/hooks/normal/0700-onscreen-keyboard.hook.chroot @@ -0,0 +1,22 @@ +#!/bin/sh +# On-screen keyboard (docs/project-plan.md Phase 16), toggled from the touch dock's +# Keyboard button. Same apt-first/documented-placeholder pattern as eww and the +# headless Spotify Connect receiver: wvkbd is not in Debian bookworm main (it's a +# small C wlroots layer-shell client, normally built from source). +set -eu + +if apt-get install -y --no-install-recommends wvkbd 2>/dev/null; then + echo "0700-onscreen-keyboard: installed wvkbd from apt." +elif command -v wvkbd-mobintl >/dev/null 2>&1; then + echo "0700-onscreen-keyboard: wvkbd-mobintl already present." +else + echo "0700-onscreen-keyboard: wvkbd is not available from the configured apt sources." + echo " Typing into Spotify search, the HA dashboard, or the web browser will have no" + echo " on-screen keyboard until it's installed:" + echo " a) build from source: https://github.com/jjsullivan5196/wvkbd" + echo " (needs libwayland-dev, wayland-protocols, cairo, pango — add a" + echo " build-dependencies block to this hook if you go this route), or" + echo " b) fetch a distro package once one exists for your release." + echo " configs/keyboard/toggle-keyboard already handles a missing binary as a no-op" + echo " (the dock button just does nothing), so the rest of the session is unaffected." +fi diff --git a/hosts/touch-panel/live-build/config/package-lists/touch-panel.list.chroot b/hosts/touch-panel/live-build/config/package-lists/touch-panel.list.chroot new file mode 100644 index 0000000..5289963 --- /dev/null +++ b/hosts/touch-panel/live-build/config/package-lists/touch-panel.list.chroot @@ -0,0 +1,66 @@ +# Touch-panel package list (live-build .list.chroot format: one package per line). +# Phase 16, docs/project-plan.md. + +# --- Wayland compositor / session --- +sway +swayidle +swaybg +greetd +foot + +# --- Browsers --- +firefox-esr +chromium +playerctl + +# --- Audio --- +pipewire +pipewire-pulse +wireplumber +alsa-utils + +# --- Spotify GUI client and Steam-Link-style apps come from Flathub, not apt +# (see 0300-flatpak-spotify.hook.chroot) --- +flatpak + +# --- Agent runtime --- +python3 +python3-venv +# Debian bookworm ships paho-mqtt 1.6.x. touchpanel_agent is written to work against +# both the 1.x and 2.x callback APIs, so this package is used as-is rather than pip'd. +python3-paho-mqtt +procps + +# --- Networking --- +network-manager + +# --- Out-of-band admin. This image has no wayvnc (unlike hosts/thin-client) — see +# the README's scope note — so SSH is the only remote admin path. --- +openssh-server +sudo + +# --- Touch dock (0500-eww-dock.hook.chroot tries apt, falls back to a documented +# placeholder if not packaged) --- +eww +jq + +# --- On-screen keyboard (0700-onscreen-keyboard.hook.chroot, same apt-first/ +# documented-fallback pattern) --- +wvkbd + +# --- Firmware for real hardware (needs contrib + non-free-firmware archive areas, +# which build-touch-panel-iso.sh passes to `lb config`) --- +firmware-linux +firmware-iwlwifi +firmware-realtek +firmware-misc-nonfree + +# --- Misc --- +ca-certificates +curl +openssl +less +vim-tiny + +# NOTE: Spotify (com.spotify.Client) is Flatpak-only — see +# 0300-flatpak-spotify.hook.chroot. There is no apt package. diff --git a/hosts/touch-panel/live-build/config/preseed.cfg b/hosts/touch-panel/live-build/config/preseed.cfg new file mode 100644 index 0000000..c209bae --- /dev/null +++ b/hosts/touch-panel/live-build/config/preseed.cfg @@ -0,0 +1,7 @@ +# Only relevant when ENABLE_INSTALLER="true" in build-touch-panel-iso.sh (off by +# default — the normal path is live-boot straight into Sway, no disk install at all). +# Identical mechanism/caveat as hosts/thin-client/live-build/config/preseed.cfg. +# +# No "keyboard-configuration/xkb-keymap seen true" line on purpose: this preseeds the +# *default* answer, it does not skip the question. +d-i keyboard-configuration/xkb-keymap select de diff --git a/hosts/touch-panel/scripts/build-touch-panel-iso.sh b/hosts/touch-panel/scripts/build-touch-panel-iso.sh new file mode 100755 index 0000000..65a5c93 --- /dev/null +++ b/hosts/touch-panel/scripts/build-touch-panel-iso.sh @@ -0,0 +1,297 @@ +#!/usr/bin/env bash +# +# Smart Home Touch-Panel ISO Builder +# Target: builds a Debian 12 (Bookworm) live ISO on a Debian/Ubuntu build machine +# +# Drives `lb config && lb build` over hosts/touch-panel/live-build/ to produce the +# Sway touch-panel image described in docs/project-plan.md Phase 16: +# - greetd autologin straight into a kiosk Sway session (no greeter UI) +# - touchpanel-agent (Python, systemd) — HA MQTT-discovery entities + swaymsg control +# - workspaces 1:spotify (full Spotify GUI, Flathub) / 2:home (Chromium kiosk +# pointed at Home Assistant) / 3:web (general Firefox browsing) +# - an always-visible eww touch dock for on-screen app switching +# - a toggleable on-screen keyboard (wvkbd) +# +# Reuses hosts/thin-client's own live-build toolchain and directory-split convention, +# NOT its live-build tree — this is a different device (touched directly, not +# remote-controlled from the couch) with a different app set, so it gets its own +# smaller config/, agent/, and live-build/, same relationship as +# hosts/audio-endpoint's amd64 build to the thin client's. +# +# This script is also the single point that keeps configs/ (the human-edited source of +# truth, reviewed in git) in sync with live-build/config/includes.chroot/ (the +# generated tree that actually gets baked into the image). Never hand-edit anything +# under includes.chroot — it is wiped and regenerated on every run. +# +# Run as: sudo ./build-touch-panel-iso.sh +# +# EDIT THE VARIABLES BELOW BEFORE RUNNING. + +set -euo pipefail + +# --------------------------------------------------------------------------- +# CONFIGURATION — edit these before running +# --------------------------------------------------------------------------- +DEBIAN_RELEASE="bookworm" # Matches the container host's OS +KIOSK_USERNAME="kiosk" # The autologin account the whole image is built around +IMAGE_HOSTNAME="touch-panel" # Hostname baked into the image +TOUCHPANEL_NAME="Kitchen touch panel" # Friendly name shown on the HA device + +KEYBOARD_LAYOUT="de" # xkb layout name (`localectl list-x11-keymap-layouts`) + +ENABLE_INSTALLER="false" # "true" adds a debian-installer to the ISO (install to disk) + +# --- Where the touch panel talks to ----------------------------------------- +MQTT_BROKER_HOST="192.168.1.10" # <-- EDIT: container-host IP running Mosquitto +MQTT_BROKER_PORT="1883" +MQTT_USERNAME="" # Leave empty while Mosquitto runs allow_anonymous +MQTT_PASSWORD="" # Never commit a real value here +HA_URL="http://192.168.1.10:8123" # <-- EDIT: Home Assistant URL — the "Home" workspace + +# Optional: an SSH public key to bake into the kiosk account for out-of-band admin. +# The image ships with password auth disabled and no wayvnc (see README's scope note), +# so without this the only admin path is the local console. +SSH_AUTHORIZED_KEY="" + +# --------------------------------------------------------------------------- +# Paths +# --------------------------------------------------------------------------- +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +TOUCH_PANEL_DIR="$(dirname "$SCRIPT_DIR")" +CONFIGS_DIR="${TOUCH_PANEL_DIR}/configs" +AGENT_DIR="${TOUCH_PANEL_DIR}/agent" +LIVE_BUILD_DIR="${TOUCH_PANEL_DIR}/live-build" +INCLUDES="${LIVE_BUILD_DIR}/config/includes.chroot" +PACKAGE_LIST="${LIVE_BUILD_DIR}/config/package-lists/touch-panel.list.chroot" + +# --------------------------------------------------------------------------- +# Sanity checks +# --------------------------------------------------------------------------- +if [[ $EUID -ne 0 ]]; then + echo "Warning: not running as root. 'lb build' needs root to bootstrap and chroot," + echo " and will fail partway through. Re-run with: sudo $0" + echo " Continuing anyway so you can at least regenerate includes.chroot..." +fi + +if ! grep -qi "debian\|ubuntu" /etc/os-release; then + echo "Warning: live-build targets a Debian/Ubuntu build host. Proceeding anyway..." +fi + +if ! command -v lb &> /dev/null; then + if [[ $EUID -eq 0 ]]; then + echo "--- Installing live-build ---" + apt-get update + apt-get install -y live-build + else + echo "live-build is not installed and this script is not running as root." >&2 + echo " Install it first: sudo apt-get install live-build" >&2 + exit 1 + fi +else + echo "--- live-build already installed, skipping ---" +fi + +if [[ ! -f "$PACKAGE_LIST" ]]; then + echo "Missing package list: $PACKAGE_LIST" >&2 + exit 1 +fi + +if [[ "$MQTT_BROKER_HOST" == "192.168.1.10" ]]; then + echo "Warning: MQTT_BROKER_HOST is still the placeholder IP." + echo " Edit it at the top of this script to your container host's real LAN address," + echo " or the touch panel won't show up as Home Assistant entities." +fi + +if [[ "$HA_URL" == "http://192.168.1.10:8123" ]]; then + echo "Warning: HA_URL is still the placeholder." + echo " The image builds and boots fine, but the Home workspace will show a" + echo " connection error until this points at a real Home Assistant." +fi + +echo +echo "=== Smart Home Touch-Panel ISO Builder ===" +echo "Debian release : $DEBIAN_RELEASE" +echo "Kiosk user : $KIOSK_USERNAME" +echo "Image hostname : $IMAGE_HOSTNAME" +echo "MQTT broker : ${MQTT_BROKER_HOST}:${MQTT_BROKER_PORT}" +echo "Home Assistant : $HA_URL" +echo "Keyboard layout : $KEYBOARD_LAYOUT" +echo + +# --------------------------------------------------------------------------- +# 1. Regenerate includes.chroot from configs/ and agent/ +# --------------------------------------------------------------------------- +echo "--- Regenerating $INCLUDES ---" +rm -rf "$INCLUDES" +mkdir -p \ + "$INCLUDES/etc/greetd" \ + "$INCLUDES/etc/touchpanel-agent" \ + "$INCLUDES/etc/firefox/policies" \ + "$INCLUDES/etc/touchpanel-firefox" \ + "$INCLUDES/usr/local/bin" \ + "$INCLUDES/opt/touchpanel-agent" \ + "$INCLUDES/home/${KIOSK_USERNAME}/.config/sway" \ + "$INCLUDES/home/${KIOSK_USERNAME}/.config/eww" \ + "$INCLUDES/home/${KIOSK_USERNAME}/.ssh" + +# @KIOSK_USERNAME@ and @KEYBOARD_LAYOUT@ are the only templated tokens in the configs. +# Everything else the hooks need is read at build time from +# /etc/touchpanel-agent/config.env (written below), which live-build copies in +# (chroot_local-includes) before it runs the hooks (chroot_local-hooks). +subst() { + sed -e "s/@KIOSK_USERNAME@/${KIOSK_USERNAME}/g" \ + -e "s/@KEYBOARD_LAYOUT@/${KEYBOARD_LAYOUT}/g" "$1" > "$2" +} + +subst "${CONFIGS_DIR}/greetd/config.toml" "$INCLUDES/etc/greetd/config.toml" +subst "${CONFIGS_DIR}/sway/config" "$INCLUDES/home/${KIOSK_USERNAME}/.config/sway/config" +subst "${AGENT_DIR}/touchpanel-agent.service" "$INCLUDES/opt/touchpanel-agent/touchpanel-agent.service" + +install -m 0755 "${CONFIGS_DIR}/greetd/kiosk-session" "$INCLUDES/usr/local/bin/kiosk-session" +install -m 0755 "${CONFIGS_DIR}/sway/ha-kiosk" "$INCLUDES/usr/local/bin/ha-kiosk" +install -m 0755 "${CONFIGS_DIR}/spotify/spotify-launch" "$INCLUDES/usr/local/bin/spotify-launch" +install -m 0755 "${CONFIGS_DIR}/keyboard/toggle-keyboard" "$INCLUDES/usr/local/bin/toggle-keyboard" + +# Console (VT/TTY) keymap — separate from Sway's own xkb_layout, since greetd briefly +# owns the console before Sway starts. +mkdir -p "$INCLUDES/etc/default" +cat > "$INCLUDES/etc/default/keyboard" < "$INCLUDES/home/${KIOSK_USERNAME}/.ssh/authorized_keys" + chmod 600 "$INCLUDES/home/${KIOSK_USERNAME}/.ssh/authorized_keys" + echo " Baked an SSH authorized_keys entry for ${KIOSK_USERNAME}." +else + echo " No SSH_AUTHORIZED_KEY set — SSH admin access will not be possible on this image." +fi + +# --------------------------------------------------------------------------- +# 2. Runtime config, read by touchpanel-agent, the sway session wrapper, and hooks +# --------------------------------------------------------------------------- +echo "--- Writing /etc/touchpanel-agent/config.env into includes.chroot ---" +cat > "$INCLUDES/etc/touchpanel-agent/config.env" < touchpanel-agent's MPRIS bridge — needs Spotify's own MPRIS bus name to" +echo " actually be 'spotify', unverified against the real Flatpak build)." +echo " 7. Tap the dock's Keyboard button over a text field (Spotify search, the browser" +echo " address bar) and confirm wvkbd shows/hides. There is no auto-show — see README." +echo +echo "Then pull the power on the container host and re-check: the panel must still boot" +echo "and show Spotify/the browser (Home will show a connection error, which is" +echo "expected) with the container host powered off — same reactive-path philosophy as" +echo "hosts/thin-client's Phase 11.10." +echo +echo "Rebuilding later: edit hosts/touch-panel/configs/* or agent/*, then re-run this" +echo "script — includes.chroot is regenerated from them every time." diff --git a/pantry-vision/Dockerfile b/pantry-vision/Dockerfile new file mode 100644 index 0000000..51b1492 --- /dev/null +++ b/pantry-vision/Dockerfile @@ -0,0 +1,15 @@ +# pantry-vision — camera-driven grocery cataloguing API (docs/project-plan.md Phase +# 17). `restart: unless-stopped`, same as admin-canvas: it has to be up whenever the +# kitchen display might hold an item up to the camera. +FROM python:3.11-slim + +ENV PYTHONUNBUFFERED=1 \ + PYTHONDONTWRITEBYTECODE=1 + +WORKDIR /app + +# stdlib only (urllib for the Ollama/Grocy calls) — no requirements.txt, same call as +# admin-canvas/server.py. +COPY server.py ./ + +CMD ["python", "server.py"] diff --git a/pantry-vision/README.md b/pantry-vision/README.md new file mode 100644 index 0000000..310f1a9 --- /dev/null +++ b/pantry-vision/README.md @@ -0,0 +1,125 @@ +# pantry-vision + +The kitchen display's backend, from [Phase 17 of the project plan](../docs/project-plan.md). + +The workflow this exists for: **come home, put down the shopping bag, hold one item +up to the kitchen display's camera, the system proposes what it is and roughly how +long it keeps, you confirm (editing anything it got wrong), put it away.** The same +display then shows the resulting inventory ordered by what expires soonest, and +Grocy's recipes, on request. + +- **`pantry-vision`** (this directory) — a small always-on Python HTTP service. + `POST /identify` (a photo → a proposal via an Ollama vision model), `POST /confirm` + (a human-reviewed proposal → written into Grocy stock), `GET /inventory` and + `GET /recipes` (proxy Grocy, reshaped for the frontend). All four endpoints are + bearer-token gated. +- **`frontend/`** — the static single-page app the kitchen display's kiosk browser + loads: Scan / Inventory / Recipes, vanilla JS, no build step, no framework — same + "vendored, dependency-free" choice as the digest/admin canvas SDKs. Served + read-only by a `pantry-web` nginx container (`setup-container-host.sh`), the same + role `digest-web`/`admin-web` already play for their own hosts. +- **`hosts/kitchen-display/`** — the touch kiosk image that runs the frontend. See + that directory's own README for the device side of this. + +## A real network listener, unlike admin-canvas + +`admin-canvas` deliberately has **no published port** — only Home Assistant, on the +same compose network, ever calls it. `pantry-vision` is different on purpose: the +kitchen display is a separate physical device on the LAN and has to reach this +service directly (there is no HA-mediation step between "hold item up to camera" and +"get an identification back" — that has to be fast and synchronous). So +`PANTRY_VISION_PORT` **is** published, and every request — including the two GETs — +requires the bearer token, as the actual boundary instead of network placement. + +The same token has to be baked into the kitchen display's own build config +(`hosts/kitchen-display/scripts/build-kitchen-display-iso.sh`), not just Home +Assistant's — see that host's README. + +## `/identify` never writes anything by itself + +This is the one guardrail that matters most in this whole phase. The vision model's +guess — name, category, how many days until it likely goes bad — is a **proposal**, +shown on screen for the person to review and edit before anything is confirmed. Only +`/confirm`, a separate call the frontend makes after the person taps "Confirm & add," +ever writes to Grocy. This is the same "propose, never auto-commit" rule this project +already applies to identity-merge confirmation (see the *Identity store* row in +`docs/project-plan.md` §2) — a wrong camera guess costs one tap to fix, not a wrong +fact silently written into the household's inventory. + +## Configure + +```sh +cp pantry-vision/pantry-vision.env.example /opt/smart-home/pantry-vision/pantry-vision.env +openssl rand -hex 32 # put the result in PANTRY_VISION_TOKEN +chmod 600 /opt/smart-home/pantry-vision/pantry-vision.env +$EDITOR /opt/smart-home/pantry-vision/pantry-vision.env +``` + +You also need, inside Grocy's own UI (it's already running as the always-on `grocy` +container regardless of this phase): **Settings → Manage API keys** for +`GROCY_API_KEY`, and to confirm **Settings → Locations / Quantity units** actually +match `GROCY_DEFAULT_LOCATION_ID`/`GROCY_DEFAULT_QU_ID` (fresh-install defaults, not +guaranteed to match a Grocy that's already been customised). + +## Pick a vision model + +`OLLAMA_VISION_MODEL` defaults to `llava`, but **nothing here has confirmed that name +against a real pull** — pick a vision-capable model (`llava`, `qwen2.5vl`, or whatever +your LLM host's GPU/CPU tier can run at acceptable latency for someone standing at +the counter holding a can of beans) and: + +```sh +ollama pull llava # on the LLM host, or whichever model you picked +``` + +Plain text models (`qwen2.5:14b-instruct`, used elsewhere in this project for +digest/Assist) **cannot see images at all** — pointing `OLLAMA_VISION_MODEL` at one +of those will not error clearly, it will just produce a useless/hallucinated +response for every photo. Latency is unmeasured; a vision pass on a CPU-only LLM +host could easily be too slow for a "hold item up to camera" interaction to feel +responsive — this needs to be measured on real hardware, not assumed. + +## Grocy API assumptions — unverified against a real instance + +`server.py`'s Grocy calls (`_find_or_create_product`, `_add_to_stock`, the `/inventory` +and `/recipes` proxies) are written against Grocy's *documented* API shape, not +checked against a running instance. In particular: + +- Whether `GET /api/stock` rows carry a nested `product` object with a `name` field + by default, or need an explicit embed/expand parameter — `_handle_inventory` + degrades to `Product #` if not, rather than dropping the row, but that's a + fallback, not a fix. +- Whether `POST /api/objects/products` with just + `name`/`location_id`/`qu_id_purchase`/`qu_id_stock` is actually enough to create a + minimal product on your Grocy version, or whether it requires more fields. +- Whether the Recipes feature (`GET /api/objects/recipes`, + `GET /api/recipes/{id}/fulfillment`) needs to be explicitly enabled/populated + before it returns anything meaningful — `_handle_recipes` degrades to + `"fulfilled": null` per-recipe on any failure rather than breaking the whole list. + +Grocy exposes a live OpenAPI spec at `http://:9283/api/openapi/specification` +once it's running — read that against a real instance before trusting any of the +above, and adjust `server.py` if the shapes differ. + +## Deploy + +Wired into `hosts/container-host/scripts/setup-container-host.sh` behind +`ENABLE_PANTRY_VISION` (off by default) — see that script's `# CONFIGURATION` block +and its own README. It builds two containers: `pantry-vision` (this API) and +`pantry-web` (nginx, serves `frontend/` read-only). + +## Manual verification still outstanding + +1. All of the Grocy API assumptions above. +2. Real-world vision-model accuracy and latency for grocery items — untested with + any actual model or camera. +3. Whether Ollama's `/api/generate` `images` field is still the right call shape for + whichever vision model you pick — some multimodal models are only exposed through + Ollama's newer `/api/chat` with a `images` field per-message instead; this was + written against `/api/generate`'s documented multimodal support and not run + against a real model. +4. CORS: `_respond`'s blanket `Access-Control-Allow-Origin: *` is fine for a + LAN-only, bearer-token-gated service with no cookies, but hasn't been checked + against a real browser's preflight behavior for the raw-image-bytes `POST + /identify` call in particular (some browsers preflight non-simple `Content-Type`s + like `image/jpeg` — `do_OPTIONS` is written to handle that but is untested). diff --git a/pantry-vision/frontend/app.js b/pantry-vision/frontend/app.js new file mode 100644 index 0000000..a3e44e6 --- /dev/null +++ b/pantry-vision/frontend/app.js @@ -0,0 +1,227 @@ +// pantry-vision frontend logic (docs/project-plan.md Phase 17). Vanilla JS, no +// framework, no build step — same "vendored, dependency-free" choice as the digest/ +// admin canvas SDKs' render.js. +// +// Config comes from URL query params (?api=...&token=...), set by the +// kitchen-display kiosk's own launch command, NOT hardcoded here — this file is a +// generic static asset with no secret in it, served read-only by pantry-web to +// whatever device points a browser at it. +"use strict"; + +const params = new URLSearchParams(location.search); +const API = (params.get("api") || "").replace(/\/$/, ""); +const TOKEN = params.get("token") || ""; + +if (!API || !TOKEN) { + document.body.innerHTML = + '

pantry-vision not configured — missing ' + + "?api=&token= in the URL. See hosts/kitchen-display/configs/sway/pantry-kiosk.

"; + throw new Error("pantry-vision: missing ?api=/&token= query params"); +} + +function api(path, options) { + options = options || {}; + options.headers = Object.assign({ Authorization: `Bearer ${TOKEN}` }, options.headers || {}); + return fetch(`${API}${path}`, options).then((res) => { + if (!res.ok) { + return res.json().catch(() => ({})).then((body) => { + throw new Error(body.error || `${res.status} ${res.statusText}`); + }); + } + return res.json(); + }); +} + +// --- Tabs -------------------------------------------------------------------- +const tabs = document.querySelectorAll(".tab"); +const panels = document.querySelectorAll(".panel"); + +function activateTab(name) { + tabs.forEach((t) => t.classList.toggle("active", t.dataset.tab === name)); + panels.forEach((p) => p.classList.toggle("active", p.id === name)); + if (name === "inventory") loadInventory(); + if (name === "recipes") loadRecipes(); + if (name === "scan") resetScan(); +} + +tabs.forEach((t) => t.addEventListener("click", () => activateTab(t.dataset.tab))); + +// A "Show " MQTT command from touchpanel-style HA control arrives as a URL +// fragment reload (hosts/kitchen-display/agent kills and relaunches Chromium at +// index.html#recipes, same pattern as the thin client's digest-browser) — honour it +// on load, same as any manual tap. +if (location.hash) { + const initial = location.hash.slice(1); + if (["scan", "inventory", "recipes"].includes(initial)) activateTab(initial); +} + +// --- Scan ---------------------------------------------------------------------- +const video = document.getElementById("camera-preview"); +const captureBtn = document.getElementById("capture-btn"); +const cameraError = document.getElementById("camera-error"); +const scanCamera = document.getElementById("scan-camera"); +const scanResult = document.getElementById("scan-result"); +const capturedFrame = document.getElementById("captured-frame"); +const confirmForm = document.getElementById("confirm-form"); +const confirmStatus = document.getElementById("confirm-status"); +const retakeBtn = document.getElementById("retake-btn"); + +let stream = null; + +function startCamera() { + if (stream) return; + navigator.mediaDevices + .getUserMedia({ video: { facingMode: "environment" }, audio: false }) + .then((s) => { + stream = s; + video.srcObject = s; + cameraError.hidden = true; + }) + .catch((err) => { + cameraError.textContent = `Camera unavailable: ${err.message}. See hosts/kitchen-display/README.md.`; + cameraError.hidden = false; + }); +} + +function resetScan() { + scanResult.hidden = true; + scanCamera.hidden = false; + confirmStatus.textContent = ""; + startCamera(); +} + +captureBtn.addEventListener("click", () => { + if (!stream) return; + capturedFrame.width = video.videoWidth; + capturedFrame.height = video.videoHeight; + capturedFrame.getContext("2d").drawImage(video, 0, 0); + + scanCamera.hidden = true; + scanResult.hidden = false; + confirmStatus.textContent = "Identifying…"; + document.getElementById("f-confidence").textContent = ""; + + capturedFrame.toBlob( + (blob) => { + api("/identify", { method: "POST", body: blob, headers: { "Content-Type": "image/jpeg" } }) + .then((proposal) => { + document.getElementById("f-name").value = proposal.name || ""; + document.getElementById("f-category").value = proposal.category || "other"; + const days = Number.isFinite(proposal.estimated_shelf_life_days) + ? proposal.estimated_shelf_life_days + : 7; + const due = new Date(); + due.setDate(due.getDate() + days); + document.getElementById("f-date").value = due.toISOString().slice(0, 10); + document.getElementById("f-confidence").textContent = + `Model confidence: ${proposal.confidence || "unknown"}` + + (proposal.note ? ` — ${proposal.note}` : "") + + ". Review before confirming."; + confirmStatus.textContent = ""; + }) + .catch((err) => { + confirmStatus.textContent = `Could not identify: ${err.message}. Fill in manually.`; + }); + }, + "image/jpeg", + 0.85 + ); +}); + +retakeBtn.addEventListener("click", resetScan); + +confirmForm.addEventListener("submit", (event) => { + event.preventDefault(); + confirmStatus.textContent = "Adding…"; + api("/confirm", { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ + name: document.getElementById("f-name").value.trim(), + category: document.getElementById("f-category").value, + best_before_date: document.getElementById("f-date").value, + quantity: Number(document.getElementById("f-quantity").value) || 1, + }), + }) + .then(() => { + confirmStatus.textContent = "Added. Put it away!"; + setTimeout(resetScan, 1500); + }) + .catch((err) => { + confirmStatus.textContent = `Could not add: ${err.message}`; + }); +}); + +// --- Inventory ------------------------------------------------------------------- +function urgencyClass(daysLeft) { + if (daysLeft === null || daysLeft === undefined) return ""; + if (daysLeft < 0) return "urgent-expired"; + if (daysLeft <= 2) return "urgent-soon"; + if (daysLeft <= 7) return "urgent-week"; + return ""; +} + +function loadInventory() { + const el = document.getElementById("inventory-list"); + api("/inventory") + .then((data) => { + const items = data.items || []; + if (!items.length) { + el.innerHTML = '

Nothing in stock yet — scan something!

'; + return; + } + el.innerHTML = items + .map((item) => { + const days = item.days_left; + const label = + days === null || days === undefined + ? "no date" + : days < 0 + ? `expired ${-days}d ago` + : days === 0 + ? "expires today" + : `${days}d left`; + return `
+ ${escapeHtml(item.name)} + ×${escapeHtml(String(item.amount ?? ""))} + ${escapeHtml(label)} +
`; + }) + .join(""); + }) + .catch((err) => { + el.innerHTML = `

Could not load inventory: ${escapeHtml(err.message)}

`; + }); +} + +// --- Recipes ----------------------------------------------------------------- +function loadRecipes() { + const el = document.getElementById("recipes-list"); + api("/recipes") + .then((data) => { + const recipes = data.recipes || []; + if (!recipes.length) { + el.innerHTML = '

No recipes in Grocy yet.

'; + return; + } + el.innerHTML = recipes + .map((r) => { + const badge = r.fulfilled === true ? "✓ can make now" : r.fulfilled === false ? "missing items" : ""; + return `
+ ${escapeHtml(r.name || "")} + ${escapeHtml(badge)} +
`; + }) + .join(""); + }) + .catch((err) => { + el.innerHTML = `

Could not load recipes: ${escapeHtml(err.message)}

`; + }); +} + +function escapeHtml(s) { + return String(s).replace(/[&<>"']/g, (c) => ({ "&": "&", "<": "<", ">": ">", '"': """, "'": "'" }[c])); +} + +// Start on whatever tab is active (default: scan). +resetScan(); diff --git a/pantry-vision/frontend/index.html b/pantry-vision/frontend/index.html new file mode 100644 index 0000000..d3d58a9 --- /dev/null +++ b/pantry-vision/frontend/index.html @@ -0,0 +1,75 @@ + + + + + +Pantry + + + + + + +
+
+
+ + + +
+ + +
+ +
+
+

Loading…

+
+
+ +
+
+

Loading…

+
+
+
+ + + + diff --git a/pantry-vision/frontend/style.css b/pantry-vision/frontend/style.css new file mode 100644 index 0000000..5d308ec --- /dev/null +++ b/pantry-vision/frontend/style.css @@ -0,0 +1,192 @@ +/* pantry-vision frontend styling (docs/project-plan.md Phase 17). Touch targets are + * deliberately large — this is a kitchen-counter finger interface. */ + +* { + box-sizing: border-box; +} + +html, body { + margin: 0; + height: 100%; + background: #101014; + color: #e8e8ec; + font-family: sans-serif; + overflow: hidden; +} + +#tabs { + display: flex; + height: 84px; + background: rgba(16, 16, 20, 0.96); + border-bottom: 1px solid rgba(255, 255, 255, 0.08); +} + +.tab { + flex: 1; + display: flex; + flex-direction: column; + align-items: center; + justify-content: center; + gap: 2px; + background: transparent; + border: none; + color: #9a9aa6; + font-size: 26px; +} + +.tab span { + font-size: 13px; + font-weight: 500; +} + +.tab.active { + color: #e8e8ec; + background: rgba(110, 168, 254, 0.16); +} + +main { + height: calc(100% - 84px); + overflow-y: auto; +} + +.panel { + display: none; + padding: 16px; +} + +.panel.active { + display: block; +} + +/* --- Scan ------------------------------------------------------------------- */ +#scan-camera, #scan-result { + display: flex; + flex-direction: column; + align-items: center; + gap: 16px; +} + +#camera-preview, #captured-frame { + width: 100%; + max-width: 640px; + border-radius: 12px; + background: #000; +} + +.big-btn { + min-height: 64px; + min-width: 200px; + font-size: 18px; + font-weight: 600; + border-radius: 12px; + border: none; + color: #101014; + background: #6ea8fe; +} + +.big-btn.secondary { + background: rgba(255, 255, 255, 0.14); + color: #e8e8ec; +} + +.big-btn.primary { + background: #6ea8fe; +} + +#confirm-form { + width: 100%; + max-width: 640px; + display: flex; + flex-direction: column; + gap: 12px; +} + +#confirm-form label { + display: flex; + flex-direction: column; + gap: 4px; + font-size: 14px; + color: #9a9aa6; +} + +#confirm-form input, #confirm-form select { + min-height: 48px; + font-size: 16px; + padding: 8px 12px; + border-radius: 8px; + border: 1px solid rgba(255, 255, 255, 0.16); + background: #1a1a20; + color: #e8e8ec; +} + +.form-actions { + display: flex; + gap: 12px; + justify-content: center; +} + +.hint { + color: #9a9aa6; + font-size: 14px; +} + +.error { + color: #ff8080; + font-size: 14px; +} + +/* --- Cards (inventory + recipes) --------------------------------------------- */ +.card-list { + display: flex; + flex-direction: column; + gap: 8px; +} + +.card { + display: flex; + align-items: center; + gap: 12px; + min-height: 56px; + padding: 8px 16px; + border-radius: 10px; + background: rgba(255, 255, 255, 0.06); +} + +.card-name { + flex: 1; + font-size: 16px; + font-weight: 500; +} + +.card-amount { + color: #9a9aa6; + font-size: 14px; +} + +.card-due { + font-size: 14px; + color: #9a9aa6; + min-width: 110px; + text-align: right; +} + +.urgent-week { + background: rgba(255, 200, 80, 0.10); +} +.urgent-week .card-due { + color: #ffc850; +} + +.urgent-soon { + background: rgba(255, 140, 80, 0.14); +} +.urgent-soon .card-due { + color: #ff8c50; +} + +.urgent-expired { + background: rgba(255, 90, 90, 0.16); +} +.urgent-expired .card-due { + color: #ff5a5a; +} diff --git a/pantry-vision/pantry-vision.env.example b/pantry-vision/pantry-vision.env.example new file mode 100644 index 0000000..bd9f51b --- /dev/null +++ b/pantry-vision/pantry-vision.env.example @@ -0,0 +1,48 @@ +# pantry-vision configuration template. +# +# Copy this to the container host as (for example) +# /opt/smart-home/pantry-vision/pantry-vision.env, fill in real values, and chmod 600 +# it. Same never-commit handling as admin-canvas.env / digest-engine.env. + +# --------------------------------------------------------------------------- +# Auth — required. pantry-vision fails closed (rejects every request) while this is +# empty. Unlike admin-canvas's token (which only ever needs to be known by Home +# Assistant), this one also has to be baked into the kitchen-display kiosk's own +# build config (hosts/kitchen-display/scripts/build-kitchen-display-iso.sh), since +# the kiosk's frontend calls this API directly, not through HA. Generate one with: +# openssl rand -hex 32 +# --------------------------------------------------------------------------- +PANTRY_VISION_TOKEN= + +# --------------------------------------------------------------------------- +# Vision LLM (Ollama, the same LLM host as digest-engine/Assist) +# --------------------------------------------------------------------------- +OLLAMA_HOST=http://llm-host:11434 +# MUST be a vision-capable model, and MUST already be pulled on the LLM host +# (`ollama pull llava`, or whatever you pick) — plain text models like the +# qwen2.5:14b-instruct used elsewhere in this project cannot see images at all. +# Not verified against a real pull/run — see README.md. +OLLAMA_VISION_MODEL=llava +OLLAMA_TIMEOUT=120 + +# --------------------------------------------------------------------------- +# Grocy — already running as the always-on `grocy` container from +# hosts/container-host/scripts/setup-container-host.sh, reachable on the compose +# network at this internal name/port regardless of ENABLE_PANTRY_VISION. +# --------------------------------------------------------------------------- +GROCY_URL=http://grocy:80 +# Settings -> Manage API keys, inside Grocy's own UI. Required for /confirm (writes); +# /inventory and /recipes (reads) use the same key. +GROCY_API_KEY= +# VERIFY against your own Grocy instance (Settings -> Locations / Quantity units) — +# these are fresh-install defaults ("Default" location, "Piece" unit), not guaranteed +# to match a Grocy that's already been customised. +GROCY_DEFAULT_LOCATION_ID=1 +GROCY_DEFAULT_QU_ID=2 + +# --------------------------------------------------------------------------- +# Run behaviour +# --------------------------------------------------------------------------- +PANTRY_VISION_PORT=8095 +PANTRY_VISION_MAX_IMAGE_MB=15 +LOG_LEVEL=INFO diff --git a/pantry-vision/server.py b/pantry-vision/server.py new file mode 100644 index 0000000..05ab617 --- /dev/null +++ b/pantry-vision/server.py @@ -0,0 +1,391 @@ +"""pantry-vision — camera-driven grocery cataloguing for the kitchen display, from +docs/project-plan.md Phase 17. + +The workflow this exists for: come home, put down the shopping bag, hold one item up +to the kitchen display's camera, the system proposes what it is and how long it keeps, +you confirm (editing anything it got wrong) before it's written anywhere, then put the +item away. The display also shows the resulting inventory sorted by what expires +soonest, and Grocy's recipes. + +SECURITY BOUNDARY, same shape as admin-canvas/server.py's, with one deliberate +difference: this service DOES get a published port (see the README and +setup-container-host.sh's PANTRY_VISION_PORT), because unlike admin-canvas — which is +only ever called by Home Assistant on the compose network — this one is called +directly by the kitchen-display kiosk, a separate physical device on the LAN. Every +request still requires the bearer token below; an unset token fails closed (rejects +everything), never "auth optional". + +Four endpoints, all token-gated: + - POST /identify raw image bytes -> Ollama vision model -> a *proposal* + - POST /confirm JSON, human-reviewed/edited -> writes to Grocy stock + - GET /inventory proxies Grocy stock, sorted by soonest-expiring first + - GET /recipes proxies Grocy recipes (+ fulfillment, best-effort) + +/identify NEVER writes anything by itself — same "propose, never auto-commit" rule +this project already applies to identity-merge confirmation (see the Identity store +row in docs/project-plan.md §2): the vision model's guess is shown to the person +standing at the display, who can edit the name/category/date/location before /confirm +is ever called. A wrong guess costs a tap to fix, not a wrong fact silently written +into the household's inventory. +""" + +from __future__ import annotations + +import base64 +import json +import logging +import os +import re +import sys +import urllib.error +import urllib.request +from datetime import date, datetime +from http import HTTPStatus +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer +from urllib.parse import urlsplit + +LOG = logging.getLogger("pantry-vision") + +TOKEN = os.environ.get("PANTRY_VISION_TOKEN", "") + +OLLAMA_HOST = os.environ.get("OLLAMA_HOST", "http://llm-host:11434").rstrip("/") +# TBD/unverified — see README.md's "Pick a vision model" section. Not every Ollama +# model can see images; this must be one that can (llava, qwen2.5vl, etc.) and must +# actually be pulled on the LLM host (`ollama pull `) before /identify works. +OLLAMA_VISION_MODEL = os.environ.get("OLLAMA_VISION_MODEL", "llava") +OLLAMA_TIMEOUT = float(os.environ.get("OLLAMA_TIMEOUT", "120")) + +GROCY_URL = os.environ.get("GROCY_URL", "http://grocy:80").rstrip("/") +GROCY_API_KEY = os.environ.get("GROCY_API_KEY", "") +# Fresh-install defaults ("Default" location, "Piece" quantity unit) — VERIFY against +# your own Grocy instance (Settings -> Locations / Quantity units) and override if +# they differ. Same "don't build against a guess" flag as every other cross-service ID +# in this project (ADMIN_WEB_URL placeholders, weather_entity_id, etc.). +GROCY_DEFAULT_LOCATION_ID = os.environ.get("GROCY_DEFAULT_LOCATION_ID", "1") +GROCY_DEFAULT_QU_ID = os.environ.get("GROCY_DEFAULT_QU_ID", "2") + +MAX_IMAGE_BYTES = int(os.environ.get("PANTRY_VISION_MAX_IMAGE_MB", "15")) * 1024 * 1024 +MAX_JSON_BYTES = 64 * 1024 + +_IDENTIFY_PROMPT = """You are looking at a single grocery item held up to a kitchen \ +camera, on a plain background. Identify it and answer ONLY with a JSON object, no \ +other text, in exactly this shape: +{"name": "", "category": "", \ +"estimated_shelf_life_days": , "confidence": ""} +If you cannot identify the item, still return the JSON shape with your best guess and \ +"confidence": "low".""" + +_JSON_OBJECT_RE = re.compile(r"\{.*\}", re.DOTALL) + + +def _http_json(method: str, url: str, payload: dict | None = None, headers: dict | None = None, + timeout: float = 30.0) -> dict: + data = json.dumps(payload).encode("utf-8") if payload is not None else None + req = urllib.request.Request(url, data=data, method=method) + req.add_header("Content-Type", "application/json") + for key, value in (headers or {}).items(): + req.add_header(key, value) + with urllib.request.urlopen(req, timeout=timeout) as resp: + body = resp.read() + return json.loads(body) if body else {} + + +def _grocy_headers() -> dict: + return {"GROCY-API-KEY": GROCY_API_KEY} + + +def _grocy_get(path: str): + return _http_json("GET", f"{GROCY_URL}{path}", headers=_grocy_headers(), timeout=15) + + +def _grocy_post(path: str, payload: dict): + return _http_json("POST", f"{GROCY_URL}{path}", payload=payload, headers=_grocy_headers(), timeout=15) + + +def _identify_via_ollama(image_bytes: bytes) -> dict: + """Calls the vision model and returns a best-effort proposal — degrades to a + low-confidence placeholder rather than raising, mirroring digest-engine's + llm_client._fallback_document: a broken/blank result is worse than one flagged + as unreliable that a human has to review anyway (which they always do — see the + module docstring, /identify never writes anything by itself). + """ + payload = { + "model": OLLAMA_VISION_MODEL, + "prompt": _IDENTIFY_PROMPT, + "images": [base64.b64encode(image_bytes).decode("ascii")], + "stream": False, + "options": {"temperature": 0.2}, + } + try: + result = _http_json("POST", f"{OLLAMA_HOST}/api/generate", payload=payload, timeout=OLLAMA_TIMEOUT) + text = result.get("response", "") + except Exception: + LOG.warning("pantry-vision: Ollama call failed", exc_info=True) + return { + "name": "Unknown item", + "category": "other", + "estimated_shelf_life_days": 7, + "confidence": "low", + "note": f"Could not reach the vision model at {OLLAMA_HOST} — filled in manually.", + } + + match = _JSON_OBJECT_RE.search(text) + if match: + try: + proposal = json.loads(match.group(0)) + proposal.setdefault("name", "Unknown item") + proposal.setdefault("category", "other") + proposal.setdefault("estimated_shelf_life_days", 7) + proposal.setdefault("confidence", "low") + return proposal + except ValueError: + pass + + LOG.warning("pantry-vision: could not parse a JSON proposal out of the model's response") + return { + "name": "Unknown item", + "category": "other", + "estimated_shelf_life_days": 7, + "confidence": "low", + "note": "The vision model's response wasn't valid JSON — filled in manually.", + "raw_response": text[:500], + } + + +def _find_or_create_product(name: str) -> int: + """Best-effort product lookup/create against Grocy's object API. + + VERIFY against a real Grocy instance — this assumes GET /api/objects/products + returns a flat list of {id, name, ...} and that POST to the same path with just + name/location_id/qu_id_purchase/qu_id_stock is enough to create a minimal product. + Grocy's actual required-field set depends on its own settings and has not been + checked against a live instance; see README.md's verification list. + """ + products = _grocy_get("/api/objects/products") + needle = name.strip().lower() + for product in products: + if str(product.get("name", "")).strip().lower() == needle: + return int(product["id"]) + + created = _grocy_post( + "/api/objects/products", + { + "name": name, + "location_id": int(GROCY_DEFAULT_LOCATION_ID), + "qu_id_purchase": int(GROCY_DEFAULT_QU_ID), + "qu_id_stock": int(GROCY_DEFAULT_QU_ID), + }, + ) + return int(created["created_object_id"]) + + +def _add_to_stock(product_id: int, amount: float, best_before_date: str) -> dict: + return _grocy_post( + f"/api/stock/products/{product_id}/add", + {"amount": amount, "best_before_date": best_before_date, "transaction_type": "purchase"}, + ) + + +def _days_left(best_before: str | None) -> int | None: + if not best_before: + return None + try: + due = datetime.strptime(best_before, "%Y-%m-%d").date() + except ValueError: + return None + return (due - date.today()).days + + +class Handler(BaseHTTPRequestHandler): + server_version = "pantry-vision/1" + + def log_message(self, format, *args): # noqa: A002 + LOG.info("%s - %s", self.address_string(), format % args) + + def _authorized(self) -> bool: + if not TOKEN: + return False + return self.headers.get("Authorization", "") == f"Bearer {TOKEN}" + + def _respond(self, status: HTTPStatus, payload) -> None: + body = json.dumps(payload).encode("utf-8") + self.send_response(status) + self.send_header("Content-Type", "application/json") + self.send_header("Content-Length", str(len(body))) + self.send_header("Access-Control-Allow-Origin", "*") + self.end_headers() + self.wfile.write(body) + + def _read_body(self, max_bytes: int) -> bytes: + try: + length = int(self.headers.get("Content-Length") or 0) + except ValueError: + raise ValueError("missing or invalid Content-Length") from None + if length <= 0: + return b"" + if length > max_bytes: + raise ValueError(f"body too large ({length} > {max_bytes} bytes)") + return self.rfile.read(length) + + def do_OPTIONS(self): # noqa: N802 + # The frontend (pantry-web, a separate origin/port) fetches this API directly + # from the browser, so a plain CORS preflight has to succeed. + self.send_response(HTTPStatus.NO_CONTENT) + self.send_header("Access-Control-Allow-Origin", "*") + self.send_header("Access-Control-Allow-Headers", "Authorization, Content-Type") + self.send_header("Access-Control-Allow-Methods", "GET, POST, OPTIONS") + self.end_headers() + + def do_GET(self): # noqa: N802 + if not self._authorized(): + self._respond(HTTPStatus.UNAUTHORIZED, {"error": "missing or invalid bearer token"}) + return + path = urlsplit(self.path).path + if path == "/inventory": + self._handle_inventory() + elif path == "/recipes": + self._handle_recipes() + else: + self._respond(HTTPStatus.NOT_FOUND, {"error": "no such endpoint"}) + + def do_POST(self): # noqa: N802 + if not self._authorized(): + self._respond(HTTPStatus.UNAUTHORIZED, {"error": "missing or invalid bearer token"}) + return + path = urlsplit(self.path).path + if path == "/identify": + self._handle_identify() + elif path == "/confirm": + self._handle_confirm() + else: + self._respond(HTTPStatus.NOT_FOUND, {"error": "no such endpoint"}) + + def _handle_identify(self) -> None: + try: + image = self._read_body(MAX_IMAGE_BYTES) + except ValueError as exc: + self._respond(HTTPStatus.REQUEST_ENTITY_TOO_LARGE, {"error": str(exc)}) + return + if not image: + self._respond(HTTPStatus.BAD_REQUEST, {"error": "empty image body"}) + return + + proposal = _identify_via_ollama(image) + self._respond(HTTPStatus.OK, proposal) + + def _handle_confirm(self) -> None: + try: + raw = self._read_body(MAX_JSON_BYTES) + payload = json.loads(raw or b"{}") + except (ValueError, json.JSONDecodeError) as exc: + self._respond(HTTPStatus.BAD_REQUEST, {"error": f"bad request body: {exc}"}) + return + + name = str(payload.get("name", "")).strip() + best_before = str(payload.get("best_before_date", "")).strip() + amount = float(payload.get("quantity") or 1) + + if not name: + self._respond(HTTPStatus.BAD_REQUEST, {"error": "'name' is required"}) + return + if not re.match(r"^\d{4}-\d{2}-\d{2}$", best_before): + self._respond(HTTPStatus.BAD_REQUEST, {"error": "'best_before_date' must be YYYY-MM-DD"}) + return + if not GROCY_API_KEY: + self._respond(HTTPStatus.SERVICE_UNAVAILABLE, {"error": "GROCY_API_KEY is not configured"}) + return + + try: + product_id = _find_or_create_product(name) + _add_to_stock(product_id, amount, best_before) + except (urllib.error.URLError, urllib.error.HTTPError, KeyError, ValueError) as exc: + LOG.warning("pantry-vision: Grocy write failed", exc_info=True) + self._respond(HTTPStatus.BAD_GATEWAY, {"error": f"Grocy write failed: {exc}"}) + return + + LOG.info("pantry-vision: added %s x%s (best before %s) to Grocy", name, amount, best_before) + self._respond(HTTPStatus.OK, {"ok": True, "product_id": product_id}) + + def _handle_inventory(self) -> None: + try: + stock = _grocy_get("/api/stock") + except (urllib.error.URLError, urllib.error.HTTPError) as exc: + LOG.warning("pantry-vision: Grocy /api/stock unreachable", exc_info=True) + self._respond(HTTPStatus.BAD_GATEWAY, {"error": f"Grocy unreachable: {exc}"}) + return + + # VERIFY: assumes each stock row carries a nested "product" object with at + # least a "name" (Grocy's documented default `?embed` behaviour varies by + # version) — falls back to "product_id" as the display name if not, rather + # than dropping the row, same "degrade, don't blank" rule as everywhere else + # in this project's renderers. + items = [] + for row in stock if isinstance(stock, list) else []: + product = row.get("product") or {} + name = product.get("name") or f"Product #{row.get('product_id')}" + best_before = row.get("best_before_date") + items.append( + { + "name": name, + "amount": row.get("amount"), + "best_before_date": best_before, + "days_left": _days_left(best_before), + } + ) + + items.sort(key=lambda i: (i["days_left"] is None, i["days_left"])) + self._respond(HTTPStatus.OK, {"items": items}) + + def _handle_recipes(self) -> None: + try: + recipes = _grocy_get("/api/objects/recipes") + except (urllib.error.URLError, urllib.error.HTTPError) as exc: + LOG.warning("pantry-vision: Grocy recipes unreachable", exc_info=True) + self._respond(HTTPStatus.BAD_GATEWAY, {"error": f"Grocy unreachable: {exc}"}) + return + + results = [] + for recipe in recipes if isinstance(recipes, list) else []: + entry = {"id": recipe.get("id"), "name": recipe.get("name"), "fulfilled": None} + # Best-effort only: Grocy's fulfillment endpoint needs the recipes feature + # actually set up (ingredients entered per recipe) to mean anything, and + # its exact response shape is unverified — see README.md. A failure here + # degrades to "fulfilled: null" (rendered as "unknown" client-side), never + # a broken page. + try: + fulfillment = _grocy_get(f"/api/recipes/{entry['id']}/fulfillment") + entry["fulfilled"] = bool(fulfillment.get("need_fulfilled")) + except Exception: + pass + results.append(entry) + + self._respond(HTTPStatus.OK, {"recipes": results}) + + +def main() -> int: + logging.basicConfig( + level=os.environ.get("LOG_LEVEL", "INFO").upper(), + format="%(asctime)s %(levelname)-7s %(name)s: %(message)s", + ) + + if not TOKEN: + LOG.error( + "PANTRY_VISION_TOKEN is not set — every request will be rejected until it is. " + "See pantry-vision.env.example." + ) + if not GROCY_API_KEY: + LOG.warning("GROCY_API_KEY is not set — /confirm will fail until it is configured.") + + port = int(os.environ.get("PANTRY_VISION_PORT", "8095")) + server = ThreadingHTTPServer(("0.0.0.0", port), Handler) + LOG.info("pantry-vision listening on :%d (Ollama: %s, Grocy: %s)", port, OLLAMA_HOST, GROCY_URL) + try: + server.serve_forever() + except KeyboardInterrupt: + pass + return 0 + + +if __name__ == "__main__": + sys.exit(main())