Add Phase 16 touch panel and Phase 17 kitchen/fridge display endpoints
hosts/touch-panel/: a touch-driven Sway kiosk with a full Spotify GUI client (Flathub), a dedicated Home Assistant Chromium kiosk window, a general web browser, an always-on eww touch dock for app switching, an on-screen keyboard (wvkbd), and touchpanel-agent giving HA/the LLM the same MQTT-mediated control the thin client has. hosts/kitchen-display/ + pantry-vision/: a camera-vision grocery cataloguing endpoint for the fridge/pantry. The kiosk's browser captures a photo via getUserMedia and sends it to the new pantry-vision service, which asks an Ollama vision model to identify the item and estimate shelf life; the proposal is always human-confirmed before being written into Grocy stock. The same display shows inventory sorted by soonest-to-expire and Grocy's recipes. Wires ENABLE_PANTRY_VISION into setup-container-host.sh and documents both phases in docs/project-plan.md and README.md, matching this project's existing per-host hardware/software/guardrail/open-decision conventions. Nothing here has been run against real hardware, a real camera, or a real Grocy/vision-model instance — see each new README's verification list. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01K6CrKjW3yVptUnjG35HjC1digest-per-person-and-agendas
parent
bfca1c7eb0
commit
f3f870c4b4
17
README.md
17
README.md
|
|
@ -3,7 +3,8 @@
|
||||||
Local-first, open-source smart home: Home Assistant + Zigbee + RuView (CSI presence)
|
Local-first, open-source smart home: Home Assistant + Zigbee + RuView (CSI presence)
|
||||||
+ Bermuda (BLE identity) + local LLM (Ollama) + Frigate (peephole face recognition)
|
+ Bermuda (BLE identity) + local LLM (Ollama) + Frigate (peephole face recognition)
|
||||||
+ Grocy (kitchen kiosk) + Nextcloud calendar sync + a Sway thin-client media station
|
+ Grocy (kitchen kiosk) + Nextcloud calendar sync + a Sway thin-client media station
|
||||||
+ a quarter-daily LLM-generated digest.
|
+ a Sway touch panel + a camera-vision kitchen/fridge display + a quarter-daily
|
||||||
|
LLM-generated digest.
|
||||||
|
|
||||||
See [`docs/project-plan.md`](docs/project-plan.md) for the full hardware list,
|
See [`docs/project-plan.md`](docs/project-plan.md) for the full hardware list,
|
||||||
software stack, and phased implementation plan.
|
software stack, and phased implementation plan.
|
||||||
|
|
@ -21,6 +22,13 @@ hosts/
|
||||||
thin-client/ Sway kiosk/media-station ISO (live-build) + thinclient-agent
|
thin-client/ Sway kiosk/media-station ISO (live-build) + thinclient-agent
|
||||||
audio-endpoint/ Headless Spotify Connect appliance for rooms with no thin
|
audio-endpoint/ Headless Spotify Connect appliance for rooms with no thin
|
||||||
client — arm64 (rpi-image-gen) + amd64 (live-build) images
|
client — arm64 (rpi-image-gen) + amd64 (live-build) images
|
||||||
|
touch-panel/ Touch-driven Sway panel: full Spotify GUI, a dedicated
|
||||||
|
Home Assistant Chromium kiosk window, and a general web
|
||||||
|
browser, switched via an always-on touch dock or by HA/
|
||||||
|
the local LLM over MQTT
|
||||||
|
kitchen-display/ Single-purpose Sway kiosk for the fridge/pantry: one
|
||||||
|
Chromium window showing pantry-vision's Scan/Inventory/
|
||||||
|
Recipes frontend, camera capture via the browser itself
|
||||||
firmware/
|
firmware/
|
||||||
ruview/ RuView ESP32-S3 CSI presence node configs
|
ruview/ RuView ESP32-S3 CSI presence node configs
|
||||||
esphome-ble-proxy/ ESPHome configs for Bermuda BLE proxy nodes
|
esphome-ble-proxy/ ESPHome configs for Bermuda BLE proxy nodes
|
||||||
|
|
@ -35,6 +43,11 @@ admin-canvas/ On-demand sys-admin-llm display surface for the thin
|
||||||
clients: stats/graphics/media, pushed on demand
|
clients: stats/graphics/media, pushed on demand
|
||||||
rather than on a schedule (write API + admin-web
|
rather than on a schedule (write API + admin-web
|
||||||
static serving)
|
static serving)
|
||||||
|
pantry-vision/ Kitchen-display backend: a photo held up to the camera
|
||||||
|
-> an Ollama vision-model proposal -> human-confirmed
|
||||||
|
write into Grocy stock; also proxies Grocy's inventory
|
||||||
|
(soonest-expiring first) and recipes to the kiosk
|
||||||
|
frontend (write API + frontend/ static serving)
|
||||||
```
|
```
|
||||||
|
|
||||||
## Status
|
## Status
|
||||||
|
|
@ -56,6 +69,8 @@ admin-canvas/ On-demand sys-admin-llm display surface for the thin
|
||||||
- [x] admin-canvas + admin-web (sys-admin-llm on-demand display surface for the thin clients) — built and wired into `setup-container-host.sh` (`ENABLE_ADMIN_CANVAS`, off by default); the HA-side tool/rest_command wiring and the specific entities it surfaces (e.g. power-monitoring) are still undecided, see `docs/project-plan.md` §4
|
- [x] admin-canvas + admin-web (sys-admin-llm on-demand display surface for the thin clients) — built and wired into `setup-container-host.sh` (`ENABLE_ADMIN_CANVAS`, off by default); the HA-side tool/rest_command wiring and the specific entities it surfaces (e.g. power-monitoring) are still undecided, see `docs/project-plan.md` §4
|
||||||
- [ ] ESP32-S3-Touch-LCD-1.85C-V2 voice satellite + status display (`firmware/esp32-s3-touch-lcd-1.85c/`) — ESPHome config written and passes `esphome config`, not yet flashed to real hardware; `media_player`/`weather` entity IDs still need to be chosen, see `docs/project-plan.md` §4
|
- [ ] ESP32-S3-Touch-LCD-1.85C-V2 voice satellite + status display (`firmware/esp32-s3-touch-lcd-1.85c/`) — ESPHome config written and passes `esphome config`, not yet flashed to real hardware; `media_player`/`weather` entity IDs still need to be chosen, see `docs/project-plan.md` §4
|
||||||
- [ ] Headless audio endpoint (`hosts/audio-endpoint/`) — per-room independent Spotify Connect appliance for rooms without a thin client, arm64 (Raspberry Pi + HiFiBerry Amp2, rpi-image-gen) and amd64 (mini PC + USB DAC/amp, live-build) build pipelines written, **neither built/flashed/booted on real hardware** — rpi-image-gen's exact config schema in particular is unverified, see `hosts/audio-endpoint/README.md`
|
- [ ] Headless audio endpoint (`hosts/audio-endpoint/`) — per-room independent Spotify Connect appliance for rooms without a thin client, arm64 (Raspberry Pi + HiFiBerry Amp2, rpi-image-gen) and amd64 (mini PC + USB DAC/amp, live-build) build pipelines written, **neither built/flashed/booted on real hardware** — rpi-image-gen's exact config schema in particular is unverified, see `hosts/audio-endpoint/README.md`
|
||||||
|
- [ ] Sway touch panel (`hosts/touch-panel/`) — touch-driven Sway image: full Spotify GUI (Flathub), a dedicated Home Assistant Chromium kiosk window, a general web browser, an always-on touch dock for app switching, an on-screen keyboard (toggled manually, no auto-show), and `touchpanel-agent` (HA MQTT control, same LLM-mediated-through-HA security model as the thin client) — built, **no touch-panel hardware chosen and nothing booted on real metal**, see `hosts/touch-panel/README.md`
|
||||||
|
- [ ] Kitchen/fridge display + `pantry-vision` (`hosts/kitchen-display/`, `pantry-vision/`) — hold a grocery item up to the camera, an Ollama vision model proposes what it is and roughly how long it keeps, a human confirms (never auto-committed) before it's written into Grocy stock; the display then shows inventory sorted by soonest-to-expire and Grocy's recipes — built and wired into `setup-container-host.sh` (`ENABLE_PANTRY_VISION`, off by default), **nothing run against a real camera, vision model, or Grocy instance** — the Grocy API call shapes in particular are written from documentation only, see `pantry-vision/README.md` and `hosts/kitchen-display/README.md`
|
||||||
|
|
||||||
## Quick start
|
## Quick start
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,6 +1,6 @@
|
||||||
# AI-Managed Smart Home — Full Build Plan (v2)
|
# AI-Managed Smart Home — Full Build Plan (v2)
|
||||||
|
|
||||||
Local-first, open-source stack: Home Assistant + RuView presence + Bermuda BLE identity + local LLM (Ollama) + Zigbee sensors/lighting + Frigate (peephole face recognition) + Grocy kitchen kiosk + Nextcloud calendar sync + Node-RED glue logic + a Sway thin-client media station + a quarter-daily LLM digest.
|
Local-first, open-source stack: Home Assistant + RuView presence + Bermuda BLE identity + local LLM (Ollama) + Zigbee sensors/lighting + Frigate (peephole face recognition) + Grocy kitchen kiosk + Nextcloud calendar sync + Node-RED glue logic + a Sway thin-client media station + a Sway touch panel + a camera-vision kitchen/fridge display + a quarter-daily LLM digest.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|
@ -98,6 +98,25 @@ Already covered — using your existing Haozee CC2652P USB dongle. No coordinato
|
||||||
| Passive bookshelf speakers (e.g. Micca MB42X G2) | ~€100/pair | Same pairing already recommended for the thin-client rooms |
|
| Passive bookshelf speakers (e.g. Micca MB42X G2) | ~€100/pair | Same pairing already recommended for the thin-client rooms |
|
||||||
| *(amd64 alternative)* Spare x86 mini PC + USB DAC/amp (e.g. Fosi Audio V3, ~€130) | €0–130 | No GPIO HAT involved — a USB Audio Class device instead, for rooms getting a repurposed mini PC rather than a Raspberry Pi |
|
| *(amd64 alternative)* Spare x86 mini PC + USB DAC/amp (e.g. Fosi Audio V3, ~€130) | €0–130 | No GPIO HAT involved — a USB Audio Class device instead, for rooms getting a repurposed mini PC rather than a Raspberry Pi |
|
||||||
|
|
||||||
|
### 1.14 Touch panel hardware (Phase 16)
|
||||||
|
| Item | Est. Price (EUR) | Notes |
|
||||||
|
|---|---|---|
|
||||||
|
| Mini PC or SBC with a capacitive touchscreen (e.g. 10–15" USB-touch monitor + small x86 mini PC, or an all-in-one touch panel PC) | €150–300 | **No specific unit has been chosen** — the one hard requirement is that the touchscreen exposes itself as a native Wayland `wl_touch` device under libinput, not merely an emulated-mouse HID interface, since `hosts/touch-panel/` has no gesture/pointer-translation fallback the way the thin client does |
|
||||||
|
| Speakers or a wired connection to existing room audio | €0–100 | For local Spotify GUI-client playback — same speaker guidance as the thin-client/audio-endpoint rooms |
|
||||||
|
|
||||||
|
*(No container-host changes — `touchpanel-agent` talks to the existing Mosquitto/HA
|
||||||
|
from Phase 1, same as every other MQTT-connected host in this plan.)*
|
||||||
|
|
||||||
|
### 1.15 Kitchen/fridge display hardware (Phase 17)
|
||||||
|
| Item | Est. Price (EUR) | Notes |
|
||||||
|
|---|---|---|
|
||||||
|
| Mini PC or SBC with a touchscreen, mounted near the fridge/pantry | €150–300 | Same "must be a native Wayland `wl_touch` device" requirement as §1.14 — `hosts/kitchen-display/` has no pointer-emulation fallback either |
|
||||||
|
| USB webcam, pointed at wherever items get held up for scanning | €15–40 | Either built into the panel or on a short cable/gooseneck so its angle can be set independently of the screen. Any UVC webcam works — this is the same "no depth camera, no accelerator needed" bar as the thin client's gesture-control camera (§1.11), just used for a photo instead of continuous tracking |
|
||||||
|
|
||||||
|
*(No new container-host hardware — `pantry-vision` is a container on the existing
|
||||||
|
Phase 1 host, calling the existing Phase 3/9 LLM host for vision inference and the
|
||||||
|
already-running `grocy` container for storage.)*
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 2. Software (all open source / self-hosted)
|
## 2. Software (all open source / self-hosted)
|
||||||
|
|
@ -155,6 +174,21 @@ Already covered — using your existing Haozee CC2652P USB dongle. No coordinato
|
||||||
| Headless audio endpoint (arm64) | **rpi-image-gen** (custom config, `hosts/audio-endpoint/rpi-image-gen/`) | Raspberry Pi Foundation's current custom-image tool; builds a flashable `.img` with a per-room Spotify Connect receiver + HiFiBerry Amp2 audio |
|
| Headless audio endpoint (arm64) | **rpi-image-gen** (custom config, `hosts/audio-endpoint/rpi-image-gen/`) | Raspberry Pi Foundation's current custom-image tool; builds a flashable `.img` with a per-room Spotify Connect receiver + HiFiBerry Amp2 audio |
|
||||||
| Headless audio endpoint (amd64) | **live-build** (custom config, `hosts/audio-endpoint/live-build-amd64/`) | Reuses the thin client's own build tool, stripped of the whole graphical/kiosk stack — headless boot straight to the same per-room Spotify Connect receiver |
|
| Headless audio endpoint (amd64) | **live-build** (custom config, `hosts/audio-endpoint/live-build-amd64/`) | Reuses the thin client's own build tool, stripped of the whole graphical/kiosk stack — headless boot straight to the same per-room Spotify Connect receiver |
|
||||||
| Headless audio endpoint software | **librespot** or **spotifyd** (apt-first, documented fallback) | Independent, per-room Spotify Connect — no Snapcast, no shared/synced stream; each endpoint is its own device in Spotify's picker, same model as the thin client's own per-room spotifyd/librespot (Phase 11.6) |
|
| Headless audio endpoint software | **librespot** or **spotifyd** (apt-first, documented fallback) | Independent, per-room Spotify Connect — no Snapcast, no shared/synced stream; each endpoint is its own device in Spotify's picker, same model as the thin client's own per-room spotifyd/librespot (Phase 11.6) |
|
||||||
|
| Touch-panel OS build | **live-build** (custom config, `hosts/touch-panel/live-build/`) | Reuses the thin client's own build tool and directory-split convention, not its live-build tree — a different, smaller image for a different device |
|
||||||
|
| Touch-panel compositor | **Sway** | Same choice as the thin client, configured for direct touch instead of remote control: an always-on dock instead of no bars, no gesture-camera/wayvnc machinery |
|
||||||
|
| Touch-panel scripted control | **touchpanel-agent** (custom) | HA MQTT-discovery entity + `swaymsg`/app-process control, identical security shape to `thinclient-agent` — LLM tool call → HA service call → MQTT → this agent, never a direct path |
|
||||||
|
| Touch-panel music | **Spotify (official Linux client, Flathub `com.spotify.Client`)** | Full GUI client, not a headless Connect receiver — the point of a touch panel is a screen you interact with directly (Premium required, same as the headless receivers) |
|
||||||
|
| Touch-panel home dashboard | **Chromium (kiosk mode, `--app=`)** | Dedicated, always-open window pointed at Home Assistant, auto-restarted if it crashes |
|
||||||
|
| Touch-panel browser | **Firefox** | General browsing, minimal (not kiosk-locked) chrome — back/forward/reload/address bar |
|
||||||
|
| Touch-panel on-screen keyboard | **wvkbd** (apt-first, documented fallback) | Manually toggled from the touch dock — no shell-level input-method integration on this image, so no automatic show-on-focus |
|
||||||
|
| Touch-panel navigation | **eww** (apt-first, documented fallback) | An always-on, layer-shell-reserved touch dock (Spotify / Home / Web / Keyboard) — the touch-first equivalent of the thin client's HA-only **Screen** select |
|
||||||
|
| Kitchen-display OS build | **live-build** (custom config, `hosts/kitchen-display/live-build/`) | Same toolchain/convention reuse as the touch panel, a third and simpler sibling image — one workspace, one app |
|
||||||
|
| Kitchen-display camera capture | Browser-native `getUserMedia()` (Chromium) | No native capture app on-device at all — the kiosk page itself talks to the webcam and hands a frame straight to `pantry-vision` |
|
||||||
|
| Kitchen-display scripted control | **kitchen-display-agent** (custom) | HA MQTT-discovery entity for **Show scan/inventory/recipes** only — the actual camera/Grocy read-write path is `pantry-vision`'s own published API, called directly by the kiosk browser, not mediated through this agent |
|
||||||
|
| Grocery vision recognition | **Ollama** (a vision-capable model, e.g. `llava`/`qwen2.5vl` — TBD, not yet pulled or benchmarked) | Identifies a grocery item from one photo and estimates its shelf life; the proposal is always human-reviewed before anything is written (see `pantry-vision/README.md`) |
|
||||||
|
| Grocery inventory backend | **pantry-vision** (custom Python, stdlib `http.server`) | `POST /identify` (photo → proposal), `POST /confirm` (human-reviewed proposal → Grocy stock write), `GET /inventory`/`GET /recipes` (proxy Grocy, reshaped). Bearer-token gated, **and — unlike admin-canvas — LAN-published**, since the kitchen display's kiosk browser calls it directly rather than through Home Assistant |
|
||||||
|
| Grocery inventory storage | **Grocy** (already running, Phase 1/7) | The single source of truth for stock/best-before-dates/recipes; `pantry-vision` is a client of Grocy's own REST API, not a replacement for it |
|
||||||
|
| Kitchen-display static serving | **pantry-web** (nginx:alpine) | Serves `pantry-vision/frontend/`'s Scan/Inventory/Recipes single-page app read-only to the kitchen display — same role `digest-web`/`admin-web` play for their own hosts |
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|
@ -296,6 +330,131 @@ New hardware: §1.13. Two separate build pipelines, arm64 and amd64 — see belo
|
||||||
7. Neither image gives Home Assistant any control or visibility over these endpoints — no MQTT, no HA entities. Deliberate scope decision matching the "per-room independent" choice, not an oversight; revisit as a separate addition if remote control/monitoring from HA is wanted later.
|
7. Neither image gives Home Assistant any control or visibility over these endpoints — no MQTT, no HA entities. Deliberate scope decision matching the "per-room independent" choice, not an oversight; revisit as a separate addition if remote control/monitoring from HA is wanted later.
|
||||||
8. Nothing built or flashed on real hardware — no Raspberry Pi, no HiFiBerry Amp2, no x86 test box available. See the itemized unverified list in `hosts/audio-endpoint/README.md`.
|
8. Nothing built or flashed on real hardware — no Raspberry Pi, no HiFiBerry Amp2, no x86 test box available. See the itemized unverified list in `hosts/audio-endpoint/README.md`.
|
||||||
|
|
||||||
|
### Phase 16 — Sway touch panel
|
||||||
|
|
||||||
|
New hardware: §1.14. No container-host changes.
|
||||||
|
|
||||||
|
1. New top-level `hosts/touch-panel/` directory — **a different device, not a variant
|
||||||
|
of the Phase 11 thin client**: that image is a couch-distance media station
|
||||||
|
deliberately built around *remote* control (HA/MQTT, wayvnc) with no on-screen
|
||||||
|
bars; this one is touched directly, so it inverts that — an always-on touch dock,
|
||||||
|
native Wayland touch input, no gesture-camera/remote-view machinery. It reuses the
|
||||||
|
thin client's live-build toolchain and `configs/`+`agent/` split, not its
|
||||||
|
live-build tree, exactly the relationship `hosts/audio-endpoint/`'s amd64 build
|
||||||
|
already has to the thin client's.
|
||||||
|
2. **Three fixed workspaces, no on-demand ones**: `1:spotify` (the real, official
|
||||||
|
Spotify Linux GUI client via Flathub `com.spotify.Client` — not the thin client's
|
||||||
|
headless `spotifyd`/`librespot` Connect receiver, since the whole point of a touch
|
||||||
|
panel is a screen you interact with directly), `2:home` (a persistent Chromium
|
||||||
|
`--kiosk --app=$HA_URL` window, auto-restarted on crash by its own supervising
|
||||||
|
loop rather than by sway or systemd), and `3:web` (Firefox, minimal chrome —
|
||||||
|
general browsing, not kiosk-locked, mirroring the thin client's own `web_browser`
|
||||||
|
app in shape but on a separate, duplicated profile).
|
||||||
|
3. **Touch navigation is a persistent on-screen dock (`eww`, layer-shell
|
||||||
|
`:exclusive true`), not a keyboard/mouse-driven `bar {}`**: reserving a strip at
|
||||||
|
the bottom means the dock can never be covered by, or lose screen area behind,
|
||||||
|
whatever app is on screen — the opposite trade-off from the thin client's
|
||||||
|
deliberately bar-less, overlay-only widgets. Same session-scoped-not-agent
|
||||||
|
ownership rule as the thin client's now-playing widget: this is compositor UI, not
|
||||||
|
device control, and dies with sway rather than opening a second inbound channel
|
||||||
|
into the security-sensitive agent.
|
||||||
|
4. **`touchpanel-agent` mirrors `thinclient-agent`'s exact security boundary**: LLM
|
||||||
|
tool call → HA service call → MQTT → this agent, never a direct network path. It
|
||||||
|
publishes a **Screen** select (the touch dock's HA-reachable equivalent) and
|
||||||
|
**Show Spotify** / **Show Home** / **Show web browser** buttons — the latter two
|
||||||
|
focus the already-running app rather than relaunching it (both are meant to stay
|
||||||
|
open and stateful), unlike the thin client's stateless kill-and-relaunch digest/
|
||||||
|
admin kiosk pages. Also publishes a `media_player`-shaped entity set bridged from
|
||||||
|
Spotify's own MPRIS interface, identical in shape to the thin client's MPRIS
|
||||||
|
bridge, pointed at the `spotify` player name instead of `mpv`/`spotifyd`.
|
||||||
|
5. **An on-screen keyboard (`wvkbd`) with no auto-show, by explicit choice**: proper
|
||||||
|
show-on-text-field-focus needs the `text-input-v3`/`virtual-keyboard-v1` protocols
|
||||||
|
wired through a shell component (the way Phosh does it for squeekboard), which this
|
||||||
|
image doesn't run. Rather than build that integration now, the dock's Keyboard
|
||||||
|
button is a manual toggle — a documented, accepted limitation, not a gap hidden
|
||||||
|
from the README.
|
||||||
|
6. **No wayvnc on this image**, unlike the thin client — a deliberate scope decision,
|
||||||
|
not an oversight: the primary interaction model here is a finger at the panel, not
|
||||||
|
a remote viewer, so SSH alone covers the "something's broken, fix it from a shell"
|
||||||
|
case. Revisit only if a real need for remote *view* shows up once this runs on real
|
||||||
|
hardware; adding wayvnc back would be a small, isolated change (see
|
||||||
|
`hosts/touch-panel/README.md`).
|
||||||
|
7. **Both apt-first/documented-fallback installs already established elsewhere in this
|
||||||
|
repo are reused, not reinvented**: `eww` (same as the thin client's now-playing
|
||||||
|
widget) and `wvkbd` — neither is in Debian bookworm main, and both hooks stop at a
|
||||||
|
documented placeholder rather than a hardcoded release URL if apt doesn't have them.
|
||||||
|
8. Nothing built or flashed on real hardware — no touch-panel unit has been chosen
|
||||||
|
(§1.14), and in particular whether the touchscreen exposes itself as a native
|
||||||
|
Wayland `wl_touch` device (this image's one hard assumption, since there is no
|
||||||
|
pointer-emulation fallback the way the thin client's gesture control has one) is
|
||||||
|
unverified. See the itemized list in `hosts/touch-panel/README.md`.
|
||||||
|
|
||||||
|
### Phase 17 — Kitchen/fridge display (camera-vision grocery cataloguing)
|
||||||
|
|
||||||
|
New hardware: §1.15. New top-level `pantry-vision/` directory (container-host
|
||||||
|
service) plus `hosts/kitchen-display/` (a third, simpler kiosk image).
|
||||||
|
|
||||||
|
1. **The workflow this phase is built around, stated once because everything below
|
||||||
|
serves it**: come home, put down the shopping bag, hold one item up to the
|
||||||
|
kitchen display's camera, the system proposes what it is and roughly how long it
|
||||||
|
keeps, the person confirms (editing anything wrong) before anything is written
|
||||||
|
anywhere, put the item away. The same display then shows inventory ordered by
|
||||||
|
what expires soonest, and Grocy's recipes, on request.
|
||||||
|
2. **`pantry-vision` never writes to Grocy from `/identify` alone — only `/confirm`,
|
||||||
|
a separate human-reviewed call, ever does.** This is the single guardrail the
|
||||||
|
whole phase is designed around, the same "propose, never auto-commit" rule this
|
||||||
|
project already applies to identity-merge confirmation (the *Identity store* row
|
||||||
|
in §2). A wrong camera guess costs one tap to fix on the kiosk screen, not a wrong
|
||||||
|
fact silently written into the household's inventory.
|
||||||
|
3. **A real, LAN-published network listener — the one deliberate exception to this
|
||||||
|
project's usual "the LLM only ever reaches a device through HA→MQTT" shape.**
|
||||||
|
`admin-canvas` (Phase 13) has no published port because only Home Assistant calls
|
||||||
|
it; `pantry-vision` is different on purpose because the kitchen display is a
|
||||||
|
separate physical device that has to get an identification back synchronously,
|
||||||
|
with no HA round-trip in the loop for that specific call. The bearer token — not
|
||||||
|
network placement — is the actual boundary here; see
|
||||||
|
`pantry-vision/README.md`'s "A real network listener, unlike admin-canvas"
|
||||||
|
section. Controlling *which screen is showing* (Scan/Inventory/Recipes) is a
|
||||||
|
separate, narrower path that does stay HA/MQTT-mediated: `kitchen-display-agent`,
|
||||||
|
identical in shape to every other host's agent in this project.
|
||||||
|
4. **Grocy is the system of record, not a new inventory store** — `pantry-vision` is
|
||||||
|
a client of Grocy's own REST API (already running unconditionally since Phase 1/7)
|
||||||
|
for both reads (`/inventory`, `/recipes`) and the one write (`/confirm` → Grocy
|
||||||
|
stock). Nothing here duplicates or replaces Grocy's own data model.
|
||||||
|
5. **The vision-identification prompt asks for one photo, not a live video stream**,
|
||||||
|
and degrades to a low-confidence placeholder proposal (never an error page) if the
|
||||||
|
model call fails or its response isn't parseable JSON — same "degrade, don't
|
||||||
|
blank" rule as the digest/admin canvas renderers and `llm_client.py`'s own
|
||||||
|
`_fallback_document`.
|
||||||
|
6. **The kitchen display has no native camera-capture app at all** — Chromium's own
|
||||||
|
`getUserMedia()`, called from `pantry-vision/frontend/app.js`, talks to the webcam
|
||||||
|
directly inside the kiosk page and hands a captured frame straight to
|
||||||
|
`POST /identify` as a blob. `--use-fake-ui-for-media-stream` on the kiosk launch
|
||||||
|
auto-accepts the permission prompt that would otherwise sit unanswered on an
|
||||||
|
unattended screen.
|
||||||
|
7. **`hosts/kitchen-display/` is one workspace, one app — no touch dock, no
|
||||||
|
multi-app switching** the way `hosts/touch-panel` has: Scan/Inventory/Recipes is
|
||||||
|
in-page tab navigation inside `pantry-vision/frontend/`'s single-page app, since
|
||||||
|
there's only one thing this device does. `kitchen-display-agent`'s MQTT surface
|
||||||
|
is correspondingly narrow: three "Show X" buttons and nothing else — no
|
||||||
|
media_player, no capture/audio/remote-desktop entities.
|
||||||
|
8. **The Grocy API integration is written from documentation, not verified against a
|
||||||
|
live instance** — the exact `GET /api/stock` response shape (whether product
|
||||||
|
names arrive nested by default), the minimum required fields for
|
||||||
|
`POST /api/objects/products`, and whether the Recipes/fulfillment endpoints need
|
||||||
|
explicit setup before they return anything meaningful are all flagged in
|
||||||
|
`pantry-vision/README.md` alongside the live OpenAPI spec URL every real Grocy
|
||||||
|
instance exposes (`/api/openapi/specification`) as the way to actually check.
|
||||||
|
9. **The vision model choice is unpicked and unbenchmarked** (`OLLAMA_VISION_MODEL`
|
||||||
|
defaults to `llava`, not confirmed pulled or even correct for a given LLM host's
|
||||||
|
GPU/CPU tier) — same "don't build against a guess" flag as every other
|
||||||
|
LLM-model/hardware pairing in this plan. Latency in particular is unmeasured and
|
||||||
|
is the number that decides whether "hold it up, wait, confirm" feels usable at
|
||||||
|
the counter or like standing there for 30 seconds.
|
||||||
|
10. Nothing built or run against real hardware, a real camera, a real vision model,
|
||||||
|
or a real Grocy instance. See the itemized lists in `pantry-vision/README.md`
|
||||||
|
and `hosts/kitchen-display/README.md`.
|
||||||
|
|
||||||
### Testing checklist before calling any phase "done"
|
### Testing checklist before calling any phase "done"
|
||||||
- Does the reactive path (presence → light on) work with the LLM host powered off? (It must.)
|
- Does the reactive path (presence → light on) work with the LLM host powered off? (It must.)
|
||||||
- Does a bad/slow LLM response ever block a light switch? (It must not.)
|
- Does a bad/slow LLM response ever block a light switch? (It must not.)
|
||||||
|
|
@ -327,10 +486,17 @@ New hardware: §1.13. Two separate build pipelines, arm64 and amd64 — see belo
|
||||||
- Is the ESP32-S3-Touch-LCD-1.85C firmware ever flashed onto a V1 board? (It must not be — V1 has no AEC circuit and different audio pins; this phase's config assumes V2 throughout.)
|
- Is the ESP32-S3-Touch-LCD-1.85C firmware ever flashed onto a V1 board? (It must not be — V1 has no AEC circuit and different audio pins; this phase's config assumes V2 throughout.)
|
||||||
- Does a headless audio endpoint's Spotify Connect device name ever fail to match the room it's actually in? (It must not — `spotify-connect-start` always reads `$(hostname)`, which is set per-room by Raspberry Pi Imager on arm64 or `IMAGE_HOSTNAME` at build time on amd64; a copy-pasted/reused image for two rooms is a configuration mistake this software has no way to detect, same class of risk already called out for the ESP32 firmware's `media_player_entity_id`.)
|
- Does a headless audio endpoint's Spotify Connect device name ever fail to match the room it's actually in? (It must not — `spotify-connect-start` always reads `$(hostname)`, which is set per-room by Raspberry Pi Imager on arm64 or `IMAGE_HOSTNAME` at build time on amd64; a copy-pasted/reused image for two rooms is a configuration mistake this software has no way to detect, same class of risk already called out for the ESP32 firmware's `media_player_entity_id`.)
|
||||||
- Does the headless audio endpoint ever gain a shared/synced audio path, an MQTT connection, or an HA entity without a deliberate new decision to add one? (It must not — Phase 15 is scoped to per-room independent Spotify Connect only, on purpose.)
|
- Does the headless audio endpoint ever gain a shared/synced audio path, an MQTT connection, or an HA entity without a deliberate new decision to add one? (It must not — Phase 15 is scoped to per-room independent Spotify Connect only, on purpose.)
|
||||||
|
- Can the LLM reach the touch panel through any path other than HA service call → MQTT → `touchpanel-agent`? (It must not — identical boundary to the thin client's.)
|
||||||
|
- Does the touch panel boot to a usable session (Spotify + the browser) with the container host powered off? (It must not hang — Home showing a connection error is the expected/acceptable degraded state, since unlike the thin client this device has no local media source to fall back to.)
|
||||||
|
- Does tapping the touch dock's Home/Spotify buttons ever relaunch (rather than just focus) an already-running window? (It must not — both are stateful apps; relaunching would throw away login/scroll/playback state for no reason.)
|
||||||
|
- Does `pantry-vision`'s `/identify` endpoint ever write to Grocy by itself? (It must not — only a separate, human-reviewed `/confirm` call ever writes anything.)
|
||||||
|
- Does `pantry-vision` ever accept a request without a valid bearer token, on any of its four endpoints — including the two GETs? (It must not — unlike admin-canvas, this service is LAN-published, so the token is the actual boundary, not network placement.)
|
||||||
|
- Can the LLM reach the kitchen display through any path other than HA service call → MQTT → `kitchen-display-agent`, for *which screen is showing*? (It must not — reading/writing the actual inventory is a separate, intentionally-published path through `pantry-vision` itself, not a violation of this rule.)
|
||||||
|
- If the vision model's response is unparseable or the call fails outright, does `/identify` ever return a broken/blank result instead of a flagged, low-confidence placeholder proposal? (It must not.)
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 4. Open decisions (Phases 11–15)
|
## 4. Open decisions (Phases 11–17)
|
||||||
|
|
||||||
These need a decision before their respective implementation steps can be built — everything above is written to accommodate any answer, but nothing should be built against an unresolved item.
|
These need a decision before their respective implementation steps can be built — everything above is written to accommodate any answer, but nothing should be built against an unresolved item.
|
||||||
|
|
||||||
|
|
@ -349,3 +515,7 @@ These need a decision before their respective implementation steps can be built
|
||||||
13. **The sys-admin-llm's HA-side wiring has no real data source or tool definition yet** (new, Phase 13) — same shape as open decision #9 above: `admin-canvas/README.md` documents the expected `rest_command:`/tool-call contract and worked example JSON, but which HA entities/history back something like "the kitchen outlet's power draw" is unresolved, and no metering-capable Zigbee smart plug is in this plan's hardware list (§1.4) yet. Needs a hardware decision (a power-monitoring outlet) and an actual HA tool/script, neither of which exists in this repo by design — see the Phase 13 "nothing under this repo builds the HA side" note.
|
13. **The sys-admin-llm's HA-side wiring has no real data source or tool definition yet** (new, Phase 13) — same shape as open decision #9 above: `admin-canvas/README.md` documents the expected `rest_command:`/tool-call contract and worked example JSON, but which HA entities/history back something like "the kitchen outlet's power draw" is unresolved, and no metering-capable Zigbee smart plug is in this plan's hardware list (§1.4) yet. Needs a hardware decision (a power-monitoring outlet) and an actual HA tool/script, neither of which exists in this repo by design — see the Phase 13 "nothing under this repo builds the HA side" note.
|
||||||
14. **The voice/display satellite has no real hardware verification, and two of its data sources are unpicked** (new, Phase 14) — `firmware/esp32-s3-touch-lcd-1.85c/voice-display.yaml` passes ESPHome's own config validator but has never been flashed to a physical unit; the display init sequence, the AEC audio path, and wake-word sensitivity are all adapted/assumed, not measured (see the itemized list in that directory's README). Separately, `media_player_entity_id` and `weather_entity_id` are placeholders — which media player this unit should mirror needs a decision if the household ends up with more than one active at a time.
|
14. **The voice/display satellite has no real hardware verification, and two of its data sources are unpicked** (new, Phase 14) — `firmware/esp32-s3-touch-lcd-1.85c/voice-display.yaml` passes ESPHome's own config validator but has never been flashed to a physical unit; the display init sequence, the AEC audio path, and wake-word sensitivity are all adapted/assumed, not measured (see the itemized list in that directory's README). Separately, `media_player_entity_id` and `weather_entity_id` are placeholders — which media player this unit should mirror needs a decision if the household ends up with more than one active at a time.
|
||||||
15. **The headless audio endpoint has no real hardware verification on either architecture** (new, Phase 15) — neither the arm64 (`rpi-image-gen`) nor amd64 (`live-build`) pipeline has been run to completion, let alone flashed and booted. rpi-image-gen's exact config/layer YAML schema in particular was written from documentation only, with no ARM build environment available to validate it against the real tool — see `hosts/audio-endpoint/README.md`'s prominent risk callout. Also still open: whether `spotifyd` or `librespot` actually installs cleanly from current bookworm apt sources at all (the fallback list is documented, not exercised), and how stable USB DAC/amp ALSA enumeration turns out to be on a real amd64 mini PC.
|
15. **The headless audio endpoint has no real hardware verification on either architecture** (new, Phase 15) — neither the arm64 (`rpi-image-gen`) nor amd64 (`live-build`) pipeline has been run to completion, let alone flashed and booted. rpi-image-gen's exact config/layer YAML schema in particular was written from documentation only, with no ARM build environment available to validate it against the real tool — see `hosts/audio-endpoint/README.md`'s prominent risk callout. Also still open: whether `spotifyd` or `librespot` actually installs cleanly from current bookworm apt sources at all (the fallback list is documented, not exercised), and how stable USB DAC/amp ALSA enumeration turns out to be on a real amd64 mini PC.
|
||||||
|
16. **No touch-panel hardware has been chosen** (new, Phase 16) — nothing in `hosts/touch-panel/` has been booted on real metal or a real touchscreen. In particular, whether the panel's touch input actually surfaces as a native Wayland `wl_touch` device (this image's one hard assumption) is unverified — see `hosts/touch-panel/README.md`'s verification list, item 3.
|
||||||
|
17. **The touch panel's Spotify/Chromium integration details are assumed, not confirmed** (new, Phase 16) — the Flathub app ID `com.spotify.Client`, the Spotify Flatpak's real MPRIS bus name (assumed `spotify`), and Chromium's Wayland `app_id` for a `--app=` kiosk window (assumed to start with `chromium`) are all flagged for on-hardware verification in `hosts/touch-panel/README.md`.
|
||||||
|
18. **No vision-capable Ollama model has been picked, pulled, or benchmarked** (new, Phase 17) — `pantry-vision`'s `OLLAMA_VISION_MODEL` defaults to `llava` with no confirmation it's the right choice for any given LLM host's hardware tier, and grocery-item identification latency/accuracy is completely unmeasured. This is the single highest-risk unknown in Phase 17: if it's too slow or too inaccurate, "hold item up to camera" stops being a usable interaction and the phase needs a different approach (a smaller/faster model, a barcode-scan fallback via Grocy's existing scanning support, or both).
|
||||||
|
19. **`pantry-vision`'s Grocy API integration is written from documentation, not a live instance** (new, Phase 17) — `GET /api/stock`'s response shape, `POST /api/objects/products`'s minimum required fields, and the Recipes/fulfillment endpoints' setup requirements are all assumed; `pantry-vision/README.md` points at each real Grocy instance's own live OpenAPI spec (`/api/openapi/specification`) as the way to check before trusting any of it. `GROCY_DEFAULT_LOCATION_ID`/`GROCY_DEFAULT_QU_ID` are fresh-install-default guesses that need confirming against Settings → Locations/Quantity units on the real instance too.
|
||||||
|
|
|
||||||
|
|
@ -28,6 +28,10 @@
|
||||||
# - admin-canvas + admin-web (Phase 13 sys-admin-llm display surface, optional,
|
# - admin-canvas + admin-web (Phase 13 sys-admin-llm display surface, optional,
|
||||||
# off by default — needs admin-canvas/ from this repo checked out on this
|
# off by default — needs admin-canvas/ from this repo checked out on this
|
||||||
# host, see ADMIN_CANVAS_SRC below and admin-canvas/README.md)
|
# host, see ADMIN_CANVAS_SRC below and admin-canvas/README.md)
|
||||||
|
# - pantry-vision + pantry-web (Phase 17 kitchen-display camera cataloguing
|
||||||
|
# backend, optional, off by default — needs pantry-vision/ from this repo
|
||||||
|
# checked out on this host, see PANTRY_VISION_SRC below and
|
||||||
|
# pantry-vision/README.md)
|
||||||
#
|
#
|
||||||
# Run as: sudo ./setup-container-host.sh
|
# Run as: sudo ./setup-container-host.sh
|
||||||
#
|
#
|
||||||
|
|
@ -94,6 +98,19 @@ ADMIN_CANVAS_SRC="/opt/smart-home/src/admin-canvas"
|
||||||
ADMIN_CANVAS_PORT="8092" # internal only — no `ports:` mapping, HA-reachable only
|
ADMIN_CANVAS_PORT="8092" # internal only — no `ports:` mapping, HA-reachable only
|
||||||
ADMIN_WEB_PORT="8094" # LAN-facing read-only static serving
|
ADMIN_WEB_PORT="8094" # LAN-facing read-only static serving
|
||||||
|
|
||||||
|
# --- Kitchen-display camera cataloguing backend (Phase 17) — off by default until
|
||||||
|
# --- PANTRY_VISION_TOKEN and GROCY_API_KEY are provisioned. See pantry-vision/README.md.
|
||||||
|
# Unlike admin-canvas, this one IS meant to be reached directly by a separate physical
|
||||||
|
# device (hosts/kitchen-display's kiosk browser), not only by Home Assistant — so,
|
||||||
|
# deliberately unlike ADMIN_CANVAS_PORT above, it DOES get a `ports:` mapping. The
|
||||||
|
# bearer token is the actual boundary here, not network placement — see
|
||||||
|
# pantry-vision/README.md's "A real network listener, unlike admin-canvas" section.
|
||||||
|
ENABLE_PANTRY_VISION="false"
|
||||||
|
# Where this repo's pantry-vision/ directory lives on THIS host (build context).
|
||||||
|
PANTRY_VISION_SRC="/opt/smart-home/src/pantry-vision"
|
||||||
|
PANTRY_VISION_PORT="8095" # LAN-facing — the kitchen display's kiosk browser calls this directly
|
||||||
|
PANTRY_WEB_PORT="8096" # LAN-facing read-only static serving (the kiosk's frontend)
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# Sanity checks
|
# Sanity checks
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
|
|
@ -130,6 +147,12 @@ if [[ "$ENABLE_ADMIN_CANVAS" == "true" && ! -d "$ADMIN_CANVAS_SRC" ]]; then
|
||||||
echo " build context for the admin-canvas image — then re-run."
|
echo " build context for the admin-canvas image — then re-run."
|
||||||
fi
|
fi
|
||||||
|
|
||||||
|
if [[ "$ENABLE_PANTRY_VISION" == "true" && ! -d "$PANTRY_VISION_SRC" ]]; then
|
||||||
|
echo "Warning: ENABLE_PANTRY_VISION=true but $PANTRY_VISION_SRC does not exist."
|
||||||
|
echo " Copy or clone this repo's pantry-vision/ directory there — it is the"
|
||||||
|
echo " build context for the pantry-vision image — then re-run."
|
||||||
|
fi
|
||||||
|
|
||||||
if [[ "$ENABLE_WHATSAPP_INGEST" == "true" ]]; then
|
if [[ "$ENABLE_WHATSAPP_INGEST" == "true" ]]; then
|
||||||
echo "WARNING: WhatsApp ingestion is enabled."
|
echo "WARNING: WhatsApp ingestion is enabled."
|
||||||
echo " There is no officially sanctioned way to read WhatsApp programmatically."
|
echo " There is no officially sanctioned way to read WhatsApp programmatically."
|
||||||
|
|
@ -252,6 +275,17 @@ if [[ "$ENABLE_ADMIN_CANVAS" == "true" ]]; then
|
||||||
echo " Seeded $BASE_DIR/admin-canvas/admin-canvas.env from the template — fill in a real ADMIN_CANVAS_TOKEN."
|
echo " Seeded $BASE_DIR/admin-canvas/admin-canvas.env from the template — fill in a real ADMIN_CANVAS_TOKEN."
|
||||||
fi
|
fi
|
||||||
fi
|
fi
|
||||||
|
if [[ "$ENABLE_PANTRY_VISION" == "true" ]]; then
|
||||||
|
mkdir -p "$BASE_DIR"/pantry-vision
|
||||||
|
# Same handling as admin-canvas.env above: seed from the committed template on
|
||||||
|
# first run, 600, never committed (repo .gitignore covers *.env).
|
||||||
|
if [[ ! -f "$BASE_DIR/pantry-vision/pantry-vision.env" ]]; then
|
||||||
|
cp "$PANTRY_VISION_SRC/pantry-vision.env.example" "$BASE_DIR/pantry-vision/pantry-vision.env"
|
||||||
|
chmod 600 "$BASE_DIR/pantry-vision/pantry-vision.env"
|
||||||
|
echo " Seeded $BASE_DIR/pantry-vision/pantry-vision.env from the template — fill in a real"
|
||||||
|
echo " PANTRY_VISION_TOKEN and GROCY_API_KEY (Grocy's own UI: Settings -> Manage API keys)."
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# 4. Mosquitto config
|
# 4. Mosquitto config
|
||||||
|
|
@ -625,6 +659,46 @@ if [[ "$ENABLE_ADMIN_CANVAS" == "true" ]]; then
|
||||||
"
|
"
|
||||||
fi
|
fi
|
||||||
|
|
||||||
|
# pantry-vision (Phase 17) — the kitchen display's camera-cataloguing backend. Same
|
||||||
|
# build/write-service shape as admin-canvas above, but see this block's own comment
|
||||||
|
# where it differs (a published port, since a separate physical device calls it
|
||||||
|
# directly) and pantry-vision/README.md for the full reasoning. `depends_on: grocy`
|
||||||
|
# is a startup-order hint only — pantry-vision's own outbound calls already retry per
|
||||||
|
# request, so a slow-starting Grocy delays the first real call, not the container.
|
||||||
|
PANTRY_VISION_BLOCK=""
|
||||||
|
PANTRY_WEB_BLOCK=""
|
||||||
|
if [[ "$ENABLE_PANTRY_VISION" == "true" ]]; then
|
||||||
|
PANTRY_VISION_BLOCK="
|
||||||
|
pantry-vision:
|
||||||
|
build: ${PANTRY_VISION_SRC}
|
||||||
|
image: smart-home/pantry-vision:local
|
||||||
|
container_name: pantry-vision
|
||||||
|
restart: unless-stopped
|
||||||
|
depends_on:
|
||||||
|
- grocy
|
||||||
|
ports:
|
||||||
|
- \"${PANTRY_VISION_PORT}:${PANTRY_VISION_PORT}\"
|
||||||
|
env_file:
|
||||||
|
- ${BASE_DIR}/pantry-vision/pantry-vision.env
|
||||||
|
environment:
|
||||||
|
- PANTRY_VISION_PORT=${PANTRY_VISION_PORT}
|
||||||
|
- TZ=${TIMEZONE}
|
||||||
|
"
|
||||||
|
|
||||||
|
PANTRY_WEB_BLOCK="
|
||||||
|
pantry-web:
|
||||||
|
image: nginx:alpine
|
||||||
|
container_name: pantry-web
|
||||||
|
restart: unless-stopped
|
||||||
|
ports:
|
||||||
|
- \"${PANTRY_WEB_PORT}:80\"
|
||||||
|
volumes:
|
||||||
|
- ${PANTRY_VISION_SRC}/frontend:/usr/share/nginx/html:ro
|
||||||
|
environment:
|
||||||
|
- TZ=${TIMEZONE}
|
||||||
|
"
|
||||||
|
fi
|
||||||
|
|
||||||
if [[ "$ENABLE_DIGEST_ENGINE" == "true" && "$ENABLE_WHATSAPP_INGEST" == "true" ]]; then
|
if [[ "$ENABLE_DIGEST_ENGINE" == "true" && "$ENABLE_WHATSAPP_INGEST" == "true" ]]; then
|
||||||
# Long-lived, unlike digest-engine: holds the logged-in WhatsApp Web session
|
# Long-lived, unlike digest-engine: holds the logged-in WhatsApp Web session
|
||||||
# open and appends to /data/messages.jsonl, which digest-engine drains each
|
# open and appends to /data/messages.jsonl, which digest-engine drains each
|
||||||
|
|
@ -721,7 +795,7 @@ ${FRIGATE_DEVICES}
|
||||||
- PUID=1000
|
- PUID=1000
|
||||||
- PGID=1000
|
- PGID=1000
|
||||||
- TZ=${TIMEZONE}
|
- TZ=${TIMEZONE}
|
||||||
${MEALIE_BLOCK}${NODERED_BLOCK}${NETDATA_BLOCK}${HOMEPAGE_BLOCK}${NTFY_BLOCK}${PORTAINER_BLOCK}${GALLERY_SMB_BLOCK}${DIGEST_ENGINE_BLOCK}${DIGEST_WEB_BLOCK}${WHATSAPP_BRIDGE_BLOCK}${ADMIN_CANVAS_BLOCK}${ADMIN_WEB_BLOCK}
|
${MEALIE_BLOCK}${NODERED_BLOCK}${NETDATA_BLOCK}${HOMEPAGE_BLOCK}${NTFY_BLOCK}${PORTAINER_BLOCK}${GALLERY_SMB_BLOCK}${DIGEST_ENGINE_BLOCK}${DIGEST_WEB_BLOCK}${WHATSAPP_BRIDGE_BLOCK}${ADMIN_CANVAS_BLOCK}${ADMIN_WEB_BLOCK}${PANTRY_VISION_BLOCK}${PANTRY_WEB_BLOCK}
|
||||||
EOF
|
EOF
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
|
|
@ -880,6 +954,10 @@ fi
|
||||||
if [[ "$ENABLE_ADMIN_CANVAS" == "true" ]]; then
|
if [[ "$ENABLE_ADMIN_CANVAS" == "true" ]]; then
|
||||||
echo " Admin canvas : http://${HOST_IP}:${ADMIN_WEB_PORT}/canvas.html"
|
echo " Admin canvas : http://${HOST_IP}:${ADMIN_WEB_PORT}/canvas.html"
|
||||||
fi
|
fi
|
||||||
|
if [[ "$ENABLE_PANTRY_VISION" == "true" ]]; then
|
||||||
|
echo " Pantry API : http://${HOST_IP}:${PANTRY_VISION_PORT} (bearer-token gated)"
|
||||||
|
echo " Pantry display : http://${HOST_IP}:${PANTRY_WEB_PORT}/index.html?api=http://${HOST_IP}:${PANTRY_VISION_PORT}&token=<PANTRY_VISION_TOKEN>"
|
||||||
|
fi
|
||||||
if [[ "$ENABLE_GALLERY_SMB" == "true" ]]; then
|
if [[ "$ENABLE_GALLERY_SMB" == "true" ]]; then
|
||||||
echo " Gallery SMB : \\\\${HOST_IP}\\gallery (user: ${GALLERY_SMB_USERNAME})"
|
echo " Gallery SMB : \\\\${HOST_IP}\\gallery (user: ${GALLERY_SMB_USERNAME})"
|
||||||
fi
|
fi
|
||||||
|
|
@ -927,6 +1005,15 @@ if [[ "$ENABLE_ADMIN_CANVAS" == "true" ]]; then
|
||||||
echo " admin-canvas/README.md for the worked example and the 'Show admin canvas'"
|
echo " admin-canvas/README.md for the worked example and the 'Show admin canvas'"
|
||||||
echo " entity documented in hosts/thin-client/README.md."
|
echo " entity documented in hosts/thin-client/README.md."
|
||||||
fi
|
fi
|
||||||
|
if [[ "$ENABLE_PANTRY_VISION" == "true" ]]; then
|
||||||
|
echo " 15. Fill in $BASE_DIR/pantry-vision/pantry-vision.env before the kitchen display can"
|
||||||
|
echo " identify anything: PANTRY_VISION_TOKEN (also goes into"
|
||||||
|
echo " hosts/kitchen-display/scripts/build-kitchen-display-iso.sh — both sides need the"
|
||||||
|
echo " SAME value) and GROCY_API_KEY (Grocy's own UI: Settings -> Manage API keys, at"
|
||||||
|
echo " http://${HOST_IP}:9283). Also pull a vision-capable Ollama model on the LLM host"
|
||||||
|
echo " (e.g. 'ollama pull llava') — OLLAMA_VISION_MODEL defaults to one that is NOT"
|
||||||
|
echo " confirmed pulled or even correct for your setup. See pantry-vision/README.md."
|
||||||
|
fi
|
||||||
echo
|
echo
|
||||||
echo "Updating later: cd $BASE_DIR && docker compose pull && docker compose up -d"
|
echo "Updating later: cd $BASE_DIR && docker compose pull && docker compose up -d"
|
||||||
echo "Backing up manually: sudo $BASE_DIR/backup.sh (requires ENABLE_BACKUPS=true was run once)"
|
echo "Backing up manually: sudo $BASE_DIR/backup.sh (requires ENABLE_BACKUPS=true was run once)"
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,125 @@
|
||||||
|
# Sway kitchen display
|
||||||
|
|
||||||
|
Phase 17 of `docs/project-plan.md`. Builds a Debian 12 live ISO for a single-purpose
|
||||||
|
fridge/pantry display: hold a grocery item up to its camera, confirm what the vision
|
||||||
|
model thinks it is and when it'll go bad, put it away — then browse the resulting
|
||||||
|
inventory (soonest-to-expire first) and Grocy's recipes, all touch-driven.
|
||||||
|
|
||||||
|
**The camera-vision and Grocy read/write logic does not live in this directory** —
|
||||||
|
that's [`../../pantry-vision/`](../../pantry-vision/README.md), a container-host
|
||||||
|
service this device's kiosk browser calls directly. This host is deliberately thin:
|
||||||
|
a Sway session, one Chromium kiosk window, and `kitchen-display-agent`, which only
|
||||||
|
ever controls *which screen is showing* (Scan / Inventory / Recipes) over MQTT — the
|
||||||
|
same "the agent controls the surface, a separate write API owns the content" split
|
||||||
|
`hosts/thin-client`'s admin canvas already established.
|
||||||
|
|
||||||
|
**A third, simpler sibling of `../thin-client/` and `../touch-panel/`**, not a variant
|
||||||
|
of either — one workspace, one app, no dock, no music, no general browsing. It reuses
|
||||||
|
the thin client's live-build toolchain and `configs/`+`agent/` split.
|
||||||
|
|
||||||
|
| | |
|
||||||
|
|---|---|
|
||||||
|
| Compositor | Sway, one workspace, one Chromium kiosk window |
|
||||||
|
| Autologin | greetd, straight into `/usr/local/bin/kiosk-session` |
|
||||||
|
| Touch input | Native Wayland `wl_touch`, same assumption as `../touch-panel/` |
|
||||||
|
| Camera | Browser-native `getUserMedia()` inside the Chromium kiosk page — no separate native capture app on this device at all |
|
||||||
|
| Content | `pantry-vision`'s frontend (`../../pantry-vision/frontend/`), served by `pantry-web`; in-page tabs, not separate Sway workspaces |
|
||||||
|
| Remote control (HA/LLM) | `kitchen-display-agent` — **Show scan / Show inventory / Show recipes** buttons only |
|
||||||
|
| Remote control (human) | SSH only — no wayvnc, same scope decision as `../touch-panel/` |
|
||||||
|
|
||||||
|
## Hardware
|
||||||
|
|
||||||
|
**No specific unit has been chosen.** What's assumed: a touchscreen (native
|
||||||
|
`wl_touch`, same caveat as `../touch-panel/README.md`) mounted somewhere near the
|
||||||
|
fridge/pantry, with a USB webcam pointed at wherever items get held up — either
|
||||||
|
built into the panel or a separate camera on a short cable/gooseneck mount so the
|
||||||
|
angle can be adjusted independently of the screen. See §1.15 of
|
||||||
|
`docs/project-plan.md` for the (unverified, non-specific) hardware line item.
|
||||||
|
|
||||||
|
## Before you build
|
||||||
|
|
||||||
|
Deploy `pantry-vision`/`pantry-web` first (`ENABLE_PANTRY_VISION` in
|
||||||
|
`hosts/container-host/scripts/setup-container-host.sh`) — this image builds and boots
|
||||||
|
fine without it, but every tab will show a connection error until it exists. Then edit
|
||||||
|
the `# CONFIGURATION` block at the top of
|
||||||
|
[`scripts/build-kitchen-display-iso.sh`](scripts/build-kitchen-display-iso.sh):
|
||||||
|
|
||||||
|
| Variable | What to put in it |
|
||||||
|
|---|---|
|
||||||
|
| `MQTT_BROKER_HOST` | LAN IP of the container host running Mosquitto |
|
||||||
|
| `PANTRY_WEB_URL` | The `pantry-web` service's URL, e.g. `http://192.168.1.10:8096` |
|
||||||
|
| `PANTRY_VISION_URL` | The `pantry-vision` API's URL, e.g. `http://192.168.1.10:8095` — read by `app.js` from the kiosk window's own URL query params (see `pantry-kiosk`'s comment), never baked into the served frontend files |
|
||||||
|
| `PANTRY_VISION_TOKEN` | **Must exactly match** `PANTRY_VISION_TOKEN` in `pantry-vision/pantry-vision.env` on the container host — there's no way for this repo to push that value between the two hosts for you |
|
||||||
|
| `KIOSK_USERNAME` / `IMAGE_HOSTNAME` / `KITCHEN_DISPLAY_NAME` | Per-device identity |
|
||||||
|
| `SSH_AUTHORIZED_KEY` | Optional — password auth is disabled and there is no wayvnc |
|
||||||
|
|
||||||
|
## Build
|
||||||
|
|
||||||
|
```sh
|
||||||
|
sudo ./scripts/build-kitchen-display-iso.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
Same directory-split convention as every other host in this project: `configs/` and
|
||||||
|
`agent/` are human-edited and git-tracked; `live-build/config/includes.chroot/` is
|
||||||
|
generated, gitignored, never hand-edited.
|
||||||
|
|
||||||
|
## Why the camera has no native capture app
|
||||||
|
|
||||||
|
`../touch-panel/` and `../thin-client/` both shell out to native processes for their
|
||||||
|
media (mpv, the Spotify Flatpak). This device doesn't need that: Chromium's own
|
||||||
|
`getUserMedia()` API, called from `pantry-vision/frontend/app.js`, talks to the
|
||||||
|
webcam directly and hands a captured frame straight to `POST /identify` as a blob —
|
||||||
|
no intermediate script, no temp file, no separate process to supervise. The one thing
|
||||||
|
this needs from the kiosk launch itself is `--use-fake-ui-for-media-stream`
|
||||||
|
(`configs/sway/pantry-kiosk`), which auto-accepts the camera permission prompt that
|
||||||
|
would otherwise sit unanswered on a screen nobody is there to click "Allow" on.
|
||||||
|
|
||||||
|
## Home Assistant entities
|
||||||
|
|
||||||
|
`kitchen-display-agent` publishes three buttons on connect: **Show scan**, **Show
|
||||||
|
inventory**, **Show recipes** — each kills and relaunches the kiosk Chromium window
|
||||||
|
at `PANTRY_WEB_URL/index.html?...#<tab>`. That's the entire MQTT surface; there is no
|
||||||
|
media_player, no workspace select (one workspace), no capture/audio/remote-desktop
|
||||||
|
entities the way the thin client has. Reading or writing the actual inventory (from
|
||||||
|
voice, e.g. "what's about to expire?" or "add three eggs") is a **separate** path —
|
||||||
|
Home Assistant/the LLM calling `pantry-vision`'s own published API directly, not
|
||||||
|
through this agent, exactly mirroring the split `hosts/thin-client`'s admin canvas
|
||||||
|
already established between "which screen is showing" (MQTT, this agent) and "what's
|
||||||
|
actually on it" (a dedicated write API).
|
||||||
|
|
||||||
|
### Security boundary
|
||||||
|
|
||||||
|
Same shape and reasoning as every other host's agent in this project:
|
||||||
|
`kitchen_display_agent/mqtt_discovery.py` is the entire inbound MQTT control surface
|
||||||
|
of this machine — LLM tool call → HA service call → MQTT → this agent, no HTTP
|
||||||
|
listener, no websocket, no exposed Sway IPC socket, no VNC. `pantry-vision` is a
|
||||||
|
**separate, deliberately published** service with its own bearer-token boundary — see
|
||||||
|
`../../pantry-vision/README.md`'s "A real network listener, unlike admin-canvas"
|
||||||
|
section for why that one's trust model is different on purpose.
|
||||||
|
|
||||||
|
## Manual verification still outstanding
|
||||||
|
|
||||||
|
None of this has been run on hardware. In rough order — on top of everything already
|
||||||
|
flagged as unverified in `../../pantry-vision/README.md` (the Grocy API assumptions,
|
||||||
|
the vision model's accuracy/latency, which this device's whole usefulness rides on):
|
||||||
|
|
||||||
|
1. The ISO builds at all.
|
||||||
|
2. greetd lands in Sway with no login prompt.
|
||||||
|
3. **Touch input as native `wl_touch`** — same open item as `../touch-panel/`.
|
||||||
|
4. **Camera enumeration and `--use-fake-ui-for-media-stream`** — whether Chromium
|
||||||
|
picks the right `/dev/videoN` when more than one video device is present (a
|
||||||
|
built-in laptop-style webcam plus a dedicated one, say), and whether that flag
|
||||||
|
actually suppresses the permission prompt on this Chromium/Wayland combination
|
||||||
|
specifically (it's a well-known flag for automated testing, not confirmed here
|
||||||
|
against a real kiosk session).
|
||||||
|
5. `kitchen-display-agent` connects to Mosquitto and the device appears in HA.
|
||||||
|
6. End-to-end scan latency: camera capture → `/identify` → a usable proposal on
|
||||||
|
screen, with a real vision model on real LLM-host hardware. This is the number
|
||||||
|
that decides whether the whole "hold it up, wait, confirm" workflow feels usable
|
||||||
|
or like standing at the fridge for 30 seconds waiting on an LLM.
|
||||||
|
7. Whether `?api=...&token=...` survives being passed through `--app=` URL escaping
|
||||||
|
correctly for every character that can appear in `PANTRY_VISION_URL` — the
|
||||||
|
`sed`-based percent-encoding in `configs/sway/pantry-kiosk` handles `/` and `:`
|
||||||
|
only, which is enough for a bare `http://host:port` but not for a URL with a path
|
||||||
|
or query string of its own.
|
||||||
|
8. Idle-blank timeout (20 minutes, `configs/sway/config`) and touch-resume — untested.
|
||||||
|
|
@ -0,0 +1,20 @@
|
||||||
|
[Unit]
|
||||||
|
Description=Kitchen display agent (Home Assistant MQTT control surface)
|
||||||
|
Documentation=file:///opt/kitchen-display-agent
|
||||||
|
After=network.target
|
||||||
|
|
||||||
|
[Service]
|
||||||
|
Type=simple
|
||||||
|
User=@KIOSK_USERNAME@
|
||||||
|
Group=@KIOSK_USERNAME@
|
||||||
|
WorkingDirectory=/opt/kitchen-display-agent
|
||||||
|
Environment=PYTHONPATH=/opt/kitchen-display-agent
|
||||||
|
Environment=PYTHONUNBUFFERED=1
|
||||||
|
EnvironmentFile=-/etc/kitchen-display-agent/config.env
|
||||||
|
ExecStart=/usr/bin/python3 -m kitchen_display_agent.main
|
||||||
|
Restart=always
|
||||||
|
RestartSec=5
|
||||||
|
NoNewPrivileges=true
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=multi-user.target
|
||||||
|
|
@ -0,0 +1,3 @@
|
||||||
|
"""kitchen-display-agent — Home Assistant MQTT control surface for the kitchen display."""
|
||||||
|
|
||||||
|
__version__ = "0.1.0"
|
||||||
|
|
@ -0,0 +1,137 @@
|
||||||
|
"""kitchen-display-agent entrypoint."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import logging
|
||||||
|
import os
|
||||||
|
import signal
|
||||||
|
import socket
|
||||||
|
import sys
|
||||||
|
import threading
|
||||||
|
|
||||||
|
import paho.mqtt.client as mqtt
|
||||||
|
|
||||||
|
from .mqtt_discovery import Discovery
|
||||||
|
from .session import launch_pantry_kiosk
|
||||||
|
|
||||||
|
CONFIG_PATH = os.environ.get("KITCHEN_DISPLAY_AGENT_CONFIG", "/etc/kitchen-display-agent/config.env")
|
||||||
|
|
||||||
|
CONFIG_KEYS = (
|
||||||
|
"MQTT_BROKER_HOST",
|
||||||
|
"MQTT_BROKER_PORT",
|
||||||
|
"MQTT_USERNAME",
|
||||||
|
"MQTT_PASSWORD",
|
||||||
|
"PANTRY_WEB_URL",
|
||||||
|
"PANTRY_VISION_URL",
|
||||||
|
"PANTRY_VISION_TOKEN",
|
||||||
|
"KIOSK_USERNAME",
|
||||||
|
"KITCHEN_DISPLAY_NAME",
|
||||||
|
)
|
||||||
|
|
||||||
|
log = logging.getLogger("kitchen-display-agent")
|
||||||
|
|
||||||
|
|
||||||
|
def load_config(path: str = CONFIG_PATH) -> dict[str, str]:
|
||||||
|
values: dict[str, str] = {}
|
||||||
|
try:
|
||||||
|
with open(path, encoding="utf-8") as handle:
|
||||||
|
for line in handle:
|
||||||
|
line = line.strip()
|
||||||
|
if not line or line.startswith("#") or "=" not in line:
|
||||||
|
continue
|
||||||
|
key, _, value = line.partition("=")
|
||||||
|
values[key.strip()] = value.strip().strip('"').strip("'")
|
||||||
|
except OSError as exc:
|
||||||
|
log.warning("could not read %s (%s); falling back to the environment", path, exc)
|
||||||
|
|
||||||
|
for key in CONFIG_KEYS:
|
||||||
|
if key in os.environ:
|
||||||
|
values[key] = os.environ[key]
|
||||||
|
|
||||||
|
return values
|
||||||
|
|
||||||
|
|
||||||
|
def make_client(client_id: str) -> mqtt.Client:
|
||||||
|
callback_api = getattr(mqtt, "CallbackAPIVersion", None)
|
||||||
|
if callback_api is not None:
|
||||||
|
return mqtt.Client(callback_api.VERSION1, client_id=client_id)
|
||||||
|
return mqtt.Client(client_id=client_id)
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
logging.basicConfig(
|
||||||
|
level=logging.INFO,
|
||||||
|
format="%(asctime)s %(levelname)s %(name)s: %(message)s",
|
||||||
|
stream=sys.stdout,
|
||||||
|
)
|
||||||
|
|
||||||
|
config = load_config()
|
||||||
|
hostname = socket.gethostname()
|
||||||
|
node_id = "".join(c if c.isalnum() else "_" for c in hostname).strip("_") or "kitchendisplay"
|
||||||
|
friendly_name = config.get("KITCHEN_DISPLAY_NAME") or f"Kitchen display ({hostname})"
|
||||||
|
|
||||||
|
broker_host = config.get("MQTT_BROKER_HOST", "")
|
||||||
|
broker_port = int(config.get("MQTT_BROKER_PORT") or 1883)
|
||||||
|
|
||||||
|
client = make_client(f"kitchen-display-agent-{node_id}")
|
||||||
|
if config.get("MQTT_USERNAME"):
|
||||||
|
client.username_pw_set(config["MQTT_USERNAME"], config.get("MQTT_PASSWORD") or None)
|
||||||
|
|
||||||
|
discovery = Discovery(client, node_id, friendly_name)
|
||||||
|
|
||||||
|
def on_show(fragment: str) -> None:
|
||||||
|
launch_pantry_kiosk(fragment)
|
||||||
|
|
||||||
|
def on_connect(_client, _userdata, _flags, rc):
|
||||||
|
if rc != 0:
|
||||||
|
log.error("MQTT connection refused (rc=%s)", rc)
|
||||||
|
return
|
||||||
|
log.info("connected to MQTT broker %s:%s", broker_host, broker_port)
|
||||||
|
discovery.register_screens(on_show)
|
||||||
|
discovery.subscribe_all()
|
||||||
|
discovery.publish_available(True)
|
||||||
|
|
||||||
|
def on_disconnect(_client, _userdata, rc):
|
||||||
|
log.warning("disconnected from MQTT broker (rc=%s); paho will retry", rc)
|
||||||
|
|
||||||
|
def on_message(_client, _userdata, message):
|
||||||
|
discovery.dispatch(message.topic, message.payload.decode("utf-8", "replace"))
|
||||||
|
|
||||||
|
client.on_connect = on_connect
|
||||||
|
client.on_disconnect = on_disconnect
|
||||||
|
client.on_message = on_message
|
||||||
|
client.will_set(discovery.availability_topic, "offline", qos=1, retain=True)
|
||||||
|
|
||||||
|
stop_event = threading.Event()
|
||||||
|
|
||||||
|
def handle_signal(_signum, _frame):
|
||||||
|
stop_event.set()
|
||||||
|
|
||||||
|
signal.signal(signal.SIGTERM, handle_signal)
|
||||||
|
signal.signal(signal.SIGINT, handle_signal)
|
||||||
|
|
||||||
|
if not broker_host:
|
||||||
|
log.error("MQTT_BROKER_HOST is not set in %s — running without HA control", CONFIG_PATH)
|
||||||
|
else:
|
||||||
|
# connect_async + loop_start, never a blocking connect(): same "reactive path
|
||||||
|
# never depends on a remote service" rule as every other host's agent — the
|
||||||
|
# kitchen display's local scan/inventory/recipes flow must work with the
|
||||||
|
# container host powered off, only the HA "Show X" buttons need it.
|
||||||
|
client.connect_async(broker_host, broker_port, keepalive=60)
|
||||||
|
client.loop_start()
|
||||||
|
|
||||||
|
log.info("kitchen-display-agent %s started (node_id=%s)", node_id, node_id)
|
||||||
|
try:
|
||||||
|
stop_event.wait()
|
||||||
|
finally:
|
||||||
|
log.info("shutting down")
|
||||||
|
if broker_host:
|
||||||
|
discovery.publish_available(False)
|
||||||
|
client.loop_stop()
|
||||||
|
client.disconnect()
|
||||||
|
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main())
|
||||||
|
|
@ -0,0 +1,105 @@
|
||||||
|
"""Home Assistant MQTT Discovery payloads and command dispatch.
|
||||||
|
|
||||||
|
SECURITY BOUNDARY — this module is the entire remote-control API of the kitchen
|
||||||
|
display. Same principle as every other host's agent in this project (Phase 11.4):
|
||||||
|
the local LLM never gets a network path to this machine directly. The only chain is
|
||||||
|
LLM tool call -> Home Assistant service call -> MQTT -> this dispatcher. There is no
|
||||||
|
HTTP listener, no websocket server, no exposed Sway IPC socket here.
|
||||||
|
|
||||||
|
This is deliberately NOT how the LLM reaches pantry-vision's actual inventory data —
|
||||||
|
that's a separate, already-published API (pantry-vision/server.py) that Home
|
||||||
|
Assistant or the LLM can call directly for reads/writes to Grocy. This agent only
|
||||||
|
ever controls what the physical screen is showing, exactly the same division of
|
||||||
|
labour as hosts/thin-client's "Show admin canvas" button vs. admin-canvas's own
|
||||||
|
write API.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import logging
|
||||||
|
from typing import Callable
|
||||||
|
|
||||||
|
from . import __version__
|
||||||
|
|
||||||
|
log = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
DISCOVERY_PREFIX = "homeassistant"
|
||||||
|
|
||||||
|
|
||||||
|
class Discovery:
|
||||||
|
def __init__(self, client, node_id: str, friendly_name: str):
|
||||||
|
self.client = client
|
||||||
|
self.node_id = node_id
|
||||||
|
self.friendly_name = friendly_name
|
||||||
|
self.base = f"kitchendisplay/{node_id}"
|
||||||
|
self.availability_topic = f"{self.base}/availability"
|
||||||
|
self._handlers: dict[str, Callable[[str], None]] = {}
|
||||||
|
|
||||||
|
self.device = {
|
||||||
|
"identifiers": [f"kitchendisplay_{node_id}"],
|
||||||
|
"name": friendly_name,
|
||||||
|
"manufacturer": "SmartestHome",
|
||||||
|
"model": "Sway kitchen display",
|
||||||
|
"sw_version": __version__,
|
||||||
|
}
|
||||||
|
|
||||||
|
def _publish_config(self, component: str, object_id: str, payload: dict) -> None:
|
||||||
|
payload = {
|
||||||
|
"availability_topic": self.availability_topic,
|
||||||
|
"device": self.device,
|
||||||
|
"unique_id": f"{self.node_id}_{object_id}",
|
||||||
|
**payload,
|
||||||
|
}
|
||||||
|
topic = f"{DISCOVERY_PREFIX}/{component}/{self.node_id}/{object_id}/config"
|
||||||
|
self.client.publish(topic, json.dumps(payload), qos=1, retain=True)
|
||||||
|
|
||||||
|
def _command_topic(self, suffix: str, handler) -> str:
|
||||||
|
topic = f"{self.base}/{suffix}"
|
||||||
|
self._handlers[topic] = handler
|
||||||
|
return topic
|
||||||
|
|
||||||
|
def subscribe_all(self) -> None:
|
||||||
|
for topic in self._handlers:
|
||||||
|
self.client.subscribe(topic, qos=1)
|
||||||
|
|
||||||
|
def dispatch(self, topic: str, payload: str) -> None:
|
||||||
|
handler = self._handlers.get(topic)
|
||||||
|
if handler is None:
|
||||||
|
log.warning("no handler for %s", topic)
|
||||||
|
return
|
||||||
|
try:
|
||||||
|
handler(payload)
|
||||||
|
except Exception:
|
||||||
|
log.exception("handler for %s failed", topic)
|
||||||
|
|
||||||
|
def publish_available(self, available: bool = True) -> None:
|
||||||
|
self.client.publish(
|
||||||
|
self.availability_topic,
|
||||||
|
"online" if available else "offline",
|
||||||
|
qos=1,
|
||||||
|
retain=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
def register_screens(self, on_show) -> None:
|
||||||
|
"""One button per screen — payload is the fixed page fragment (scan/
|
||||||
|
inventory/recipes), an enumerated constant handled in main.py, never passed
|
||||||
|
through as a URL/argv element unchanged (same rule as every other agent's
|
||||||
|
mqtt_discovery.py in this project).
|
||||||
|
"""
|
||||||
|
for object_id, name, fragment, icon in (
|
||||||
|
("show_scan", "Show scan", "scan", "mdi:camera"),
|
||||||
|
("show_inventory", "Show inventory", "inventory", "mdi:fridge-outline"),
|
||||||
|
("show_recipes", "Show recipes", "recipes", "mdi:chef-hat"),
|
||||||
|
):
|
||||||
|
self._publish_config(
|
||||||
|
"button",
|
||||||
|
object_id,
|
||||||
|
{
|
||||||
|
"name": name,
|
||||||
|
"command_topic": self._command_topic(
|
||||||
|
f"show/{fragment}", lambda _payload, fragment=fragment: on_show(fragment)
|
||||||
|
),
|
||||||
|
"icon": icon,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
@ -0,0 +1,47 @@
|
||||||
|
"""Session-environment resolution and process launching.
|
||||||
|
|
||||||
|
A trimmed version of hosts/thin-client's/hosts/touch-panel's sway_control.py: this
|
||||||
|
device has only one Sway workspace and one app (the pantry-kiosk Chromium window), so
|
||||||
|
there is no workspace-switching or window-focus surface to wrap — just enough to run
|
||||||
|
`/usr/local/bin/pantry-kiosk [fragment]` with the graphical session's environment.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import glob
|
||||||
|
import logging
|
||||||
|
import os
|
||||||
|
import subprocess
|
||||||
|
|
||||||
|
log = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
def runtime_dir() -> str:
|
||||||
|
return os.environ.get("XDG_RUNTIME_DIR") or f"/run/user/{os.getuid()}"
|
||||||
|
|
||||||
|
|
||||||
|
def session_env() -> dict[str, str]:
|
||||||
|
env = dict(os.environ)
|
||||||
|
env["XDG_RUNTIME_DIR"] = runtime_dir()
|
||||||
|
env.setdefault("DBUS_SESSION_BUS_ADDRESS", f"unix:path={runtime_dir()}/bus")
|
||||||
|
env.setdefault("WAYLAND_DISPLAY", "wayland-1")
|
||||||
|
matches = sorted(glob.glob(os.path.join(runtime_dir(), "sway-ipc.*.sock")))
|
||||||
|
if matches:
|
||||||
|
env["SWAYSOCK"] = matches[-1]
|
||||||
|
return env
|
||||||
|
|
||||||
|
|
||||||
|
def launch_pantry_kiosk(fragment: str) -> None:
|
||||||
|
command = ["/usr/local/bin/pantry-kiosk", fragment] if fragment else ["/usr/local/bin/pantry-kiosk"]
|
||||||
|
log.info("launching %s", " ".join(command))
|
||||||
|
try:
|
||||||
|
subprocess.Popen(
|
||||||
|
command,
|
||||||
|
env=session_env(),
|
||||||
|
stdin=subprocess.DEVNULL,
|
||||||
|
stdout=subprocess.DEVNULL,
|
||||||
|
stderr=subprocess.DEVNULL,
|
||||||
|
start_new_session=True,
|
||||||
|
)
|
||||||
|
except OSError as exc:
|
||||||
|
log.error("could not launch pantry-kiosk: %s", exc)
|
||||||
|
|
@ -0,0 +1,4 @@
|
||||||
|
# The ISO installs this from apt (python3-paho-mqtt) rather than pip — see
|
||||||
|
# live-build/config/package-lists/kitchen-display.list.chroot. This file is for
|
||||||
|
# running the agent outside the image (development, a venv on a test box).
|
||||||
|
paho-mqtt>=1.6
|
||||||
|
|
@ -0,0 +1,9 @@
|
||||||
|
# greetd — autologin straight into the kiosk Sway session, no greeter UI. Same
|
||||||
|
# shape as hosts/touch-panel's and hosts/thin-client's own config.toml.
|
||||||
|
|
||||||
|
[terminal]
|
||||||
|
vt = 1
|
||||||
|
|
||||||
|
[default_session]
|
||||||
|
command = "/usr/local/bin/kiosk-session"
|
||||||
|
user = "@KIOSK_USERNAME@"
|
||||||
|
|
@ -0,0 +1,17 @@
|
||||||
|
#!/bin/sh
|
||||||
|
# greetd's default_session command. Installed to /usr/local/bin/kiosk-session.
|
||||||
|
set -eu
|
||||||
|
|
||||||
|
[ -r /etc/kitchen-display-agent/config.env ] && . /etc/kitchen-display-agent/config.env
|
||||||
|
export MQTT_BROKER_HOST PANTRY_WEB_URL PANTRY_VISION_URL PANTRY_VISION_TOKEN
|
||||||
|
|
||||||
|
export XDG_CURRENT_DESKTOP=sway
|
||||||
|
export XDG_SESSION_TYPE=wayland
|
||||||
|
export XDG_SESSION_DESKTOP=sway
|
||||||
|
export MOZ_ENABLE_WAYLAND=1
|
||||||
|
export OZONE_PLATFORM=wayland
|
||||||
|
|
||||||
|
: "${XDG_RUNTIME_DIR:=/run/user/$(id -u)}"
|
||||||
|
export XDG_RUNTIME_DIR
|
||||||
|
|
||||||
|
exec sway
|
||||||
|
|
@ -0,0 +1,56 @@
|
||||||
|
# Sway kiosk session for the kitchen display (docs/project-plan.md Phase 17).
|
||||||
|
#
|
||||||
|
# Installed to /home/<KIOSK_USERNAME>/.config/sway/config by
|
||||||
|
# build-kitchen-display-iso.sh. A single-purpose device — one workspace, one
|
||||||
|
# Chromium kiosk window pointed at pantry-web's frontend (Scan / Inventory / Recipes
|
||||||
|
# is in-page tab navigation there, not separate Sway workspaces the way
|
||||||
|
# hosts/touch-panel switches between three different apps).
|
||||||
|
|
||||||
|
set $mod Mod4
|
||||||
|
set $ws_pantry 1:pantry
|
||||||
|
|
||||||
|
input type:keyboard {
|
||||||
|
xkb_layout @KEYBOARD_LAYOUT@
|
||||||
|
}
|
||||||
|
|
||||||
|
# Real touchscreen hardware — native Wayland wl_touch, same assumption and same
|
||||||
|
# caveat as hosts/touch-panel/configs/sway/config (see that host's README
|
||||||
|
# verification list item 3).
|
||||||
|
input type:touch {
|
||||||
|
map_to_output "*"
|
||||||
|
}
|
||||||
|
|
||||||
|
output * bg #101014 solid_color
|
||||||
|
|
||||||
|
default_border none
|
||||||
|
default_floating_border none
|
||||||
|
hide_edge_borders both
|
||||||
|
gaps inner 0
|
||||||
|
gaps outer 0
|
||||||
|
|
||||||
|
# kitchen-display-agent is NOT started here — systemd owns it, same reasoning as
|
||||||
|
# every other agent in this project (reachable over MQTT whether or not a graphical
|
||||||
|
# session is up, survives a sway restart).
|
||||||
|
|
||||||
|
for_window [app_id="chromium.*"] fullscreen enable
|
||||||
|
for_window [app_id="chromium.*"] inhibit_idle fullscreen
|
||||||
|
|
||||||
|
exec sh -c '[ -n "$PANTRY_WEB_URL" ] && { swaymsg workspace $ws_pantry; /usr/local/bin/pantry-kiosk; }'
|
||||||
|
|
||||||
|
# Blank after 20 minutes; any touch resumes it. No lock screen — same reasoning as
|
||||||
|
# every other kiosk image in this project: a shared kitchen display has nobody to
|
||||||
|
# unlock it.
|
||||||
|
exec swayidle -w \
|
||||||
|
timeout 1200 'swaymsg "output * power off"' \
|
||||||
|
resume 'swaymsg "output * power on"'
|
||||||
|
|
||||||
|
bindsym $mod+Shift+Ctrl+m exec foot --title maintenance-shell
|
||||||
|
for_window [title="maintenance-shell"] floating enable, resize set width 800 height 500, move position center
|
||||||
|
|
||||||
|
bindsym $mod+Return exec foot
|
||||||
|
bindsym $mod+q kill
|
||||||
|
bindsym $mod+Shift+c reload
|
||||||
|
|
||||||
|
# Deliberately no exit binding — same reasoning as the thin client/touch panel.
|
||||||
|
|
||||||
|
workspace $ws_pantry
|
||||||
|
|
@ -0,0 +1,58 @@
|
||||||
|
#!/bin/sh
|
||||||
|
# Chromium kiosk window pointed at pantry-vision's frontend (pantry-web). Installed
|
||||||
|
# to /usr/local/bin/pantry-kiosk. Called both at session start and by
|
||||||
|
# kitchen-display-agent when a "Show scan/inventory/recipes" MQTT command arrives.
|
||||||
|
#
|
||||||
|
# Kill-and-relaunch, not focus-and-navigate — same reasoning as hosts/thin-client's
|
||||||
|
# digest-browser: Chromium's --kiosk window has no address bar to drive remotely, and
|
||||||
|
# the page underneath (pantry-vision/frontend/) is cheap to reload. Camera capture is
|
||||||
|
# momentary (hold item up, tap Capture) so losing an in-progress scan to a "Show
|
||||||
|
# inventory" command is an acceptable, rare edge case, not a design flaw to solve here.
|
||||||
|
set -eu
|
||||||
|
|
||||||
|
PROFILE_DIR="${HOME:-/home/$(id -un)}/.config/kitchen-display-chromium"
|
||||||
|
|
||||||
|
BASE_URL="${PANTRY_WEB_URL:-}"
|
||||||
|
[ -n "$BASE_URL" ] || { echo "pantry-kiosk: PANTRY_WEB_URL is unset" >&2; exit 1; }
|
||||||
|
[ -n "${PANTRY_VISION_URL:-}" ] || { echo "pantry-kiosk: PANTRY_VISION_URL is unset" >&2; exit 1; }
|
||||||
|
[ -n "${PANTRY_VISION_TOKEN:-}" ] || { echo "pantry-kiosk: PANTRY_VISION_TOKEN is unset" >&2; exit 1; }
|
||||||
|
|
||||||
|
# api=/token= are read by frontend/app.js — see that file's top comment for why
|
||||||
|
# config travels this way instead of being baked into the served static files.
|
||||||
|
FRAGMENT="${1:-}"
|
||||||
|
URL="${BASE_URL%/}/index.html?api=$(printf '%s' "$PANTRY_VISION_URL" | sed 's/\//%2F/g; s/:/%3A/g')&token=${PANTRY_VISION_TOKEN}"
|
||||||
|
[ -n "$FRAGMENT" ] && URL="${URL}#${FRAGMENT}"
|
||||||
|
|
||||||
|
if command -v chromium >/dev/null 2>&1; then
|
||||||
|
CHROMIUM=chromium
|
||||||
|
elif command -v chromium-browser >/dev/null 2>&1; then
|
||||||
|
CHROMIUM=chromium-browser
|
||||||
|
else
|
||||||
|
echo "pantry-kiosk: no chromium/chromium-browser binary found" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
mkdir -p "$PROFILE_DIR"
|
||||||
|
|
||||||
|
pkill -u "$(id -u)" -f "$CHROMIUM .*--user-data-dir=$PROFILE_DIR" 2>/dev/null || true
|
||||||
|
i=0
|
||||||
|
while pgrep -u "$(id -u)" -f "$CHROMIUM .*--user-data-dir=$PROFILE_DIR" >/dev/null 2>&1 && [ "$i" -lt 20 ]; do
|
||||||
|
sleep 0.25
|
||||||
|
i=$((i + 1))
|
||||||
|
done
|
||||||
|
|
||||||
|
# --use-fake-ui-for-media-stream auto-accepts the getUserMedia camera prompt instead
|
||||||
|
# of leaving it sitting unanswered on a screen nobody is there to click "Allow" on —
|
||||||
|
# VERIFY: the simplest documented way to do this: the alternative,
|
||||||
|
# VideoCaptureAllowedUrls enterprise policy, needs a policies.json this image doesn't
|
||||||
|
# currently ship, so start with the flag and only add that if it proves insufficient.
|
||||||
|
exec "$CHROMIUM" \
|
||||||
|
--user-data-dir="$PROFILE_DIR" \
|
||||||
|
--ozone-platform=wayland \
|
||||||
|
--kiosk --app="$URL" \
|
||||||
|
--use-fake-ui-for-media-stream \
|
||||||
|
--start-fullscreen \
|
||||||
|
--noerrdialogs --disable-infobars --disable-session-crashed-bubble \
|
||||||
|
--overscroll-history-navigation=0 \
|
||||||
|
--touch-events=enabled \
|
||||||
|
--check-for-update-interval=31536000
|
||||||
|
|
@ -0,0 +1,42 @@
|
||||||
|
#!/bin/sh
|
||||||
|
# Creates the kiosk account. Identical logic to hosts/touch-panel's/hosts/thin-client's
|
||||||
|
# own 0100-user-setup.hook.chroot.
|
||||||
|
set -eu
|
||||||
|
|
||||||
|
. /etc/kitchen-display-agent/config.env
|
||||||
|
|
||||||
|
if ! id "$KIOSK_USERNAME" >/dev/null 2>&1; then
|
||||||
|
useradd --create-home --shell /bin/bash --comment "Kitchen display kiosk session" "$KIOSK_USERNAME"
|
||||||
|
fi
|
||||||
|
|
||||||
|
for grp in audio video input render dialout netdev plugdev seat _seatd; do
|
||||||
|
if getent group "$grp" >/dev/null 2>&1; then
|
||||||
|
adduser "$KIOSK_USERNAME" "$grp" >/dev/null
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
passwd --lock "$KIOSK_USERNAME" >/dev/null
|
||||||
|
adduser "$KIOSK_USERNAME" sudo >/dev/null
|
||||||
|
|
||||||
|
cat > "/etc/sudoers.d/010-${KIOSK_USERNAME}" <<EOF
|
||||||
|
${KIOSK_USERNAME} ALL=(ALL) NOPASSWD: ALL
|
||||||
|
EOF
|
||||||
|
chmod 0440 "/etc/sudoers.d/010-${KIOSK_USERNAME}"
|
||||||
|
|
||||||
|
mkdir -p /etc/ssh/sshd_config.d
|
||||||
|
cat > /etc/ssh/sshd_config.d/10-kitchen-display.conf <<'EOF'
|
||||||
|
PermitRootLogin no
|
||||||
|
PasswordAuthentication no
|
||||||
|
KbdInteractiveAuthentication no
|
||||||
|
PubkeyAuthentication yes
|
||||||
|
EOF
|
||||||
|
|
||||||
|
if [ -d "/home/${KIOSK_USERNAME}/.ssh" ]; then
|
||||||
|
chmod 700 "/home/${KIOSK_USERNAME}/.ssh"
|
||||||
|
[ -f "/home/${KIOSK_USERNAME}/.ssh/authorized_keys" ] && \
|
||||||
|
chmod 600 "/home/${KIOSK_USERNAME}/.ssh/authorized_keys"
|
||||||
|
fi
|
||||||
|
|
||||||
|
chown -R "${KIOSK_USERNAME}:${KIOSK_USERNAME}" "/home/${KIOSK_USERNAME}"
|
||||||
|
|
||||||
|
systemctl enable ssh >/dev/null 2>&1 || true
|
||||||
|
|
@ -0,0 +1,19 @@
|
||||||
|
#!/bin/sh
|
||||||
|
# Makes greetd the boot target. Identical logic to hosts/touch-panel's own
|
||||||
|
# 0200-greetd.hook.chroot.
|
||||||
|
set -eu
|
||||||
|
|
||||||
|
. /etc/kitchen-display-agent/config.env
|
||||||
|
|
||||||
|
chmod 0755 /usr/local/bin/kiosk-session
|
||||||
|
|
||||||
|
if ! id greeter >/dev/null 2>&1; then
|
||||||
|
useradd --system --create-home --home-dir /var/lib/greetd --shell /usr/sbin/nologin greeter
|
||||||
|
fi
|
||||||
|
|
||||||
|
systemctl enable greetd
|
||||||
|
systemctl set-default graphical.target
|
||||||
|
systemctl mask getty@tty1.service
|
||||||
|
|
||||||
|
mkdir -p "/home/${KIOSK_USERNAME}/.config/sway"
|
||||||
|
chown -R "${KIOSK_USERNAME}:${KIOSK_USERNAME}" "/home/${KIOSK_USERNAME}/.config"
|
||||||
|
|
@ -0,0 +1,12 @@
|
||||||
|
#!/bin/sh
|
||||||
|
# Installs the kitchen-display-agent systemd unit. Identical logic to
|
||||||
|
# hosts/touch-panel's own 0400-touchpanel-agent.hook.chroot.
|
||||||
|
set -eu
|
||||||
|
|
||||||
|
install -m 0644 /opt/kitchen-display-agent/kitchen-display-agent.service \
|
||||||
|
/etc/systemd/system/kitchen-display-agent.service
|
||||||
|
|
||||||
|
chmod 0644 /etc/kitchen-display-agent/config.env
|
||||||
|
chown -R root:root /opt/kitchen-display-agent
|
||||||
|
|
||||||
|
systemctl enable kitchen-display-agent
|
||||||
|
|
@ -0,0 +1,46 @@
|
||||||
|
# Kitchen-display package list (live-build .list.chroot format: one package per
|
||||||
|
# line). Phase 17, docs/project-plan.md.
|
||||||
|
|
||||||
|
# --- Wayland compositor / session ---
|
||||||
|
sway
|
||||||
|
swayidle
|
||||||
|
swaybg
|
||||||
|
greetd
|
||||||
|
foot
|
||||||
|
|
||||||
|
# --- Browser (the only app on this device — a Chromium kiosk window pointed at
|
||||||
|
# pantry-web's frontend, which uses getUserMedia for camera capture directly in
|
||||||
|
# the browser rather than a native capture app) ---
|
||||||
|
chromium
|
||||||
|
|
||||||
|
# --- Agent runtime ---
|
||||||
|
python3
|
||||||
|
python3-venv
|
||||||
|
python3-paho-mqtt
|
||||||
|
procps
|
||||||
|
|
||||||
|
# --- Networking ---
|
||||||
|
network-manager
|
||||||
|
|
||||||
|
# --- Out-of-band admin — no wayvnc on this image either, same scope decision as
|
||||||
|
# hosts/touch-panel (see that host's README) ---
|
||||||
|
openssh-server
|
||||||
|
sudo
|
||||||
|
|
||||||
|
# --- Camera troubleshooting (`v4l2-ctl --list-devices` to confirm the webcam
|
||||||
|
# enumerates correctly) — Chromium/getUserMedia talks to it directly, nothing
|
||||||
|
# here shells out to v4l2 itself ---
|
||||||
|
v4l-utils
|
||||||
|
|
||||||
|
# --- Firmware for real hardware ---
|
||||||
|
firmware-linux
|
||||||
|
firmware-iwlwifi
|
||||||
|
firmware-realtek
|
||||||
|
firmware-misc-nonfree
|
||||||
|
|
||||||
|
# --- Misc ---
|
||||||
|
ca-certificates
|
||||||
|
curl
|
||||||
|
openssl
|
||||||
|
less
|
||||||
|
vim-tiny
|
||||||
|
|
@ -0,0 +1,3 @@
|
||||||
|
# Only relevant when ENABLE_INSTALLER="true" in build-kitchen-display-iso.sh (off by
|
||||||
|
# default). Identical mechanism/caveat as the other hosts' preseed.cfg.
|
||||||
|
d-i keyboard-configuration/xkb-keymap select de
|
||||||
|
|
@ -0,0 +1,265 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
#
|
||||||
|
# Smart Home Kitchen-Display ISO Builder
|
||||||
|
# Target: builds a Debian 12 (Bookworm) live ISO on a Debian/Ubuntu build machine
|
||||||
|
#
|
||||||
|
# Drives `lb config && lb build` over hosts/kitchen-display/live-build/ to produce
|
||||||
|
# the fridge/pantry camera-cataloguing display described in docs/project-plan.md
|
||||||
|
# Phase 17:
|
||||||
|
# - greetd autologin straight into a kiosk Sway session (no greeter UI)
|
||||||
|
# - ONE Chromium kiosk window pointed at pantry-web's frontend (Scan / Inventory /
|
||||||
|
# Recipes is in-page tab navigation, not separate Sway workspaces)
|
||||||
|
# - kitchen-display-agent (Python, systemd) — HA MQTT "Show scan/inventory/recipes"
|
||||||
|
# buttons only; the actual camera capture and Grocy read/write path is
|
||||||
|
# pantry-vision's own published API, called directly by the browser
|
||||||
|
#
|
||||||
|
# Reuses hosts/thin-client's live-build toolchain and directory-split convention, not
|
||||||
|
# its live-build tree — same relationship hosts/touch-panel and hosts/audio-endpoint
|
||||||
|
# already have to the thin client's.
|
||||||
|
#
|
||||||
|
# Run as: sudo ./build-kitchen-display-iso.sh
|
||||||
|
#
|
||||||
|
# EDIT THE VARIABLES BELOW BEFORE RUNNING.
|
||||||
|
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# CONFIGURATION — edit these before running
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
DEBIAN_RELEASE="bookworm"
|
||||||
|
KIOSK_USERNAME="kiosk"
|
||||||
|
IMAGE_HOSTNAME="kitchen-display"
|
||||||
|
KITCHEN_DISPLAY_NAME="Kitchen fridge display"
|
||||||
|
|
||||||
|
KEYBOARD_LAYOUT="de"
|
||||||
|
|
||||||
|
ENABLE_INSTALLER="false"
|
||||||
|
|
||||||
|
# --- Where the kitchen display talks to -------------------------------------
|
||||||
|
MQTT_BROKER_HOST="192.168.1.10" # <-- EDIT: container-host IP running Mosquitto
|
||||||
|
MQTT_BROKER_PORT="1883"
|
||||||
|
MQTT_USERNAME=""
|
||||||
|
MQTT_PASSWORD=""
|
||||||
|
|
||||||
|
# pantry-web (nginx, serves pantry-vision/frontend/) and pantry-vision (the API the
|
||||||
|
# frontend calls directly from the browser) — both from ENABLE_PANTRY_VISION in
|
||||||
|
# setup-container-host.sh. Placeholders until that's deployed; the image builds and
|
||||||
|
# boots fine without it, the kiosk window just shows a connection error.
|
||||||
|
PANTRY_WEB_URL="http://192.168.1.10:8096" # <-- EDIT once pantry-web is deployed
|
||||||
|
PANTRY_VISION_URL="http://192.168.1.10:8095" # <-- EDIT once pantry-vision is deployed
|
||||||
|
# Same value as PANTRY_VISION_TOKEN in pantry-vision/pantry-vision.env — there is no
|
||||||
|
# way for this repo to push it between the two hosts for you, same as every other
|
||||||
|
# credential pair that spans two machines in this project.
|
||||||
|
PANTRY_VISION_TOKEN="" # <-- EDIT: must match pantry-vision's own token
|
||||||
|
|
||||||
|
SSH_AUTHORIZED_KEY=""
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Paths
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
KITCHEN_DISPLAY_DIR="$(dirname "$SCRIPT_DIR")"
|
||||||
|
CONFIGS_DIR="${KITCHEN_DISPLAY_DIR}/configs"
|
||||||
|
AGENT_DIR="${KITCHEN_DISPLAY_DIR}/agent"
|
||||||
|
LIVE_BUILD_DIR="${KITCHEN_DISPLAY_DIR}/live-build"
|
||||||
|
INCLUDES="${LIVE_BUILD_DIR}/config/includes.chroot"
|
||||||
|
PACKAGE_LIST="${LIVE_BUILD_DIR}/config/package-lists/kitchen-display.list.chroot"
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Sanity checks
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
if [[ $EUID -ne 0 ]]; then
|
||||||
|
echo "Warning: not running as root. 'lb build' needs root; re-run with: sudo $0"
|
||||||
|
echo " Continuing anyway so you can at least regenerate includes.chroot..."
|
||||||
|
fi
|
||||||
|
|
||||||
|
if ! command -v lb &> /dev/null; then
|
||||||
|
if [[ $EUID -eq 0 ]]; then
|
||||||
|
echo "--- Installing live-build ---"
|
||||||
|
apt-get update
|
||||||
|
apt-get install -y live-build
|
||||||
|
else
|
||||||
|
echo "live-build is not installed and this script is not running as root." >&2
|
||||||
|
echo " Install it first: sudo apt-get install live-build" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
echo "--- live-build already installed, skipping ---"
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [[ ! -f "$PACKAGE_LIST" ]]; then
|
||||||
|
echo "Missing package list: $PACKAGE_LIST" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [[ "$MQTT_BROKER_HOST" == "192.168.1.10" ]]; then
|
||||||
|
echo "Warning: MQTT_BROKER_HOST is still the placeholder IP — edit it before building."
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [[ -z "$PANTRY_VISION_TOKEN" ]]; then
|
||||||
|
echo "Warning: PANTRY_VISION_TOKEN is empty. The kiosk window will load but every"
|
||||||
|
echo " call to pantry-vision will fail (401) until this matches the token in"
|
||||||
|
echo " pantry-vision/pantry-vision.env on the container host."
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo
|
||||||
|
echo "=== Smart Home Kitchen-Display ISO Builder ==="
|
||||||
|
echo "Debian release : $DEBIAN_RELEASE"
|
||||||
|
echo "Kiosk user : $KIOSK_USERNAME"
|
||||||
|
echo "Image hostname : $IMAGE_HOSTNAME"
|
||||||
|
echo "MQTT broker : ${MQTT_BROKER_HOST}:${MQTT_BROKER_PORT}"
|
||||||
|
echo "pantry-web : $PANTRY_WEB_URL"
|
||||||
|
echo "pantry-vision : $PANTRY_VISION_URL"
|
||||||
|
echo "Keyboard layout : $KEYBOARD_LAYOUT"
|
||||||
|
echo
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# 1. Regenerate includes.chroot from configs/ and agent/
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
echo "--- Regenerating $INCLUDES ---"
|
||||||
|
rm -rf "$INCLUDES"
|
||||||
|
mkdir -p \
|
||||||
|
"$INCLUDES/etc/greetd" \
|
||||||
|
"$INCLUDES/etc/kitchen-display-agent" \
|
||||||
|
"$INCLUDES/usr/local/bin" \
|
||||||
|
"$INCLUDES/opt/kitchen-display-agent" \
|
||||||
|
"$INCLUDES/home/${KIOSK_USERNAME}/.config/sway" \
|
||||||
|
"$INCLUDES/home/${KIOSK_USERNAME}/.ssh"
|
||||||
|
|
||||||
|
subst() {
|
||||||
|
sed -e "s/@KIOSK_USERNAME@/${KIOSK_USERNAME}/g" \
|
||||||
|
-e "s/@KEYBOARD_LAYOUT@/${KEYBOARD_LAYOUT}/g" "$1" > "$2"
|
||||||
|
}
|
||||||
|
|
||||||
|
subst "${CONFIGS_DIR}/greetd/config.toml" "$INCLUDES/etc/greetd/config.toml"
|
||||||
|
subst "${CONFIGS_DIR}/sway/config" "$INCLUDES/home/${KIOSK_USERNAME}/.config/sway/config"
|
||||||
|
subst "${AGENT_DIR}/kitchen-display-agent.service" "$INCLUDES/opt/kitchen-display-agent/kitchen-display-agent.service"
|
||||||
|
|
||||||
|
install -m 0755 "${CONFIGS_DIR}/greetd/kiosk-session" "$INCLUDES/usr/local/bin/kiosk-session"
|
||||||
|
install -m 0755 "${CONFIGS_DIR}/sway/pantry-kiosk" "$INCLUDES/usr/local/bin/pantry-kiosk"
|
||||||
|
|
||||||
|
mkdir -p "$INCLUDES/etc/default"
|
||||||
|
cat > "$INCLUDES/etc/default/keyboard" <<EOF
|
||||||
|
XKBMODEL="pc105"
|
||||||
|
XKBLAYOUT="${KEYBOARD_LAYOUT}"
|
||||||
|
XKBVARIANT=""
|
||||||
|
XKBOPTIONS=""
|
||||||
|
|
||||||
|
BACKSPACE="guess"
|
||||||
|
EOF
|
||||||
|
|
||||||
|
cp -r "${AGENT_DIR}/kitchen_display_agent" "$INCLUDES/opt/kitchen-display-agent/"
|
||||||
|
install -m 0644 "${AGENT_DIR}/requirements.txt" "$INCLUDES/opt/kitchen-display-agent/requirements.txt"
|
||||||
|
find "$INCLUDES/opt/kitchen-display-agent" -name '__pycache__' -type d -prune -exec rm -rf {} +
|
||||||
|
|
||||||
|
if [[ -n "$SSH_AUTHORIZED_KEY" ]]; then
|
||||||
|
echo "$SSH_AUTHORIZED_KEY" > "$INCLUDES/home/${KIOSK_USERNAME}/.ssh/authorized_keys"
|
||||||
|
chmod 600 "$INCLUDES/home/${KIOSK_USERNAME}/.ssh/authorized_keys"
|
||||||
|
echo " Baked an SSH authorized_keys entry for ${KIOSK_USERNAME}."
|
||||||
|
else
|
||||||
|
echo " No SSH_AUTHORIZED_KEY set — SSH admin access will not be possible on this image."
|
||||||
|
fi
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# 2. Runtime config
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
echo "--- Writing /etc/kitchen-display-agent/config.env into includes.chroot ---"
|
||||||
|
cat > "$INCLUDES/etc/kitchen-display-agent/config.env" <<EOF
|
||||||
|
# Generated by hosts/kitchen-display/scripts/build-kitchen-display-iso.sh — do not
|
||||||
|
# hand-edit here; edit the CONFIGURATION block in that script and rebuild.
|
||||||
|
KIOSK_USERNAME=${KIOSK_USERNAME}
|
||||||
|
KITCHEN_DISPLAY_NAME=${KITCHEN_DISPLAY_NAME}
|
||||||
|
|
||||||
|
MQTT_BROKER_HOST=${MQTT_BROKER_HOST}
|
||||||
|
MQTT_BROKER_PORT=${MQTT_BROKER_PORT}
|
||||||
|
MQTT_USERNAME=${MQTT_USERNAME}
|
||||||
|
MQTT_PASSWORD=${MQTT_PASSWORD}
|
||||||
|
|
||||||
|
PANTRY_WEB_URL=${PANTRY_WEB_URL}
|
||||||
|
PANTRY_VISION_URL=${PANTRY_VISION_URL}
|
||||||
|
PANTRY_VISION_TOKEN=${PANTRY_VISION_TOKEN}
|
||||||
|
EOF
|
||||||
|
chmod 0644 "$INCLUDES/etc/kitchen-display-agent/config.env"
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# 3. lb config
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
cd "$LIVE_BUILD_DIR"
|
||||||
|
|
||||||
|
chmod +x config/hooks/normal/*.hook.chroot
|
||||||
|
|
||||||
|
if [[ -e .build ]]; then
|
||||||
|
echo "--- Previous build found, running 'lb clean' (package cache is kept) ---"
|
||||||
|
lb clean
|
||||||
|
fi
|
||||||
|
|
||||||
|
INSTALLER_MODE="none"
|
||||||
|
if [[ "$ENABLE_INSTALLER" == "true" ]]; then
|
||||||
|
INSTALLER_MODE="live"
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "--- Running lb config ---"
|
||||||
|
lb config \
|
||||||
|
--distribution "$DEBIAN_RELEASE" \
|
||||||
|
--architectures amd64 \
|
||||||
|
--linux-flavours amd64 \
|
||||||
|
--archive-areas "main contrib non-free-firmware" \
|
||||||
|
--binary-images iso-hybrid \
|
||||||
|
--debian-installer "$INSTALLER_MODE" \
|
||||||
|
--iso-application "SmartestHome Kitchen Display" \
|
||||||
|
--iso-publisher "SmartestHome" \
|
||||||
|
--iso-volume "smarthome-kitchen-display" \
|
||||||
|
--memtest none \
|
||||||
|
--bootappend-live "boot=live components quiet splash noautologin username=${KIOSK_USERNAME} hostname=${IMAGE_HOSTNAME}"
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# 4. lb build
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
echo "--- Running lb build (this takes a while and needs network) ---"
|
||||||
|
lb build
|
||||||
|
|
||||||
|
ISO_PATH="$(find "$LIVE_BUILD_DIR" -maxdepth 1 -name '*.iso' -print -quit)"
|
||||||
|
ISO_PATH="${ISO_PATH:-${LIVE_BUILD_DIR}/live-image-amd64.hybrid.iso}"
|
||||||
|
|
||||||
|
echo
|
||||||
|
echo "=== Done ==="
|
||||||
|
echo "ISO written to:"
|
||||||
|
echo " ${ISO_PATH}"
|
||||||
|
echo
|
||||||
|
echo "What's in it:"
|
||||||
|
echo " greetd : autologin as '${KIOSK_USERNAME}' straight into Sway on vt1"
|
||||||
|
echo " Sway : one workspace, one Chromium kiosk window"
|
||||||
|
echo " Kiosk page : ${PANTRY_WEB_URL}/index.html (Scan / Inventory / Recipes tabs)"
|
||||||
|
echo " Camera : getUserMedia in Chromium (--use-fake-ui-for-media-stream"
|
||||||
|
echo " auto-accepts the permission prompt), not a native capture app"
|
||||||
|
echo " kitchen-display-agent: system service, MQTT ${MQTT_BROKER_HOST}:${MQTT_BROKER_PORT}"
|
||||||
|
echo " Keyboard layout : ${KEYBOARD_LAYOUT} (console + Sway)"
|
||||||
|
echo " Maintenance shell : Super+Shift+Ctrl+M opens a floating terminal locally"
|
||||||
|
echo " Remote admin : SSH only — no wayvnc, same scope decision as hosts/touch-panel"
|
||||||
|
echo
|
||||||
|
echo "Next steps:"
|
||||||
|
echo " 1. Write the ISO to a USB stick:"
|
||||||
|
echo " sudo dd if=${ISO_PATH} of=/dev/sdX bs=4M status=progress oflag=sync"
|
||||||
|
echo " (double-check /dev/sdX with 'lsblk' first)"
|
||||||
|
echo " 2. Boot the target machine — a webcam pointed at the counter/fridge front is"
|
||||||
|
echo " required hardware, not built in; see the README for what's assumed about it."
|
||||||
|
echo " 3. Confirm greetd autologin lands in Sway with the Chromium kiosk window up."
|
||||||
|
echo " 4. Grant/verify camera access: --use-fake-ui-for-media-stream should skip the"
|
||||||
|
echo " prompt entirely; if the Scan tab shows a camera error, check"
|
||||||
|
echo " 'v4l2-ctl --list-devices' and that Chromium picked the right one (Chromium"
|
||||||
|
echo " defaults to the system default video input — untested on real hardware)."
|
||||||
|
echo " 5. Confirm kitchen-display-agent connected and registered:"
|
||||||
|
echo " systemctl status kitchen-display-agent"
|
||||||
|
echo " In Home Assistant, a '${KITCHEN_DISPLAY_NAME}' device should appear under the"
|
||||||
|
echo " MQTT integration with: Show scan / Show inventory / Show recipes (buttons)."
|
||||||
|
echo " 6. Hold a grocery item up to the camera, tap Capture, and confirm a proposal"
|
||||||
|
echo " comes back from pantry-vision within a few seconds — first checkpoint for"
|
||||||
|
echo " whether the chosen Ollama vision model's latency is workable at all."
|
||||||
|
echo
|
||||||
|
echo "Then pull the power on the container host and re-check: the kiosk must still"
|
||||||
|
echo "boot (Scan/Inventory/Recipes will show connection errors, which is the expected"
|
||||||
|
echo "degraded state — this device has no local fallback data source, same acceptance"
|
||||||
|
echo "already documented for hosts/touch-panel's Home workspace)."
|
||||||
|
echo
|
||||||
|
echo "Rebuilding later: edit hosts/kitchen-display/configs/* or agent/*, then re-run"
|
||||||
|
echo "this script — includes.chroot is regenerated from them every time."
|
||||||
|
|
@ -0,0 +1,203 @@
|
||||||
|
# Sway touch panel
|
||||||
|
|
||||||
|
Phase 16 of `docs/project-plan.md`. Builds a Debian 12 live ISO for a touch-driven
|
||||||
|
wall/counter panel: autologin into Sway, three fixed apps (Spotify, Home Assistant,
|
||||||
|
a general web browser) switched with a finger via an always-on dock, and also
|
||||||
|
controllable by Home Assistant/the local LLM over MQTT.
|
||||||
|
|
||||||
|
**A different device from `../thin-client/`**, not a variant of it. The thin client is
|
||||||
|
a couch-distance media station whose primary control surface is HA/MQTT and wayvnc,
|
||||||
|
deliberately with no on-screen bars. This one is touched directly, so it needs the
|
||||||
|
opposite: a persistent, finger-sized on-screen dock, native touchscreen input, and no
|
||||||
|
gesture/remote-control machinery. It reuses the thin client's live-build toolchain and
|
||||||
|
`configs/`+`agent/` directory-split convention, not its live-build tree — same
|
||||||
|
relationship as `../audio-endpoint/`'s amd64 build to the thin client's.
|
||||||
|
|
||||||
|
What ends up on the image:
|
||||||
|
|
||||||
|
| | |
|
||||||
|
|---|---|
|
||||||
|
| Compositor | Sway, workspaces `1:spotify` / `2:home` / `3:web`, always-on touch dock |
|
||||||
|
| Autologin | greetd, `default_session` straight into `/usr/local/bin/kiosk-session` |
|
||||||
|
| Touch input | Native Wayland `wl_touch` (libinput), no pointer-emulation layer |
|
||||||
|
| Music | Full Spotify GUI client (Flathub `com.spotify.Client`) — not a headless Connect receiver |
|
||||||
|
| Home dashboard | Chromium in kiosk mode, `--app=$HA_URL`, auto-restart if it crashes |
|
||||||
|
| Web browser | Firefox, minimal chrome (back/forward/reload/address bar), general browsing |
|
||||||
|
| On-screen keyboard | wvkbd, toggled from the dock — see the caveat below, no auto-show |
|
||||||
|
| Remote control (HA/LLM) | `touchpanel-agent`, a systemd service publishing HA MQTT-discovery entities |
|
||||||
|
| Remote control (human) | SSH only — **no wayvnc on this image**, see below |
|
||||||
|
|
||||||
|
## Before you build
|
||||||
|
|
||||||
|
> **No touch-panel hardware has been chosen.** Nothing here has been booted on real
|
||||||
|
> metal or a real touchscreen. Touch-as-`wl_touch` (as opposed to some panels that only
|
||||||
|
> expose an emulated-mouse HID interface) is assumed, not confirmed.
|
||||||
|
|
||||||
|
Edit the `# CONFIGURATION` block at the top of
|
||||||
|
[`scripts/build-touch-panel-iso.sh`](scripts/build-touch-panel-iso.sh):
|
||||||
|
|
||||||
|
| Variable | What to put in it |
|
||||||
|
|---|---|
|
||||||
|
| `MQTT_BROKER_HOST` | LAN IP of the container host running Mosquitto (Phase 1) |
|
||||||
|
| `HA_URL` | Home Assistant URL, e.g. `http://192.168.1.10:8123` — this is what loads in the Home workspace |
|
||||||
|
| `KIOSK_USERNAME` | Autologin account name (`kiosk`) |
|
||||||
|
| `TOUCHPANEL_NAME` / `IMAGE_HOSTNAME` | Per-panel identity; each touch panel needs its own |
|
||||||
|
| `SSH_AUTHORIZED_KEY` | Optional. Password auth is disabled and there is no wayvnc, so without a key the only admin path is the local console. |
|
||||||
|
|
||||||
|
## Build
|
||||||
|
|
||||||
|
```sh
|
||||||
|
sudo ./scripts/build-touch-panel-iso.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
Same directory-split convention as the thin client: `configs/` and `agent/` are the
|
||||||
|
human-edited, git-tracked source of truth; `live-build/config/includes.chroot/` is
|
||||||
|
**generated** — wiped and rebuilt on every run, gitignored, never hand-edited. The only
|
||||||
|
templated tokens are `@KIOSK_USERNAME@`/`@KEYBOARD_LAYOUT@`; everything else the
|
||||||
|
in-chroot hooks need comes from `/etc/touchpanel-agent/config.env`.
|
||||||
|
|
||||||
|
## Why there's no wayvnc on this image
|
||||||
|
|
||||||
|
The thin client's primary control surface is deliberately remote (HA/MQTT + wayvnc)
|
||||||
|
because nobody is standing at a couch-distance media station. This panel is the
|
||||||
|
opposite: it's mounted somewhere and touched directly, so remote *view/control* isn't
|
||||||
|
the primary need the way it is for the thin client — SSH already covers "something's
|
||||||
|
broken, let me fix it from a shell." If a real need for a remote-view channel shows up
|
||||||
|
once this is on real hardware, adding wayvnc back is a small, isolated change (copy
|
||||||
|
`../thin-client/configs/wayvnc/` and its two hooks) — deliberately left out for now
|
||||||
|
rather than built against a guess.
|
||||||
|
|
||||||
|
## Home Assistant login
|
||||||
|
|
||||||
|
The Chromium kiosk window's profile (`~/.config/touchpanel-chromium-ha`) is persistent
|
||||||
|
across restarts specifically so logging into Home Assistant once survives a reboot —
|
||||||
|
unlike the thin client's digest/admin kiosk Firefox profiles, which are stateless
|
||||||
|
pages re-copied on every launch. Log in on first boot (tap the dock's Home button if
|
||||||
|
the window doesn't already have focus) and it should stay logged in after that,
|
||||||
|
**unverified** — depends on HA's own session cookie lifetime and whether Chromium's
|
||||||
|
Wayland profile persistence behaves the way assumed.
|
||||||
|
|
||||||
|
## Spotify: full GUI, not a headless Connect receiver
|
||||||
|
|
||||||
|
`../thin-client/` and `../audio-endpoint/` both run `spotifyd`/`librespot` — a
|
||||||
|
Spotify Connect *receiver* with no interface of its own, controlled from the phone
|
||||||
|
app or HA. This panel runs the real, official Spotify Linux client instead
|
||||||
|
(`configs/spotify/spotify-launch`, Flathub `com.spotify.Client`), because the whole
|
||||||
|
point of a touch panel is a screen you interact with directly: browsing your library,
|
||||||
|
searching, picking a playlist with a finger. **Spotify Premium is required for
|
||||||
|
playback either way** — same requirement as the headless receivers.
|
||||||
|
|
||||||
|
Login is interactive, on the device, on first launch, and persists in the Flatpak's
|
||||||
|
own data directory (`~/.var/app/com.spotify.Client`) — nothing here wipes or reseeds
|
||||||
|
it.
|
||||||
|
|
||||||
|
`touchpanel_agent/mpris_bridge.py` bridges Spotify's MPRIS interface (via
|
||||||
|
`playerctl`) into an HA `media_player`-shaped set of entities, identical in shape to
|
||||||
|
the thin client's own bridge, just pointed at `spotify` instead of `mpv`/`spotifyd` as
|
||||||
|
the player name — **unverified that the Spotify Flatpak's MPRIS bus name is actually
|
||||||
|
`spotify`** (some Flatpak-sandboxed apps register under a `flatpak.<app-id>`-shaped
|
||||||
|
name instead). If Playback state never updates, check `playerctl -l` on the booted
|
||||||
|
panel and adjust `PLAYER_PRIORITY` in `mpris_bridge.py`.
|
||||||
|
|
||||||
|
## Touch dock
|
||||||
|
|
||||||
|
An always-visible bar reserved at the bottom of the screen (`configs/eww/`, layer-shell
|
||||||
|
`:exclusive true` so nothing ever tiles under or draws over it) with four buttons:
|
||||||
|
**Spotify**, **Home**, **Web**, and **Keyboard**. The first three call `swaymsg
|
||||||
|
workspace` directly — fixed constants in `eww.yuck`, nothing from MQTT/HA is ever
|
||||||
|
interpolated into them, same rule as the thin client's now-playing widget. This is the
|
||||||
|
touch-first equivalent of the thin client's HA **Screen** select entity — both exist,
|
||||||
|
one for a finger, one for the LLM.
|
||||||
|
|
||||||
|
Session-scoped (`exec_always` in `configs/sway/config`), not part of
|
||||||
|
`touchpanel-agent`, for the identical reason the thin client's now-playing widget
|
||||||
|
isn't: compositor UI should die with the compositor, not need a restart-independent
|
||||||
|
lifecycle or open a second inbound control channel into the security-sensitive agent.
|
||||||
|
|
||||||
|
**If `eww` isn't installed** (it's not in Debian bookworm main —
|
||||||
|
`0500-eww-dock.hook.chroot` tries apt and otherwise leaves a documented placeholder),
|
||||||
|
there is no on-screen app switcher at all. The panel still boots and shows Home; HA/
|
||||||
|
LLM-driven app switching via `touchpanel-agent`'s **Screen** select and **Show
|
||||||
|
\<app\>** buttons still works either way, since that's a separate control path.
|
||||||
|
|
||||||
|
## On-screen keyboard — no auto-show, by design
|
||||||
|
|
||||||
|
`configs/keyboard/toggle-keyboard` toggles `wvkbd` on and off. There is deliberately
|
||||||
|
**no automatic show-on-text-field-focus** — that needs the `text-input-v3`/
|
||||||
|
`virtual-keyboard-v1` Wayland protocols wired all the way through each app via a shell
|
||||||
|
component (the way Phosh does it for squeekboard), which this image doesn't run.
|
||||||
|
Instead: tap the dock's Keyboard button before typing, tap it again to dismiss. This
|
||||||
|
is a known, accepted limitation, not a bug to chase before the panel is otherwise
|
||||||
|
useful — see the verification list below if you want to pursue proper auto-show later.
|
||||||
|
|
||||||
|
`wvkbd`, not `squeekboard`, was picked specifically because it doesn't assume that
|
||||||
|
shell integration exists; see `0700-onscreen-keyboard.hook.chroot`'s comment. It is
|
||||||
|
also not in Debian bookworm main, so that hook has the same apt-first/documented-
|
||||||
|
placeholder shape as `eww` and the thin client's `spotifyd`/`librespot` install.
|
||||||
|
|
||||||
|
## Home Assistant entities
|
||||||
|
|
||||||
|
`touchpanel-agent` publishes MQTT-discovery configs on connect. Under the MQTT
|
||||||
|
integration you should get one device per touch panel with:
|
||||||
|
|
||||||
|
- **Show Spotify**, **Show Home**, **Show web browser** (buttons) — switch workspace
|
||||||
|
and, for Spotify/Home, focus the already-running app rather than relaunching it
|
||||||
|
(both are meant to stay open and stateful, unlike the thin client's stateless
|
||||||
|
digest/admin kiosk pages)
|
||||||
|
- **Screen** (select) — `1:spotify` / `2:home` / `3:web`
|
||||||
|
- **Playback state** (sensor, with track metadata as attributes), **Volume**
|
||||||
|
(number), and play/pause / next / previous / stop (buttons), bridged from
|
||||||
|
Spotify's own MPRIS interface
|
||||||
|
- A **media_player** discovery payload — see the thin client README's identical
|
||||||
|
caveat: stock Home Assistant's MQTT integration has no `media_player` platform: the
|
||||||
|
button/number/sensor entities above are what give playback control on a plain
|
||||||
|
install; the HACS *MQTT Media Player* integration is what picks up this one.
|
||||||
|
|
||||||
|
### Security boundary
|
||||||
|
|
||||||
|
Identical shape and reasoning to `../thin-client/README.md`'s section of the same
|
||||||
|
name: `touchpanel_agent/mqtt_discovery.py` is the **entire** inbound control surface
|
||||||
|
of this machine. The LLM never gets a direct network path here — the only chain is
|
||||||
|
*LLM tool call → HA service call → MQTT → touchpanel-agent*. No HTTP listener, no
|
||||||
|
websocket server, no exposed Sway IPC socket, no VNC (this image has none at all, see
|
||||||
|
above). New control features belong as additional MQTT entities, not as a second
|
||||||
|
listener.
|
||||||
|
|
||||||
|
## Manual verification still outstanding
|
||||||
|
|
||||||
|
None of this has been run on hardware. In rough order:
|
||||||
|
|
||||||
|
1. The ISO builds at all (`lb build` is network-heavy and can fail on mirror hiccups).
|
||||||
|
2. greetd lands in Sway with no login prompt, on vt1, with getty@tty1 masked.
|
||||||
|
3. **Touch input actually arrives as `wl_touch`**, not an emulated pointer — the whole
|
||||||
|
design (native touch in Chromium/Firefox/Spotify, no gesture-translation layer)
|
||||||
|
assumes a real Wayland touchscreen driver. If it only shows up as a mouse, tapping
|
||||||
|
will still mostly work by accident (a single point maps to a click) but pinch/
|
||||||
|
multi-touch in Chromium or Spotify won't.
|
||||||
|
4. `touchpanel-agent` connects to Mosquitto and the device appears in HA.
|
||||||
|
5. **Flathub app ID `com.spotify.Client` is correct** — flagged for verification in
|
||||||
|
`0300-flatpak-spotify.hook.chroot`, same as the thin client's Steam Link ID.
|
||||||
|
6. Spotify login persists in `~/.var/app/com.spotify.Client` across a reboot.
|
||||||
|
7. Spotify's real MPRIS bus name — assumed `spotify`, not confirmed (see the Spotify
|
||||||
|
section above); `playerctl -l` on the booted panel is the way to check.
|
||||||
|
8. **Chromium's Wayland `app_id` for a `--app=` kiosk window** — assumed to start with
|
||||||
|
`chromium` (`configs/sway/config`'s `for_window` rule and
|
||||||
|
`touchpanel_agent/main.py`'s `focus_criteria` both match on it). If focusing the
|
||||||
|
Home workspace's window from the dock/HA stops working, check the real app_id with
|
||||||
|
`swaymsg -t get_tree` and correct both places.
|
||||||
|
9. HA login persistence in the Chromium kiosk profile across a reboot (see above).
|
||||||
|
10. `eww` and `wvkbd` are not in Debian bookworm main — both hooks try apt and
|
||||||
|
otherwise leave a documented placeholder (dock absent / keyboard absent
|
||||||
|
respectively); neither has been checked against the real bookworm archive.
|
||||||
|
11. **The Firefox extension IDs in `configs/firefox/policies.json`** — same unverified
|
||||||
|
caveat as the thin client's copy; see that README's identical item.
|
||||||
|
12. **Phase 16's own "reactive path" check**: power off the container host and confirm
|
||||||
|
the panel still boots to Spotify + the browser (Home will show a connection
|
||||||
|
error, which is the expected/acceptable failure mode — unlike the thin client,
|
||||||
|
there's no local-media fallback to fall back to here since this device has no
|
||||||
|
local media source at all).
|
||||||
|
13. Keyboard layout defaults to German (`KEYBOARD_LAYOUT="de"`) — same per-image,
|
||||||
|
no-runtime-override handling as the thin client's identical setting.
|
||||||
|
14. Dock touch-target sizing (`configs/eww/eww.scss`, ~84px buttons) — a starting
|
||||||
|
guess, not measured against a real finger on real glass at the real screen DPI
|
||||||
|
chosen once hardware exists.
|
||||||
|
|
@ -0,0 +1,8 @@
|
||||||
|
# The ISO installs this from apt (python3-paho-mqtt, 1.6.x on bookworm) rather than
|
||||||
|
# pip — see live-build/config/package-lists/touch-panel.list.chroot. This file is for
|
||||||
|
# running the agent outside the image (development, a venv on a test box). The code
|
||||||
|
# works against both the 1.x and 2.x callback APIs.
|
||||||
|
paho-mqtt>=1.6
|
||||||
|
|
||||||
|
# Config is read from /etc/touchpanel-agent/config.env by a small parser in main.py,
|
||||||
|
# so there is no python-dotenv dependency.
|
||||||
|
|
@ -0,0 +1,31 @@
|
||||||
|
[Unit]
|
||||||
|
Description=Touch panel agent (Home Assistant MQTT control surface for Sway)
|
||||||
|
Documentation=file:///opt/touchpanel-agent
|
||||||
|
# A system unit, not a `systemctl --user` unit — see thinclient-agent.service's
|
||||||
|
# identical reasoning: this must be reachable over MQTT whether or not a graphical
|
||||||
|
# session is up, and must survive sway restarting. Session-scoped bits (SWAYSOCK,
|
||||||
|
# XDG_RUNTIME_DIR, DBUS_SESSION_BUS_ADDRESS) are derived at call time in
|
||||||
|
# sway_control.SwayControl.session_env().
|
||||||
|
#
|
||||||
|
# Deliberately NOT After=network-online.target: the panel must boot to a usable
|
||||||
|
# session (Spotify/HA/browser) with the container host powered off.
|
||||||
|
After=network.target
|
||||||
|
|
||||||
|
[Service]
|
||||||
|
Type=simple
|
||||||
|
User=@KIOSK_USERNAME@
|
||||||
|
Group=@KIOSK_USERNAME@
|
||||||
|
WorkingDirectory=/opt/touchpanel-agent
|
||||||
|
Environment=PYTHONPATH=/opt/touchpanel-agent
|
||||||
|
Environment=PYTHONUNBUFFERED=1
|
||||||
|
EnvironmentFile=-/etc/touchpanel-agent/config.env
|
||||||
|
ExecStart=/usr/bin/python3 -m touchpanel_agent.main
|
||||||
|
Restart=always
|
||||||
|
RestartSec=5
|
||||||
|
# Sandboxing stops here on purpose: this unit's whole job is to spawn GUI child
|
||||||
|
# processes (Chromium, the Spotify flatpak, Firefox) that need the real /tmp, the
|
||||||
|
# user's home, and the session bus.
|
||||||
|
NoNewPrivileges=true
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=multi-user.target
|
||||||
|
|
@ -0,0 +1,3 @@
|
||||||
|
"""touchpanel-agent — Home Assistant MQTT control surface for a Sway touch panel."""
|
||||||
|
|
||||||
|
__version__ = "0.1.0"
|
||||||
|
|
@ -0,0 +1,211 @@
|
||||||
|
"""touchpanel-agent entrypoint."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import logging
|
||||||
|
import os
|
||||||
|
import signal
|
||||||
|
import socket
|
||||||
|
import sys
|
||||||
|
import threading
|
||||||
|
from dataclasses import dataclass
|
||||||
|
|
||||||
|
import paho.mqtt.client as mqtt
|
||||||
|
|
||||||
|
from .mpris_bridge import MprisBridge
|
||||||
|
from .mqtt_discovery import Discovery
|
||||||
|
from .sway_control import WS_HOME, WS_SPOTIFY, WS_WEB, SwayControl
|
||||||
|
|
||||||
|
CONFIG_PATH = os.environ.get("TOUCHPANEL_AGENT_CONFIG", "/etc/touchpanel-agent/config.env")
|
||||||
|
|
||||||
|
CONFIG_KEYS = (
|
||||||
|
"MQTT_BROKER_HOST",
|
||||||
|
"MQTT_BROKER_PORT",
|
||||||
|
"MQTT_USERNAME",
|
||||||
|
"MQTT_PASSWORD",
|
||||||
|
"HA_URL",
|
||||||
|
"KIOSK_USERNAME",
|
||||||
|
"TOUCHPANEL_NAME",
|
||||||
|
)
|
||||||
|
|
||||||
|
WORKSPACES = (WS_SPOTIFY, WS_HOME, WS_WEB)
|
||||||
|
|
||||||
|
log = logging.getLogger("touchpanel-agent")
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class App:
|
||||||
|
name: str
|
||||||
|
command: list[str]
|
||||||
|
process_pattern: str | None = None
|
||||||
|
workspace: str | None = None
|
||||||
|
focus_criteria: str | None = None
|
||||||
|
icon: str = "mdi:application"
|
||||||
|
|
||||||
|
|
||||||
|
def load_config(path: str = CONFIG_PATH) -> dict[str, str]:
|
||||||
|
values: dict[str, str] = {}
|
||||||
|
try:
|
||||||
|
with open(path, encoding="utf-8") as handle:
|
||||||
|
for line in handle:
|
||||||
|
line = line.strip()
|
||||||
|
if not line or line.startswith("#") or "=" not in line:
|
||||||
|
continue
|
||||||
|
key, _, value = line.partition("=")
|
||||||
|
values[key.strip()] = value.strip().strip('"').strip("'")
|
||||||
|
except OSError as exc:
|
||||||
|
log.warning("could not read %s (%s); falling back to the environment", path, exc)
|
||||||
|
|
||||||
|
for key in CONFIG_KEYS:
|
||||||
|
if key in os.environ:
|
||||||
|
values[key] = os.environ[key]
|
||||||
|
|
||||||
|
return values
|
||||||
|
|
||||||
|
|
||||||
|
def build_apps(_config: dict[str, str]) -> dict[str, App]:
|
||||||
|
return {
|
||||||
|
"spotify": App(
|
||||||
|
name="Spotify",
|
||||||
|
command=["/usr/local/bin/spotify-launch"],
|
||||||
|
process_pattern="com.spotify.Client",
|
||||||
|
workspace=WS_SPOTIFY,
|
||||||
|
focus_criteria='app_id="spotify"',
|
||||||
|
icon="mdi:spotify",
|
||||||
|
),
|
||||||
|
"home": App(
|
||||||
|
name="Home",
|
||||||
|
# ha-kiosk is already running (started at session boot, see
|
||||||
|
# configs/sway/config) and supervises its own restart loop — this just
|
||||||
|
# switches workspace and focuses the existing window rather than
|
||||||
|
# relaunching it, since the HA dashboard is stateful. See ha-kiosk's own
|
||||||
|
# comment for why this is deliberately not a kill-and-relaunch like the
|
||||||
|
# thin client's digest canvas.
|
||||||
|
command=["/usr/local/bin/ha-kiosk"],
|
||||||
|
process_pattern="ha-kiosk",
|
||||||
|
workspace=WS_HOME,
|
||||||
|
focus_criteria='app_id="chromium.*"',
|
||||||
|
icon="mdi:home-assistant",
|
||||||
|
),
|
||||||
|
"web_browser": App(
|
||||||
|
name="web browser",
|
||||||
|
command=["/usr/local/bin/web-browser"],
|
||||||
|
workspace=WS_WEB,
|
||||||
|
icon="mdi:web",
|
||||||
|
),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def make_client(client_id: str) -> mqtt.Client:
|
||||||
|
# paho-mqtt 2.x requires an explicit callback API version; bookworm's
|
||||||
|
# python3-paho-mqtt is 1.6.x and has no such argument. VERSION1 is requested when
|
||||||
|
# available so the callback signatures below are identical under both.
|
||||||
|
callback_api = getattr(mqtt, "CallbackAPIVersion", None)
|
||||||
|
if callback_api is not None:
|
||||||
|
return mqtt.Client(callback_api.VERSION1, client_id=client_id)
|
||||||
|
return mqtt.Client(client_id=client_id)
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
logging.basicConfig(
|
||||||
|
level=logging.INFO,
|
||||||
|
format="%(asctime)s %(levelname)s %(name)s: %(message)s",
|
||||||
|
stream=sys.stdout,
|
||||||
|
)
|
||||||
|
|
||||||
|
config = load_config()
|
||||||
|
hostname = socket.gethostname()
|
||||||
|
node_id = "".join(c if c.isalnum() else "_" for c in hostname).strip("_") or "touchpanel"
|
||||||
|
friendly_name = config.get("TOUCHPANEL_NAME") or f"Touch panel ({hostname})"
|
||||||
|
|
||||||
|
broker_host = config.get("MQTT_BROKER_HOST", "")
|
||||||
|
broker_port = int(config.get("MQTT_BROKER_PORT") or 1883)
|
||||||
|
|
||||||
|
sway = SwayControl()
|
||||||
|
apps = build_apps(config)
|
||||||
|
|
||||||
|
client = make_client(f"touchpanel-agent-{node_id}")
|
||||||
|
if config.get("MQTT_USERNAME"):
|
||||||
|
client.username_pw_set(config["MQTT_USERNAME"], config.get("MQTT_PASSWORD") or None)
|
||||||
|
|
||||||
|
discovery = Discovery(client, node_id, friendly_name)
|
||||||
|
mpris = MprisBridge(discovery.publish_media_state, sway.session_env)
|
||||||
|
|
||||||
|
def on_launch(key: str) -> None:
|
||||||
|
app = apps[key]
|
||||||
|
sway.launch_app(
|
||||||
|
app.command,
|
||||||
|
process_pattern=app.process_pattern,
|
||||||
|
workspace=app.workspace,
|
||||||
|
focus_criteria=app.focus_criteria,
|
||||||
|
)
|
||||||
|
if app.workspace:
|
||||||
|
discovery.publish_workspace(app.workspace)
|
||||||
|
|
||||||
|
def on_workspace(payload: str) -> None:
|
||||||
|
name = payload.strip()
|
||||||
|
# Enumerated, never passed through: see the security note in mqtt_discovery.py.
|
||||||
|
if name not in WORKSPACES:
|
||||||
|
log.warning("ignoring unknown workspace %r", name)
|
||||||
|
return
|
||||||
|
sway.switch_workspace(name)
|
||||||
|
discovery.publish_workspace(name)
|
||||||
|
|
||||||
|
def on_connect(_client, _userdata, _flags, rc):
|
||||||
|
if rc != 0:
|
||||||
|
log.error("MQTT connection refused (rc=%s)", rc)
|
||||||
|
return
|
||||||
|
log.info("connected to MQTT broker %s:%s", broker_host, broker_port)
|
||||||
|
discovery.register_media_player(mpris.handle_command, mpris.set_volume)
|
||||||
|
discovery.register_app_launchers(apps, on_launch)
|
||||||
|
discovery.register_workspace_select(WORKSPACES, on_workspace, WS_HOME)
|
||||||
|
discovery.subscribe_all()
|
||||||
|
discovery.publish_available(True)
|
||||||
|
|
||||||
|
def on_disconnect(_client, _userdata, rc):
|
||||||
|
log.warning("disconnected from MQTT broker (rc=%s); paho will retry", rc)
|
||||||
|
|
||||||
|
def on_message(_client, _userdata, message):
|
||||||
|
discovery.dispatch(message.topic, message.payload.decode("utf-8", "replace"))
|
||||||
|
|
||||||
|
client.on_connect = on_connect
|
||||||
|
client.on_disconnect = on_disconnect
|
||||||
|
client.on_message = on_message
|
||||||
|
client.will_set(discovery.availability_topic, "offline", qos=1, retain=True)
|
||||||
|
|
||||||
|
stop_event = threading.Event()
|
||||||
|
|
||||||
|
def handle_signal(_signum, _frame):
|
||||||
|
stop_event.set()
|
||||||
|
|
||||||
|
signal.signal(signal.SIGTERM, handle_signal)
|
||||||
|
signal.signal(signal.SIGINT, handle_signal)
|
||||||
|
|
||||||
|
if not broker_host:
|
||||||
|
log.error("MQTT_BROKER_HOST is not set in %s — running without HA control", CONFIG_PATH)
|
||||||
|
else:
|
||||||
|
# connect_async + loop_start, never a blocking connect(): the panel must come
|
||||||
|
# up and show Spotify/HA/the browser with the container host powered off,
|
||||||
|
# same "reactive path never depends on a remote service" rule as the thin
|
||||||
|
# client's Phase 11.10.
|
||||||
|
client.connect_async(broker_host, broker_port, keepalive=60)
|
||||||
|
client.loop_start()
|
||||||
|
|
||||||
|
log.info("touchpanel-agent %s started (node_id=%s)", node_id, node_id)
|
||||||
|
if not config.get("HA_URL"):
|
||||||
|
log.warning("HA_URL is not set — the Home workspace will have nothing to show")
|
||||||
|
|
||||||
|
try:
|
||||||
|
mpris.run_forever(stop_event)
|
||||||
|
finally:
|
||||||
|
log.info("shutting down")
|
||||||
|
if broker_host:
|
||||||
|
discovery.publish_available(False)
|
||||||
|
client.loop_stop()
|
||||||
|
client.disconnect()
|
||||||
|
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main())
|
||||||
|
|
@ -0,0 +1,127 @@
|
||||||
|
"""Bridges the Spotify client's MPRIS state to the HA media_player entity via playerctl.
|
||||||
|
|
||||||
|
Identical in shape and reasoning to
|
||||||
|
hosts/thin-client/agent/thinclient_agent/mpris_bridge.py, with PLAYER_PRIORITY pointed
|
||||||
|
at the Spotify GUI client's own MPRIS bus name instead of mpv/spotifyd — see that
|
||||||
|
file's docstring for why this polls via subprocess rather than holding a dbus
|
||||||
|
connection open.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import logging
|
||||||
|
import subprocess
|
||||||
|
import threading
|
||||||
|
|
||||||
|
log = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
PLAYER_PRIORITY = "spotify,%any"
|
||||||
|
|
||||||
|
_STATUS_TO_HA = {
|
||||||
|
"Playing": "playing",
|
||||||
|
"Paused": "paused",
|
||||||
|
"Stopped": "idle",
|
||||||
|
}
|
||||||
|
|
||||||
|
_METADATA_FORMAT = "{{title}}\x1f{{artist}}\x1f{{album}}\x1f{{mpris:length}}\x1f{{mpris:artUrl}}"
|
||||||
|
|
||||||
|
|
||||||
|
class MprisBridge:
|
||||||
|
def __init__(self, publish_state, env_provider, poll_interval: float = 2.0):
|
||||||
|
self._publish_state = publish_state
|
||||||
|
# playerctl needs DBUS_SESSION_BUS_ADDRESS, which this system service does not
|
||||||
|
# inherit; SwayControl.session_env() derives it from the kiosk user's runtime dir.
|
||||||
|
self._env_provider = env_provider
|
||||||
|
self._poll_interval = poll_interval
|
||||||
|
self._last_state: dict | None = None
|
||||||
|
|
||||||
|
def _playerctl(self, *args: str) -> str | None:
|
||||||
|
try:
|
||||||
|
result = subprocess.run(
|
||||||
|
["playerctl", "-p", PLAYER_PRIORITY, *args],
|
||||||
|
env=self._env_provider(),
|
||||||
|
capture_output=True,
|
||||||
|
text=True,
|
||||||
|
timeout=5,
|
||||||
|
check=False,
|
||||||
|
)
|
||||||
|
except (OSError, subprocess.TimeoutExpired) as exc:
|
||||||
|
log.debug("playerctl %s failed: %s", " ".join(args), exc)
|
||||||
|
return None
|
||||||
|
if result.returncode != 0:
|
||||||
|
return None
|
||||||
|
return result.stdout.strip()
|
||||||
|
|
||||||
|
def read_state(self) -> dict:
|
||||||
|
status = self._playerctl("status")
|
||||||
|
if status is None:
|
||||||
|
return {"state": "off"}
|
||||||
|
|
||||||
|
state = {"state": _STATUS_TO_HA.get(status, "idle")}
|
||||||
|
|
||||||
|
metadata = self._playerctl("metadata", "--format", _METADATA_FORMAT)
|
||||||
|
if metadata:
|
||||||
|
title, artist, album, length, art_url = (metadata.split("\x1f") + [""] * 5)[:5]
|
||||||
|
state["title"] = title
|
||||||
|
state["artist"] = artist
|
||||||
|
state["album"] = album
|
||||||
|
state["art_url"] = art_url
|
||||||
|
if length.isdigit():
|
||||||
|
state["duration"] = int(length) // 1_000_000
|
||||||
|
|
||||||
|
position = self._playerctl("position")
|
||||||
|
if position:
|
||||||
|
try:
|
||||||
|
state["position"] = int(float(position))
|
||||||
|
except ValueError:
|
||||||
|
pass
|
||||||
|
|
||||||
|
volume = self._playerctl("volume")
|
||||||
|
if volume:
|
||||||
|
try:
|
||||||
|
state["volume"] = round(float(volume), 3)
|
||||||
|
except ValueError:
|
||||||
|
pass
|
||||||
|
|
||||||
|
return state
|
||||||
|
|
||||||
|
def poll_once(self) -> None:
|
||||||
|
state = self.read_state()
|
||||||
|
if state != self._last_state:
|
||||||
|
self._last_state = state
|
||||||
|
self._publish_state(state)
|
||||||
|
|
||||||
|
def run_forever(self, stop_event: threading.Event) -> None:
|
||||||
|
while not stop_event.is_set():
|
||||||
|
try:
|
||||||
|
self.poll_once()
|
||||||
|
except Exception:
|
||||||
|
log.exception("MPRIS poll failed")
|
||||||
|
stop_event.wait(self._poll_interval)
|
||||||
|
|
||||||
|
# --- command side -------------------------------------------------------
|
||||||
|
def handle_command(self, command: str) -> None:
|
||||||
|
command = command.strip().upper()
|
||||||
|
action = {
|
||||||
|
"PLAY": ("play",),
|
||||||
|
"PAUSE": ("pause",),
|
||||||
|
"PLAY_PAUSE": ("play-pause",),
|
||||||
|
"TOGGLE": ("play-pause",),
|
||||||
|
"STOP": ("stop",),
|
||||||
|
"NEXT": ("next",),
|
||||||
|
"PREVIOUS": ("previous",),
|
||||||
|
"PREV": ("previous",),
|
||||||
|
}.get(command)
|
||||||
|
|
||||||
|
if action is None:
|
||||||
|
log.warning("ignoring unknown media command %r", command)
|
||||||
|
return
|
||||||
|
|
||||||
|
log.info("media command %s", command)
|
||||||
|
self._playerctl(*action)
|
||||||
|
self.poll_once()
|
||||||
|
|
||||||
|
def set_volume(self, level: float) -> None:
|
||||||
|
level = max(0.0, min(1.0, level))
|
||||||
|
self._playerctl("volume", f"{level:.3f}")
|
||||||
|
self.poll_once()
|
||||||
|
|
@ -0,0 +1,193 @@
|
||||||
|
"""Home Assistant MQTT Discovery payloads and command dispatch.
|
||||||
|
|
||||||
|
SECURITY BOUNDARY — this module is the entire remote-control API of the touch panel.
|
||||||
|
|
||||||
|
Same principle as hosts/thin-client/agent/thinclient_agent/mqtt_discovery.py (Phase
|
||||||
|
11.4), applied to this device: the local LLM never gets a network path to this
|
||||||
|
machine. The only chain is: LLM tool call -> Home Assistant service call -> MQTT ->
|
||||||
|
this dispatcher. That property holds only as long as this stays the sole inbound
|
||||||
|
control surface: no HTTP listener, no websocket server, no exposed Sway IPC socket, no
|
||||||
|
shell endpoint. A new feature belongs as another entity below, not as another
|
||||||
|
listener.
|
||||||
|
|
||||||
|
Every command handler here is a fixed, enumerated action. A payload never becomes an
|
||||||
|
argv element, a shell string, or a URL host — see the launch table in main.py, which
|
||||||
|
builds every command from local constants and uses the payload only to pick between
|
||||||
|
known values.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import logging
|
||||||
|
from typing import Callable
|
||||||
|
|
||||||
|
from . import __version__
|
||||||
|
|
||||||
|
log = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
DISCOVERY_PREFIX = "homeassistant"
|
||||||
|
|
||||||
|
|
||||||
|
class Discovery:
|
||||||
|
def __init__(self, client, node_id: str, friendly_name: str):
|
||||||
|
self.client = client
|
||||||
|
self.node_id = node_id
|
||||||
|
self.friendly_name = friendly_name
|
||||||
|
self.base = f"touchpanel/{node_id}"
|
||||||
|
self.availability_topic = f"{self.base}/availability"
|
||||||
|
self.media_state_topic = f"{self.base}/media/state"
|
||||||
|
self._handlers: dict[str, Callable[[str], None]] = {}
|
||||||
|
|
||||||
|
self.device = {
|
||||||
|
"identifiers": [f"touchpanel_{node_id}"],
|
||||||
|
"name": friendly_name,
|
||||||
|
"manufacturer": "SmartestHome",
|
||||||
|
"model": "Sway touch panel",
|
||||||
|
"sw_version": __version__,
|
||||||
|
}
|
||||||
|
|
||||||
|
# --- plumbing -----------------------------------------------------------
|
||||||
|
def _publish_config(self, component: str, object_id: str, payload: dict) -> None:
|
||||||
|
payload = {
|
||||||
|
"availability_topic": self.availability_topic,
|
||||||
|
"device": self.device,
|
||||||
|
"unique_id": f"{self.node_id}_{object_id}",
|
||||||
|
**payload,
|
||||||
|
}
|
||||||
|
topic = f"{DISCOVERY_PREFIX}/{component}/{self.node_id}/{object_id}/config"
|
||||||
|
self.client.publish(topic, json.dumps(payload), qos=1, retain=True)
|
||||||
|
|
||||||
|
def _command_topic(self, suffix: str, handler) -> str:
|
||||||
|
topic = f"{self.base}/{suffix}"
|
||||||
|
self._handlers[topic] = handler
|
||||||
|
return topic
|
||||||
|
|
||||||
|
def subscribe_all(self) -> None:
|
||||||
|
for topic in self._handlers:
|
||||||
|
self.client.subscribe(topic, qos=1)
|
||||||
|
|
||||||
|
def dispatch(self, topic: str, payload: str) -> None:
|
||||||
|
handler = self._handlers.get(topic)
|
||||||
|
if handler is None:
|
||||||
|
log.warning("no handler for %s", topic)
|
||||||
|
return
|
||||||
|
try:
|
||||||
|
handler(payload)
|
||||||
|
except Exception:
|
||||||
|
log.exception("handler for %s failed", topic)
|
||||||
|
|
||||||
|
def publish_available(self, available: bool = True) -> None:
|
||||||
|
self.client.publish(
|
||||||
|
self.availability_topic,
|
||||||
|
"online" if available else "offline",
|
||||||
|
qos=1,
|
||||||
|
retain=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
def publish_media_state(self, state: dict) -> None:
|
||||||
|
self.client.publish(self.media_state_topic, json.dumps(state), qos=0, retain=True)
|
||||||
|
|
||||||
|
# --- entities -----------------------------------------------------------
|
||||||
|
def register_media_player(self, on_command, on_volume) -> None:
|
||||||
|
command_topic = self._command_topic("media/command", on_command)
|
||||||
|
volume_topic = self._command_topic("media/volume/set", lambda p: on_volume(float(p)))
|
||||||
|
|
||||||
|
# Core Home Assistant's MQTT integration has NO media_player platform — see
|
||||||
|
# hosts/thin-client/README.md's identical caveat. The button/number entities
|
||||||
|
# below give the same transport control with stock HA.
|
||||||
|
self._publish_config(
|
||||||
|
"media_player",
|
||||||
|
"media",
|
||||||
|
{
|
||||||
|
"name": "Media",
|
||||||
|
"state_topic": self.media_state_topic,
|
||||||
|
"state_template": "{{ value_json.state }}",
|
||||||
|
"command_topic": command_topic,
|
||||||
|
"volume_command_topic": volume_topic,
|
||||||
|
"volume_state_topic": self.media_state_topic,
|
||||||
|
"volume_template": "{{ value_json.volume }}",
|
||||||
|
"title_template": "{{ value_json.title }}",
|
||||||
|
"artist_template": "{{ value_json.artist }}",
|
||||||
|
"album_template": "{{ value_json.album }}",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
for object_id, name, payload, icon in (
|
||||||
|
("media_play_pause", "Play/pause", "PLAY_PAUSE", "mdi:play-pause"),
|
||||||
|
("media_next", "Next track", "NEXT", "mdi:skip-next"),
|
||||||
|
("media_previous", "Previous track", "PREVIOUS", "mdi:skip-previous"),
|
||||||
|
("media_stop", "Stop", "STOP", "mdi:stop"),
|
||||||
|
):
|
||||||
|
self._publish_config(
|
||||||
|
"button",
|
||||||
|
object_id,
|
||||||
|
{
|
||||||
|
"name": name,
|
||||||
|
"command_topic": command_topic,
|
||||||
|
"payload_press": payload,
|
||||||
|
"icon": icon,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
self._publish_config(
|
||||||
|
"sensor",
|
||||||
|
"media_state",
|
||||||
|
{
|
||||||
|
"name": "Playback state",
|
||||||
|
"state_topic": self.media_state_topic,
|
||||||
|
"value_template": "{{ value_json.state }}",
|
||||||
|
"json_attributes_topic": self.media_state_topic,
|
||||||
|
"icon": "mdi:spotify",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
self._publish_config(
|
||||||
|
"number",
|
||||||
|
"media_volume",
|
||||||
|
{
|
||||||
|
"name": "Volume",
|
||||||
|
"command_topic": volume_topic,
|
||||||
|
"state_topic": self.media_state_topic,
|
||||||
|
"value_template": "{{ value_json.volume }}",
|
||||||
|
"min": 0,
|
||||||
|
"max": 1,
|
||||||
|
"step": 0.05,
|
||||||
|
"mode": "slider",
|
||||||
|
"icon": "mdi:volume-high",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
def register_app_launchers(self, apps, on_launch) -> None:
|
||||||
|
for key, app in apps.items():
|
||||||
|
self._publish_config(
|
||||||
|
"button",
|
||||||
|
f"launch_{key}",
|
||||||
|
{
|
||||||
|
"name": f"Show {app.name}",
|
||||||
|
"command_topic": self._command_topic(
|
||||||
|
f"app/{key}/launch",
|
||||||
|
lambda _payload, key=key: on_launch(key),
|
||||||
|
),
|
||||||
|
"icon": app.icon,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
def register_workspace_select(self, workspaces, on_workspace, state_topic_value) -> None:
|
||||||
|
self._publish_config(
|
||||||
|
"select",
|
||||||
|
"workspace",
|
||||||
|
{
|
||||||
|
"name": "Screen",
|
||||||
|
"command_topic": self._command_topic("workspace/set", on_workspace),
|
||||||
|
"state_topic": f"{self.base}/workspace/state",
|
||||||
|
"options": list(workspaces),
|
||||||
|
"icon": "mdi:view-dashboard",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
self.client.publish(
|
||||||
|
f"{self.base}/workspace/state", state_topic_value, qos=1, retain=True
|
||||||
|
)
|
||||||
|
|
||||||
|
def publish_workspace(self, name: str) -> None:
|
||||||
|
self.client.publish(f"{self.base}/workspace/state", name, qos=1, retain=True)
|
||||||
|
|
@ -0,0 +1,115 @@
|
||||||
|
"""Thin wrapper around swaymsg and local process launching.
|
||||||
|
|
||||||
|
Identical in shape to hosts/thin-client/agent/thinclient_agent/sway_control.py —
|
||||||
|
duplicated rather than imported across hosts, same convention as the digest/admin
|
||||||
|
canvas SDK split in Phase 13.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import glob
|
||||||
|
import logging
|
||||||
|
import os
|
||||||
|
import subprocess
|
||||||
|
|
||||||
|
log = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
# Contract with configs/sway/config — these strings must match the `set $ws_*` lines.
|
||||||
|
WS_SPOTIFY = "1:spotify"
|
||||||
|
WS_HOME = "2:home"
|
||||||
|
WS_WEB = "3:web"
|
||||||
|
|
||||||
|
|
||||||
|
def runtime_dir() -> str:
|
||||||
|
return os.environ.get("XDG_RUNTIME_DIR") or f"/run/user/{os.getuid()}"
|
||||||
|
|
||||||
|
|
||||||
|
class SwayControl:
|
||||||
|
def _socket_path(self) -> str | None:
|
||||||
|
path = os.environ.get("SWAYSOCK")
|
||||||
|
if path and os.path.exists(path):
|
||||||
|
return path
|
||||||
|
# sway names the socket sway-ipc.<uid>.<pid>.sock, so the path changes every
|
||||||
|
# time sway restarts. touchpanel-agent is a system service that outlives the
|
||||||
|
# session, so the socket is re-resolved per call instead of cached at startup.
|
||||||
|
matches = sorted(glob.glob(os.path.join(runtime_dir(), "sway-ipc.*.sock")))
|
||||||
|
return matches[-1] if matches else None
|
||||||
|
|
||||||
|
def session_env(self) -> dict[str, str]:
|
||||||
|
env = dict(os.environ)
|
||||||
|
env["XDG_RUNTIME_DIR"] = runtime_dir()
|
||||||
|
env.setdefault("DBUS_SESSION_BUS_ADDRESS", f"unix:path={runtime_dir()}/bus")
|
||||||
|
env.setdefault("WAYLAND_DISPLAY", "wayland-1")
|
||||||
|
sock = self._socket_path()
|
||||||
|
if sock:
|
||||||
|
env["SWAYSOCK"] = sock
|
||||||
|
return env
|
||||||
|
|
||||||
|
def swaymsg(self, *args: str) -> str | None:
|
||||||
|
if self._socket_path() is None:
|
||||||
|
log.warning("no sway IPC socket found; dropping command %s", " ".join(args))
|
||||||
|
return None
|
||||||
|
try:
|
||||||
|
result = subprocess.run(
|
||||||
|
["swaymsg", *args],
|
||||||
|
env=self.session_env(),
|
||||||
|
capture_output=True,
|
||||||
|
text=True,
|
||||||
|
timeout=10,
|
||||||
|
check=False,
|
||||||
|
)
|
||||||
|
except (OSError, subprocess.TimeoutExpired) as exc:
|
||||||
|
log.warning("swaymsg %s failed: %s", " ".join(args), exc)
|
||||||
|
return None
|
||||||
|
if result.returncode != 0:
|
||||||
|
log.warning("swaymsg %s: %s", " ".join(args), result.stderr.strip())
|
||||||
|
return None
|
||||||
|
return result.stdout
|
||||||
|
|
||||||
|
def switch_workspace(self, name: str) -> None:
|
||||||
|
log.info("switching to workspace %s", name)
|
||||||
|
self.swaymsg("workspace", name)
|
||||||
|
|
||||||
|
def focus_window(self, criteria: str) -> None:
|
||||||
|
self.swaymsg(f"[{criteria}] focus")
|
||||||
|
|
||||||
|
def is_running(self, pattern: str) -> bool:
|
||||||
|
try:
|
||||||
|
result = subprocess.run(
|
||||||
|
["pgrep", "-u", str(os.getuid()), "-f", pattern],
|
||||||
|
capture_output=True,
|
||||||
|
timeout=5,
|
||||||
|
check=False,
|
||||||
|
)
|
||||||
|
except (OSError, subprocess.TimeoutExpired):
|
||||||
|
return False
|
||||||
|
return result.returncode == 0
|
||||||
|
|
||||||
|
def launch_app(
|
||||||
|
self,
|
||||||
|
command: list[str],
|
||||||
|
process_pattern: str | None = None,
|
||||||
|
workspace: str | None = None,
|
||||||
|
focus_criteria: str | None = None,
|
||||||
|
) -> None:
|
||||||
|
if workspace:
|
||||||
|
self.switch_workspace(workspace)
|
||||||
|
|
||||||
|
if process_pattern and self.is_running(process_pattern):
|
||||||
|
log.info("%s already running; focusing instead of launching", command[0])
|
||||||
|
if focus_criteria:
|
||||||
|
self.focus_window(focus_criteria)
|
||||||
|
return
|
||||||
|
|
||||||
|
log.info("launching %s", " ".join(command))
|
||||||
|
try:
|
||||||
|
subprocess.Popen(
|
||||||
|
command,
|
||||||
|
env=self.session_env(),
|
||||||
|
stdin=subprocess.DEVNULL,
|
||||||
|
stdout=subprocess.DEVNULL,
|
||||||
|
stderr=subprocess.DEVNULL,
|
||||||
|
start_new_session=True,
|
||||||
|
)
|
||||||
|
except OSError as exc:
|
||||||
|
log.error("could not launch %s: %s", " ".join(command), exc)
|
||||||
|
|
@ -0,0 +1,50 @@
|
||||||
|
// Styling for the touch dock. Installed to
|
||||||
|
// /home/<KIOSK_USERNAME>/.config/eww/eww.scss by build-touch-panel-iso.sh.
|
||||||
|
//
|
||||||
|
// Touch targets are deliberately large (min ~72px tap area) — this is a finger
|
||||||
|
// interface, not a mouse one.
|
||||||
|
|
||||||
|
$bg: rgba(16, 16, 20, 0.96);
|
||||||
|
$fg: #e8e8ec;
|
||||||
|
$muted: #9a9aa6;
|
||||||
|
$accent: #6ea8fe;
|
||||||
|
|
||||||
|
.dock-root {
|
||||||
|
background-color: $bg;
|
||||||
|
padding: 6px 12px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.dock-btn {
|
||||||
|
min-width: 84px;
|
||||||
|
min-height: 68px;
|
||||||
|
background-color: transparent;
|
||||||
|
border: none;
|
||||||
|
border-radius: 12px;
|
||||||
|
padding: 4px 10px;
|
||||||
|
color: $muted;
|
||||||
|
|
||||||
|
&:active {
|
||||||
|
background-color: rgba(255, 255, 255, 0.14);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
.dock-btn-active {
|
||||||
|
background-color: rgba(110, 168, 254, 0.16);
|
||||||
|
color: $fg;
|
||||||
|
}
|
||||||
|
|
||||||
|
.dock-icon {
|
||||||
|
font-size: 26px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.dock-label {
|
||||||
|
font-size: 13px;
|
||||||
|
font-weight: 500;
|
||||||
|
color: inherit;
|
||||||
|
}
|
||||||
|
|
||||||
|
.dock-sep {
|
||||||
|
min-width: 1px;
|
||||||
|
background-color: rgba(255, 255, 255, 0.12);
|
||||||
|
margin: 8px 4px;
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,44 @@
|
||||||
|
;; The touch panel's on-screen dock — the primary local navigation surface (this
|
||||||
|
;; device is touched directly, unlike hosts/thin-client, which deliberately has no
|
||||||
|
;; bars because its primary control is HA/MQTT and wayvnc). Installed to
|
||||||
|
;; /home/<KIOSK_USERNAME>/.config/eww/eww.yuck by build-touch-panel-iso.sh.
|
||||||
|
;;
|
||||||
|
;; Session-scoped, not part of touchpanel-agent, same reasoning as the thin client's
|
||||||
|
;; now-playing widget: this is compositor UI, not device control, and dies with sway
|
||||||
|
;; rather than needing a restart-independent lifecycle. The :onclick strings below are
|
||||||
|
;; fixed constants in this file — nothing from MQTT, HA, or player metadata is ever
|
||||||
|
;; interpolated into them.
|
||||||
|
|
||||||
|
(defpoll active_workspace :interval "1s" :initial "2:home"
|
||||||
|
"swaymsg -t get_workspaces | jq -r '.[] | select(.focused) | .name' 2>/dev/null || echo 2:home")
|
||||||
|
|
||||||
|
(defwidget dock-btn [ws icon label]
|
||||||
|
(button :class {active_workspace == ws ? "dock-btn dock-btn-active" : "dock-btn"}
|
||||||
|
:onclick "swaymsg workspace ${ws}"
|
||||||
|
(box :orientation "vertical" :space-evenly false :spacing 2
|
||||||
|
(label :class "dock-icon" :text icon)
|
||||||
|
(label :class "dock-label" :text label))))
|
||||||
|
|
||||||
|
(defwidget dock-root []
|
||||||
|
(box :class "dock-root" :orientation "horizontal" :space-evenly false :spacing 8
|
||||||
|
:halign "center" :valign "center"
|
||||||
|
(dock-btn :ws "1:spotify" :icon "🎵" :label "Spotify")
|
||||||
|
(dock-btn :ws "2:home" :icon "🏠" :label "Home")
|
||||||
|
(dock-btn :ws "3:web" :icon "🌐" :label "Web")
|
||||||
|
(box :class "dock-sep")
|
||||||
|
(button :class "dock-btn" :onclick "toggle-keyboard"
|
||||||
|
(box :orientation "vertical" :space-evenly false :spacing 2
|
||||||
|
(label :class "dock-icon" :text "⌨")
|
||||||
|
(label :class "dock-label" :text "Keyboard")))))
|
||||||
|
|
||||||
|
;; :exclusive true reserves this strip so it is never overlapped by, and never steals
|
||||||
|
;; area from behind, the app on screen — unlike the thin client's overlay widgets,
|
||||||
|
;; which deliberately float over fullscreen video. A dock a finger can miss because
|
||||||
|
;; something else drew over it is worse than a smaller app area.
|
||||||
|
(defwindow touch-dock
|
||||||
|
:monitor 0
|
||||||
|
:geometry (geometry :width "100%" :height "84px" :anchor "bottom center")
|
||||||
|
:stacking "overlay"
|
||||||
|
:exclusive true
|
||||||
|
:focusable false
|
||||||
|
(dock-root))
|
||||||
|
|
@ -0,0 +1,63 @@
|
||||||
|
{
|
||||||
|
"_comment_path": "Installed to /etc/firefox/policies/policies.json, the documented Linux location for Firefox enterprise policy. 0600-firefox.hook.chroot also links it into firefox-esr's distribution/ directory, an older location some builds read instead; whichever the installed build honours wins and the other is ignored. Every key below lives OUTSIDE the 'policies' object on purpose — Firefox flags unrecognised keys *inside* it as invalid policies in about:policies.",
|
||||||
|
|
||||||
|
"_comment_extensions": "Same two extensions and the same unverified-ID caveat as hosts/thin-client/configs/firefox/policies.json — see that file's comment and the touch-panel README's verification list. Duplicated, not shared, per this project's established cross-host convention (Phase 13's admin-canvas/digest-canvas SDK split).",
|
||||||
|
|
||||||
|
"_comment_kiosk_ui": "The Disable*/UserMessaging/FirefoxHome blocks all remove a prompt, tour, or nag that would otherwise sit on an unattended panel with nobody there to dismiss it.",
|
||||||
|
|
||||||
|
"_comment_updates": "The browser is part of the image and is replaced by rebuilding and reflashing it, so in-browser updates would only produce version drift between panels plus a restart banner nobody is there to click.",
|
||||||
|
|
||||||
|
"policies": {
|
||||||
|
"ExtensionSettings": {
|
||||||
|
"uBlock0@raymondhill.net": {
|
||||||
|
"installation_mode": "force_installed",
|
||||||
|
"install_url": "https://addons.mozilla.org/firefox/downloads/latest/ublock-origin/latest.xpi",
|
||||||
|
"default_area": "menupanel"
|
||||||
|
},
|
||||||
|
"sponsorBlocker@ajay.app": {
|
||||||
|
"installation_mode": "force_installed",
|
||||||
|
"install_url": "https://addons.mozilla.org/firefox/downloads/latest/sponsorblock/latest.xpi",
|
||||||
|
"default_area": "menupanel"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
|
||||||
|
"DisableProfileImport": true,
|
||||||
|
"DisableProfileRefresh": true,
|
||||||
|
"DisableFirefoxAccounts": true,
|
||||||
|
"DisableFirefoxStudies": true,
|
||||||
|
"DisableTelemetry": true,
|
||||||
|
"DisablePocket": true,
|
||||||
|
"DisableFeedbackCommands": true,
|
||||||
|
"DisableSetDesktopBackground": true,
|
||||||
|
"DontCheckDefaultBrowser": true,
|
||||||
|
"NoDefaultBookmarks": true,
|
||||||
|
"OfferToSaveLogins": false,
|
||||||
|
"PasswordManagerEnabled": false,
|
||||||
|
"PromptForDownloadLocation": false,
|
||||||
|
|
||||||
|
"AppAutoUpdate": false,
|
||||||
|
"DisableAppUpdate": true,
|
||||||
|
|
||||||
|
"UserMessaging": {
|
||||||
|
"WhatsNew": false,
|
||||||
|
"ExtensionRecommendations": false,
|
||||||
|
"FeatureRecommendations": false,
|
||||||
|
"UrlbarInterventions": false,
|
||||||
|
"SkipOnboarding": true,
|
||||||
|
"MoreFromMozilla": false
|
||||||
|
},
|
||||||
|
|
||||||
|
"FirefoxHome": {
|
||||||
|
"Search": true,
|
||||||
|
"TopSites": true,
|
||||||
|
"SponsoredTopSites": false,
|
||||||
|
"Highlights": false,
|
||||||
|
"Pocket": false,
|
||||||
|
"SponsoredPocket": false,
|
||||||
|
"Snippets": false
|
||||||
|
},
|
||||||
|
|
||||||
|
"OverrideFirstRunPage": "",
|
||||||
|
"OverridePostUpdatePage": ""
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,45 @@
|
||||||
|
// Prefs for the touch panel's general-browsing Firefox profile. Installed by
|
||||||
|
// build-touch-panel-iso.sh into /home/<KIOSK_USERNAME>/.mozilla/firefox/web.
|
||||||
|
//
|
||||||
|
// Same shape as hosts/thin-client/configs/firefox/user.js — see that file's comment
|
||||||
|
// for why user.js (re-applied on every startup) rather than a one-shot prefs.js seed,
|
||||||
|
// and why anything expressible as enterprise policy lives in policies.json instead.
|
||||||
|
// Duplicated rather than shared between the two hosts on purpose — same call as
|
||||||
|
// admin-canvas/digest-canvas's SDK split in Phase 13: this profile is free to drift
|
||||||
|
// (touch-target padding, no ydotool-specific concerns) without touching the thin
|
||||||
|
// client's.
|
||||||
|
|
||||||
|
user_pref("toolkit.legacyUserProfileCustomizations.stylesheets", true);
|
||||||
|
|
||||||
|
// First-run / post-update interstitials. An unattended panel has nobody to close them.
|
||||||
|
user_pref("browser.startup.homepage_override.mstone", "ignore");
|
||||||
|
user_pref("browser.aboutwelcome.enabled", false);
|
||||||
|
user_pref("browser.messaging-system.whatsNewPanel.enabled", false);
|
||||||
|
user_pref("trailhead.firstrun.didSeeAboutWelcome", true);
|
||||||
|
user_pref("datareporting.policy.firstRunURL", "");
|
||||||
|
user_pref("datareporting.policy.dataSubmissionPolicyBypassNotification", true);
|
||||||
|
|
||||||
|
// Session restore prompts after a power cut would leave the panel stuck on a
|
||||||
|
// "restore your session?" page nobody is there to click through.
|
||||||
|
user_pref("browser.sessionstore.resume_from_crash", false);
|
||||||
|
user_pref("browser.tabs.warnOnClose", false);
|
||||||
|
user_pref("browser.tabs.warnOnCloseOtherTabs", false);
|
||||||
|
user_pref("browser.warnOnQuit", false);
|
||||||
|
|
||||||
|
// No permission doorhangers on a shared touch panel. Autoplay is left blocked
|
||||||
|
// (unlike the thin client's digest canvas) — general browsing has no reason to want
|
||||||
|
// arbitrary pages making sound on a household wall panel.
|
||||||
|
user_pref("permissions.default.desktop-notification", 2);
|
||||||
|
user_pref("permissions.default.geo", 2);
|
||||||
|
|
||||||
|
user_pref("gfx.webrender.all", true);
|
||||||
|
|
||||||
|
// Touch-scrolling tuning: Firefox's Wayland/GTK touch handling is on by default, but
|
||||||
|
// the overlay/kinetic scrollbar reads better at panel viewing distance than the
|
||||||
|
// default desktop one.
|
||||||
|
user_pref("widget.gtk.overlay-scrollbars.enabled", true);
|
||||||
|
|
||||||
|
user_pref("browser.urlbar.suggest.topsites", false);
|
||||||
|
user_pref("browser.urlbar.suggest.quicksuggest.sponsored", false);
|
||||||
|
user_pref("browser.newtabpage.activity-stream.showSponsored", false);
|
||||||
|
user_pref("browser.newtabpage.activity-stream.showSponsoredTopSites", false);
|
||||||
|
|
@ -0,0 +1,64 @@
|
||||||
|
/* Minimal browser chrome for the touch panel's general-browsing Firefox window.
|
||||||
|
* Installed to <profile>/chrome/userChrome.css by build-touch-panel-iso.sh, and only
|
||||||
|
* read because user.js sets toolkit.legacyUserProfileCustomizations.stylesheets=true.
|
||||||
|
*
|
||||||
|
* What survives: back, forward, reload, and the address bar — the four things a
|
||||||
|
* finger actually needs to drive this window, same reasoning as
|
||||||
|
* hosts/thin-client/configs/firefox/userChrome.css (adapted here for a touchscreen
|
||||||
|
* instead of ydotool/wayvnc remote input). Every extra toolbar button is one more
|
||||||
|
* small target to mis-tap.
|
||||||
|
*/
|
||||||
|
|
||||||
|
/* Tab strip. One page at a time; a tab bar on a shared panel just accumulates
|
||||||
|
* whatever the last person left open. */
|
||||||
|
#TabsToolbar,
|
||||||
|
#tabbrowser-tabs,
|
||||||
|
#alltabs-button {
|
||||||
|
visibility: collapse !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Menu bar and bookmarks bar. */
|
||||||
|
#toolbar-menubar,
|
||||||
|
#PersonalToolbar,
|
||||||
|
#PlacesToolbar {
|
||||||
|
visibility: collapse !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Everything on the nav bar except back / forward / reload / the address bar. */
|
||||||
|
#home-button,
|
||||||
|
#library-button,
|
||||||
|
#sidebar-button,
|
||||||
|
#fxa-toolbar-menu-button,
|
||||||
|
#unified-extensions-button,
|
||||||
|
#PanelUI-button,
|
||||||
|
#downloads-button,
|
||||||
|
#save-to-pocket-button,
|
||||||
|
#pageActionButton,
|
||||||
|
#star-button-box,
|
||||||
|
#reader-mode-button,
|
||||||
|
#customizableui-special-spring1,
|
||||||
|
#customizableui-special-spring2 {
|
||||||
|
display: none !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
#urlbar-container {
|
||||||
|
min-width: 0 !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
#nav-bar {
|
||||||
|
border: none !important;
|
||||||
|
box-shadow: none !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
.titlebar-buttonbox-container,
|
||||||
|
.titlebar-spacer {
|
||||||
|
display: none !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Touch targets: the remaining nav-bar buttons default to a mouse-sized hit area.
|
||||||
|
* Bumping padding here is cheaper than a full touch-optimised theme. */
|
||||||
|
#nav-bar toolbarbutton {
|
||||||
|
padding: 8px !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Findbar and notification popups stay: they are transient and user-initiated. */
|
||||||
|
|
@ -0,0 +1,34 @@
|
||||||
|
#!/bin/sh
|
||||||
|
# Opens the general-browsing Firefox window — minimal chrome, not kiosk mode.
|
||||||
|
# Installed to /usr/local/bin/web-browser. Its own profile, separate from any other
|
||||||
|
# Firefox instance on this machine (there is only this one on the touch panel, but the
|
||||||
|
# profile is still named explicitly for the same reason hosts/thin-client keeps three
|
||||||
|
# separate ones: a Firefox profile can only be open in one process).
|
||||||
|
set -eu
|
||||||
|
|
||||||
|
PROFILE_DIR="${HOME:-/home/$(id -un)}/.mozilla/firefox/web"
|
||||||
|
|
||||||
|
if command -v firefox-esr >/dev/null 2>&1; then
|
||||||
|
FIREFOX=firefox-esr
|
||||||
|
else
|
||||||
|
FIREFOX=firefox
|
||||||
|
fi
|
||||||
|
|
||||||
|
mkdir -p "$PROFILE_DIR/chrome"
|
||||||
|
|
||||||
|
# Re-copied on every launch rather than once at build time: this is a locked-down
|
||||||
|
# profile with no interactive customisation expected, so keeping it in lockstep with
|
||||||
|
# /etc/touchpanel-firefox/ (edited by rebuilding the image) is simpler than a one-shot
|
||||||
|
# seed that could drift after a userChrome.css update.
|
||||||
|
cp /etc/touchpanel-firefox/userChrome.css "$PROFILE_DIR/chrome/userChrome.css" 2>/dev/null || true
|
||||||
|
cp /etc/touchpanel-firefox/user.js "$PROFILE_DIR/user.js" 2>/dev/null || true
|
||||||
|
|
||||||
|
# Focus an already-open window rather than stacking a second one: this window holds
|
||||||
|
# state (history, a half-typed URL, a logged-in page) that a kill-and-relaunch would
|
||||||
|
# throw away.
|
||||||
|
if pgrep -u "$(id -u)" -f "$FIREFOX .*--profile $PROFILE_DIR" >/dev/null 2>&1; then
|
||||||
|
[ -n "${1:-}" ] && exec "$FIREFOX" --profile "$PROFILE_DIR" --new-tab "$1"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
exec "$FIREFOX" --profile "$PROFILE_DIR" --new-instance --new-window "${1:-about:blank}"
|
||||||
|
|
@ -0,0 +1,14 @@
|
||||||
|
# greetd — autologin straight into the kiosk Sway session, no greeter UI.
|
||||||
|
# Installed to /etc/greetd/config.toml by build-touch-panel-iso.sh, which substitutes
|
||||||
|
# @KIOSK_USERNAME@ with its own KIOSK_USERNAME variable on the way in.
|
||||||
|
#
|
||||||
|
# Same shape as hosts/thin-client/configs/greetd/config.toml — see that file's comment
|
||||||
|
# for why [default_session] (not [initial_session]) is what makes autologin permanent
|
||||||
|
# and self-healing after a crash or a `swaymsg exit`.
|
||||||
|
|
||||||
|
[terminal]
|
||||||
|
vt = 1
|
||||||
|
|
||||||
|
[default_session]
|
||||||
|
command = "/usr/local/bin/kiosk-session"
|
||||||
|
user = "@KIOSK_USERNAME@"
|
||||||
|
|
@ -0,0 +1,25 @@
|
||||||
|
#!/bin/sh
|
||||||
|
# greetd's default_session command. Installed to /usr/local/bin/kiosk-session.
|
||||||
|
#
|
||||||
|
# Same purpose as the thin client's own kiosk-session: gets MQTT_BROKER_HOST / HA_URL /
|
||||||
|
# SPOTIFY_APP_ID / etc into Sway's environment, since sway's config file has no way to
|
||||||
|
# read an env file itself but every `exec` line it runs inherits this process's
|
||||||
|
# environment.
|
||||||
|
set -eu
|
||||||
|
|
||||||
|
[ -r /etc/touchpanel-agent/config.env ] && . /etc/touchpanel-agent/config.env
|
||||||
|
export MQTT_BROKER_HOST HA_URL
|
||||||
|
|
||||||
|
export XDG_CURRENT_DESKTOP=sway
|
||||||
|
export XDG_SESSION_TYPE=wayland
|
||||||
|
export XDG_SESSION_DESKTOP=sway
|
||||||
|
export MOZ_ENABLE_WAYLAND=1
|
||||||
|
# Chromium's Wayland/Ozone backend, needed for native touch events in the HA kiosk
|
||||||
|
# window (see configs/sway/ha-kiosk) — without it Chromium falls back to Xwayland,
|
||||||
|
# where touch input is translated through XTest and loses multi-touch gestures.
|
||||||
|
export OZONE_PLATFORM=wayland
|
||||||
|
|
||||||
|
: "${XDG_RUNTIME_DIR:=/run/user/$(id -u)}"
|
||||||
|
export XDG_RUNTIME_DIR
|
||||||
|
|
||||||
|
exec sway
|
||||||
|
|
@ -0,0 +1,25 @@
|
||||||
|
#!/bin/sh
|
||||||
|
# Toggles the on-screen keyboard. Installed to /usr/local/bin/toggle-keyboard, run
|
||||||
|
# from the dock's Keyboard button (configs/eww/eww.yuck) and bindable to any other
|
||||||
|
# tap target later.
|
||||||
|
#
|
||||||
|
# wvkbd, not squeekboard: squeekboard's auto-show behaviour is wired through Phosh's
|
||||||
|
# session/DBus setup, which this image doesn't run, so under plain Sway it would need
|
||||||
|
# the same manual-toggle treatment anyway while pulling in more of the GNOME-mobile
|
||||||
|
# stack. wvkbd is a small layer-shell surface built for exactly this — a wlroots
|
||||||
|
# kiosk with no shell-level input-method integration.
|
||||||
|
#
|
||||||
|
# KNOWN LIMITATION, documented rather than hidden: there is no automatic show-on-
|
||||||
|
# text-field-focus here (that needs the text-input-v3/virtual-keyboard-v1 protocols
|
||||||
|
# wired all the way through each app, which is inconsistent across Chromium/Firefox/
|
||||||
|
# Spotify under plain wlroots). The dock's Keyboard button is therefore the only way
|
||||||
|
# to open it — tap before typing, tap again to dismiss.
|
||||||
|
set -eu
|
||||||
|
|
||||||
|
if pgrep -u "$(id -u)" -x wvkbd-mobintl >/dev/null 2>&1; then
|
||||||
|
pkill -u "$(id -u)" -x wvkbd-mobintl
|
||||||
|
else
|
||||||
|
# -L: keyboard layer height in px. 280 is a starting point for a touch-panel-sized
|
||||||
|
# screen, not measured against real hardware — see the README's verification list.
|
||||||
|
wvkbd-mobintl -L 280 &
|
||||||
|
fi
|
||||||
|
|
@ -0,0 +1,20 @@
|
||||||
|
#!/bin/sh
|
||||||
|
# Full Spotify GUI client (docs/project-plan.md Phase 16). Installed to
|
||||||
|
# /usr/local/bin/spotify-launch.
|
||||||
|
#
|
||||||
|
# Deliberately NOT spotifyd/librespot — those are the thin client's and audio-endpoint's
|
||||||
|
# headless Spotify Connect *receivers*, with no GUI at all. This device is touched
|
||||||
|
# directly, so it runs the real, official Spotify Linux client: its own library
|
||||||
|
# browser, search, and playlists. Spotify ships no apt package for bookworm, so this
|
||||||
|
# comes from Flathub (0300-flatpak.hook.chroot installs it) — same distribution
|
||||||
|
# channel already used for the thin client's Steam Link, and the only maintained one
|
||||||
|
# for Spotify on Linux.
|
||||||
|
#
|
||||||
|
# Login is interactive, on-device, on first launch (Spotify account/Premium required
|
||||||
|
# for playback — Premium is also what the thin client's headless receivers need). The
|
||||||
|
# Flatpak's own persistent data dir (~/.var/app/com.spotify.Client) keeps that session
|
||||||
|
# across restarts; nothing here wipes or reseeds it the way the kiosk Firefox profiles
|
||||||
|
# are re-copied on every launch, since staying logged in is the whole point here.
|
||||||
|
set -eu
|
||||||
|
|
||||||
|
exec flatpak run com.spotify.Client "$@"
|
||||||
|
|
@ -0,0 +1,126 @@
|
||||||
|
# Sway kiosk session for the touch panel (docs/project-plan.md Phase 16).
|
||||||
|
#
|
||||||
|
# Installed to /home/<KIOSK_USERNAME>/.config/sway/config by build-touch-panel-iso.sh.
|
||||||
|
# Every `exec` below inherits the environment set by /usr/local/bin/kiosk-session,
|
||||||
|
# which sources /etc/touchpanel-agent/config.env — that is how $HA_URL gets here
|
||||||
|
# without this file being templated.
|
||||||
|
#
|
||||||
|
# This is a DIFFERENT device from hosts/thin-client: that one is a couch-distance
|
||||||
|
# media station whose primary control surface is HA/MQTT and wayvnc, deliberately with
|
||||||
|
# "no bars" (see its config's own comment). This one is touched directly, so the whole
|
||||||
|
# point is an always-visible on-screen dock a finger can hit — see the eww bar below.
|
||||||
|
|
||||||
|
set $mod Mod4
|
||||||
|
set $ws_spotify 1:spotify
|
||||||
|
set $ws_home 2:home
|
||||||
|
set $ws_web 3:web
|
||||||
|
|
||||||
|
# Workspace names are a contract with touchpanel_agent/sway_control.py — changing one
|
||||||
|
# side means changing the other.
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Output / input
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
output * bg #101014 solid_color
|
||||||
|
|
||||||
|
input type:keyboard {
|
||||||
|
xkb_layout @KEYBOARD_LAYOUT@
|
||||||
|
}
|
||||||
|
|
||||||
|
# Real touchscreen hardware, not the thin client's couch-distance camera-gesture
|
||||||
|
# trick: libinput/Wayland surfaces (GTK, Chromium, Firefox) all speak wl_touch
|
||||||
|
# natively, so taps/scrolls/pinches just work without a pointer-emulation layer.
|
||||||
|
input type:touch {
|
||||||
|
map_to_output "*"
|
||||||
|
}
|
||||||
|
|
||||||
|
input type:tablet_tool {
|
||||||
|
map_to_output "*"
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Look
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
default_border none
|
||||||
|
default_floating_border none
|
||||||
|
hide_edge_borders both
|
||||||
|
gaps inner 0
|
||||||
|
gaps outer 0
|
||||||
|
|
||||||
|
# Reserve a strip at the bottom for the touch dock (bar/app-switcher), so it is
|
||||||
|
# never covered by, and never steals area from behind, whatever app is on screen.
|
||||||
|
# :exclusive is the eww window's own setting (configs/eww/eww.yuck) — sway just
|
||||||
|
# needs to know nothing else should tile into that space, which layer-shell
|
||||||
|
# exclusivity already guarantees without a `bar {}` block here.
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Session-scoped helpers
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# touchpanel-agent is NOT started here — see thin-client's identical note. systemd
|
||||||
|
# owns it so it is reachable over MQTT whether or not a graphical session is up, and
|
||||||
|
# survives a sway restart. Starting it from sway too would give two competing MQTT
|
||||||
|
# clients.
|
||||||
|
|
||||||
|
# The touch dock (eww bar) and the on-screen keyboard toggle both live for the
|
||||||
|
# session, not the system, for the same reason the thin client's now-playing widget
|
||||||
|
# does: they are compositor UI, not device control, and should die with sway rather
|
||||||
|
# than need their own restart-independent lifecycle.
|
||||||
|
exec_always eww daemon
|
||||||
|
exec_always eww open touch-dock
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Applications
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# No `assign` rules, same reasoning as the thin client: every launch path is explicit
|
||||||
|
# about its own workspace via touchpanel_agent.sway_control.launch_app(), so a
|
||||||
|
# separately-matched `assign` rule can't race it.
|
||||||
|
for_window [app_id="chromium.*"] fullscreen enable
|
||||||
|
for_window [app_id="spotify"] fullscreen enable
|
||||||
|
|
||||||
|
# HA kiosk window and Spotify both auto-launch at session start — this is a fixed
|
||||||
|
# 3-app panel, not an on-demand surface like the thin client's digest/admin
|
||||||
|
# workspaces, so there is no "nothing to show yet" state to guard against beyond an
|
||||||
|
# unset HA_URL.
|
||||||
|
exec sh -c '[ -n "$HA_URL" ] && { swaymsg workspace $ws_home; /usr/local/bin/ha-kiosk; }'
|
||||||
|
exec sh -c 'swaymsg workspace $ws_spotify; /usr/local/bin/spotify-launch'
|
||||||
|
exec swaymsg workspace $ws_home
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Idle
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Blank after 10 minutes to save the panel; any touch (or local input) resumes it.
|
||||||
|
# No lock screen, same reasoning as the thin client: a lock screen would make a wall
|
||||||
|
# panel unusable to anyone in the room who isn't holding a passcode.
|
||||||
|
exec swayidle -w \
|
||||||
|
timeout 600 'swaymsg "output * power off"' \
|
||||||
|
resume 'swaymsg "output * power on"'
|
||||||
|
|
||||||
|
for_window [app_id="chromium.*"] inhibit_idle fullscreen
|
||||||
|
for_window [app_id="spotify"] inhibit_idle fullscreen
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Local override keys — a fallback for standing in front of the machine (or a
|
||||||
|
# connected keyboard), not the primary control surface, which is touch + HA/MQTT.
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
bindsym $mod+Shift+Ctrl+m exec foot --title maintenance-shell
|
||||||
|
for_window [title="maintenance-shell"] floating enable, resize set width 800 height 500, move position center
|
||||||
|
|
||||||
|
bindsym $mod+Return exec foot
|
||||||
|
bindsym $mod+q kill
|
||||||
|
bindsym $mod+f fullscreen toggle
|
||||||
|
bindsym $mod+Shift+c reload
|
||||||
|
bindsym $mod+1 workspace $ws_spotify
|
||||||
|
bindsym $mod+2 workspace $ws_home
|
||||||
|
bindsym $mod+3 workspace $ws_web
|
||||||
|
|
||||||
|
bindsym XF86AudioPlay exec playerctl -p spotify play-pause
|
||||||
|
bindsym XF86AudioNext exec playerctl -p spotify next
|
||||||
|
bindsym XF86AudioPrev exec playerctl -p spotify previous
|
||||||
|
bindsym XF86AudioRaiseVolume exec wpctl set-volume @DEFAULT_AUDIO_SINK@ 5%+
|
||||||
|
bindsym XF86AudioLowerVolume exec wpctl set-volume @DEFAULT_AUDIO_SINK@ 5%-
|
||||||
|
bindsym XF86AudioMute exec wpctl set-mute @DEFAULT_AUDIO_SINK@ toggle
|
||||||
|
|
||||||
|
# Deliberately no exit binding — see the thin client's identical note: `swaymsg exit`
|
||||||
|
# would just drop to a black VT and greetd would autologin straight back in.
|
||||||
|
|
||||||
|
workspace $ws_home
|
||||||
|
|
@ -0,0 +1,47 @@
|
||||||
|
#!/bin/sh
|
||||||
|
# Persistent Chromium kiosk window pointed at Home Assistant. Installed to
|
||||||
|
# /usr/local/bin/ha-kiosk. Launched once at session start (configs/sway/config) and
|
||||||
|
# supervised right here with a restart loop, not by sway (its `exec` has no
|
||||||
|
# restart-on-crash) or systemd (this needs the graphical session's env, and a system
|
||||||
|
# unit chasing a per-session Wayland socket is more trouble than a two-line loop).
|
||||||
|
#
|
||||||
|
# Unlike hosts/thin-client's digest-browser, this is never killed-and-relaunched on
|
||||||
|
# an MQTT command — touchpanel_agent.sway_control.launch_app()'s is_running/
|
||||||
|
# focus_criteria path just switches workspace and focuses the existing window, since
|
||||||
|
# the HA dashboard is stateful (login session, scroll position, which view you left
|
||||||
|
# it on) and reloading it on every "show home" tap would throw that away for nothing.
|
||||||
|
set -eu
|
||||||
|
|
||||||
|
PROFILE_DIR="${HOME:-/home/$(id -un)}/.config/touchpanel-chromium-ha"
|
||||||
|
URL="${HA_URL:-}"
|
||||||
|
[ -n "$URL" ] || { echo "ha-kiosk: HA_URL is unset" >&2; exit 1; }
|
||||||
|
|
||||||
|
if command -v chromium >/dev/null 2>&1; then
|
||||||
|
CHROMIUM=chromium
|
||||||
|
elif command -v chromium-browser >/dev/null 2>&1; then
|
||||||
|
CHROMIUM=chromium-browser
|
||||||
|
else
|
||||||
|
echo "ha-kiosk: no chromium/chromium-browser binary found" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
mkdir -p "$PROFILE_DIR"
|
||||||
|
|
||||||
|
# VERIFY: Chromium's Wayland app_id for a --app= kiosk window is assumed to start
|
||||||
|
# with "chromium" (matched by configs/sway/config's [app_id="chromium.*"] and by
|
||||||
|
# touchpanel_agent's focus_criteria) — not confirmed against a real build. If the
|
||||||
|
# fullscreen/for_window rule or the "Home" dock button's focus-instead-of-relaunch
|
||||||
|
# stops working, check the real app_id with `swaymsg -t get_tree`.
|
||||||
|
while true; do
|
||||||
|
"$CHROMIUM" \
|
||||||
|
--user-data-dir="$PROFILE_DIR" \
|
||||||
|
--ozone-platform=wayland \
|
||||||
|
--kiosk --app="$URL" \
|
||||||
|
--start-fullscreen \
|
||||||
|
--noerrdialogs --disable-infobars --disable-session-crashed-bubble \
|
||||||
|
--overscroll-history-navigation=0 \
|
||||||
|
--touch-events=enabled \
|
||||||
|
--check-for-update-interval=31536000 \
|
||||||
|
>/tmp/ha-kiosk.log 2>&1 || true
|
||||||
|
sleep 2
|
||||||
|
done
|
||||||
|
|
@ -0,0 +1,50 @@
|
||||||
|
#!/bin/sh
|
||||||
|
# Creates the kiosk account the whole image is built around. Identical logic to
|
||||||
|
# hosts/thin-client's 0100-user-setup.hook.chroot — see that file's comments for the
|
||||||
|
# reasoning behind the locked password, passwordless sudo, and key-only SSH.
|
||||||
|
set -eu
|
||||||
|
|
||||||
|
. /etc/touchpanel-agent/config.env
|
||||||
|
|
||||||
|
if ! id "$KIOSK_USERNAME" >/dev/null 2>&1; then
|
||||||
|
useradd --create-home --shell /bin/bash --comment "Touch panel kiosk session" "$KIOSK_USERNAME"
|
||||||
|
fi
|
||||||
|
|
||||||
|
for grp in audio video input render dialout netdev plugdev seat _seatd; do
|
||||||
|
if getent group "$grp" >/dev/null 2>&1; then
|
||||||
|
adduser "$KIOSK_USERNAME" "$grp" >/dev/null
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
# No password is baked in: the account is locked so it can never be used to log in
|
||||||
|
# remotely, while the physical console still autologins via greetd.
|
||||||
|
passwd --lock "$KIOSK_USERNAME" >/dev/null
|
||||||
|
|
||||||
|
adduser "$KIOSK_USERNAME" sudo >/dev/null
|
||||||
|
|
||||||
|
# Passwordless sudo: this image autologins to an unattended interactive Sway session
|
||||||
|
# at the physical console, so anyone standing in front of the panel already has the
|
||||||
|
# equivalent of a root shell. The boundary that actually matters is key-only SSH below
|
||||||
|
# — there is no wayvnc on this image (see the README for that scope decision).
|
||||||
|
cat > "/etc/sudoers.d/010-${KIOSK_USERNAME}" <<EOF
|
||||||
|
${KIOSK_USERNAME} ALL=(ALL) NOPASSWD: ALL
|
||||||
|
EOF
|
||||||
|
chmod 0440 "/etc/sudoers.d/010-${KIOSK_USERNAME}"
|
||||||
|
|
||||||
|
mkdir -p /etc/ssh/sshd_config.d
|
||||||
|
cat > /etc/ssh/sshd_config.d/10-touch-panel.conf <<'EOF'
|
||||||
|
PermitRootLogin no
|
||||||
|
PasswordAuthentication no
|
||||||
|
KbdInteractiveAuthentication no
|
||||||
|
PubkeyAuthentication yes
|
||||||
|
EOF
|
||||||
|
|
||||||
|
if [ -d "/home/${KIOSK_USERNAME}/.ssh" ]; then
|
||||||
|
chmod 700 "/home/${KIOSK_USERNAME}/.ssh"
|
||||||
|
[ -f "/home/${KIOSK_USERNAME}/.ssh/authorized_keys" ] && \
|
||||||
|
chmod 600 "/home/${KIOSK_USERNAME}/.ssh/authorized_keys"
|
||||||
|
fi
|
||||||
|
|
||||||
|
chown -R "${KIOSK_USERNAME}:${KIOSK_USERNAME}" "/home/${KIOSK_USERNAME}"
|
||||||
|
|
||||||
|
systemctl enable ssh >/dev/null 2>&1 || true
|
||||||
|
|
@ -0,0 +1,23 @@
|
||||||
|
#!/bin/sh
|
||||||
|
# Makes greetd the boot target so the machine comes up straight in the kiosk Sway
|
||||||
|
# session. Identical logic to hosts/thin-client's 0200-greetd.hook.chroot.
|
||||||
|
set -eu
|
||||||
|
|
||||||
|
. /etc/touchpanel-agent/config.env
|
||||||
|
|
||||||
|
chmod 0755 /usr/local/bin/kiosk-session
|
||||||
|
|
||||||
|
if ! id greeter >/dev/null 2>&1; then
|
||||||
|
useradd --system --create-home --home-dir /var/lib/greetd --shell /usr/sbin/nologin greeter
|
||||||
|
fi
|
||||||
|
|
||||||
|
systemctl enable greetd
|
||||||
|
systemctl set-default graphical.target
|
||||||
|
|
||||||
|
# live-config would otherwise autologin its own account on tty1 and fight greetd for
|
||||||
|
# the VT. build-touch-panel-iso.sh passes `noautologin` on the kernel command line;
|
||||||
|
# masking the getty on vt1 makes that robust even if someone edits the boot args.
|
||||||
|
systemctl mask getty@tty1.service
|
||||||
|
|
||||||
|
mkdir -p "/home/${KIOSK_USERNAME}/.config/sway"
|
||||||
|
chown -R "${KIOSK_USERNAME}:${KIOSK_USERNAME}" "/home/${KIOSK_USERNAME}/.config"
|
||||||
|
|
@ -0,0 +1,24 @@
|
||||||
|
#!/bin/sh
|
||||||
|
# Flathub remote + the full Spotify GUI client (docs/project-plan.md Phase 16).
|
||||||
|
#
|
||||||
|
# Spotify ships no apt package for bookworm, so this comes from Flathub — the same
|
||||||
|
# distribution channel already used for hosts/thin-client's Steam Link. No release
|
||||||
|
# version is pinned, same "don't hardcode a URL that will just rot" reasoning as that
|
||||||
|
# hook and as 0500-spotify-connect.hook.chroot's headless-receiver install.
|
||||||
|
set -eu
|
||||||
|
|
||||||
|
flatpak remote-add --if-not-exists flathub https://dl.flathub.org/repo/flathub.flatpakrepo
|
||||||
|
|
||||||
|
# VERIFY BEFORE THE FIRST REAL BUILD: confirm this application ID against the live
|
||||||
|
# Flathub listing (`flatpak search Spotify`). Believed correct (it's Spotify's own
|
||||||
|
# published app on Flathub) but not checked from this environment.
|
||||||
|
SPOTIFY_APP_ID="com.spotify.Client"
|
||||||
|
|
||||||
|
if flatpak install -y --noninteractive flathub "$SPOTIFY_APP_ID"; then
|
||||||
|
echo "0300-flatpak-spotify: installed ${SPOTIFY_APP_ID}."
|
||||||
|
else
|
||||||
|
echo "0300-flatpak-spotify: WARNING — could not install ${SPOTIFY_APP_ID} during the"
|
||||||
|
echo " build (no network in the chroot, or the app ID is wrong). Run this on the"
|
||||||
|
echo " booted image instead: flatpak install -y flathub ${SPOTIFY_APP_ID}"
|
||||||
|
echo " The Spotify workspace will show a blank window until this is done."
|
||||||
|
fi
|
||||||
|
|
@ -0,0 +1,14 @@
|
||||||
|
#!/bin/sh
|
||||||
|
# Installs the touchpanel-agent systemd unit. Identical logic to hosts/thin-client's
|
||||||
|
# 0700-thinclient-agent.hook.chroot — the package arrives under /opt/touchpanel-agent
|
||||||
|
# via includes.chroot and runs on the system interpreter against apt's
|
||||||
|
# python3-paho-mqtt, so there is nothing to pip-install here.
|
||||||
|
set -eu
|
||||||
|
|
||||||
|
install -m 0644 /opt/touchpanel-agent/touchpanel-agent.service \
|
||||||
|
/etc/systemd/system/touchpanel-agent.service
|
||||||
|
|
||||||
|
chmod 0644 /etc/touchpanel-agent/config.env
|
||||||
|
chown -R root:root /opt/touchpanel-agent
|
||||||
|
|
||||||
|
systemctl enable touchpanel-agent
|
||||||
|
|
@ -0,0 +1,20 @@
|
||||||
|
#!/bin/sh
|
||||||
|
# The touch dock widget (docs/project-plan.md Phase 16). Identical apt-first/
|
||||||
|
# documented-placeholder pattern as hosts/thin-client's 0800-eww-widget.hook.chroot —
|
||||||
|
# eww is not in Debian bookworm main (a Rust binary, normally cargo-built or fetched
|
||||||
|
# as a GitHub release).
|
||||||
|
set -eu
|
||||||
|
|
||||||
|
if apt-get install -y --no-install-recommends eww 2>/dev/null; then
|
||||||
|
echo "0500-eww-dock: installed eww from apt."
|
||||||
|
else
|
||||||
|
echo "0500-eww-dock: eww is not available from the configured apt sources."
|
||||||
|
echo " The touch dock (the on-screen Spotify/Home/Web/Keyboard bar) will not appear"
|
||||||
|
echo " on this image until eww is installed:"
|
||||||
|
echo " a) cargo install eww (needs a Rust toolchain in a build hook), or"
|
||||||
|
echo " b) fetch a release binary from https://github.com/elkowar/eww/releases"
|
||||||
|
echo " (check the current tag yourself) into /usr/local/bin/eww."
|
||||||
|
echo " Without it there is NO on-screen app switcher — the panel still boots and"
|
||||||
|
echo " shows Home, but touch navigation between apps needs the dock. Home Assistant/"
|
||||||
|
echo " MQTT app switching (touchpanel_agent) still works either way."
|
||||||
|
fi
|
||||||
|
|
@ -0,0 +1,17 @@
|
||||||
|
#!/bin/sh
|
||||||
|
# Firefox enterprise policy plumbing for the general-browsing window. Identical logic
|
||||||
|
# to hosts/thin-client's 0900-firefox.hook.chroot.
|
||||||
|
set -eu
|
||||||
|
|
||||||
|
FIREFOX_LIB_DIR="/usr/lib/firefox-esr"
|
||||||
|
|
||||||
|
if [ -d "$FIREFOX_LIB_DIR" ]; then
|
||||||
|
mkdir -p "$FIREFOX_LIB_DIR/distribution"
|
||||||
|
ln -sf /etc/firefox/policies/policies.json "$FIREFOX_LIB_DIR/distribution/policies.json"
|
||||||
|
echo "0600-firefox: linked policies.json into $FIREFOX_LIB_DIR/distribution/"
|
||||||
|
else
|
||||||
|
echo "0600-firefox: $FIREFOX_LIB_DIR not found (firefox-esr not installed yet, or a"
|
||||||
|
echo " different path in this Debian release) — /etc/firefox/policies/policies.json"
|
||||||
|
echo " still applies if the package's install order runs after this hook; verify"
|
||||||
|
echo " about:policies shows uBlock Origin/SponsorBlock as force-installed on first boot."
|
||||||
|
fi
|
||||||
|
|
@ -0,0 +1,22 @@
|
||||||
|
#!/bin/sh
|
||||||
|
# On-screen keyboard (docs/project-plan.md Phase 16), toggled from the touch dock's
|
||||||
|
# Keyboard button. Same apt-first/documented-placeholder pattern as eww and the
|
||||||
|
# headless Spotify Connect receiver: wvkbd is not in Debian bookworm main (it's a
|
||||||
|
# small C wlroots layer-shell client, normally built from source).
|
||||||
|
set -eu
|
||||||
|
|
||||||
|
if apt-get install -y --no-install-recommends wvkbd 2>/dev/null; then
|
||||||
|
echo "0700-onscreen-keyboard: installed wvkbd from apt."
|
||||||
|
elif command -v wvkbd-mobintl >/dev/null 2>&1; then
|
||||||
|
echo "0700-onscreen-keyboard: wvkbd-mobintl already present."
|
||||||
|
else
|
||||||
|
echo "0700-onscreen-keyboard: wvkbd is not available from the configured apt sources."
|
||||||
|
echo " Typing into Spotify search, the HA dashboard, or the web browser will have no"
|
||||||
|
echo " on-screen keyboard until it's installed:"
|
||||||
|
echo " a) build from source: https://github.com/jjsullivan5196/wvkbd"
|
||||||
|
echo " (needs libwayland-dev, wayland-protocols, cairo, pango — add a"
|
||||||
|
echo " build-dependencies block to this hook if you go this route), or"
|
||||||
|
echo " b) fetch a distro package once one exists for your release."
|
||||||
|
echo " configs/keyboard/toggle-keyboard already handles a missing binary as a no-op"
|
||||||
|
echo " (the dock button just does nothing), so the rest of the session is unaffected."
|
||||||
|
fi
|
||||||
|
|
@ -0,0 +1,66 @@
|
||||||
|
# Touch-panel package list (live-build .list.chroot format: one package per line).
|
||||||
|
# Phase 16, docs/project-plan.md.
|
||||||
|
|
||||||
|
# --- Wayland compositor / session ---
|
||||||
|
sway
|
||||||
|
swayidle
|
||||||
|
swaybg
|
||||||
|
greetd
|
||||||
|
foot
|
||||||
|
|
||||||
|
# --- Browsers ---
|
||||||
|
firefox-esr
|
||||||
|
chromium
|
||||||
|
playerctl
|
||||||
|
|
||||||
|
# --- Audio ---
|
||||||
|
pipewire
|
||||||
|
pipewire-pulse
|
||||||
|
wireplumber
|
||||||
|
alsa-utils
|
||||||
|
|
||||||
|
# --- Spotify GUI client and Steam-Link-style apps come from Flathub, not apt
|
||||||
|
# (see 0300-flatpak-spotify.hook.chroot) ---
|
||||||
|
flatpak
|
||||||
|
|
||||||
|
# --- Agent runtime ---
|
||||||
|
python3
|
||||||
|
python3-venv
|
||||||
|
# Debian bookworm ships paho-mqtt 1.6.x. touchpanel_agent is written to work against
|
||||||
|
# both the 1.x and 2.x callback APIs, so this package is used as-is rather than pip'd.
|
||||||
|
python3-paho-mqtt
|
||||||
|
procps
|
||||||
|
|
||||||
|
# --- Networking ---
|
||||||
|
network-manager
|
||||||
|
|
||||||
|
# --- Out-of-band admin. This image has no wayvnc (unlike hosts/thin-client) — see
|
||||||
|
# the README's scope note — so SSH is the only remote admin path. ---
|
||||||
|
openssh-server
|
||||||
|
sudo
|
||||||
|
|
||||||
|
# --- Touch dock (0500-eww-dock.hook.chroot tries apt, falls back to a documented
|
||||||
|
# placeholder if not packaged) ---
|
||||||
|
eww
|
||||||
|
jq
|
||||||
|
|
||||||
|
# --- On-screen keyboard (0700-onscreen-keyboard.hook.chroot, same apt-first/
|
||||||
|
# documented-fallback pattern) ---
|
||||||
|
wvkbd
|
||||||
|
|
||||||
|
# --- Firmware for real hardware (needs contrib + non-free-firmware archive areas,
|
||||||
|
# which build-touch-panel-iso.sh passes to `lb config`) ---
|
||||||
|
firmware-linux
|
||||||
|
firmware-iwlwifi
|
||||||
|
firmware-realtek
|
||||||
|
firmware-misc-nonfree
|
||||||
|
|
||||||
|
# --- Misc ---
|
||||||
|
ca-certificates
|
||||||
|
curl
|
||||||
|
openssl
|
||||||
|
less
|
||||||
|
vim-tiny
|
||||||
|
|
||||||
|
# NOTE: Spotify (com.spotify.Client) is Flatpak-only — see
|
||||||
|
# 0300-flatpak-spotify.hook.chroot. There is no apt package.
|
||||||
|
|
@ -0,0 +1,7 @@
|
||||||
|
# Only relevant when ENABLE_INSTALLER="true" in build-touch-panel-iso.sh (off by
|
||||||
|
# default — the normal path is live-boot straight into Sway, no disk install at all).
|
||||||
|
# Identical mechanism/caveat as hosts/thin-client/live-build/config/preseed.cfg.
|
||||||
|
#
|
||||||
|
# No "keyboard-configuration/xkb-keymap seen true" line on purpose: this preseeds the
|
||||||
|
# *default* answer, it does not skip the question.
|
||||||
|
d-i keyboard-configuration/xkb-keymap select de
|
||||||
|
|
@ -0,0 +1,297 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
#
|
||||||
|
# Smart Home Touch-Panel ISO Builder
|
||||||
|
# Target: builds a Debian 12 (Bookworm) live ISO on a Debian/Ubuntu build machine
|
||||||
|
#
|
||||||
|
# Drives `lb config && lb build` over hosts/touch-panel/live-build/ to produce the
|
||||||
|
# Sway touch-panel image described in docs/project-plan.md Phase 16:
|
||||||
|
# - greetd autologin straight into a kiosk Sway session (no greeter UI)
|
||||||
|
# - touchpanel-agent (Python, systemd) — HA MQTT-discovery entities + swaymsg control
|
||||||
|
# - workspaces 1:spotify (full Spotify GUI, Flathub) / 2:home (Chromium kiosk
|
||||||
|
# pointed at Home Assistant) / 3:web (general Firefox browsing)
|
||||||
|
# - an always-visible eww touch dock for on-screen app switching
|
||||||
|
# - a toggleable on-screen keyboard (wvkbd)
|
||||||
|
#
|
||||||
|
# Reuses hosts/thin-client's own live-build toolchain and directory-split convention,
|
||||||
|
# NOT its live-build tree — this is a different device (touched directly, not
|
||||||
|
# remote-controlled from the couch) with a different app set, so it gets its own
|
||||||
|
# smaller config/, agent/, and live-build/, same relationship as
|
||||||
|
# hosts/audio-endpoint's amd64 build to the thin client's.
|
||||||
|
#
|
||||||
|
# This script is also the single point that keeps configs/ (the human-edited source of
|
||||||
|
# truth, reviewed in git) in sync with live-build/config/includes.chroot/ (the
|
||||||
|
# generated tree that actually gets baked into the image). Never hand-edit anything
|
||||||
|
# under includes.chroot — it is wiped and regenerated on every run.
|
||||||
|
#
|
||||||
|
# Run as: sudo ./build-touch-panel-iso.sh
|
||||||
|
#
|
||||||
|
# EDIT THE VARIABLES BELOW BEFORE RUNNING.
|
||||||
|
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# CONFIGURATION — edit these before running
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
DEBIAN_RELEASE="bookworm" # Matches the container host's OS
|
||||||
|
KIOSK_USERNAME="kiosk" # The autologin account the whole image is built around
|
||||||
|
IMAGE_HOSTNAME="touch-panel" # Hostname baked into the image
|
||||||
|
TOUCHPANEL_NAME="Kitchen touch panel" # Friendly name shown on the HA device
|
||||||
|
|
||||||
|
KEYBOARD_LAYOUT="de" # xkb layout name (`localectl list-x11-keymap-layouts`)
|
||||||
|
|
||||||
|
ENABLE_INSTALLER="false" # "true" adds a debian-installer to the ISO (install to disk)
|
||||||
|
|
||||||
|
# --- Where the touch panel talks to -----------------------------------------
|
||||||
|
MQTT_BROKER_HOST="192.168.1.10" # <-- EDIT: container-host IP running Mosquitto
|
||||||
|
MQTT_BROKER_PORT="1883"
|
||||||
|
MQTT_USERNAME="" # Leave empty while Mosquitto runs allow_anonymous
|
||||||
|
MQTT_PASSWORD="" # Never commit a real value here
|
||||||
|
HA_URL="http://192.168.1.10:8123" # <-- EDIT: Home Assistant URL — the "Home" workspace
|
||||||
|
|
||||||
|
# Optional: an SSH public key to bake into the kiosk account for out-of-band admin.
|
||||||
|
# The image ships with password auth disabled and no wayvnc (see README's scope note),
|
||||||
|
# so without this the only admin path is the local console.
|
||||||
|
SSH_AUTHORIZED_KEY=""
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Paths
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
TOUCH_PANEL_DIR="$(dirname "$SCRIPT_DIR")"
|
||||||
|
CONFIGS_DIR="${TOUCH_PANEL_DIR}/configs"
|
||||||
|
AGENT_DIR="${TOUCH_PANEL_DIR}/agent"
|
||||||
|
LIVE_BUILD_DIR="${TOUCH_PANEL_DIR}/live-build"
|
||||||
|
INCLUDES="${LIVE_BUILD_DIR}/config/includes.chroot"
|
||||||
|
PACKAGE_LIST="${LIVE_BUILD_DIR}/config/package-lists/touch-panel.list.chroot"
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Sanity checks
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
if [[ $EUID -ne 0 ]]; then
|
||||||
|
echo "Warning: not running as root. 'lb build' needs root to bootstrap and chroot,"
|
||||||
|
echo " and will fail partway through. Re-run with: sudo $0"
|
||||||
|
echo " Continuing anyway so you can at least regenerate includes.chroot..."
|
||||||
|
fi
|
||||||
|
|
||||||
|
if ! grep -qi "debian\|ubuntu" /etc/os-release; then
|
||||||
|
echo "Warning: live-build targets a Debian/Ubuntu build host. Proceeding anyway..."
|
||||||
|
fi
|
||||||
|
|
||||||
|
if ! command -v lb &> /dev/null; then
|
||||||
|
if [[ $EUID -eq 0 ]]; then
|
||||||
|
echo "--- Installing live-build ---"
|
||||||
|
apt-get update
|
||||||
|
apt-get install -y live-build
|
||||||
|
else
|
||||||
|
echo "live-build is not installed and this script is not running as root." >&2
|
||||||
|
echo " Install it first: sudo apt-get install live-build" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
echo "--- live-build already installed, skipping ---"
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [[ ! -f "$PACKAGE_LIST" ]]; then
|
||||||
|
echo "Missing package list: $PACKAGE_LIST" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [[ "$MQTT_BROKER_HOST" == "192.168.1.10" ]]; then
|
||||||
|
echo "Warning: MQTT_BROKER_HOST is still the placeholder IP."
|
||||||
|
echo " Edit it at the top of this script to your container host's real LAN address,"
|
||||||
|
echo " or the touch panel won't show up as Home Assistant entities."
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [[ "$HA_URL" == "http://192.168.1.10:8123" ]]; then
|
||||||
|
echo "Warning: HA_URL is still the placeholder."
|
||||||
|
echo " The image builds and boots fine, but the Home workspace will show a"
|
||||||
|
echo " connection error until this points at a real Home Assistant."
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo
|
||||||
|
echo "=== Smart Home Touch-Panel ISO Builder ==="
|
||||||
|
echo "Debian release : $DEBIAN_RELEASE"
|
||||||
|
echo "Kiosk user : $KIOSK_USERNAME"
|
||||||
|
echo "Image hostname : $IMAGE_HOSTNAME"
|
||||||
|
echo "MQTT broker : ${MQTT_BROKER_HOST}:${MQTT_BROKER_PORT}"
|
||||||
|
echo "Home Assistant : $HA_URL"
|
||||||
|
echo "Keyboard layout : $KEYBOARD_LAYOUT"
|
||||||
|
echo
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# 1. Regenerate includes.chroot from configs/ and agent/
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
echo "--- Regenerating $INCLUDES ---"
|
||||||
|
rm -rf "$INCLUDES"
|
||||||
|
mkdir -p \
|
||||||
|
"$INCLUDES/etc/greetd" \
|
||||||
|
"$INCLUDES/etc/touchpanel-agent" \
|
||||||
|
"$INCLUDES/etc/firefox/policies" \
|
||||||
|
"$INCLUDES/etc/touchpanel-firefox" \
|
||||||
|
"$INCLUDES/usr/local/bin" \
|
||||||
|
"$INCLUDES/opt/touchpanel-agent" \
|
||||||
|
"$INCLUDES/home/${KIOSK_USERNAME}/.config/sway" \
|
||||||
|
"$INCLUDES/home/${KIOSK_USERNAME}/.config/eww" \
|
||||||
|
"$INCLUDES/home/${KIOSK_USERNAME}/.ssh"
|
||||||
|
|
||||||
|
# @KIOSK_USERNAME@ and @KEYBOARD_LAYOUT@ are the only templated tokens in the configs.
|
||||||
|
# Everything else the hooks need is read at build time from
|
||||||
|
# /etc/touchpanel-agent/config.env (written below), which live-build copies in
|
||||||
|
# (chroot_local-includes) before it runs the hooks (chroot_local-hooks).
|
||||||
|
subst() {
|
||||||
|
sed -e "s/@KIOSK_USERNAME@/${KIOSK_USERNAME}/g" \
|
||||||
|
-e "s/@KEYBOARD_LAYOUT@/${KEYBOARD_LAYOUT}/g" "$1" > "$2"
|
||||||
|
}
|
||||||
|
|
||||||
|
subst "${CONFIGS_DIR}/greetd/config.toml" "$INCLUDES/etc/greetd/config.toml"
|
||||||
|
subst "${CONFIGS_DIR}/sway/config" "$INCLUDES/home/${KIOSK_USERNAME}/.config/sway/config"
|
||||||
|
subst "${AGENT_DIR}/touchpanel-agent.service" "$INCLUDES/opt/touchpanel-agent/touchpanel-agent.service"
|
||||||
|
|
||||||
|
install -m 0755 "${CONFIGS_DIR}/greetd/kiosk-session" "$INCLUDES/usr/local/bin/kiosk-session"
|
||||||
|
install -m 0755 "${CONFIGS_DIR}/sway/ha-kiosk" "$INCLUDES/usr/local/bin/ha-kiosk"
|
||||||
|
install -m 0755 "${CONFIGS_DIR}/spotify/spotify-launch" "$INCLUDES/usr/local/bin/spotify-launch"
|
||||||
|
install -m 0755 "${CONFIGS_DIR}/keyboard/toggle-keyboard" "$INCLUDES/usr/local/bin/toggle-keyboard"
|
||||||
|
|
||||||
|
# Console (VT/TTY) keymap — separate from Sway's own xkb_layout, since greetd briefly
|
||||||
|
# owns the console before Sway starts.
|
||||||
|
mkdir -p "$INCLUDES/etc/default"
|
||||||
|
cat > "$INCLUDES/etc/default/keyboard" <<EOF
|
||||||
|
XKBMODEL="pc105"
|
||||||
|
XKBLAYOUT="${KEYBOARD_LAYOUT}"
|
||||||
|
XKBVARIANT=""
|
||||||
|
XKBOPTIONS=""
|
||||||
|
|
||||||
|
BACKSPACE="guess"
|
||||||
|
EOF
|
||||||
|
|
||||||
|
# Touch dock (eww) — session-scoped files under the kiosk user's own config.
|
||||||
|
install -m 0644 "${CONFIGS_DIR}/eww/eww.yuck" "$INCLUDES/home/${KIOSK_USERNAME}/.config/eww/eww.yuck"
|
||||||
|
install -m 0644 "${CONFIGS_DIR}/eww/eww.scss" "$INCLUDES/home/${KIOSK_USERNAME}/.config/eww/eww.scss"
|
||||||
|
|
||||||
|
# Firefox: the general-browsing window launcher, plus the shared chrome/prefs template
|
||||||
|
# it copies into its own profile at launch. policies.json is global and goes straight
|
||||||
|
# into includes.chroot.
|
||||||
|
install -m 0755 "${CONFIGS_DIR}/firefox/web-browser" "$INCLUDES/usr/local/bin/web-browser"
|
||||||
|
install -m 0644 "${CONFIGS_DIR}/firefox/policies.json" "$INCLUDES/etc/firefox/policies/policies.json"
|
||||||
|
install -m 0644 "${CONFIGS_DIR}/firefox/userChrome.css" "$INCLUDES/etc/touchpanel-firefox/userChrome.css"
|
||||||
|
install -m 0644 "${CONFIGS_DIR}/firefox/user.js" "$INCLUDES/etc/touchpanel-firefox/user.js"
|
||||||
|
|
||||||
|
cp -r "${AGENT_DIR}/touchpanel_agent" "$INCLUDES/opt/touchpanel-agent/"
|
||||||
|
install -m 0644 "${AGENT_DIR}/requirements.txt" "$INCLUDES/opt/touchpanel-agent/requirements.txt"
|
||||||
|
find "$INCLUDES/opt/touchpanel-agent" -name '__pycache__' -type d -prune -exec rm -rf {} +
|
||||||
|
|
||||||
|
if [[ -n "$SSH_AUTHORIZED_KEY" ]]; then
|
||||||
|
echo "$SSH_AUTHORIZED_KEY" > "$INCLUDES/home/${KIOSK_USERNAME}/.ssh/authorized_keys"
|
||||||
|
chmod 600 "$INCLUDES/home/${KIOSK_USERNAME}/.ssh/authorized_keys"
|
||||||
|
echo " Baked an SSH authorized_keys entry for ${KIOSK_USERNAME}."
|
||||||
|
else
|
||||||
|
echo " No SSH_AUTHORIZED_KEY set — SSH admin access will not be possible on this image."
|
||||||
|
fi
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# 2. Runtime config, read by touchpanel-agent, the sway session wrapper, and hooks
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
echo "--- Writing /etc/touchpanel-agent/config.env into includes.chroot ---"
|
||||||
|
cat > "$INCLUDES/etc/touchpanel-agent/config.env" <<EOF
|
||||||
|
# Generated by hosts/touch-panel/scripts/build-touch-panel-iso.sh — do not hand-edit
|
||||||
|
# here; edit the CONFIGURATION block in that script and rebuild.
|
||||||
|
KIOSK_USERNAME=${KIOSK_USERNAME}
|
||||||
|
TOUCHPANEL_NAME=${TOUCHPANEL_NAME}
|
||||||
|
|
||||||
|
MQTT_BROKER_HOST=${MQTT_BROKER_HOST}
|
||||||
|
MQTT_BROKER_PORT=${MQTT_BROKER_PORT}
|
||||||
|
MQTT_USERNAME=${MQTT_USERNAME}
|
||||||
|
MQTT_PASSWORD=${MQTT_PASSWORD}
|
||||||
|
|
||||||
|
HA_URL=${HA_URL}
|
||||||
|
EOF
|
||||||
|
chmod 0644 "$INCLUDES/etc/touchpanel-agent/config.env"
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# 3. lb config
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
cd "$LIVE_BUILD_DIR"
|
||||||
|
|
||||||
|
chmod +x config/hooks/normal/*.hook.chroot
|
||||||
|
|
||||||
|
if [[ -e .build ]]; then
|
||||||
|
echo "--- Previous build found, running 'lb clean' (package cache is kept) ---"
|
||||||
|
lb clean
|
||||||
|
fi
|
||||||
|
|
||||||
|
INSTALLER_MODE="none"
|
||||||
|
if [[ "$ENABLE_INSTALLER" == "true" ]]; then
|
||||||
|
INSTALLER_MODE="live"
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "--- Running lb config ---"
|
||||||
|
lb config \
|
||||||
|
--distribution "$DEBIAN_RELEASE" \
|
||||||
|
--architectures amd64 \
|
||||||
|
--linux-flavours amd64 \
|
||||||
|
--archive-areas "main contrib non-free-firmware" \
|
||||||
|
--binary-images iso-hybrid \
|
||||||
|
--debian-installer "$INSTALLER_MODE" \
|
||||||
|
--iso-application "SmartestHome Touch Panel" \
|
||||||
|
--iso-publisher "SmartestHome" \
|
||||||
|
--iso-volume "smarthome-touch-panel" \
|
||||||
|
--memtest none \
|
||||||
|
--bootappend-live "boot=live components quiet splash noautologin username=${KIOSK_USERNAME} hostname=${IMAGE_HOSTNAME}"
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# 4. lb build
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
echo "--- Running lb build (this takes a while and needs network) ---"
|
||||||
|
lb build
|
||||||
|
|
||||||
|
ISO_PATH="$(find "$LIVE_BUILD_DIR" -maxdepth 1 -name '*.iso' -print -quit)"
|
||||||
|
ISO_PATH="${ISO_PATH:-${LIVE_BUILD_DIR}/live-image-amd64.hybrid.iso}"
|
||||||
|
|
||||||
|
echo
|
||||||
|
echo "=== Done ==="
|
||||||
|
echo "ISO written to:"
|
||||||
|
echo " ${ISO_PATH}"
|
||||||
|
echo
|
||||||
|
echo "What's in it:"
|
||||||
|
echo " greetd : autologin as '${KIOSK_USERNAME}' straight into Sway on vt1"
|
||||||
|
echo " Sway : workspaces 1:spotify / 2:home / 3:web, always-on touch dock"
|
||||||
|
echo " touchpanel-agent : system service, MQTT ${MQTT_BROKER_HOST}:${MQTT_BROKER_PORT}"
|
||||||
|
echo " Spotify : full GUI client (Flathub com.spotify.Client), workspace 1"
|
||||||
|
echo " Home Assistant : Chromium kiosk, ${HA_URL}, workspace 2, auto-restart-on-crash"
|
||||||
|
echo " Web browser : Firefox, minimal chrome, uBlock Origin + SponsorBlock, workspace 3"
|
||||||
|
echo " Touch dock : bottom bar (eww) — Spotify / Home / Web / Keyboard buttons"
|
||||||
|
echo " On-screen keyboard: wvkbd, toggled from the dock's Keyboard button"
|
||||||
|
echo " Keyboard layout : ${KEYBOARD_LAYOUT} (console + Sway)"
|
||||||
|
echo " Maintenance shell : Super+Shift+Ctrl+M opens a floating terminal locally"
|
||||||
|
echo " Remote admin : SSH only — no wayvnc on this image, see README"
|
||||||
|
echo
|
||||||
|
echo "Next steps:"
|
||||||
|
echo " 1. Write the ISO to a USB stick:"
|
||||||
|
echo " sudo dd if=${ISO_PATH} of=/dev/sdX bs=4M status=progress oflag=sync"
|
||||||
|
echo " (double-check /dev/sdX with 'lsblk' first — dd will happily eat the wrong disk)"
|
||||||
|
echo " 2. Boot the target touchscreen machine from it. No touch-panel hardware has been"
|
||||||
|
echo " chosen yet (see the README) — confirm touch input is recognised as a Wayland"
|
||||||
|
echo " wl_touch device (not emulated as a pointer) before treating this as validated."
|
||||||
|
echo " 3. Confirm greetd autologin lands in Sway: dark background, the touch dock at"
|
||||||
|
echo " the bottom, and the Home Assistant kiosk window on workspace 2:home."
|
||||||
|
echo " 4. Tap through the dock: Spotify (log in on first launch — Premium required),"
|
||||||
|
echo " Home, Web."
|
||||||
|
echo " 5. Confirm touchpanel-agent connected and registered:"
|
||||||
|
echo " systemctl status touchpanel-agent"
|
||||||
|
echo " In Home Assistant, a '${TOUCHPANEL_NAME}' device should appear under the MQTT"
|
||||||
|
echo " integration with: Show Spotify / Show Home / Show web browser (buttons),"
|
||||||
|
echo " Screen (select), Volume (number), Playback state (sensor), and"
|
||||||
|
echo " play/pause/next/prev/stop buttons."
|
||||||
|
echo " 6. Start something in Spotify and confirm Playback state follows it (playerctl"
|
||||||
|
echo " -> touchpanel-agent's MPRIS bridge — needs Spotify's own MPRIS bus name to"
|
||||||
|
echo " actually be 'spotify', unverified against the real Flatpak build)."
|
||||||
|
echo " 7. Tap the dock's Keyboard button over a text field (Spotify search, the browser"
|
||||||
|
echo " address bar) and confirm wvkbd shows/hides. There is no auto-show — see README."
|
||||||
|
echo
|
||||||
|
echo "Then pull the power on the container host and re-check: the panel must still boot"
|
||||||
|
echo "and show Spotify/the browser (Home will show a connection error, which is"
|
||||||
|
echo "expected) with the container host powered off — same reactive-path philosophy as"
|
||||||
|
echo "hosts/thin-client's Phase 11.10."
|
||||||
|
echo
|
||||||
|
echo "Rebuilding later: edit hosts/touch-panel/configs/* or agent/*, then re-run this"
|
||||||
|
echo "script — includes.chroot is regenerated from them every time."
|
||||||
|
|
@ -0,0 +1,15 @@
|
||||||
|
# pantry-vision — camera-driven grocery cataloguing API (docs/project-plan.md Phase
|
||||||
|
# 17). `restart: unless-stopped`, same as admin-canvas: it has to be up whenever the
|
||||||
|
# kitchen display might hold an item up to the camera.
|
||||||
|
FROM python:3.11-slim
|
||||||
|
|
||||||
|
ENV PYTHONUNBUFFERED=1 \
|
||||||
|
PYTHONDONTWRITEBYTECODE=1
|
||||||
|
|
||||||
|
WORKDIR /app
|
||||||
|
|
||||||
|
# stdlib only (urllib for the Ollama/Grocy calls) — no requirements.txt, same call as
|
||||||
|
# admin-canvas/server.py.
|
||||||
|
COPY server.py ./
|
||||||
|
|
||||||
|
CMD ["python", "server.py"]
|
||||||
|
|
@ -0,0 +1,125 @@
|
||||||
|
# pantry-vision
|
||||||
|
|
||||||
|
The kitchen display's backend, from [Phase 17 of the project plan](../docs/project-plan.md).
|
||||||
|
|
||||||
|
The workflow this exists for: **come home, put down the shopping bag, hold one item
|
||||||
|
up to the kitchen display's camera, the system proposes what it is and roughly how
|
||||||
|
long it keeps, you confirm (editing anything it got wrong), put it away.** The same
|
||||||
|
display then shows the resulting inventory ordered by what expires soonest, and
|
||||||
|
Grocy's recipes, on request.
|
||||||
|
|
||||||
|
- **`pantry-vision`** (this directory) — a small always-on Python HTTP service.
|
||||||
|
`POST /identify` (a photo → a proposal via an Ollama vision model), `POST /confirm`
|
||||||
|
(a human-reviewed proposal → written into Grocy stock), `GET /inventory` and
|
||||||
|
`GET /recipes` (proxy Grocy, reshaped for the frontend). All four endpoints are
|
||||||
|
bearer-token gated.
|
||||||
|
- **`frontend/`** — the static single-page app the kitchen display's kiosk browser
|
||||||
|
loads: Scan / Inventory / Recipes, vanilla JS, no build step, no framework — same
|
||||||
|
"vendored, dependency-free" choice as the digest/admin canvas SDKs. Served
|
||||||
|
read-only by a `pantry-web` nginx container (`setup-container-host.sh`), the same
|
||||||
|
role `digest-web`/`admin-web` already play for their own hosts.
|
||||||
|
- **`hosts/kitchen-display/`** — the touch kiosk image that runs the frontend. See
|
||||||
|
that directory's own README for the device side of this.
|
||||||
|
|
||||||
|
## A real network listener, unlike admin-canvas
|
||||||
|
|
||||||
|
`admin-canvas` deliberately has **no published port** — only Home Assistant, on the
|
||||||
|
same compose network, ever calls it. `pantry-vision` is different on purpose: the
|
||||||
|
kitchen display is a separate physical device on the LAN and has to reach this
|
||||||
|
service directly (there is no HA-mediation step between "hold item up to camera" and
|
||||||
|
"get an identification back" — that has to be fast and synchronous). So
|
||||||
|
`PANTRY_VISION_PORT` **is** published, and every request — including the two GETs —
|
||||||
|
requires the bearer token, as the actual boundary instead of network placement.
|
||||||
|
|
||||||
|
The same token has to be baked into the kitchen display's own build config
|
||||||
|
(`hosts/kitchen-display/scripts/build-kitchen-display-iso.sh`), not just Home
|
||||||
|
Assistant's — see that host's README.
|
||||||
|
|
||||||
|
## `/identify` never writes anything by itself
|
||||||
|
|
||||||
|
This is the one guardrail that matters most in this whole phase. The vision model's
|
||||||
|
guess — name, category, how many days until it likely goes bad — is a **proposal**,
|
||||||
|
shown on screen for the person to review and edit before anything is confirmed. Only
|
||||||
|
`/confirm`, a separate call the frontend makes after the person taps "Confirm & add,"
|
||||||
|
ever writes to Grocy. This is the same "propose, never auto-commit" rule this project
|
||||||
|
already applies to identity-merge confirmation (see the *Identity store* row in
|
||||||
|
`docs/project-plan.md` §2) — a wrong camera guess costs one tap to fix, not a wrong
|
||||||
|
fact silently written into the household's inventory.
|
||||||
|
|
||||||
|
## Configure
|
||||||
|
|
||||||
|
```sh
|
||||||
|
cp pantry-vision/pantry-vision.env.example /opt/smart-home/pantry-vision/pantry-vision.env
|
||||||
|
openssl rand -hex 32 # put the result in PANTRY_VISION_TOKEN
|
||||||
|
chmod 600 /opt/smart-home/pantry-vision/pantry-vision.env
|
||||||
|
$EDITOR /opt/smart-home/pantry-vision/pantry-vision.env
|
||||||
|
```
|
||||||
|
|
||||||
|
You also need, inside Grocy's own UI (it's already running as the always-on `grocy`
|
||||||
|
container regardless of this phase): **Settings → Manage API keys** for
|
||||||
|
`GROCY_API_KEY`, and to confirm **Settings → Locations / Quantity units** actually
|
||||||
|
match `GROCY_DEFAULT_LOCATION_ID`/`GROCY_DEFAULT_QU_ID` (fresh-install defaults, not
|
||||||
|
guaranteed to match a Grocy that's already been customised).
|
||||||
|
|
||||||
|
## Pick a vision model
|
||||||
|
|
||||||
|
`OLLAMA_VISION_MODEL` defaults to `llava`, but **nothing here has confirmed that name
|
||||||
|
against a real pull** — pick a vision-capable model (`llava`, `qwen2.5vl`, or whatever
|
||||||
|
your LLM host's GPU/CPU tier can run at acceptable latency for someone standing at
|
||||||
|
the counter holding a can of beans) and:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
ollama pull llava # on the LLM host, or whichever model you picked
|
||||||
|
```
|
||||||
|
|
||||||
|
Plain text models (`qwen2.5:14b-instruct`, used elsewhere in this project for
|
||||||
|
digest/Assist) **cannot see images at all** — pointing `OLLAMA_VISION_MODEL` at one
|
||||||
|
of those will not error clearly, it will just produce a useless/hallucinated
|
||||||
|
response for every photo. Latency is unmeasured; a vision pass on a CPU-only LLM
|
||||||
|
host could easily be too slow for a "hold item up to camera" interaction to feel
|
||||||
|
responsive — this needs to be measured on real hardware, not assumed.
|
||||||
|
|
||||||
|
## Grocy API assumptions — unverified against a real instance
|
||||||
|
|
||||||
|
`server.py`'s Grocy calls (`_find_or_create_product`, `_add_to_stock`, the `/inventory`
|
||||||
|
and `/recipes` proxies) are written against Grocy's *documented* API shape, not
|
||||||
|
checked against a running instance. In particular:
|
||||||
|
|
||||||
|
- Whether `GET /api/stock` rows carry a nested `product` object with a `name` field
|
||||||
|
by default, or need an explicit embed/expand parameter — `_handle_inventory`
|
||||||
|
degrades to `Product #<id>` if not, rather than dropping the row, but that's a
|
||||||
|
fallback, not a fix.
|
||||||
|
- Whether `POST /api/objects/products` with just
|
||||||
|
`name`/`location_id`/`qu_id_purchase`/`qu_id_stock` is actually enough to create a
|
||||||
|
minimal product on your Grocy version, or whether it requires more fields.
|
||||||
|
- Whether the Recipes feature (`GET /api/objects/recipes`,
|
||||||
|
`GET /api/recipes/{id}/fulfillment`) needs to be explicitly enabled/populated
|
||||||
|
before it returns anything meaningful — `_handle_recipes` degrades to
|
||||||
|
`"fulfilled": null` per-recipe on any failure rather than breaking the whole list.
|
||||||
|
|
||||||
|
Grocy exposes a live OpenAPI spec at `http://<grocy-host>:9283/api/openapi/specification`
|
||||||
|
once it's running — read that against a real instance before trusting any of the
|
||||||
|
above, and adjust `server.py` if the shapes differ.
|
||||||
|
|
||||||
|
## Deploy
|
||||||
|
|
||||||
|
Wired into `hosts/container-host/scripts/setup-container-host.sh` behind
|
||||||
|
`ENABLE_PANTRY_VISION` (off by default) — see that script's `# CONFIGURATION` block
|
||||||
|
and its own README. It builds two containers: `pantry-vision` (this API) and
|
||||||
|
`pantry-web` (nginx, serves `frontend/` read-only).
|
||||||
|
|
||||||
|
## Manual verification still outstanding
|
||||||
|
|
||||||
|
1. All of the Grocy API assumptions above.
|
||||||
|
2. Real-world vision-model accuracy and latency for grocery items — untested with
|
||||||
|
any actual model or camera.
|
||||||
|
3. Whether Ollama's `/api/generate` `images` field is still the right call shape for
|
||||||
|
whichever vision model you pick — some multimodal models are only exposed through
|
||||||
|
Ollama's newer `/api/chat` with a `images` field per-message instead; this was
|
||||||
|
written against `/api/generate`'s documented multimodal support and not run
|
||||||
|
against a real model.
|
||||||
|
4. CORS: `_respond`'s blanket `Access-Control-Allow-Origin: *` is fine for a
|
||||||
|
LAN-only, bearer-token-gated service with no cookies, but hasn't been checked
|
||||||
|
against a real browser's preflight behavior for the raw-image-bytes `POST
|
||||||
|
/identify` call in particular (some browsers preflight non-simple `Content-Type`s
|
||||||
|
like `image/jpeg` — `do_OPTIONS` is written to handle that but is untested).
|
||||||
|
|
@ -0,0 +1,227 @@
|
||||||
|
// pantry-vision frontend logic (docs/project-plan.md Phase 17). Vanilla JS, no
|
||||||
|
// framework, no build step — same "vendored, dependency-free" choice as the digest/
|
||||||
|
// admin canvas SDKs' render.js.
|
||||||
|
//
|
||||||
|
// Config comes from URL query params (?api=...&token=...), set by the
|
||||||
|
// kitchen-display kiosk's own launch command, NOT hardcoded here — this file is a
|
||||||
|
// generic static asset with no secret in it, served read-only by pantry-web to
|
||||||
|
// whatever device points a browser at it.
|
||||||
|
"use strict";
|
||||||
|
|
||||||
|
const params = new URLSearchParams(location.search);
|
||||||
|
const API = (params.get("api") || "").replace(/\/$/, "");
|
||||||
|
const TOKEN = params.get("token") || "";
|
||||||
|
|
||||||
|
if (!API || !TOKEN) {
|
||||||
|
document.body.innerHTML =
|
||||||
|
'<p class="error" style="padding:24px">pantry-vision not configured — missing ' +
|
||||||
|
"?api=&token= in the URL. See hosts/kitchen-display/configs/sway/pantry-kiosk.</p>";
|
||||||
|
throw new Error("pantry-vision: missing ?api=/&token= query params");
|
||||||
|
}
|
||||||
|
|
||||||
|
function api(path, options) {
|
||||||
|
options = options || {};
|
||||||
|
options.headers = Object.assign({ Authorization: `Bearer ${TOKEN}` }, options.headers || {});
|
||||||
|
return fetch(`${API}${path}`, options).then((res) => {
|
||||||
|
if (!res.ok) {
|
||||||
|
return res.json().catch(() => ({})).then((body) => {
|
||||||
|
throw new Error(body.error || `${res.status} ${res.statusText}`);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
return res.json();
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
// --- Tabs --------------------------------------------------------------------
|
||||||
|
const tabs = document.querySelectorAll(".tab");
|
||||||
|
const panels = document.querySelectorAll(".panel");
|
||||||
|
|
||||||
|
function activateTab(name) {
|
||||||
|
tabs.forEach((t) => t.classList.toggle("active", t.dataset.tab === name));
|
||||||
|
panels.forEach((p) => p.classList.toggle("active", p.id === name));
|
||||||
|
if (name === "inventory") loadInventory();
|
||||||
|
if (name === "recipes") loadRecipes();
|
||||||
|
if (name === "scan") resetScan();
|
||||||
|
}
|
||||||
|
|
||||||
|
tabs.forEach((t) => t.addEventListener("click", () => activateTab(t.dataset.tab)));
|
||||||
|
|
||||||
|
// A "Show <tab>" MQTT command from touchpanel-style HA control arrives as a URL
|
||||||
|
// fragment reload (hosts/kitchen-display/agent kills and relaunches Chromium at
|
||||||
|
// index.html#recipes, same pattern as the thin client's digest-browser) — honour it
|
||||||
|
// on load, same as any manual tap.
|
||||||
|
if (location.hash) {
|
||||||
|
const initial = location.hash.slice(1);
|
||||||
|
if (["scan", "inventory", "recipes"].includes(initial)) activateTab(initial);
|
||||||
|
}
|
||||||
|
|
||||||
|
// --- Scan ----------------------------------------------------------------------
|
||||||
|
const video = document.getElementById("camera-preview");
|
||||||
|
const captureBtn = document.getElementById("capture-btn");
|
||||||
|
const cameraError = document.getElementById("camera-error");
|
||||||
|
const scanCamera = document.getElementById("scan-camera");
|
||||||
|
const scanResult = document.getElementById("scan-result");
|
||||||
|
const capturedFrame = document.getElementById("captured-frame");
|
||||||
|
const confirmForm = document.getElementById("confirm-form");
|
||||||
|
const confirmStatus = document.getElementById("confirm-status");
|
||||||
|
const retakeBtn = document.getElementById("retake-btn");
|
||||||
|
|
||||||
|
let stream = null;
|
||||||
|
|
||||||
|
function startCamera() {
|
||||||
|
if (stream) return;
|
||||||
|
navigator.mediaDevices
|
||||||
|
.getUserMedia({ video: { facingMode: "environment" }, audio: false })
|
||||||
|
.then((s) => {
|
||||||
|
stream = s;
|
||||||
|
video.srcObject = s;
|
||||||
|
cameraError.hidden = true;
|
||||||
|
})
|
||||||
|
.catch((err) => {
|
||||||
|
cameraError.textContent = `Camera unavailable: ${err.message}. See hosts/kitchen-display/README.md.`;
|
||||||
|
cameraError.hidden = false;
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
function resetScan() {
|
||||||
|
scanResult.hidden = true;
|
||||||
|
scanCamera.hidden = false;
|
||||||
|
confirmStatus.textContent = "";
|
||||||
|
startCamera();
|
||||||
|
}
|
||||||
|
|
||||||
|
captureBtn.addEventListener("click", () => {
|
||||||
|
if (!stream) return;
|
||||||
|
capturedFrame.width = video.videoWidth;
|
||||||
|
capturedFrame.height = video.videoHeight;
|
||||||
|
capturedFrame.getContext("2d").drawImage(video, 0, 0);
|
||||||
|
|
||||||
|
scanCamera.hidden = true;
|
||||||
|
scanResult.hidden = false;
|
||||||
|
confirmStatus.textContent = "Identifying…";
|
||||||
|
document.getElementById("f-confidence").textContent = "";
|
||||||
|
|
||||||
|
capturedFrame.toBlob(
|
||||||
|
(blob) => {
|
||||||
|
api("/identify", { method: "POST", body: blob, headers: { "Content-Type": "image/jpeg" } })
|
||||||
|
.then((proposal) => {
|
||||||
|
document.getElementById("f-name").value = proposal.name || "";
|
||||||
|
document.getElementById("f-category").value = proposal.category || "other";
|
||||||
|
const days = Number.isFinite(proposal.estimated_shelf_life_days)
|
||||||
|
? proposal.estimated_shelf_life_days
|
||||||
|
: 7;
|
||||||
|
const due = new Date();
|
||||||
|
due.setDate(due.getDate() + days);
|
||||||
|
document.getElementById("f-date").value = due.toISOString().slice(0, 10);
|
||||||
|
document.getElementById("f-confidence").textContent =
|
||||||
|
`Model confidence: ${proposal.confidence || "unknown"}` +
|
||||||
|
(proposal.note ? ` — ${proposal.note}` : "") +
|
||||||
|
". Review before confirming.";
|
||||||
|
confirmStatus.textContent = "";
|
||||||
|
})
|
||||||
|
.catch((err) => {
|
||||||
|
confirmStatus.textContent = `Could not identify: ${err.message}. Fill in manually.`;
|
||||||
|
});
|
||||||
|
},
|
||||||
|
"image/jpeg",
|
||||||
|
0.85
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
retakeBtn.addEventListener("click", resetScan);
|
||||||
|
|
||||||
|
confirmForm.addEventListener("submit", (event) => {
|
||||||
|
event.preventDefault();
|
||||||
|
confirmStatus.textContent = "Adding…";
|
||||||
|
api("/confirm", {
|
||||||
|
method: "POST",
|
||||||
|
headers: { "Content-Type": "application/json" },
|
||||||
|
body: JSON.stringify({
|
||||||
|
name: document.getElementById("f-name").value.trim(),
|
||||||
|
category: document.getElementById("f-category").value,
|
||||||
|
best_before_date: document.getElementById("f-date").value,
|
||||||
|
quantity: Number(document.getElementById("f-quantity").value) || 1,
|
||||||
|
}),
|
||||||
|
})
|
||||||
|
.then(() => {
|
||||||
|
confirmStatus.textContent = "Added. Put it away!";
|
||||||
|
setTimeout(resetScan, 1500);
|
||||||
|
})
|
||||||
|
.catch((err) => {
|
||||||
|
confirmStatus.textContent = `Could not add: ${err.message}`;
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
// --- Inventory -------------------------------------------------------------------
|
||||||
|
function urgencyClass(daysLeft) {
|
||||||
|
if (daysLeft === null || daysLeft === undefined) return "";
|
||||||
|
if (daysLeft < 0) return "urgent-expired";
|
||||||
|
if (daysLeft <= 2) return "urgent-soon";
|
||||||
|
if (daysLeft <= 7) return "urgent-week";
|
||||||
|
return "";
|
||||||
|
}
|
||||||
|
|
||||||
|
function loadInventory() {
|
||||||
|
const el = document.getElementById("inventory-list");
|
||||||
|
api("/inventory")
|
||||||
|
.then((data) => {
|
||||||
|
const items = data.items || [];
|
||||||
|
if (!items.length) {
|
||||||
|
el.innerHTML = '<p class="hint">Nothing in stock yet — scan something!</p>';
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
el.innerHTML = items
|
||||||
|
.map((item) => {
|
||||||
|
const days = item.days_left;
|
||||||
|
const label =
|
||||||
|
days === null || days === undefined
|
||||||
|
? "no date"
|
||||||
|
: days < 0
|
||||||
|
? `expired ${-days}d ago`
|
||||||
|
: days === 0
|
||||||
|
? "expires today"
|
||||||
|
: `${days}d left`;
|
||||||
|
return `<div class="card ${urgencyClass(days)}">
|
||||||
|
<span class="card-name">${escapeHtml(item.name)}</span>
|
||||||
|
<span class="card-amount">×${escapeHtml(String(item.amount ?? ""))}</span>
|
||||||
|
<span class="card-due">${escapeHtml(label)}</span>
|
||||||
|
</div>`;
|
||||||
|
})
|
||||||
|
.join("");
|
||||||
|
})
|
||||||
|
.catch((err) => {
|
||||||
|
el.innerHTML = `<p class="error">Could not load inventory: ${escapeHtml(err.message)}</p>`;
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
// --- Recipes -----------------------------------------------------------------
|
||||||
|
function loadRecipes() {
|
||||||
|
const el = document.getElementById("recipes-list");
|
||||||
|
api("/recipes")
|
||||||
|
.then((data) => {
|
||||||
|
const recipes = data.recipes || [];
|
||||||
|
if (!recipes.length) {
|
||||||
|
el.innerHTML = '<p class="hint">No recipes in Grocy yet.</p>';
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
el.innerHTML = recipes
|
||||||
|
.map((r) => {
|
||||||
|
const badge = r.fulfilled === true ? "✓ can make now" : r.fulfilled === false ? "missing items" : "";
|
||||||
|
return `<div class="card">
|
||||||
|
<span class="card-name">${escapeHtml(r.name || "")}</span>
|
||||||
|
<span class="card-due">${escapeHtml(badge)}</span>
|
||||||
|
</div>`;
|
||||||
|
})
|
||||||
|
.join("");
|
||||||
|
})
|
||||||
|
.catch((err) => {
|
||||||
|
el.innerHTML = `<p class="error">Could not load recipes: ${escapeHtml(err.message)}</p>`;
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
function escapeHtml(s) {
|
||||||
|
return String(s).replace(/[&<>"']/g, (c) => ({ "&": "&", "<": "<", ">": ">", '"': """, "'": "'" }[c]));
|
||||||
|
}
|
||||||
|
|
||||||
|
// Start on whatever tab is active (default: scan).
|
||||||
|
resetScan();
|
||||||
|
|
@ -0,0 +1,75 @@
|
||||||
|
<!doctype html>
|
||||||
|
<html lang="en">
|
||||||
|
<head>
|
||||||
|
<meta charset="utf-8">
|
||||||
|
<meta name="viewport" content="width=device-width, initial-scale=1, user-scalable=no">
|
||||||
|
<title>Pantry</title>
|
||||||
|
<link rel="stylesheet" href="style.css">
|
||||||
|
</head>
|
||||||
|
<body>
|
||||||
|
<!--
|
||||||
|
pantry-vision's frontend (docs/project-plan.md Phase 17). A single static page, no
|
||||||
|
build step, no framework — same "vendored, dependency-free" choice as the digest/
|
||||||
|
admin canvas SDKs. Config (the pantry-vision API URL and its bearer token) arrives
|
||||||
|
as URL query params from the kitchen-display kiosk's own launch command
|
||||||
|
(hosts/kitchen-display/configs/sway/pantry-kiosk), never baked into this file, so
|
||||||
|
this stays a plain static asset with no secret in git — see app.js's top comment.
|
||||||
|
-->
|
||||||
|
<nav id="tabs">
|
||||||
|
<button class="tab active" data-tab="scan">📷<span>Scan</span></button>
|
||||||
|
<button class="tab" data-tab="inventory">🧊<span>Inventory</span></button>
|
||||||
|
<button class="tab" data-tab="recipes">🍳<span>Recipes</span></button>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
<main>
|
||||||
|
<section id="scan" class="panel active">
|
||||||
|
<div id="scan-camera">
|
||||||
|
<video id="camera-preview" autoplay playsinline muted></video>
|
||||||
|
<button id="capture-btn" class="big-btn">Capture</button>
|
||||||
|
<p id="camera-error" class="error" hidden></p>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div id="scan-result" hidden>
|
||||||
|
<canvas id="captured-frame"></canvas>
|
||||||
|
<form id="confirm-form">
|
||||||
|
<label>Name <input id="f-name" type="text" required></label>
|
||||||
|
<label>Category
|
||||||
|
<select id="f-category">
|
||||||
|
<option value="produce">Produce</option>
|
||||||
|
<option value="dairy">Dairy</option>
|
||||||
|
<option value="meat">Meat</option>
|
||||||
|
<option value="frozen">Frozen</option>
|
||||||
|
<option value="pantry">Pantry</option>
|
||||||
|
<option value="bakery">Bakery</option>
|
||||||
|
<option value="beverage">Beverage</option>
|
||||||
|
<option value="other">Other</option>
|
||||||
|
</select>
|
||||||
|
</label>
|
||||||
|
<label>Best before <input id="f-date" type="date" required></label>
|
||||||
|
<label>Quantity <input id="f-quantity" type="number" min="1" step="1" value="1"></label>
|
||||||
|
<p id="f-confidence" class="hint"></p>
|
||||||
|
<div class="form-actions">
|
||||||
|
<button type="button" id="retake-btn" class="big-btn secondary">Retake</button>
|
||||||
|
<button type="submit" id="confirm-btn" class="big-btn primary">Confirm & add</button>
|
||||||
|
</div>
|
||||||
|
<p id="confirm-status" class="hint"></p>
|
||||||
|
</form>
|
||||||
|
</div>
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<section id="inventory" class="panel">
|
||||||
|
<div id="inventory-list" class="card-list">
|
||||||
|
<p class="hint">Loading…</p>
|
||||||
|
</div>
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<section id="recipes" class="panel">
|
||||||
|
<div id="recipes-list" class="card-list">
|
||||||
|
<p class="hint">Loading…</p>
|
||||||
|
</div>
|
||||||
|
</section>
|
||||||
|
</main>
|
||||||
|
|
||||||
|
<script src="app.js"></script>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
|
|
@ -0,0 +1,192 @@
|
||||||
|
/* pantry-vision frontend styling (docs/project-plan.md Phase 17). Touch targets are
|
||||||
|
* deliberately large — this is a kitchen-counter finger interface. */
|
||||||
|
|
||||||
|
* {
|
||||||
|
box-sizing: border-box;
|
||||||
|
}
|
||||||
|
|
||||||
|
html, body {
|
||||||
|
margin: 0;
|
||||||
|
height: 100%;
|
||||||
|
background: #101014;
|
||||||
|
color: #e8e8ec;
|
||||||
|
font-family: sans-serif;
|
||||||
|
overflow: hidden;
|
||||||
|
}
|
||||||
|
|
||||||
|
#tabs {
|
||||||
|
display: flex;
|
||||||
|
height: 84px;
|
||||||
|
background: rgba(16, 16, 20, 0.96);
|
||||||
|
border-bottom: 1px solid rgba(255, 255, 255, 0.08);
|
||||||
|
}
|
||||||
|
|
||||||
|
.tab {
|
||||||
|
flex: 1;
|
||||||
|
display: flex;
|
||||||
|
flex-direction: column;
|
||||||
|
align-items: center;
|
||||||
|
justify-content: center;
|
||||||
|
gap: 2px;
|
||||||
|
background: transparent;
|
||||||
|
border: none;
|
||||||
|
color: #9a9aa6;
|
||||||
|
font-size: 26px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.tab span {
|
||||||
|
font-size: 13px;
|
||||||
|
font-weight: 500;
|
||||||
|
}
|
||||||
|
|
||||||
|
.tab.active {
|
||||||
|
color: #e8e8ec;
|
||||||
|
background: rgba(110, 168, 254, 0.16);
|
||||||
|
}
|
||||||
|
|
||||||
|
main {
|
||||||
|
height: calc(100% - 84px);
|
||||||
|
overflow-y: auto;
|
||||||
|
}
|
||||||
|
|
||||||
|
.panel {
|
||||||
|
display: none;
|
||||||
|
padding: 16px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.panel.active {
|
||||||
|
display: block;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* --- Scan ------------------------------------------------------------------- */
|
||||||
|
#scan-camera, #scan-result {
|
||||||
|
display: flex;
|
||||||
|
flex-direction: column;
|
||||||
|
align-items: center;
|
||||||
|
gap: 16px;
|
||||||
|
}
|
||||||
|
|
||||||
|
#camera-preview, #captured-frame {
|
||||||
|
width: 100%;
|
||||||
|
max-width: 640px;
|
||||||
|
border-radius: 12px;
|
||||||
|
background: #000;
|
||||||
|
}
|
||||||
|
|
||||||
|
.big-btn {
|
||||||
|
min-height: 64px;
|
||||||
|
min-width: 200px;
|
||||||
|
font-size: 18px;
|
||||||
|
font-weight: 600;
|
||||||
|
border-radius: 12px;
|
||||||
|
border: none;
|
||||||
|
color: #101014;
|
||||||
|
background: #6ea8fe;
|
||||||
|
}
|
||||||
|
|
||||||
|
.big-btn.secondary {
|
||||||
|
background: rgba(255, 255, 255, 0.14);
|
||||||
|
color: #e8e8ec;
|
||||||
|
}
|
||||||
|
|
||||||
|
.big-btn.primary {
|
||||||
|
background: #6ea8fe;
|
||||||
|
}
|
||||||
|
|
||||||
|
#confirm-form {
|
||||||
|
width: 100%;
|
||||||
|
max-width: 640px;
|
||||||
|
display: flex;
|
||||||
|
flex-direction: column;
|
||||||
|
gap: 12px;
|
||||||
|
}
|
||||||
|
|
||||||
|
#confirm-form label {
|
||||||
|
display: flex;
|
||||||
|
flex-direction: column;
|
||||||
|
gap: 4px;
|
||||||
|
font-size: 14px;
|
||||||
|
color: #9a9aa6;
|
||||||
|
}
|
||||||
|
|
||||||
|
#confirm-form input, #confirm-form select {
|
||||||
|
min-height: 48px;
|
||||||
|
font-size: 16px;
|
||||||
|
padding: 8px 12px;
|
||||||
|
border-radius: 8px;
|
||||||
|
border: 1px solid rgba(255, 255, 255, 0.16);
|
||||||
|
background: #1a1a20;
|
||||||
|
color: #e8e8ec;
|
||||||
|
}
|
||||||
|
|
||||||
|
.form-actions {
|
||||||
|
display: flex;
|
||||||
|
gap: 12px;
|
||||||
|
justify-content: center;
|
||||||
|
}
|
||||||
|
|
||||||
|
.hint {
|
||||||
|
color: #9a9aa6;
|
||||||
|
font-size: 14px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.error {
|
||||||
|
color: #ff8080;
|
||||||
|
font-size: 14px;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* --- Cards (inventory + recipes) --------------------------------------------- */
|
||||||
|
.card-list {
|
||||||
|
display: flex;
|
||||||
|
flex-direction: column;
|
||||||
|
gap: 8px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.card {
|
||||||
|
display: flex;
|
||||||
|
align-items: center;
|
||||||
|
gap: 12px;
|
||||||
|
min-height: 56px;
|
||||||
|
padding: 8px 16px;
|
||||||
|
border-radius: 10px;
|
||||||
|
background: rgba(255, 255, 255, 0.06);
|
||||||
|
}
|
||||||
|
|
||||||
|
.card-name {
|
||||||
|
flex: 1;
|
||||||
|
font-size: 16px;
|
||||||
|
font-weight: 500;
|
||||||
|
}
|
||||||
|
|
||||||
|
.card-amount {
|
||||||
|
color: #9a9aa6;
|
||||||
|
font-size: 14px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.card-due {
|
||||||
|
font-size: 14px;
|
||||||
|
color: #9a9aa6;
|
||||||
|
min-width: 110px;
|
||||||
|
text-align: right;
|
||||||
|
}
|
||||||
|
|
||||||
|
.urgent-week {
|
||||||
|
background: rgba(255, 200, 80, 0.10);
|
||||||
|
}
|
||||||
|
.urgent-week .card-due {
|
||||||
|
color: #ffc850;
|
||||||
|
}
|
||||||
|
|
||||||
|
.urgent-soon {
|
||||||
|
background: rgba(255, 140, 80, 0.14);
|
||||||
|
}
|
||||||
|
.urgent-soon .card-due {
|
||||||
|
color: #ff8c50;
|
||||||
|
}
|
||||||
|
|
||||||
|
.urgent-expired {
|
||||||
|
background: rgba(255, 90, 90, 0.16);
|
||||||
|
}
|
||||||
|
.urgent-expired .card-due {
|
||||||
|
color: #ff5a5a;
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,48 @@
|
||||||
|
# pantry-vision configuration template.
|
||||||
|
#
|
||||||
|
# Copy this to the container host as (for example)
|
||||||
|
# /opt/smart-home/pantry-vision/pantry-vision.env, fill in real values, and chmod 600
|
||||||
|
# it. Same never-commit handling as admin-canvas.env / digest-engine.env.
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Auth — required. pantry-vision fails closed (rejects every request) while this is
|
||||||
|
# empty. Unlike admin-canvas's token (which only ever needs to be known by Home
|
||||||
|
# Assistant), this one also has to be baked into the kitchen-display kiosk's own
|
||||||
|
# build config (hosts/kitchen-display/scripts/build-kitchen-display-iso.sh), since
|
||||||
|
# the kiosk's frontend calls this API directly, not through HA. Generate one with:
|
||||||
|
# openssl rand -hex 32
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
PANTRY_VISION_TOKEN=
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Vision LLM (Ollama, the same LLM host as digest-engine/Assist)
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
OLLAMA_HOST=http://llm-host:11434
|
||||||
|
# MUST be a vision-capable model, and MUST already be pulled on the LLM host
|
||||||
|
# (`ollama pull llava`, or whatever you pick) — plain text models like the
|
||||||
|
# qwen2.5:14b-instruct used elsewhere in this project cannot see images at all.
|
||||||
|
# Not verified against a real pull/run — see README.md.
|
||||||
|
OLLAMA_VISION_MODEL=llava
|
||||||
|
OLLAMA_TIMEOUT=120
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Grocy — already running as the always-on `grocy` container from
|
||||||
|
# hosts/container-host/scripts/setup-container-host.sh, reachable on the compose
|
||||||
|
# network at this internal name/port regardless of ENABLE_PANTRY_VISION.
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
GROCY_URL=http://grocy:80
|
||||||
|
# Settings -> Manage API keys, inside Grocy's own UI. Required for /confirm (writes);
|
||||||
|
# /inventory and /recipes (reads) use the same key.
|
||||||
|
GROCY_API_KEY=
|
||||||
|
# VERIFY against your own Grocy instance (Settings -> Locations / Quantity units) —
|
||||||
|
# these are fresh-install defaults ("Default" location, "Piece" unit), not guaranteed
|
||||||
|
# to match a Grocy that's already been customised.
|
||||||
|
GROCY_DEFAULT_LOCATION_ID=1
|
||||||
|
GROCY_DEFAULT_QU_ID=2
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Run behaviour
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
PANTRY_VISION_PORT=8095
|
||||||
|
PANTRY_VISION_MAX_IMAGE_MB=15
|
||||||
|
LOG_LEVEL=INFO
|
||||||
|
|
@ -0,0 +1,391 @@
|
||||||
|
"""pantry-vision — camera-driven grocery cataloguing for the kitchen display, from
|
||||||
|
docs/project-plan.md Phase 17.
|
||||||
|
|
||||||
|
The workflow this exists for: come home, put down the shopping bag, hold one item up
|
||||||
|
to the kitchen display's camera, the system proposes what it is and how long it keeps,
|
||||||
|
you confirm (editing anything it got wrong) before it's written anywhere, then put the
|
||||||
|
item away. The display also shows the resulting inventory sorted by what expires
|
||||||
|
soonest, and Grocy's recipes.
|
||||||
|
|
||||||
|
SECURITY BOUNDARY, same shape as admin-canvas/server.py's, with one deliberate
|
||||||
|
difference: this service DOES get a published port (see the README and
|
||||||
|
setup-container-host.sh's PANTRY_VISION_PORT), because unlike admin-canvas — which is
|
||||||
|
only ever called by Home Assistant on the compose network — this one is called
|
||||||
|
directly by the kitchen-display kiosk, a separate physical device on the LAN. Every
|
||||||
|
request still requires the bearer token below; an unset token fails closed (rejects
|
||||||
|
everything), never "auth optional".
|
||||||
|
|
||||||
|
Four endpoints, all token-gated:
|
||||||
|
- POST /identify raw image bytes -> Ollama vision model -> a *proposal*
|
||||||
|
- POST /confirm JSON, human-reviewed/edited -> writes to Grocy stock
|
||||||
|
- GET /inventory proxies Grocy stock, sorted by soonest-expiring first
|
||||||
|
- GET /recipes proxies Grocy recipes (+ fulfillment, best-effort)
|
||||||
|
|
||||||
|
/identify NEVER writes anything by itself — same "propose, never auto-commit" rule
|
||||||
|
this project already applies to identity-merge confirmation (see the Identity store
|
||||||
|
row in docs/project-plan.md §2): the vision model's guess is shown to the person
|
||||||
|
standing at the display, who can edit the name/category/date/location before /confirm
|
||||||
|
is ever called. A wrong guess costs a tap to fix, not a wrong fact silently written
|
||||||
|
into the household's inventory.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import base64
|
||||||
|
import json
|
||||||
|
import logging
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import sys
|
||||||
|
import urllib.error
|
||||||
|
import urllib.request
|
||||||
|
from datetime import date, datetime
|
||||||
|
from http import HTTPStatus
|
||||||
|
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||||
|
from urllib.parse import urlsplit
|
||||||
|
|
||||||
|
LOG = logging.getLogger("pantry-vision")
|
||||||
|
|
||||||
|
TOKEN = os.environ.get("PANTRY_VISION_TOKEN", "")
|
||||||
|
|
||||||
|
OLLAMA_HOST = os.environ.get("OLLAMA_HOST", "http://llm-host:11434").rstrip("/")
|
||||||
|
# TBD/unverified — see README.md's "Pick a vision model" section. Not every Ollama
|
||||||
|
# model can see images; this must be one that can (llava, qwen2.5vl, etc.) and must
|
||||||
|
# actually be pulled on the LLM host (`ollama pull <name>`) before /identify works.
|
||||||
|
OLLAMA_VISION_MODEL = os.environ.get("OLLAMA_VISION_MODEL", "llava")
|
||||||
|
OLLAMA_TIMEOUT = float(os.environ.get("OLLAMA_TIMEOUT", "120"))
|
||||||
|
|
||||||
|
GROCY_URL = os.environ.get("GROCY_URL", "http://grocy:80").rstrip("/")
|
||||||
|
GROCY_API_KEY = os.environ.get("GROCY_API_KEY", "")
|
||||||
|
# Fresh-install defaults ("Default" location, "Piece" quantity unit) — VERIFY against
|
||||||
|
# your own Grocy instance (Settings -> Locations / Quantity units) and override if
|
||||||
|
# they differ. Same "don't build against a guess" flag as every other cross-service ID
|
||||||
|
# in this project (ADMIN_WEB_URL placeholders, weather_entity_id, etc.).
|
||||||
|
GROCY_DEFAULT_LOCATION_ID = os.environ.get("GROCY_DEFAULT_LOCATION_ID", "1")
|
||||||
|
GROCY_DEFAULT_QU_ID = os.environ.get("GROCY_DEFAULT_QU_ID", "2")
|
||||||
|
|
||||||
|
MAX_IMAGE_BYTES = int(os.environ.get("PANTRY_VISION_MAX_IMAGE_MB", "15")) * 1024 * 1024
|
||||||
|
MAX_JSON_BYTES = 64 * 1024
|
||||||
|
|
||||||
|
_IDENTIFY_PROMPT = """You are looking at a single grocery item held up to a kitchen \
|
||||||
|
camera, on a plain background. Identify it and answer ONLY with a JSON object, no \
|
||||||
|
other text, in exactly this shape:
|
||||||
|
{"name": "<short product name>", "category": "<produce|dairy|meat|frozen|pantry|bakery|beverage|other>", \
|
||||||
|
"estimated_shelf_life_days": <integer, typical days from today until it goes bad, \
|
||||||
|
assuming normal fridge/pantry storage for its category>, "confidence": "<high|medium|low>"}
|
||||||
|
If you cannot identify the item, still return the JSON shape with your best guess and \
|
||||||
|
"confidence": "low"."""
|
||||||
|
|
||||||
|
_JSON_OBJECT_RE = re.compile(r"\{.*\}", re.DOTALL)
|
||||||
|
|
||||||
|
|
||||||
|
def _http_json(method: str, url: str, payload: dict | None = None, headers: dict | None = None,
|
||||||
|
timeout: float = 30.0) -> dict:
|
||||||
|
data = json.dumps(payload).encode("utf-8") if payload is not None else None
|
||||||
|
req = urllib.request.Request(url, data=data, method=method)
|
||||||
|
req.add_header("Content-Type", "application/json")
|
||||||
|
for key, value in (headers or {}).items():
|
||||||
|
req.add_header(key, value)
|
||||||
|
with urllib.request.urlopen(req, timeout=timeout) as resp:
|
||||||
|
body = resp.read()
|
||||||
|
return json.loads(body) if body else {}
|
||||||
|
|
||||||
|
|
||||||
|
def _grocy_headers() -> dict:
|
||||||
|
return {"GROCY-API-KEY": GROCY_API_KEY}
|
||||||
|
|
||||||
|
|
||||||
|
def _grocy_get(path: str):
|
||||||
|
return _http_json("GET", f"{GROCY_URL}{path}", headers=_grocy_headers(), timeout=15)
|
||||||
|
|
||||||
|
|
||||||
|
def _grocy_post(path: str, payload: dict):
|
||||||
|
return _http_json("POST", f"{GROCY_URL}{path}", payload=payload, headers=_grocy_headers(), timeout=15)
|
||||||
|
|
||||||
|
|
||||||
|
def _identify_via_ollama(image_bytes: bytes) -> dict:
|
||||||
|
"""Calls the vision model and returns a best-effort proposal — degrades to a
|
||||||
|
low-confidence placeholder rather than raising, mirroring digest-engine's
|
||||||
|
llm_client._fallback_document: a broken/blank result is worse than one flagged
|
||||||
|
as unreliable that a human has to review anyway (which they always do — see the
|
||||||
|
module docstring, /identify never writes anything by itself).
|
||||||
|
"""
|
||||||
|
payload = {
|
||||||
|
"model": OLLAMA_VISION_MODEL,
|
||||||
|
"prompt": _IDENTIFY_PROMPT,
|
||||||
|
"images": [base64.b64encode(image_bytes).decode("ascii")],
|
||||||
|
"stream": False,
|
||||||
|
"options": {"temperature": 0.2},
|
||||||
|
}
|
||||||
|
try:
|
||||||
|
result = _http_json("POST", f"{OLLAMA_HOST}/api/generate", payload=payload, timeout=OLLAMA_TIMEOUT)
|
||||||
|
text = result.get("response", "")
|
||||||
|
except Exception:
|
||||||
|
LOG.warning("pantry-vision: Ollama call failed", exc_info=True)
|
||||||
|
return {
|
||||||
|
"name": "Unknown item",
|
||||||
|
"category": "other",
|
||||||
|
"estimated_shelf_life_days": 7,
|
||||||
|
"confidence": "low",
|
||||||
|
"note": f"Could not reach the vision model at {OLLAMA_HOST} — filled in manually.",
|
||||||
|
}
|
||||||
|
|
||||||
|
match = _JSON_OBJECT_RE.search(text)
|
||||||
|
if match:
|
||||||
|
try:
|
||||||
|
proposal = json.loads(match.group(0))
|
||||||
|
proposal.setdefault("name", "Unknown item")
|
||||||
|
proposal.setdefault("category", "other")
|
||||||
|
proposal.setdefault("estimated_shelf_life_days", 7)
|
||||||
|
proposal.setdefault("confidence", "low")
|
||||||
|
return proposal
|
||||||
|
except ValueError:
|
||||||
|
pass
|
||||||
|
|
||||||
|
LOG.warning("pantry-vision: could not parse a JSON proposal out of the model's response")
|
||||||
|
return {
|
||||||
|
"name": "Unknown item",
|
||||||
|
"category": "other",
|
||||||
|
"estimated_shelf_life_days": 7,
|
||||||
|
"confidence": "low",
|
||||||
|
"note": "The vision model's response wasn't valid JSON — filled in manually.",
|
||||||
|
"raw_response": text[:500],
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _find_or_create_product(name: str) -> int:
|
||||||
|
"""Best-effort product lookup/create against Grocy's object API.
|
||||||
|
|
||||||
|
VERIFY against a real Grocy instance — this assumes GET /api/objects/products
|
||||||
|
returns a flat list of {id, name, ...} and that POST to the same path with just
|
||||||
|
name/location_id/qu_id_purchase/qu_id_stock is enough to create a minimal product.
|
||||||
|
Grocy's actual required-field set depends on its own settings and has not been
|
||||||
|
checked against a live instance; see README.md's verification list.
|
||||||
|
"""
|
||||||
|
products = _grocy_get("/api/objects/products")
|
||||||
|
needle = name.strip().lower()
|
||||||
|
for product in products:
|
||||||
|
if str(product.get("name", "")).strip().lower() == needle:
|
||||||
|
return int(product["id"])
|
||||||
|
|
||||||
|
created = _grocy_post(
|
||||||
|
"/api/objects/products",
|
||||||
|
{
|
||||||
|
"name": name,
|
||||||
|
"location_id": int(GROCY_DEFAULT_LOCATION_ID),
|
||||||
|
"qu_id_purchase": int(GROCY_DEFAULT_QU_ID),
|
||||||
|
"qu_id_stock": int(GROCY_DEFAULT_QU_ID),
|
||||||
|
},
|
||||||
|
)
|
||||||
|
return int(created["created_object_id"])
|
||||||
|
|
||||||
|
|
||||||
|
def _add_to_stock(product_id: int, amount: float, best_before_date: str) -> dict:
|
||||||
|
return _grocy_post(
|
||||||
|
f"/api/stock/products/{product_id}/add",
|
||||||
|
{"amount": amount, "best_before_date": best_before_date, "transaction_type": "purchase"},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _days_left(best_before: str | None) -> int | None:
|
||||||
|
if not best_before:
|
||||||
|
return None
|
||||||
|
try:
|
||||||
|
due = datetime.strptime(best_before, "%Y-%m-%d").date()
|
||||||
|
except ValueError:
|
||||||
|
return None
|
||||||
|
return (due - date.today()).days
|
||||||
|
|
||||||
|
|
||||||
|
class Handler(BaseHTTPRequestHandler):
|
||||||
|
server_version = "pantry-vision/1"
|
||||||
|
|
||||||
|
def log_message(self, format, *args): # noqa: A002
|
||||||
|
LOG.info("%s - %s", self.address_string(), format % args)
|
||||||
|
|
||||||
|
def _authorized(self) -> bool:
|
||||||
|
if not TOKEN:
|
||||||
|
return False
|
||||||
|
return self.headers.get("Authorization", "") == f"Bearer {TOKEN}"
|
||||||
|
|
||||||
|
def _respond(self, status: HTTPStatus, payload) -> None:
|
||||||
|
body = json.dumps(payload).encode("utf-8")
|
||||||
|
self.send_response(status)
|
||||||
|
self.send_header("Content-Type", "application/json")
|
||||||
|
self.send_header("Content-Length", str(len(body)))
|
||||||
|
self.send_header("Access-Control-Allow-Origin", "*")
|
||||||
|
self.end_headers()
|
||||||
|
self.wfile.write(body)
|
||||||
|
|
||||||
|
def _read_body(self, max_bytes: int) -> bytes:
|
||||||
|
try:
|
||||||
|
length = int(self.headers.get("Content-Length") or 0)
|
||||||
|
except ValueError:
|
||||||
|
raise ValueError("missing or invalid Content-Length") from None
|
||||||
|
if length <= 0:
|
||||||
|
return b""
|
||||||
|
if length > max_bytes:
|
||||||
|
raise ValueError(f"body too large ({length} > {max_bytes} bytes)")
|
||||||
|
return self.rfile.read(length)
|
||||||
|
|
||||||
|
def do_OPTIONS(self): # noqa: N802
|
||||||
|
# The frontend (pantry-web, a separate origin/port) fetches this API directly
|
||||||
|
# from the browser, so a plain CORS preflight has to succeed.
|
||||||
|
self.send_response(HTTPStatus.NO_CONTENT)
|
||||||
|
self.send_header("Access-Control-Allow-Origin", "*")
|
||||||
|
self.send_header("Access-Control-Allow-Headers", "Authorization, Content-Type")
|
||||||
|
self.send_header("Access-Control-Allow-Methods", "GET, POST, OPTIONS")
|
||||||
|
self.end_headers()
|
||||||
|
|
||||||
|
def do_GET(self): # noqa: N802
|
||||||
|
if not self._authorized():
|
||||||
|
self._respond(HTTPStatus.UNAUTHORIZED, {"error": "missing or invalid bearer token"})
|
||||||
|
return
|
||||||
|
path = urlsplit(self.path).path
|
||||||
|
if path == "/inventory":
|
||||||
|
self._handle_inventory()
|
||||||
|
elif path == "/recipes":
|
||||||
|
self._handle_recipes()
|
||||||
|
else:
|
||||||
|
self._respond(HTTPStatus.NOT_FOUND, {"error": "no such endpoint"})
|
||||||
|
|
||||||
|
def do_POST(self): # noqa: N802
|
||||||
|
if not self._authorized():
|
||||||
|
self._respond(HTTPStatus.UNAUTHORIZED, {"error": "missing or invalid bearer token"})
|
||||||
|
return
|
||||||
|
path = urlsplit(self.path).path
|
||||||
|
if path == "/identify":
|
||||||
|
self._handle_identify()
|
||||||
|
elif path == "/confirm":
|
||||||
|
self._handle_confirm()
|
||||||
|
else:
|
||||||
|
self._respond(HTTPStatus.NOT_FOUND, {"error": "no such endpoint"})
|
||||||
|
|
||||||
|
def _handle_identify(self) -> None:
|
||||||
|
try:
|
||||||
|
image = self._read_body(MAX_IMAGE_BYTES)
|
||||||
|
except ValueError as exc:
|
||||||
|
self._respond(HTTPStatus.REQUEST_ENTITY_TOO_LARGE, {"error": str(exc)})
|
||||||
|
return
|
||||||
|
if not image:
|
||||||
|
self._respond(HTTPStatus.BAD_REQUEST, {"error": "empty image body"})
|
||||||
|
return
|
||||||
|
|
||||||
|
proposal = _identify_via_ollama(image)
|
||||||
|
self._respond(HTTPStatus.OK, proposal)
|
||||||
|
|
||||||
|
def _handle_confirm(self) -> None:
|
||||||
|
try:
|
||||||
|
raw = self._read_body(MAX_JSON_BYTES)
|
||||||
|
payload = json.loads(raw or b"{}")
|
||||||
|
except (ValueError, json.JSONDecodeError) as exc:
|
||||||
|
self._respond(HTTPStatus.BAD_REQUEST, {"error": f"bad request body: {exc}"})
|
||||||
|
return
|
||||||
|
|
||||||
|
name = str(payload.get("name", "")).strip()
|
||||||
|
best_before = str(payload.get("best_before_date", "")).strip()
|
||||||
|
amount = float(payload.get("quantity") or 1)
|
||||||
|
|
||||||
|
if not name:
|
||||||
|
self._respond(HTTPStatus.BAD_REQUEST, {"error": "'name' is required"})
|
||||||
|
return
|
||||||
|
if not re.match(r"^\d{4}-\d{2}-\d{2}$", best_before):
|
||||||
|
self._respond(HTTPStatus.BAD_REQUEST, {"error": "'best_before_date' must be YYYY-MM-DD"})
|
||||||
|
return
|
||||||
|
if not GROCY_API_KEY:
|
||||||
|
self._respond(HTTPStatus.SERVICE_UNAVAILABLE, {"error": "GROCY_API_KEY is not configured"})
|
||||||
|
return
|
||||||
|
|
||||||
|
try:
|
||||||
|
product_id = _find_or_create_product(name)
|
||||||
|
_add_to_stock(product_id, amount, best_before)
|
||||||
|
except (urllib.error.URLError, urllib.error.HTTPError, KeyError, ValueError) as exc:
|
||||||
|
LOG.warning("pantry-vision: Grocy write failed", exc_info=True)
|
||||||
|
self._respond(HTTPStatus.BAD_GATEWAY, {"error": f"Grocy write failed: {exc}"})
|
||||||
|
return
|
||||||
|
|
||||||
|
LOG.info("pantry-vision: added %s x%s (best before %s) to Grocy", name, amount, best_before)
|
||||||
|
self._respond(HTTPStatus.OK, {"ok": True, "product_id": product_id})
|
||||||
|
|
||||||
|
def _handle_inventory(self) -> None:
|
||||||
|
try:
|
||||||
|
stock = _grocy_get("/api/stock")
|
||||||
|
except (urllib.error.URLError, urllib.error.HTTPError) as exc:
|
||||||
|
LOG.warning("pantry-vision: Grocy /api/stock unreachable", exc_info=True)
|
||||||
|
self._respond(HTTPStatus.BAD_GATEWAY, {"error": f"Grocy unreachable: {exc}"})
|
||||||
|
return
|
||||||
|
|
||||||
|
# VERIFY: assumes each stock row carries a nested "product" object with at
|
||||||
|
# least a "name" (Grocy's documented default `?embed` behaviour varies by
|
||||||
|
# version) — falls back to "product_id" as the display name if not, rather
|
||||||
|
# than dropping the row, same "degrade, don't blank" rule as everywhere else
|
||||||
|
# in this project's renderers.
|
||||||
|
items = []
|
||||||
|
for row in stock if isinstance(stock, list) else []:
|
||||||
|
product = row.get("product") or {}
|
||||||
|
name = product.get("name") or f"Product #{row.get('product_id')}"
|
||||||
|
best_before = row.get("best_before_date")
|
||||||
|
items.append(
|
||||||
|
{
|
||||||
|
"name": name,
|
||||||
|
"amount": row.get("amount"),
|
||||||
|
"best_before_date": best_before,
|
||||||
|
"days_left": _days_left(best_before),
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
items.sort(key=lambda i: (i["days_left"] is None, i["days_left"]))
|
||||||
|
self._respond(HTTPStatus.OK, {"items": items})
|
||||||
|
|
||||||
|
def _handle_recipes(self) -> None:
|
||||||
|
try:
|
||||||
|
recipes = _grocy_get("/api/objects/recipes")
|
||||||
|
except (urllib.error.URLError, urllib.error.HTTPError) as exc:
|
||||||
|
LOG.warning("pantry-vision: Grocy recipes unreachable", exc_info=True)
|
||||||
|
self._respond(HTTPStatus.BAD_GATEWAY, {"error": f"Grocy unreachable: {exc}"})
|
||||||
|
return
|
||||||
|
|
||||||
|
results = []
|
||||||
|
for recipe in recipes if isinstance(recipes, list) else []:
|
||||||
|
entry = {"id": recipe.get("id"), "name": recipe.get("name"), "fulfilled": None}
|
||||||
|
# Best-effort only: Grocy's fulfillment endpoint needs the recipes feature
|
||||||
|
# actually set up (ingredients entered per recipe) to mean anything, and
|
||||||
|
# its exact response shape is unverified — see README.md. A failure here
|
||||||
|
# degrades to "fulfilled: null" (rendered as "unknown" client-side), never
|
||||||
|
# a broken page.
|
||||||
|
try:
|
||||||
|
fulfillment = _grocy_get(f"/api/recipes/{entry['id']}/fulfillment")
|
||||||
|
entry["fulfilled"] = bool(fulfillment.get("need_fulfilled"))
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
results.append(entry)
|
||||||
|
|
||||||
|
self._respond(HTTPStatus.OK, {"recipes": results})
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
logging.basicConfig(
|
||||||
|
level=os.environ.get("LOG_LEVEL", "INFO").upper(),
|
||||||
|
format="%(asctime)s %(levelname)-7s %(name)s: %(message)s",
|
||||||
|
)
|
||||||
|
|
||||||
|
if not TOKEN:
|
||||||
|
LOG.error(
|
||||||
|
"PANTRY_VISION_TOKEN is not set — every request will be rejected until it is. "
|
||||||
|
"See pantry-vision.env.example."
|
||||||
|
)
|
||||||
|
if not GROCY_API_KEY:
|
||||||
|
LOG.warning("GROCY_API_KEY is not set — /confirm will fail until it is configured.")
|
||||||
|
|
||||||
|
port = int(os.environ.get("PANTRY_VISION_PORT", "8095"))
|
||||||
|
server = ThreadingHTTPServer(("0.0.0.0", port), Handler)
|
||||||
|
LOG.info("pantry-vision listening on :%d (Ollama: %s, Grocy: %s)", port, OLLAMA_HOST, GROCY_URL)
|
||||||
|
try:
|
||||||
|
server.serve_forever()
|
||||||
|
except KeyboardInterrupt:
|
||||||
|
pass
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main())
|
||||||
Loading…
Reference in New Issue