Add Phase 16 touch panel and Phase 17 kitchen/fridge display endpoints

hosts/touch-panel/: a touch-driven Sway kiosk with a full Spotify GUI client
(Flathub), a dedicated Home Assistant Chromium kiosk window, a general web
browser, an always-on eww touch dock for app switching, an on-screen keyboard
(wvkbd), and touchpanel-agent giving HA/the LLM the same MQTT-mediated control
the thin client has.

hosts/kitchen-display/ + pantry-vision/: a camera-vision grocery cataloguing
endpoint for the fridge/pantry. The kiosk's browser captures a photo via
getUserMedia and sends it to the new pantry-vision service, which asks an
Ollama vision model to identify the item and estimate shelf life; the
proposal is always human-confirmed before being written into Grocy stock.
The same display shows inventory sorted by soonest-to-expire and Grocy's
recipes.

Wires ENABLE_PANTRY_VISION into setup-container-host.sh and documents both
phases in docs/project-plan.md and README.md, matching this project's
existing per-host hardware/software/guardrail/open-decision conventions.
Nothing here has been run against real hardware, a real camera, or a real
Grocy/vision-model instance — see each new README's verification list.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6CrKjW3yVptUnjG35HjC1
digest-per-person-and-agendas
Amir Alexander Abdelbaki 2026-07-30 11:33:08 +02:00
parent bfca1c7eb0
commit f3f870c4b4
57 changed files with 4305 additions and 4 deletions

View File

@ -3,7 +3,8 @@
Local-first, open-source smart home: Home Assistant + Zigbee + RuView (CSI presence) Local-first, open-source smart home: Home Assistant + Zigbee + RuView (CSI presence)
+ Bermuda (BLE identity) + local LLM (Ollama) + Frigate (peephole face recognition) + Bermuda (BLE identity) + local LLM (Ollama) + Frigate (peephole face recognition)
+ Grocy (kitchen kiosk) + Nextcloud calendar sync + a Sway thin-client media station + Grocy (kitchen kiosk) + Nextcloud calendar sync + a Sway thin-client media station
+ a quarter-daily LLM-generated digest. + a Sway touch panel + a camera-vision kitchen/fridge display + a quarter-daily
LLM-generated digest.
See [`docs/project-plan.md`](docs/project-plan.md) for the full hardware list, See [`docs/project-plan.md`](docs/project-plan.md) for the full hardware list,
software stack, and phased implementation plan. software stack, and phased implementation plan.
@ -21,6 +22,13 @@ hosts/
thin-client/ Sway kiosk/media-station ISO (live-build) + thinclient-agent thin-client/ Sway kiosk/media-station ISO (live-build) + thinclient-agent
audio-endpoint/ Headless Spotify Connect appliance for rooms with no thin audio-endpoint/ Headless Spotify Connect appliance for rooms with no thin
client — arm64 (rpi-image-gen) + amd64 (live-build) images client — arm64 (rpi-image-gen) + amd64 (live-build) images
touch-panel/ Touch-driven Sway panel: full Spotify GUI, a dedicated
Home Assistant Chromium kiosk window, and a general web
browser, switched via an always-on touch dock or by HA/
the local LLM over MQTT
kitchen-display/ Single-purpose Sway kiosk for the fridge/pantry: one
Chromium window showing pantry-vision's Scan/Inventory/
Recipes frontend, camera capture via the browser itself
firmware/ firmware/
ruview/ RuView ESP32-S3 CSI presence node configs ruview/ RuView ESP32-S3 CSI presence node configs
esphome-ble-proxy/ ESPHome configs for Bermuda BLE proxy nodes esphome-ble-proxy/ ESPHome configs for Bermuda BLE proxy nodes
@ -35,6 +43,11 @@ admin-canvas/ On-demand sys-admin-llm display surface for the thin
clients: stats/graphics/media, pushed on demand clients: stats/graphics/media, pushed on demand
rather than on a schedule (write API + admin-web rather than on a schedule (write API + admin-web
static serving) static serving)
pantry-vision/ Kitchen-display backend: a photo held up to the camera
-> an Ollama vision-model proposal -> human-confirmed
write into Grocy stock; also proxies Grocy's inventory
(soonest-expiring first) and recipes to the kiosk
frontend (write API + frontend/ static serving)
``` ```
## Status ## Status
@ -56,6 +69,8 @@ admin-canvas/ On-demand sys-admin-llm display surface for the thin
- [x] admin-canvas + admin-web (sys-admin-llm on-demand display surface for the thin clients) — built and wired into `setup-container-host.sh` (`ENABLE_ADMIN_CANVAS`, off by default); the HA-side tool/rest_command wiring and the specific entities it surfaces (e.g. power-monitoring) are still undecided, see `docs/project-plan.md` §4 - [x] admin-canvas + admin-web (sys-admin-llm on-demand display surface for the thin clients) — built and wired into `setup-container-host.sh` (`ENABLE_ADMIN_CANVAS`, off by default); the HA-side tool/rest_command wiring and the specific entities it surfaces (e.g. power-monitoring) are still undecided, see `docs/project-plan.md` §4
- [ ] ESP32-S3-Touch-LCD-1.85C-V2 voice satellite + status display (`firmware/esp32-s3-touch-lcd-1.85c/`) — ESPHome config written and passes `esphome config`, not yet flashed to real hardware; `media_player`/`weather` entity IDs still need to be chosen, see `docs/project-plan.md` §4 - [ ] ESP32-S3-Touch-LCD-1.85C-V2 voice satellite + status display (`firmware/esp32-s3-touch-lcd-1.85c/`) — ESPHome config written and passes `esphome config`, not yet flashed to real hardware; `media_player`/`weather` entity IDs still need to be chosen, see `docs/project-plan.md` §4
- [ ] Headless audio endpoint (`hosts/audio-endpoint/`) — per-room independent Spotify Connect appliance for rooms without a thin client, arm64 (Raspberry Pi + HiFiBerry Amp2, rpi-image-gen) and amd64 (mini PC + USB DAC/amp, live-build) build pipelines written, **neither built/flashed/booted on real hardware** — rpi-image-gen's exact config schema in particular is unverified, see `hosts/audio-endpoint/README.md` - [ ] Headless audio endpoint (`hosts/audio-endpoint/`) — per-room independent Spotify Connect appliance for rooms without a thin client, arm64 (Raspberry Pi + HiFiBerry Amp2, rpi-image-gen) and amd64 (mini PC + USB DAC/amp, live-build) build pipelines written, **neither built/flashed/booted on real hardware** — rpi-image-gen's exact config schema in particular is unverified, see `hosts/audio-endpoint/README.md`
- [ ] Sway touch panel (`hosts/touch-panel/`) — touch-driven Sway image: full Spotify GUI (Flathub), a dedicated Home Assistant Chromium kiosk window, a general web browser, an always-on touch dock for app switching, an on-screen keyboard (toggled manually, no auto-show), and `touchpanel-agent` (HA MQTT control, same LLM-mediated-through-HA security model as the thin client) — built, **no touch-panel hardware chosen and nothing booted on real metal**, see `hosts/touch-panel/README.md`
- [ ] Kitchen/fridge display + `pantry-vision` (`hosts/kitchen-display/`, `pantry-vision/`) — hold a grocery item up to the camera, an Ollama vision model proposes what it is and roughly how long it keeps, a human confirms (never auto-committed) before it's written into Grocy stock; the display then shows inventory sorted by soonest-to-expire and Grocy's recipes — built and wired into `setup-container-host.sh` (`ENABLE_PANTRY_VISION`, off by default), **nothing run against a real camera, vision model, or Grocy instance** — the Grocy API call shapes in particular are written from documentation only, see `pantry-vision/README.md` and `hosts/kitchen-display/README.md`
## Quick start ## Quick start

View File

@ -1,6 +1,6 @@
# AI-Managed Smart Home — Full Build Plan (v2) # AI-Managed Smart Home — Full Build Plan (v2)
Local-first, open-source stack: Home Assistant + RuView presence + Bermuda BLE identity + local LLM (Ollama) + Zigbee sensors/lighting + Frigate (peephole face recognition) + Grocy kitchen kiosk + Nextcloud calendar sync + Node-RED glue logic + a Sway thin-client media station + a quarter-daily LLM digest. Local-first, open-source stack: Home Assistant + RuView presence + Bermuda BLE identity + local LLM (Ollama) + Zigbee sensors/lighting + Frigate (peephole face recognition) + Grocy kitchen kiosk + Nextcloud calendar sync + Node-RED glue logic + a Sway thin-client media station + a Sway touch panel + a camera-vision kitchen/fridge display + a quarter-daily LLM digest.
--- ---
@ -98,6 +98,25 @@ Already covered — using your existing Haozee CC2652P USB dongle. No coordinato
| Passive bookshelf speakers (e.g. Micca MB42X G2) | ~€100/pair | Same pairing already recommended for the thin-client rooms | | Passive bookshelf speakers (e.g. Micca MB42X G2) | ~€100/pair | Same pairing already recommended for the thin-client rooms |
| *(amd64 alternative)* Spare x86 mini PC + USB DAC/amp (e.g. Fosi Audio V3, ~€130) | €0130 | No GPIO HAT involved — a USB Audio Class device instead, for rooms getting a repurposed mini PC rather than a Raspberry Pi | | *(amd64 alternative)* Spare x86 mini PC + USB DAC/amp (e.g. Fosi Audio V3, ~€130) | €0130 | No GPIO HAT involved — a USB Audio Class device instead, for rooms getting a repurposed mini PC rather than a Raspberry Pi |
### 1.14 Touch panel hardware (Phase 16)
| Item | Est. Price (EUR) | Notes |
|---|---|---|
| Mini PC or SBC with a capacitive touchscreen (e.g. 1015" USB-touch monitor + small x86 mini PC, or an all-in-one touch panel PC) | €150300 | **No specific unit has been chosen** — the one hard requirement is that the touchscreen exposes itself as a native Wayland `wl_touch` device under libinput, not merely an emulated-mouse HID interface, since `hosts/touch-panel/` has no gesture/pointer-translation fallback the way the thin client does |
| Speakers or a wired connection to existing room audio | €0100 | For local Spotify GUI-client playback — same speaker guidance as the thin-client/audio-endpoint rooms |
*(No container-host changes — `touchpanel-agent` talks to the existing Mosquitto/HA
from Phase 1, same as every other MQTT-connected host in this plan.)*
### 1.15 Kitchen/fridge display hardware (Phase 17)
| Item | Est. Price (EUR) | Notes |
|---|---|---|
| Mini PC or SBC with a touchscreen, mounted near the fridge/pantry | €150300 | Same "must be a native Wayland `wl_touch` device" requirement as §1.14 — `hosts/kitchen-display/` has no pointer-emulation fallback either |
| USB webcam, pointed at wherever items get held up for scanning | €1540 | Either built into the panel or on a short cable/gooseneck so its angle can be set independently of the screen. Any UVC webcam works — this is the same "no depth camera, no accelerator needed" bar as the thin client's gesture-control camera (§1.11), just used for a photo instead of continuous tracking |
*(No new container-host hardware — `pantry-vision` is a container on the existing
Phase 1 host, calling the existing Phase 3/9 LLM host for vision inference and the
already-running `grocy` container for storage.)*
--- ---
## 2. Software (all open source / self-hosted) ## 2. Software (all open source / self-hosted)
@ -155,6 +174,21 @@ Already covered — using your existing Haozee CC2652P USB dongle. No coordinato
| Headless audio endpoint (arm64) | **rpi-image-gen** (custom config, `hosts/audio-endpoint/rpi-image-gen/`) | Raspberry Pi Foundation's current custom-image tool; builds a flashable `.img` with a per-room Spotify Connect receiver + HiFiBerry Amp2 audio | | Headless audio endpoint (arm64) | **rpi-image-gen** (custom config, `hosts/audio-endpoint/rpi-image-gen/`) | Raspberry Pi Foundation's current custom-image tool; builds a flashable `.img` with a per-room Spotify Connect receiver + HiFiBerry Amp2 audio |
| Headless audio endpoint (amd64) | **live-build** (custom config, `hosts/audio-endpoint/live-build-amd64/`) | Reuses the thin client's own build tool, stripped of the whole graphical/kiosk stack — headless boot straight to the same per-room Spotify Connect receiver | | Headless audio endpoint (amd64) | **live-build** (custom config, `hosts/audio-endpoint/live-build-amd64/`) | Reuses the thin client's own build tool, stripped of the whole graphical/kiosk stack — headless boot straight to the same per-room Spotify Connect receiver |
| Headless audio endpoint software | **librespot** or **spotifyd** (apt-first, documented fallback) | Independent, per-room Spotify Connect — no Snapcast, no shared/synced stream; each endpoint is its own device in Spotify's picker, same model as the thin client's own per-room spotifyd/librespot (Phase 11.6) | | Headless audio endpoint software | **librespot** or **spotifyd** (apt-first, documented fallback) | Independent, per-room Spotify Connect — no Snapcast, no shared/synced stream; each endpoint is its own device in Spotify's picker, same model as the thin client's own per-room spotifyd/librespot (Phase 11.6) |
| Touch-panel OS build | **live-build** (custom config, `hosts/touch-panel/live-build/`) | Reuses the thin client's own build tool and directory-split convention, not its live-build tree — a different, smaller image for a different device |
| Touch-panel compositor | **Sway** | Same choice as the thin client, configured for direct touch instead of remote control: an always-on dock instead of no bars, no gesture-camera/wayvnc machinery |
| Touch-panel scripted control | **touchpanel-agent** (custom) | HA MQTT-discovery entity + `swaymsg`/app-process control, identical security shape to `thinclient-agent` — LLM tool call → HA service call → MQTT → this agent, never a direct path |
| Touch-panel music | **Spotify (official Linux client, Flathub `com.spotify.Client`)** | Full GUI client, not a headless Connect receiver — the point of a touch panel is a screen you interact with directly (Premium required, same as the headless receivers) |
| Touch-panel home dashboard | **Chromium (kiosk mode, `--app=`)** | Dedicated, always-open window pointed at Home Assistant, auto-restarted if it crashes |
| Touch-panel browser | **Firefox** | General browsing, minimal (not kiosk-locked) chrome — back/forward/reload/address bar |
| Touch-panel on-screen keyboard | **wvkbd** (apt-first, documented fallback) | Manually toggled from the touch dock — no shell-level input-method integration on this image, so no automatic show-on-focus |
| Touch-panel navigation | **eww** (apt-first, documented fallback) | An always-on, layer-shell-reserved touch dock (Spotify / Home / Web / Keyboard) — the touch-first equivalent of the thin client's HA-only **Screen** select |
| Kitchen-display OS build | **live-build** (custom config, `hosts/kitchen-display/live-build/`) | Same toolchain/convention reuse as the touch panel, a third and simpler sibling image — one workspace, one app |
| Kitchen-display camera capture | Browser-native `getUserMedia()` (Chromium) | No native capture app on-device at all — the kiosk page itself talks to the webcam and hands a frame straight to `pantry-vision` |
| Kitchen-display scripted control | **kitchen-display-agent** (custom) | HA MQTT-discovery entity for **Show scan/inventory/recipes** only — the actual camera/Grocy read-write path is `pantry-vision`'s own published API, called directly by the kiosk browser, not mediated through this agent |
| Grocery vision recognition | **Ollama** (a vision-capable model, e.g. `llava`/`qwen2.5vl` — TBD, not yet pulled or benchmarked) | Identifies a grocery item from one photo and estimates its shelf life; the proposal is always human-reviewed before anything is written (see `pantry-vision/README.md`) |
| Grocery inventory backend | **pantry-vision** (custom Python, stdlib `http.server`) | `POST /identify` (photo → proposal), `POST /confirm` (human-reviewed proposal → Grocy stock write), `GET /inventory`/`GET /recipes` (proxy Grocy, reshaped). Bearer-token gated, **and — unlike admin-canvas — LAN-published**, since the kitchen display's kiosk browser calls it directly rather than through Home Assistant |
| Grocery inventory storage | **Grocy** (already running, Phase 1/7) | The single source of truth for stock/best-before-dates/recipes; `pantry-vision` is a client of Grocy's own REST API, not a replacement for it |
| Kitchen-display static serving | **pantry-web** (nginx:alpine) | Serves `pantry-vision/frontend/`'s Scan/Inventory/Recipes single-page app read-only to the kitchen display — same role `digest-web`/`admin-web` play for their own hosts |
--- ---
@ -296,6 +330,131 @@ New hardware: §1.13. Two separate build pipelines, arm64 and amd64 — see belo
7. Neither image gives Home Assistant any control or visibility over these endpoints — no MQTT, no HA entities. Deliberate scope decision matching the "per-room independent" choice, not an oversight; revisit as a separate addition if remote control/monitoring from HA is wanted later. 7. Neither image gives Home Assistant any control or visibility over these endpoints — no MQTT, no HA entities. Deliberate scope decision matching the "per-room independent" choice, not an oversight; revisit as a separate addition if remote control/monitoring from HA is wanted later.
8. Nothing built or flashed on real hardware — no Raspberry Pi, no HiFiBerry Amp2, no x86 test box available. See the itemized unverified list in `hosts/audio-endpoint/README.md`. 8. Nothing built or flashed on real hardware — no Raspberry Pi, no HiFiBerry Amp2, no x86 test box available. See the itemized unverified list in `hosts/audio-endpoint/README.md`.
### Phase 16 — Sway touch panel
New hardware: §1.14. No container-host changes.
1. New top-level `hosts/touch-panel/` directory — **a different device, not a variant
of the Phase 11 thin client**: that image is a couch-distance media station
deliberately built around *remote* control (HA/MQTT, wayvnc) with no on-screen
bars; this one is touched directly, so it inverts that — an always-on touch dock,
native Wayland touch input, no gesture-camera/remote-view machinery. It reuses the
thin client's live-build toolchain and `configs/`+`agent/` split, not its
live-build tree, exactly the relationship `hosts/audio-endpoint/`'s amd64 build
already has to the thin client's.
2. **Three fixed workspaces, no on-demand ones**: `1:spotify` (the real, official
Spotify Linux GUI client via Flathub `com.spotify.Client` — not the thin client's
headless `spotifyd`/`librespot` Connect receiver, since the whole point of a touch
panel is a screen you interact with directly), `2:home` (a persistent Chromium
`--kiosk --app=$HA_URL` window, auto-restarted on crash by its own supervising
loop rather than by sway or systemd), and `3:web` (Firefox, minimal chrome —
general browsing, not kiosk-locked, mirroring the thin client's own `web_browser`
app in shape but on a separate, duplicated profile).
3. **Touch navigation is a persistent on-screen dock (`eww`, layer-shell
`:exclusive true`), not a keyboard/mouse-driven `bar {}`**: reserving a strip at
the bottom means the dock can never be covered by, or lose screen area behind,
whatever app is on screen — the opposite trade-off from the thin client's
deliberately bar-less, overlay-only widgets. Same session-scoped-not-agent
ownership rule as the thin client's now-playing widget: this is compositor UI, not
device control, and dies with sway rather than opening a second inbound channel
into the security-sensitive agent.
4. **`touchpanel-agent` mirrors `thinclient-agent`'s exact security boundary**: LLM
tool call → HA service call → MQTT → this agent, never a direct network path. It
publishes a **Screen** select (the touch dock's HA-reachable equivalent) and
**Show Spotify** / **Show Home** / **Show web browser** buttons — the latter two
focus the already-running app rather than relaunching it (both are meant to stay
open and stateful), unlike the thin client's stateless kill-and-relaunch digest/
admin kiosk pages. Also publishes a `media_player`-shaped entity set bridged from
Spotify's own MPRIS interface, identical in shape to the thin client's MPRIS
bridge, pointed at the `spotify` player name instead of `mpv`/`spotifyd`.
5. **An on-screen keyboard (`wvkbd`) with no auto-show, by explicit choice**: proper
show-on-text-field-focus needs the `text-input-v3`/`virtual-keyboard-v1` protocols
wired through a shell component (the way Phosh does it for squeekboard), which this
image doesn't run. Rather than build that integration now, the dock's Keyboard
button is a manual toggle — a documented, accepted limitation, not a gap hidden
from the README.
6. **No wayvnc on this image**, unlike the thin client — a deliberate scope decision,
not an oversight: the primary interaction model here is a finger at the panel, not
a remote viewer, so SSH alone covers the "something's broken, fix it from a shell"
case. Revisit only if a real need for remote *view* shows up once this runs on real
hardware; adding wayvnc back would be a small, isolated change (see
`hosts/touch-panel/README.md`).
7. **Both apt-first/documented-fallback installs already established elsewhere in this
repo are reused, not reinvented**: `eww` (same as the thin client's now-playing
widget) and `wvkbd` — neither is in Debian bookworm main, and both hooks stop at a
documented placeholder rather than a hardcoded release URL if apt doesn't have them.
8. Nothing built or flashed on real hardware — no touch-panel unit has been chosen
(§1.14), and in particular whether the touchscreen exposes itself as a native
Wayland `wl_touch` device (this image's one hard assumption, since there is no
pointer-emulation fallback the way the thin client's gesture control has one) is
unverified. See the itemized list in `hosts/touch-panel/README.md`.
### Phase 17 — Kitchen/fridge display (camera-vision grocery cataloguing)
New hardware: §1.15. New top-level `pantry-vision/` directory (container-host
service) plus `hosts/kitchen-display/` (a third, simpler kiosk image).
1. **The workflow this phase is built around, stated once because everything below
serves it**: come home, put down the shopping bag, hold one item up to the
kitchen display's camera, the system proposes what it is and roughly how long it
keeps, the person confirms (editing anything wrong) before anything is written
anywhere, put the item away. The same display then shows inventory ordered by
what expires soonest, and Grocy's recipes, on request.
2. **`pantry-vision` never writes to Grocy from `/identify` alone — only `/confirm`,
a separate human-reviewed call, ever does.** This is the single guardrail the
whole phase is designed around, the same "propose, never auto-commit" rule this
project already applies to identity-merge confirmation (the *Identity store* row
in §2). A wrong camera guess costs one tap to fix on the kiosk screen, not a wrong
fact silently written into the household's inventory.
3. **A real, LAN-published network listener — the one deliberate exception to this
project's usual "the LLM only ever reaches a device through HA→MQTT" shape.**
`admin-canvas` (Phase 13) has no published port because only Home Assistant calls
it; `pantry-vision` is different on purpose because the kitchen display is a
separate physical device that has to get an identification back synchronously,
with no HA round-trip in the loop for that specific call. The bearer token — not
network placement — is the actual boundary here; see
`pantry-vision/README.md`'s "A real network listener, unlike admin-canvas"
section. Controlling *which screen is showing* (Scan/Inventory/Recipes) is a
separate, narrower path that does stay HA/MQTT-mediated: `kitchen-display-agent`,
identical in shape to every other host's agent in this project.
4. **Grocy is the system of record, not a new inventory store**`pantry-vision` is
a client of Grocy's own REST API (already running unconditionally since Phase 1/7)
for both reads (`/inventory`, `/recipes`) and the one write (`/confirm` → Grocy
stock). Nothing here duplicates or replaces Grocy's own data model.
5. **The vision-identification prompt asks for one photo, not a live video stream**,
and degrades to a low-confidence placeholder proposal (never an error page) if the
model call fails or its response isn't parseable JSON — same "degrade, don't
blank" rule as the digest/admin canvas renderers and `llm_client.py`'s own
`_fallback_document`.
6. **The kitchen display has no native camera-capture app at all** — Chromium's own
`getUserMedia()`, called from `pantry-vision/frontend/app.js`, talks to the webcam
directly inside the kiosk page and hands a captured frame straight to
`POST /identify` as a blob. `--use-fake-ui-for-media-stream` on the kiosk launch
auto-accepts the permission prompt that would otherwise sit unanswered on an
unattended screen.
7. **`hosts/kitchen-display/` is one workspace, one app — no touch dock, no
multi-app switching** the way `hosts/touch-panel` has: Scan/Inventory/Recipes is
in-page tab navigation inside `pantry-vision/frontend/`'s single-page app, since
there's only one thing this device does. `kitchen-display-agent`'s MQTT surface
is correspondingly narrow: three "Show X" buttons and nothing else — no
media_player, no capture/audio/remote-desktop entities.
8. **The Grocy API integration is written from documentation, not verified against a
live instance** — the exact `GET /api/stock` response shape (whether product
names arrive nested by default), the minimum required fields for
`POST /api/objects/products`, and whether the Recipes/fulfillment endpoints need
explicit setup before they return anything meaningful are all flagged in
`pantry-vision/README.md` alongside the live OpenAPI spec URL every real Grocy
instance exposes (`/api/openapi/specification`) as the way to actually check.
9. **The vision model choice is unpicked and unbenchmarked** (`OLLAMA_VISION_MODEL`
defaults to `llava`, not confirmed pulled or even correct for a given LLM host's
GPU/CPU tier) — same "don't build against a guess" flag as every other
LLM-model/hardware pairing in this plan. Latency in particular is unmeasured and
is the number that decides whether "hold it up, wait, confirm" feels usable at
the counter or like standing there for 30 seconds.
10. Nothing built or run against real hardware, a real camera, a real vision model,
or a real Grocy instance. See the itemized lists in `pantry-vision/README.md`
and `hosts/kitchen-display/README.md`.
### Testing checklist before calling any phase "done" ### Testing checklist before calling any phase "done"
- Does the reactive path (presence → light on) work with the LLM host powered off? (It must.) - Does the reactive path (presence → light on) work with the LLM host powered off? (It must.)
- Does a bad/slow LLM response ever block a light switch? (It must not.) - Does a bad/slow LLM response ever block a light switch? (It must not.)
@ -327,10 +486,17 @@ New hardware: §1.13. Two separate build pipelines, arm64 and amd64 — see belo
- Is the ESP32-S3-Touch-LCD-1.85C firmware ever flashed onto a V1 board? (It must not be — V1 has no AEC circuit and different audio pins; this phase's config assumes V2 throughout.) - Is the ESP32-S3-Touch-LCD-1.85C firmware ever flashed onto a V1 board? (It must not be — V1 has no AEC circuit and different audio pins; this phase's config assumes V2 throughout.)
- Does a headless audio endpoint's Spotify Connect device name ever fail to match the room it's actually in? (It must not — `spotify-connect-start` always reads `$(hostname)`, which is set per-room by Raspberry Pi Imager on arm64 or `IMAGE_HOSTNAME` at build time on amd64; a copy-pasted/reused image for two rooms is a configuration mistake this software has no way to detect, same class of risk already called out for the ESP32 firmware's `media_player_entity_id`.) - Does a headless audio endpoint's Spotify Connect device name ever fail to match the room it's actually in? (It must not — `spotify-connect-start` always reads `$(hostname)`, which is set per-room by Raspberry Pi Imager on arm64 or `IMAGE_HOSTNAME` at build time on amd64; a copy-pasted/reused image for two rooms is a configuration mistake this software has no way to detect, same class of risk already called out for the ESP32 firmware's `media_player_entity_id`.)
- Does the headless audio endpoint ever gain a shared/synced audio path, an MQTT connection, or an HA entity without a deliberate new decision to add one? (It must not — Phase 15 is scoped to per-room independent Spotify Connect only, on purpose.) - Does the headless audio endpoint ever gain a shared/synced audio path, an MQTT connection, or an HA entity without a deliberate new decision to add one? (It must not — Phase 15 is scoped to per-room independent Spotify Connect only, on purpose.)
- Can the LLM reach the touch panel through any path other than HA service call → MQTT → `touchpanel-agent`? (It must not — identical boundary to the thin client's.)
- Does the touch panel boot to a usable session (Spotify + the browser) with the container host powered off? (It must not hang — Home showing a connection error is the expected/acceptable degraded state, since unlike the thin client this device has no local media source to fall back to.)
- Does tapping the touch dock's Home/Spotify buttons ever relaunch (rather than just focus) an already-running window? (It must not — both are stateful apps; relaunching would throw away login/scroll/playback state for no reason.)
- Does `pantry-vision`'s `/identify` endpoint ever write to Grocy by itself? (It must not — only a separate, human-reviewed `/confirm` call ever writes anything.)
- Does `pantry-vision` ever accept a request without a valid bearer token, on any of its four endpoints — including the two GETs? (It must not — unlike admin-canvas, this service is LAN-published, so the token is the actual boundary, not network placement.)
- Can the LLM reach the kitchen display through any path other than HA service call → MQTT → `kitchen-display-agent`, for *which screen is showing*? (It must not — reading/writing the actual inventory is a separate, intentionally-published path through `pantry-vision` itself, not a violation of this rule.)
- If the vision model's response is unparseable or the call fails outright, does `/identify` ever return a broken/blank result instead of a flagged, low-confidence placeholder proposal? (It must not.)
--- ---
## 4. Open decisions (Phases 1115) ## 4. Open decisions (Phases 1117)
These need a decision before their respective implementation steps can be built — everything above is written to accommodate any answer, but nothing should be built against an unresolved item. These need a decision before their respective implementation steps can be built — everything above is written to accommodate any answer, but nothing should be built against an unresolved item.
@ -349,3 +515,7 @@ These need a decision before their respective implementation steps can be built
13. **The sys-admin-llm's HA-side wiring has no real data source or tool definition yet** (new, Phase 13) — same shape as open decision #9 above: `admin-canvas/README.md` documents the expected `rest_command:`/tool-call contract and worked example JSON, but which HA entities/history back something like "the kitchen outlet's power draw" is unresolved, and no metering-capable Zigbee smart plug is in this plan's hardware list (§1.4) yet. Needs a hardware decision (a power-monitoring outlet) and an actual HA tool/script, neither of which exists in this repo by design — see the Phase 13 "nothing under this repo builds the HA side" note. 13. **The sys-admin-llm's HA-side wiring has no real data source or tool definition yet** (new, Phase 13) — same shape as open decision #9 above: `admin-canvas/README.md` documents the expected `rest_command:`/tool-call contract and worked example JSON, but which HA entities/history back something like "the kitchen outlet's power draw" is unresolved, and no metering-capable Zigbee smart plug is in this plan's hardware list (§1.4) yet. Needs a hardware decision (a power-monitoring outlet) and an actual HA tool/script, neither of which exists in this repo by design — see the Phase 13 "nothing under this repo builds the HA side" note.
14. **The voice/display satellite has no real hardware verification, and two of its data sources are unpicked** (new, Phase 14) — `firmware/esp32-s3-touch-lcd-1.85c/voice-display.yaml` passes ESPHome's own config validator but has never been flashed to a physical unit; the display init sequence, the AEC audio path, and wake-word sensitivity are all adapted/assumed, not measured (see the itemized list in that directory's README). Separately, `media_player_entity_id` and `weather_entity_id` are placeholders — which media player this unit should mirror needs a decision if the household ends up with more than one active at a time. 14. **The voice/display satellite has no real hardware verification, and two of its data sources are unpicked** (new, Phase 14) — `firmware/esp32-s3-touch-lcd-1.85c/voice-display.yaml` passes ESPHome's own config validator but has never been flashed to a physical unit; the display init sequence, the AEC audio path, and wake-word sensitivity are all adapted/assumed, not measured (see the itemized list in that directory's README). Separately, `media_player_entity_id` and `weather_entity_id` are placeholders — which media player this unit should mirror needs a decision if the household ends up with more than one active at a time.
15. **The headless audio endpoint has no real hardware verification on either architecture** (new, Phase 15) — neither the arm64 (`rpi-image-gen`) nor amd64 (`live-build`) pipeline has been run to completion, let alone flashed and booted. rpi-image-gen's exact config/layer YAML schema in particular was written from documentation only, with no ARM build environment available to validate it against the real tool — see `hosts/audio-endpoint/README.md`'s prominent risk callout. Also still open: whether `spotifyd` or `librespot` actually installs cleanly from current bookworm apt sources at all (the fallback list is documented, not exercised), and how stable USB DAC/amp ALSA enumeration turns out to be on a real amd64 mini PC. 15. **The headless audio endpoint has no real hardware verification on either architecture** (new, Phase 15) — neither the arm64 (`rpi-image-gen`) nor amd64 (`live-build`) pipeline has been run to completion, let alone flashed and booted. rpi-image-gen's exact config/layer YAML schema in particular was written from documentation only, with no ARM build environment available to validate it against the real tool — see `hosts/audio-endpoint/README.md`'s prominent risk callout. Also still open: whether `spotifyd` or `librespot` actually installs cleanly from current bookworm apt sources at all (the fallback list is documented, not exercised), and how stable USB DAC/amp ALSA enumeration turns out to be on a real amd64 mini PC.
16. **No touch-panel hardware has been chosen** (new, Phase 16) — nothing in `hosts/touch-panel/` has been booted on real metal or a real touchscreen. In particular, whether the panel's touch input actually surfaces as a native Wayland `wl_touch` device (this image's one hard assumption) is unverified — see `hosts/touch-panel/README.md`'s verification list, item 3.
17. **The touch panel's Spotify/Chromium integration details are assumed, not confirmed** (new, Phase 16) — the Flathub app ID `com.spotify.Client`, the Spotify Flatpak's real MPRIS bus name (assumed `spotify`), and Chromium's Wayland `app_id` for a `--app=` kiosk window (assumed to start with `chromium`) are all flagged for on-hardware verification in `hosts/touch-panel/README.md`.
18. **No vision-capable Ollama model has been picked, pulled, or benchmarked** (new, Phase 17) — `pantry-vision`'s `OLLAMA_VISION_MODEL` defaults to `llava` with no confirmation it's the right choice for any given LLM host's hardware tier, and grocery-item identification latency/accuracy is completely unmeasured. This is the single highest-risk unknown in Phase 17: if it's too slow or too inaccurate, "hold item up to camera" stops being a usable interaction and the phase needs a different approach (a smaller/faster model, a barcode-scan fallback via Grocy's existing scanning support, or both).
19. **`pantry-vision`'s Grocy API integration is written from documentation, not a live instance** (new, Phase 17) — `GET /api/stock`'s response shape, `POST /api/objects/products`'s minimum required fields, and the Recipes/fulfillment endpoints' setup requirements are all assumed; `pantry-vision/README.md` points at each real Grocy instance's own live OpenAPI spec (`/api/openapi/specification`) as the way to check before trusting any of it. `GROCY_DEFAULT_LOCATION_ID`/`GROCY_DEFAULT_QU_ID` are fresh-install-default guesses that need confirming against Settings → Locations/Quantity units on the real instance too.

View File

@ -28,6 +28,10 @@
# - admin-canvas + admin-web (Phase 13 sys-admin-llm display surface, optional, # - admin-canvas + admin-web (Phase 13 sys-admin-llm display surface, optional,
# off by default — needs admin-canvas/ from this repo checked out on this # off by default — needs admin-canvas/ from this repo checked out on this
# host, see ADMIN_CANVAS_SRC below and admin-canvas/README.md) # host, see ADMIN_CANVAS_SRC below and admin-canvas/README.md)
# - pantry-vision + pantry-web (Phase 17 kitchen-display camera cataloguing
# backend, optional, off by default — needs pantry-vision/ from this repo
# checked out on this host, see PANTRY_VISION_SRC below and
# pantry-vision/README.md)
# #
# Run as: sudo ./setup-container-host.sh # Run as: sudo ./setup-container-host.sh
# #
@ -94,6 +98,19 @@ ADMIN_CANVAS_SRC="/opt/smart-home/src/admin-canvas"
ADMIN_CANVAS_PORT="8092" # internal only — no `ports:` mapping, HA-reachable only ADMIN_CANVAS_PORT="8092" # internal only — no `ports:` mapping, HA-reachable only
ADMIN_WEB_PORT="8094" # LAN-facing read-only static serving ADMIN_WEB_PORT="8094" # LAN-facing read-only static serving
# --- Kitchen-display camera cataloguing backend (Phase 17) — off by default until
# --- PANTRY_VISION_TOKEN and GROCY_API_KEY are provisioned. See pantry-vision/README.md.
# Unlike admin-canvas, this one IS meant to be reached directly by a separate physical
# device (hosts/kitchen-display's kiosk browser), not only by Home Assistant — so,
# deliberately unlike ADMIN_CANVAS_PORT above, it DOES get a `ports:` mapping. The
# bearer token is the actual boundary here, not network placement — see
# pantry-vision/README.md's "A real network listener, unlike admin-canvas" section.
ENABLE_PANTRY_VISION="false"
# Where this repo's pantry-vision/ directory lives on THIS host (build context).
PANTRY_VISION_SRC="/opt/smart-home/src/pantry-vision"
PANTRY_VISION_PORT="8095" # LAN-facing — the kitchen display's kiosk browser calls this directly
PANTRY_WEB_PORT="8096" # LAN-facing read-only static serving (the kiosk's frontend)
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# Sanity checks # Sanity checks
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
@ -130,6 +147,12 @@ if [[ "$ENABLE_ADMIN_CANVAS" == "true" && ! -d "$ADMIN_CANVAS_SRC" ]]; then
echo " build context for the admin-canvas image — then re-run." echo " build context for the admin-canvas image — then re-run."
fi fi
if [[ "$ENABLE_PANTRY_VISION" == "true" && ! -d "$PANTRY_VISION_SRC" ]]; then
echo "Warning: ENABLE_PANTRY_VISION=true but $PANTRY_VISION_SRC does not exist."
echo " Copy or clone this repo's pantry-vision/ directory there — it is the"
echo " build context for the pantry-vision image — then re-run."
fi
if [[ "$ENABLE_WHATSAPP_INGEST" == "true" ]]; then if [[ "$ENABLE_WHATSAPP_INGEST" == "true" ]]; then
echo "WARNING: WhatsApp ingestion is enabled." echo "WARNING: WhatsApp ingestion is enabled."
echo " There is no officially sanctioned way to read WhatsApp programmatically." echo " There is no officially sanctioned way to read WhatsApp programmatically."
@ -252,6 +275,17 @@ if [[ "$ENABLE_ADMIN_CANVAS" == "true" ]]; then
echo " Seeded $BASE_DIR/admin-canvas/admin-canvas.env from the template — fill in a real ADMIN_CANVAS_TOKEN." echo " Seeded $BASE_DIR/admin-canvas/admin-canvas.env from the template — fill in a real ADMIN_CANVAS_TOKEN."
fi fi
fi fi
if [[ "$ENABLE_PANTRY_VISION" == "true" ]]; then
mkdir -p "$BASE_DIR"/pantry-vision
# Same handling as admin-canvas.env above: seed from the committed template on
# first run, 600, never committed (repo .gitignore covers *.env).
if [[ ! -f "$BASE_DIR/pantry-vision/pantry-vision.env" ]]; then
cp "$PANTRY_VISION_SRC/pantry-vision.env.example" "$BASE_DIR/pantry-vision/pantry-vision.env"
chmod 600 "$BASE_DIR/pantry-vision/pantry-vision.env"
echo " Seeded $BASE_DIR/pantry-vision/pantry-vision.env from the template — fill in a real"
echo " PANTRY_VISION_TOKEN and GROCY_API_KEY (Grocy's own UI: Settings -> Manage API keys)."
fi
fi
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# 4. Mosquitto config # 4. Mosquitto config
@ -625,6 +659,46 @@ if [[ "$ENABLE_ADMIN_CANVAS" == "true" ]]; then
" "
fi fi
# pantry-vision (Phase 17) — the kitchen display's camera-cataloguing backend. Same
# build/write-service shape as admin-canvas above, but see this block's own comment
# where it differs (a published port, since a separate physical device calls it
# directly) and pantry-vision/README.md for the full reasoning. `depends_on: grocy`
# is a startup-order hint only — pantry-vision's own outbound calls already retry per
# request, so a slow-starting Grocy delays the first real call, not the container.
PANTRY_VISION_BLOCK=""
PANTRY_WEB_BLOCK=""
if [[ "$ENABLE_PANTRY_VISION" == "true" ]]; then
PANTRY_VISION_BLOCK="
pantry-vision:
build: ${PANTRY_VISION_SRC}
image: smart-home/pantry-vision:local
container_name: pantry-vision
restart: unless-stopped
depends_on:
- grocy
ports:
- \"${PANTRY_VISION_PORT}:${PANTRY_VISION_PORT}\"
env_file:
- ${BASE_DIR}/pantry-vision/pantry-vision.env
environment:
- PANTRY_VISION_PORT=${PANTRY_VISION_PORT}
- TZ=${TIMEZONE}
"
PANTRY_WEB_BLOCK="
pantry-web:
image: nginx:alpine
container_name: pantry-web
restart: unless-stopped
ports:
- \"${PANTRY_WEB_PORT}:80\"
volumes:
- ${PANTRY_VISION_SRC}/frontend:/usr/share/nginx/html:ro
environment:
- TZ=${TIMEZONE}
"
fi
if [[ "$ENABLE_DIGEST_ENGINE" == "true" && "$ENABLE_WHATSAPP_INGEST" == "true" ]]; then if [[ "$ENABLE_DIGEST_ENGINE" == "true" && "$ENABLE_WHATSAPP_INGEST" == "true" ]]; then
# Long-lived, unlike digest-engine: holds the logged-in WhatsApp Web session # Long-lived, unlike digest-engine: holds the logged-in WhatsApp Web session
# open and appends to /data/messages.jsonl, which digest-engine drains each # open and appends to /data/messages.jsonl, which digest-engine drains each
@ -721,7 +795,7 @@ ${FRIGATE_DEVICES}
- PUID=1000 - PUID=1000
- PGID=1000 - PGID=1000
- TZ=${TIMEZONE} - TZ=${TIMEZONE}
${MEALIE_BLOCK}${NODERED_BLOCK}${NETDATA_BLOCK}${HOMEPAGE_BLOCK}${NTFY_BLOCK}${PORTAINER_BLOCK}${GALLERY_SMB_BLOCK}${DIGEST_ENGINE_BLOCK}${DIGEST_WEB_BLOCK}${WHATSAPP_BRIDGE_BLOCK}${ADMIN_CANVAS_BLOCK}${ADMIN_WEB_BLOCK} ${MEALIE_BLOCK}${NODERED_BLOCK}${NETDATA_BLOCK}${HOMEPAGE_BLOCK}${NTFY_BLOCK}${PORTAINER_BLOCK}${GALLERY_SMB_BLOCK}${DIGEST_ENGINE_BLOCK}${DIGEST_WEB_BLOCK}${WHATSAPP_BRIDGE_BLOCK}${ADMIN_CANVAS_BLOCK}${ADMIN_WEB_BLOCK}${PANTRY_VISION_BLOCK}${PANTRY_WEB_BLOCK}
EOF EOF
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
@ -880,6 +954,10 @@ fi
if [[ "$ENABLE_ADMIN_CANVAS" == "true" ]]; then if [[ "$ENABLE_ADMIN_CANVAS" == "true" ]]; then
echo " Admin canvas : http://${HOST_IP}:${ADMIN_WEB_PORT}/canvas.html" echo " Admin canvas : http://${HOST_IP}:${ADMIN_WEB_PORT}/canvas.html"
fi fi
if [[ "$ENABLE_PANTRY_VISION" == "true" ]]; then
echo " Pantry API : http://${HOST_IP}:${PANTRY_VISION_PORT} (bearer-token gated)"
echo " Pantry display : http://${HOST_IP}:${PANTRY_WEB_PORT}/index.html?api=http://${HOST_IP}:${PANTRY_VISION_PORT}&token=<PANTRY_VISION_TOKEN>"
fi
if [[ "$ENABLE_GALLERY_SMB" == "true" ]]; then if [[ "$ENABLE_GALLERY_SMB" == "true" ]]; then
echo " Gallery SMB : \\\\${HOST_IP}\\gallery (user: ${GALLERY_SMB_USERNAME})" echo " Gallery SMB : \\\\${HOST_IP}\\gallery (user: ${GALLERY_SMB_USERNAME})"
fi fi
@ -927,6 +1005,15 @@ if [[ "$ENABLE_ADMIN_CANVAS" == "true" ]]; then
echo " admin-canvas/README.md for the worked example and the 'Show admin canvas'" echo " admin-canvas/README.md for the worked example and the 'Show admin canvas'"
echo " entity documented in hosts/thin-client/README.md." echo " entity documented in hosts/thin-client/README.md."
fi fi
if [[ "$ENABLE_PANTRY_VISION" == "true" ]]; then
echo " 15. Fill in $BASE_DIR/pantry-vision/pantry-vision.env before the kitchen display can"
echo " identify anything: PANTRY_VISION_TOKEN (also goes into"
echo " hosts/kitchen-display/scripts/build-kitchen-display-iso.sh — both sides need the"
echo " SAME value) and GROCY_API_KEY (Grocy's own UI: Settings -> Manage API keys, at"
echo " http://${HOST_IP}:9283). Also pull a vision-capable Ollama model on the LLM host"
echo " (e.g. 'ollama pull llava') — OLLAMA_VISION_MODEL defaults to one that is NOT"
echo " confirmed pulled or even correct for your setup. See pantry-vision/README.md."
fi
echo echo
echo "Updating later: cd $BASE_DIR && docker compose pull && docker compose up -d" echo "Updating later: cd $BASE_DIR && docker compose pull && docker compose up -d"
echo "Backing up manually: sudo $BASE_DIR/backup.sh (requires ENABLE_BACKUPS=true was run once)" echo "Backing up manually: sudo $BASE_DIR/backup.sh (requires ENABLE_BACKUPS=true was run once)"

View File

@ -0,0 +1,125 @@
# Sway kitchen display
Phase 17 of `docs/project-plan.md`. Builds a Debian 12 live ISO for a single-purpose
fridge/pantry display: hold a grocery item up to its camera, confirm what the vision
model thinks it is and when it'll go bad, put it away — then browse the resulting
inventory (soonest-to-expire first) and Grocy's recipes, all touch-driven.
**The camera-vision and Grocy read/write logic does not live in this directory** —
that's [`../../pantry-vision/`](../../pantry-vision/README.md), a container-host
service this device's kiosk browser calls directly. This host is deliberately thin:
a Sway session, one Chromium kiosk window, and `kitchen-display-agent`, which only
ever controls *which screen is showing* (Scan / Inventory / Recipes) over MQTT — the
same "the agent controls the surface, a separate write API owns the content" split
`hosts/thin-client`'s admin canvas already established.
**A third, simpler sibling of `../thin-client/` and `../touch-panel/`**, not a variant
of either — one workspace, one app, no dock, no music, no general browsing. It reuses
the thin client's live-build toolchain and `configs/`+`agent/` split.
| | |
|---|---|
| Compositor | Sway, one workspace, one Chromium kiosk window |
| Autologin | greetd, straight into `/usr/local/bin/kiosk-session` |
| Touch input | Native Wayland `wl_touch`, same assumption as `../touch-panel/` |
| Camera | Browser-native `getUserMedia()` inside the Chromium kiosk page — no separate native capture app on this device at all |
| Content | `pantry-vision`'s frontend (`../../pantry-vision/frontend/`), served by `pantry-web`; in-page tabs, not separate Sway workspaces |
| Remote control (HA/LLM) | `kitchen-display-agent`**Show scan / Show inventory / Show recipes** buttons only |
| Remote control (human) | SSH only — no wayvnc, same scope decision as `../touch-panel/` |
## Hardware
**No specific unit has been chosen.** What's assumed: a touchscreen (native
`wl_touch`, same caveat as `../touch-panel/README.md`) mounted somewhere near the
fridge/pantry, with a USB webcam pointed at wherever items get held up — either
built into the panel or a separate camera on a short cable/gooseneck mount so the
angle can be adjusted independently of the screen. See §1.15 of
`docs/project-plan.md` for the (unverified, non-specific) hardware line item.
## Before you build
Deploy `pantry-vision`/`pantry-web` first (`ENABLE_PANTRY_VISION` in
`hosts/container-host/scripts/setup-container-host.sh`) — this image builds and boots
fine without it, but every tab will show a connection error until it exists. Then edit
the `# CONFIGURATION` block at the top of
[`scripts/build-kitchen-display-iso.sh`](scripts/build-kitchen-display-iso.sh):
| Variable | What to put in it |
|---|---|
| `MQTT_BROKER_HOST` | LAN IP of the container host running Mosquitto |
| `PANTRY_WEB_URL` | The `pantry-web` service's URL, e.g. `http://192.168.1.10:8096` |
| `PANTRY_VISION_URL` | The `pantry-vision` API's URL, e.g. `http://192.168.1.10:8095` — read by `app.js` from the kiosk window's own URL query params (see `pantry-kiosk`'s comment), never baked into the served frontend files |
| `PANTRY_VISION_TOKEN` | **Must exactly match** `PANTRY_VISION_TOKEN` in `pantry-vision/pantry-vision.env` on the container host — there's no way for this repo to push that value between the two hosts for you |
| `KIOSK_USERNAME` / `IMAGE_HOSTNAME` / `KITCHEN_DISPLAY_NAME` | Per-device identity |
| `SSH_AUTHORIZED_KEY` | Optional — password auth is disabled and there is no wayvnc |
## Build
```sh
sudo ./scripts/build-kitchen-display-iso.sh
```
Same directory-split convention as every other host in this project: `configs/` and
`agent/` are human-edited and git-tracked; `live-build/config/includes.chroot/` is
generated, gitignored, never hand-edited.
## Why the camera has no native capture app
`../touch-panel/` and `../thin-client/` both shell out to native processes for their
media (mpv, the Spotify Flatpak). This device doesn't need that: Chromium's own
`getUserMedia()` API, called from `pantry-vision/frontend/app.js`, talks to the
webcam directly and hands a captured frame straight to `POST /identify` as a blob —
no intermediate script, no temp file, no separate process to supervise. The one thing
this needs from the kiosk launch itself is `--use-fake-ui-for-media-stream`
(`configs/sway/pantry-kiosk`), which auto-accepts the camera permission prompt that
would otherwise sit unanswered on a screen nobody is there to click "Allow" on.
## Home Assistant entities
`kitchen-display-agent` publishes three buttons on connect: **Show scan**, **Show
inventory**, **Show recipes** — each kills and relaunches the kiosk Chromium window
at `PANTRY_WEB_URL/index.html?...#<tab>`. That's the entire MQTT surface; there is no
media_player, no workspace select (one workspace), no capture/audio/remote-desktop
entities the way the thin client has. Reading or writing the actual inventory (from
voice, e.g. "what's about to expire?" or "add three eggs") is a **separate** path —
Home Assistant/the LLM calling `pantry-vision`'s own published API directly, not
through this agent, exactly mirroring the split `hosts/thin-client`'s admin canvas
already established between "which screen is showing" (MQTT, this agent) and "what's
actually on it" (a dedicated write API).
### Security boundary
Same shape and reasoning as every other host's agent in this project:
`kitchen_display_agent/mqtt_discovery.py` is the entire inbound MQTT control surface
of this machine — LLM tool call → HA service call → MQTT → this agent, no HTTP
listener, no websocket, no exposed Sway IPC socket, no VNC. `pantry-vision` is a
**separate, deliberately published** service with its own bearer-token boundary — see
`../../pantry-vision/README.md`'s "A real network listener, unlike admin-canvas"
section for why that one's trust model is different on purpose.
## Manual verification still outstanding
None of this has been run on hardware. In rough order — on top of everything already
flagged as unverified in `../../pantry-vision/README.md` (the Grocy API assumptions,
the vision model's accuracy/latency, which this device's whole usefulness rides on):
1. The ISO builds at all.
2. greetd lands in Sway with no login prompt.
3. **Touch input as native `wl_touch`** — same open item as `../touch-panel/`.
4. **Camera enumeration and `--use-fake-ui-for-media-stream`** — whether Chromium
picks the right `/dev/videoN` when more than one video device is present (a
built-in laptop-style webcam plus a dedicated one, say), and whether that flag
actually suppresses the permission prompt on this Chromium/Wayland combination
specifically (it's a well-known flag for automated testing, not confirmed here
against a real kiosk session).
5. `kitchen-display-agent` connects to Mosquitto and the device appears in HA.
6. End-to-end scan latency: camera capture → `/identify` → a usable proposal on
screen, with a real vision model on real LLM-host hardware. This is the number
that decides whether the whole "hold it up, wait, confirm" workflow feels usable
or like standing at the fridge for 30 seconds waiting on an LLM.
7. Whether `?api=...&token=...` survives being passed through `--app=` URL escaping
correctly for every character that can appear in `PANTRY_VISION_URL` — the
`sed`-based percent-encoding in `configs/sway/pantry-kiosk` handles `/` and `:`
only, which is enough for a bare `http://host:port` but not for a URL with a path
or query string of its own.
8. Idle-blank timeout (20 minutes, `configs/sway/config`) and touch-resume — untested.

View File

@ -0,0 +1,20 @@
[Unit]
Description=Kitchen display agent (Home Assistant MQTT control surface)
Documentation=file:///opt/kitchen-display-agent
After=network.target
[Service]
Type=simple
User=@KIOSK_USERNAME@
Group=@KIOSK_USERNAME@
WorkingDirectory=/opt/kitchen-display-agent
Environment=PYTHONPATH=/opt/kitchen-display-agent
Environment=PYTHONUNBUFFERED=1
EnvironmentFile=-/etc/kitchen-display-agent/config.env
ExecStart=/usr/bin/python3 -m kitchen_display_agent.main
Restart=always
RestartSec=5
NoNewPrivileges=true
[Install]
WantedBy=multi-user.target

View File

@ -0,0 +1,3 @@
"""kitchen-display-agent — Home Assistant MQTT control surface for the kitchen display."""
__version__ = "0.1.0"

View File

@ -0,0 +1,137 @@
"""kitchen-display-agent entrypoint."""
from __future__ import annotations
import logging
import os
import signal
import socket
import sys
import threading
import paho.mqtt.client as mqtt
from .mqtt_discovery import Discovery
from .session import launch_pantry_kiosk
CONFIG_PATH = os.environ.get("KITCHEN_DISPLAY_AGENT_CONFIG", "/etc/kitchen-display-agent/config.env")
CONFIG_KEYS = (
"MQTT_BROKER_HOST",
"MQTT_BROKER_PORT",
"MQTT_USERNAME",
"MQTT_PASSWORD",
"PANTRY_WEB_URL",
"PANTRY_VISION_URL",
"PANTRY_VISION_TOKEN",
"KIOSK_USERNAME",
"KITCHEN_DISPLAY_NAME",
)
log = logging.getLogger("kitchen-display-agent")
def load_config(path: str = CONFIG_PATH) -> dict[str, str]:
values: dict[str, str] = {}
try:
with open(path, encoding="utf-8") as handle:
for line in handle:
line = line.strip()
if not line or line.startswith("#") or "=" not in line:
continue
key, _, value = line.partition("=")
values[key.strip()] = value.strip().strip('"').strip("'")
except OSError as exc:
log.warning("could not read %s (%s); falling back to the environment", path, exc)
for key in CONFIG_KEYS:
if key in os.environ:
values[key] = os.environ[key]
return values
def make_client(client_id: str) -> mqtt.Client:
callback_api = getattr(mqtt, "CallbackAPIVersion", None)
if callback_api is not None:
return mqtt.Client(callback_api.VERSION1, client_id=client_id)
return mqtt.Client(client_id=client_id)
def main() -> int:
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s %(levelname)s %(name)s: %(message)s",
stream=sys.stdout,
)
config = load_config()
hostname = socket.gethostname()
node_id = "".join(c if c.isalnum() else "_" for c in hostname).strip("_") or "kitchendisplay"
friendly_name = config.get("KITCHEN_DISPLAY_NAME") or f"Kitchen display ({hostname})"
broker_host = config.get("MQTT_BROKER_HOST", "")
broker_port = int(config.get("MQTT_BROKER_PORT") or 1883)
client = make_client(f"kitchen-display-agent-{node_id}")
if config.get("MQTT_USERNAME"):
client.username_pw_set(config["MQTT_USERNAME"], config.get("MQTT_PASSWORD") or None)
discovery = Discovery(client, node_id, friendly_name)
def on_show(fragment: str) -> None:
launch_pantry_kiosk(fragment)
def on_connect(_client, _userdata, _flags, rc):
if rc != 0:
log.error("MQTT connection refused (rc=%s)", rc)
return
log.info("connected to MQTT broker %s:%s", broker_host, broker_port)
discovery.register_screens(on_show)
discovery.subscribe_all()
discovery.publish_available(True)
def on_disconnect(_client, _userdata, rc):
log.warning("disconnected from MQTT broker (rc=%s); paho will retry", rc)
def on_message(_client, _userdata, message):
discovery.dispatch(message.topic, message.payload.decode("utf-8", "replace"))
client.on_connect = on_connect
client.on_disconnect = on_disconnect
client.on_message = on_message
client.will_set(discovery.availability_topic, "offline", qos=1, retain=True)
stop_event = threading.Event()
def handle_signal(_signum, _frame):
stop_event.set()
signal.signal(signal.SIGTERM, handle_signal)
signal.signal(signal.SIGINT, handle_signal)
if not broker_host:
log.error("MQTT_BROKER_HOST is not set in %s — running without HA control", CONFIG_PATH)
else:
# connect_async + loop_start, never a blocking connect(): same "reactive path
# never depends on a remote service" rule as every other host's agent — the
# kitchen display's local scan/inventory/recipes flow must work with the
# container host powered off, only the HA "Show X" buttons need it.
client.connect_async(broker_host, broker_port, keepalive=60)
client.loop_start()
log.info("kitchen-display-agent %s started (node_id=%s)", node_id, node_id)
try:
stop_event.wait()
finally:
log.info("shutting down")
if broker_host:
discovery.publish_available(False)
client.loop_stop()
client.disconnect()
return 0
if __name__ == "__main__":
raise SystemExit(main())

View File

@ -0,0 +1,105 @@
"""Home Assistant MQTT Discovery payloads and command dispatch.
SECURITY BOUNDARY this module is the entire remote-control API of the kitchen
display. Same principle as every other host's agent in this project (Phase 11.4):
the local LLM never gets a network path to this machine directly. The only chain is
LLM tool call -> Home Assistant service call -> MQTT -> this dispatcher. There is no
HTTP listener, no websocket server, no exposed Sway IPC socket here.
This is deliberately NOT how the LLM reaches pantry-vision's actual inventory data —
that's a separate, already-published API (pantry-vision/server.py) that Home
Assistant or the LLM can call directly for reads/writes to Grocy. This agent only
ever controls what the physical screen is showing, exactly the same division of
labour as hosts/thin-client's "Show admin canvas" button vs. admin-canvas's own
write API.
"""
from __future__ import annotations
import json
import logging
from typing import Callable
from . import __version__
log = logging.getLogger(__name__)
DISCOVERY_PREFIX = "homeassistant"
class Discovery:
def __init__(self, client, node_id: str, friendly_name: str):
self.client = client
self.node_id = node_id
self.friendly_name = friendly_name
self.base = f"kitchendisplay/{node_id}"
self.availability_topic = f"{self.base}/availability"
self._handlers: dict[str, Callable[[str], None]] = {}
self.device = {
"identifiers": [f"kitchendisplay_{node_id}"],
"name": friendly_name,
"manufacturer": "SmartestHome",
"model": "Sway kitchen display",
"sw_version": __version__,
}
def _publish_config(self, component: str, object_id: str, payload: dict) -> None:
payload = {
"availability_topic": self.availability_topic,
"device": self.device,
"unique_id": f"{self.node_id}_{object_id}",
**payload,
}
topic = f"{DISCOVERY_PREFIX}/{component}/{self.node_id}/{object_id}/config"
self.client.publish(topic, json.dumps(payload), qos=1, retain=True)
def _command_topic(self, suffix: str, handler) -> str:
topic = f"{self.base}/{suffix}"
self._handlers[topic] = handler
return topic
def subscribe_all(self) -> None:
for topic in self._handlers:
self.client.subscribe(topic, qos=1)
def dispatch(self, topic: str, payload: str) -> None:
handler = self._handlers.get(topic)
if handler is None:
log.warning("no handler for %s", topic)
return
try:
handler(payload)
except Exception:
log.exception("handler for %s failed", topic)
def publish_available(self, available: bool = True) -> None:
self.client.publish(
self.availability_topic,
"online" if available else "offline",
qos=1,
retain=True,
)
def register_screens(self, on_show) -> None:
"""One button per screen — payload is the fixed page fragment (scan/
inventory/recipes), an enumerated constant handled in main.py, never passed
through as a URL/argv element unchanged (same rule as every other agent's
mqtt_discovery.py in this project).
"""
for object_id, name, fragment, icon in (
("show_scan", "Show scan", "scan", "mdi:camera"),
("show_inventory", "Show inventory", "inventory", "mdi:fridge-outline"),
("show_recipes", "Show recipes", "recipes", "mdi:chef-hat"),
):
self._publish_config(
"button",
object_id,
{
"name": name,
"command_topic": self._command_topic(
f"show/{fragment}", lambda _payload, fragment=fragment: on_show(fragment)
),
"icon": icon,
},
)

View File

@ -0,0 +1,47 @@
"""Session-environment resolution and process launching.
A trimmed version of hosts/thin-client's/hosts/touch-panel's sway_control.py: this
device has only one Sway workspace and one app (the pantry-kiosk Chromium window), so
there is no workspace-switching or window-focus surface to wrap just enough to run
`/usr/local/bin/pantry-kiosk [fragment]` with the graphical session's environment.
"""
from __future__ import annotations
import glob
import logging
import os
import subprocess
log = logging.getLogger(__name__)
def runtime_dir() -> str:
return os.environ.get("XDG_RUNTIME_DIR") or f"/run/user/{os.getuid()}"
def session_env() -> dict[str, str]:
env = dict(os.environ)
env["XDG_RUNTIME_DIR"] = runtime_dir()
env.setdefault("DBUS_SESSION_BUS_ADDRESS", f"unix:path={runtime_dir()}/bus")
env.setdefault("WAYLAND_DISPLAY", "wayland-1")
matches = sorted(glob.glob(os.path.join(runtime_dir(), "sway-ipc.*.sock")))
if matches:
env["SWAYSOCK"] = matches[-1]
return env
def launch_pantry_kiosk(fragment: str) -> None:
command = ["/usr/local/bin/pantry-kiosk", fragment] if fragment else ["/usr/local/bin/pantry-kiosk"]
log.info("launching %s", " ".join(command))
try:
subprocess.Popen(
command,
env=session_env(),
stdin=subprocess.DEVNULL,
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL,
start_new_session=True,
)
except OSError as exc:
log.error("could not launch pantry-kiosk: %s", exc)

View File

@ -0,0 +1,4 @@
# The ISO installs this from apt (python3-paho-mqtt) rather than pip — see
# live-build/config/package-lists/kitchen-display.list.chroot. This file is for
# running the agent outside the image (development, a venv on a test box).
paho-mqtt>=1.6

View File

@ -0,0 +1,9 @@
# greetd — autologin straight into the kiosk Sway session, no greeter UI. Same
# shape as hosts/touch-panel's and hosts/thin-client's own config.toml.
[terminal]
vt = 1
[default_session]
command = "/usr/local/bin/kiosk-session"
user = "@KIOSK_USERNAME@"

View File

@ -0,0 +1,17 @@
#!/bin/sh
# greetd's default_session command. Installed to /usr/local/bin/kiosk-session.
set -eu
[ -r /etc/kitchen-display-agent/config.env ] && . /etc/kitchen-display-agent/config.env
export MQTT_BROKER_HOST PANTRY_WEB_URL PANTRY_VISION_URL PANTRY_VISION_TOKEN
export XDG_CURRENT_DESKTOP=sway
export XDG_SESSION_TYPE=wayland
export XDG_SESSION_DESKTOP=sway
export MOZ_ENABLE_WAYLAND=1
export OZONE_PLATFORM=wayland
: "${XDG_RUNTIME_DIR:=/run/user/$(id -u)}"
export XDG_RUNTIME_DIR
exec sway

View File

@ -0,0 +1,56 @@
# Sway kiosk session for the kitchen display (docs/project-plan.md Phase 17).
#
# Installed to /home/<KIOSK_USERNAME>/.config/sway/config by
# build-kitchen-display-iso.sh. A single-purpose device — one workspace, one
# Chromium kiosk window pointed at pantry-web's frontend (Scan / Inventory / Recipes
# is in-page tab navigation there, not separate Sway workspaces the way
# hosts/touch-panel switches between three different apps).
set $mod Mod4
set $ws_pantry 1:pantry
input type:keyboard {
xkb_layout @KEYBOARD_LAYOUT@
}
# Real touchscreen hardware — native Wayland wl_touch, same assumption and same
# caveat as hosts/touch-panel/configs/sway/config (see that host's README
# verification list item 3).
input type:touch {
map_to_output "*"
}
output * bg #101014 solid_color
default_border none
default_floating_border none
hide_edge_borders both
gaps inner 0
gaps outer 0
# kitchen-display-agent is NOT started here — systemd owns it, same reasoning as
# every other agent in this project (reachable over MQTT whether or not a graphical
# session is up, survives a sway restart).
for_window [app_id="chromium.*"] fullscreen enable
for_window [app_id="chromium.*"] inhibit_idle fullscreen
exec sh -c '[ -n "$PANTRY_WEB_URL" ] && { swaymsg workspace $ws_pantry; /usr/local/bin/pantry-kiosk; }'
# Blank after 20 minutes; any touch resumes it. No lock screen — same reasoning as
# every other kiosk image in this project: a shared kitchen display has nobody to
# unlock it.
exec swayidle -w \
timeout 1200 'swaymsg "output * power off"' \
resume 'swaymsg "output * power on"'
bindsym $mod+Shift+Ctrl+m exec foot --title maintenance-shell
for_window [title="maintenance-shell"] floating enable, resize set width 800 height 500, move position center
bindsym $mod+Return exec foot
bindsym $mod+q kill
bindsym $mod+Shift+c reload
# Deliberately no exit binding — same reasoning as the thin client/touch panel.
workspace $ws_pantry

View File

@ -0,0 +1,58 @@
#!/bin/sh
# Chromium kiosk window pointed at pantry-vision's frontend (pantry-web). Installed
# to /usr/local/bin/pantry-kiosk. Called both at session start and by
# kitchen-display-agent when a "Show scan/inventory/recipes" MQTT command arrives.
#
# Kill-and-relaunch, not focus-and-navigate — same reasoning as hosts/thin-client's
# digest-browser: Chromium's --kiosk window has no address bar to drive remotely, and
# the page underneath (pantry-vision/frontend/) is cheap to reload. Camera capture is
# momentary (hold item up, tap Capture) so losing an in-progress scan to a "Show
# inventory" command is an acceptable, rare edge case, not a design flaw to solve here.
set -eu
PROFILE_DIR="${HOME:-/home/$(id -un)}/.config/kitchen-display-chromium"
BASE_URL="${PANTRY_WEB_URL:-}"
[ -n "$BASE_URL" ] || { echo "pantry-kiosk: PANTRY_WEB_URL is unset" >&2; exit 1; }
[ -n "${PANTRY_VISION_URL:-}" ] || { echo "pantry-kiosk: PANTRY_VISION_URL is unset" >&2; exit 1; }
[ -n "${PANTRY_VISION_TOKEN:-}" ] || { echo "pantry-kiosk: PANTRY_VISION_TOKEN is unset" >&2; exit 1; }
# api=/token= are read by frontend/app.js — see that file's top comment for why
# config travels this way instead of being baked into the served static files.
FRAGMENT="${1:-}"
URL="${BASE_URL%/}/index.html?api=$(printf '%s' "$PANTRY_VISION_URL" | sed 's/\//%2F/g; s/:/%3A/g')&token=${PANTRY_VISION_TOKEN}"
[ -n "$FRAGMENT" ] && URL="${URL}#${FRAGMENT}"
if command -v chromium >/dev/null 2>&1; then
CHROMIUM=chromium
elif command -v chromium-browser >/dev/null 2>&1; then
CHROMIUM=chromium-browser
else
echo "pantry-kiosk: no chromium/chromium-browser binary found" >&2
exit 1
fi
mkdir -p "$PROFILE_DIR"
pkill -u "$(id -u)" -f "$CHROMIUM .*--user-data-dir=$PROFILE_DIR" 2>/dev/null || true
i=0
while pgrep -u "$(id -u)" -f "$CHROMIUM .*--user-data-dir=$PROFILE_DIR" >/dev/null 2>&1 && [ "$i" -lt 20 ]; do
sleep 0.25
i=$((i + 1))
done
# --use-fake-ui-for-media-stream auto-accepts the getUserMedia camera prompt instead
# of leaving it sitting unanswered on a screen nobody is there to click "Allow" on —
# VERIFY: the simplest documented way to do this: the alternative,
# VideoCaptureAllowedUrls enterprise policy, needs a policies.json this image doesn't
# currently ship, so start with the flag and only add that if it proves insufficient.
exec "$CHROMIUM" \
--user-data-dir="$PROFILE_DIR" \
--ozone-platform=wayland \
--kiosk --app="$URL" \
--use-fake-ui-for-media-stream \
--start-fullscreen \
--noerrdialogs --disable-infobars --disable-session-crashed-bubble \
--overscroll-history-navigation=0 \
--touch-events=enabled \
--check-for-update-interval=31536000

View File

@ -0,0 +1,42 @@
#!/bin/sh
# Creates the kiosk account. Identical logic to hosts/touch-panel's/hosts/thin-client's
# own 0100-user-setup.hook.chroot.
set -eu
. /etc/kitchen-display-agent/config.env
if ! id "$KIOSK_USERNAME" >/dev/null 2>&1; then
useradd --create-home --shell /bin/bash --comment "Kitchen display kiosk session" "$KIOSK_USERNAME"
fi
for grp in audio video input render dialout netdev plugdev seat _seatd; do
if getent group "$grp" >/dev/null 2>&1; then
adduser "$KIOSK_USERNAME" "$grp" >/dev/null
fi
done
passwd --lock "$KIOSK_USERNAME" >/dev/null
adduser "$KIOSK_USERNAME" sudo >/dev/null
cat > "/etc/sudoers.d/010-${KIOSK_USERNAME}" <<EOF
${KIOSK_USERNAME} ALL=(ALL) NOPASSWD: ALL
EOF
chmod 0440 "/etc/sudoers.d/010-${KIOSK_USERNAME}"
mkdir -p /etc/ssh/sshd_config.d
cat > /etc/ssh/sshd_config.d/10-kitchen-display.conf <<'EOF'
PermitRootLogin no
PasswordAuthentication no
KbdInteractiveAuthentication no
PubkeyAuthentication yes
EOF
if [ -d "/home/${KIOSK_USERNAME}/.ssh" ]; then
chmod 700 "/home/${KIOSK_USERNAME}/.ssh"
[ -f "/home/${KIOSK_USERNAME}/.ssh/authorized_keys" ] && \
chmod 600 "/home/${KIOSK_USERNAME}/.ssh/authorized_keys"
fi
chown -R "${KIOSK_USERNAME}:${KIOSK_USERNAME}" "/home/${KIOSK_USERNAME}"
systemctl enable ssh >/dev/null 2>&1 || true

View File

@ -0,0 +1,19 @@
#!/bin/sh
# Makes greetd the boot target. Identical logic to hosts/touch-panel's own
# 0200-greetd.hook.chroot.
set -eu
. /etc/kitchen-display-agent/config.env
chmod 0755 /usr/local/bin/kiosk-session
if ! id greeter >/dev/null 2>&1; then
useradd --system --create-home --home-dir /var/lib/greetd --shell /usr/sbin/nologin greeter
fi
systemctl enable greetd
systemctl set-default graphical.target
systemctl mask getty@tty1.service
mkdir -p "/home/${KIOSK_USERNAME}/.config/sway"
chown -R "${KIOSK_USERNAME}:${KIOSK_USERNAME}" "/home/${KIOSK_USERNAME}/.config"

View File

@ -0,0 +1,12 @@
#!/bin/sh
# Installs the kitchen-display-agent systemd unit. Identical logic to
# hosts/touch-panel's own 0400-touchpanel-agent.hook.chroot.
set -eu
install -m 0644 /opt/kitchen-display-agent/kitchen-display-agent.service \
/etc/systemd/system/kitchen-display-agent.service
chmod 0644 /etc/kitchen-display-agent/config.env
chown -R root:root /opt/kitchen-display-agent
systemctl enable kitchen-display-agent

View File

@ -0,0 +1,46 @@
# Kitchen-display package list (live-build .list.chroot format: one package per
# line). Phase 17, docs/project-plan.md.
# --- Wayland compositor / session ---
sway
swayidle
swaybg
greetd
foot
# --- Browser (the only app on this device — a Chromium kiosk window pointed at
# pantry-web's frontend, which uses getUserMedia for camera capture directly in
# the browser rather than a native capture app) ---
chromium
# --- Agent runtime ---
python3
python3-venv
python3-paho-mqtt
procps
# --- Networking ---
network-manager
# --- Out-of-band admin — no wayvnc on this image either, same scope decision as
# hosts/touch-panel (see that host's README) ---
openssh-server
sudo
# --- Camera troubleshooting (`v4l2-ctl --list-devices` to confirm the webcam
# enumerates correctly) — Chromium/getUserMedia talks to it directly, nothing
# here shells out to v4l2 itself ---
v4l-utils
# --- Firmware for real hardware ---
firmware-linux
firmware-iwlwifi
firmware-realtek
firmware-misc-nonfree
# --- Misc ---
ca-certificates
curl
openssl
less
vim-tiny

View File

@ -0,0 +1,3 @@
# Only relevant when ENABLE_INSTALLER="true" in build-kitchen-display-iso.sh (off by
# default). Identical mechanism/caveat as the other hosts' preseed.cfg.
d-i keyboard-configuration/xkb-keymap select de

View File

@ -0,0 +1,265 @@
#!/usr/bin/env bash
#
# Smart Home Kitchen-Display ISO Builder
# Target: builds a Debian 12 (Bookworm) live ISO on a Debian/Ubuntu build machine
#
# Drives `lb config && lb build` over hosts/kitchen-display/live-build/ to produce
# the fridge/pantry camera-cataloguing display described in docs/project-plan.md
# Phase 17:
# - greetd autologin straight into a kiosk Sway session (no greeter UI)
# - ONE Chromium kiosk window pointed at pantry-web's frontend (Scan / Inventory /
# Recipes is in-page tab navigation, not separate Sway workspaces)
# - kitchen-display-agent (Python, systemd) — HA MQTT "Show scan/inventory/recipes"
# buttons only; the actual camera capture and Grocy read/write path is
# pantry-vision's own published API, called directly by the browser
#
# Reuses hosts/thin-client's live-build toolchain and directory-split convention, not
# its live-build tree — same relationship hosts/touch-panel and hosts/audio-endpoint
# already have to the thin client's.
#
# Run as: sudo ./build-kitchen-display-iso.sh
#
# EDIT THE VARIABLES BELOW BEFORE RUNNING.
set -euo pipefail
# ---------------------------------------------------------------------------
# CONFIGURATION — edit these before running
# ---------------------------------------------------------------------------
DEBIAN_RELEASE="bookworm"
KIOSK_USERNAME="kiosk"
IMAGE_HOSTNAME="kitchen-display"
KITCHEN_DISPLAY_NAME="Kitchen fridge display"
KEYBOARD_LAYOUT="de"
ENABLE_INSTALLER="false"
# --- Where the kitchen display talks to -------------------------------------
MQTT_BROKER_HOST="192.168.1.10" # <-- EDIT: container-host IP running Mosquitto
MQTT_BROKER_PORT="1883"
MQTT_USERNAME=""
MQTT_PASSWORD=""
# pantry-web (nginx, serves pantry-vision/frontend/) and pantry-vision (the API the
# frontend calls directly from the browser) — both from ENABLE_PANTRY_VISION in
# setup-container-host.sh. Placeholders until that's deployed; the image builds and
# boots fine without it, the kiosk window just shows a connection error.
PANTRY_WEB_URL="http://192.168.1.10:8096" # <-- EDIT once pantry-web is deployed
PANTRY_VISION_URL="http://192.168.1.10:8095" # <-- EDIT once pantry-vision is deployed
# Same value as PANTRY_VISION_TOKEN in pantry-vision/pantry-vision.env — there is no
# way for this repo to push it between the two hosts for you, same as every other
# credential pair that spans two machines in this project.
PANTRY_VISION_TOKEN="" # <-- EDIT: must match pantry-vision's own token
SSH_AUTHORIZED_KEY=""
# ---------------------------------------------------------------------------
# Paths
# ---------------------------------------------------------------------------
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
KITCHEN_DISPLAY_DIR="$(dirname "$SCRIPT_DIR")"
CONFIGS_DIR="${KITCHEN_DISPLAY_DIR}/configs"
AGENT_DIR="${KITCHEN_DISPLAY_DIR}/agent"
LIVE_BUILD_DIR="${KITCHEN_DISPLAY_DIR}/live-build"
INCLUDES="${LIVE_BUILD_DIR}/config/includes.chroot"
PACKAGE_LIST="${LIVE_BUILD_DIR}/config/package-lists/kitchen-display.list.chroot"
# ---------------------------------------------------------------------------
# Sanity checks
# ---------------------------------------------------------------------------
if [[ $EUID -ne 0 ]]; then
echo "Warning: not running as root. 'lb build' needs root; re-run with: sudo $0"
echo " Continuing anyway so you can at least regenerate includes.chroot..."
fi
if ! command -v lb &> /dev/null; then
if [[ $EUID -eq 0 ]]; then
echo "--- Installing live-build ---"
apt-get update
apt-get install -y live-build
else
echo "live-build is not installed and this script is not running as root." >&2
echo " Install it first: sudo apt-get install live-build" >&2
exit 1
fi
else
echo "--- live-build already installed, skipping ---"
fi
if [[ ! -f "$PACKAGE_LIST" ]]; then
echo "Missing package list: $PACKAGE_LIST" >&2
exit 1
fi
if [[ "$MQTT_BROKER_HOST" == "192.168.1.10" ]]; then
echo "Warning: MQTT_BROKER_HOST is still the placeholder IP — edit it before building."
fi
if [[ -z "$PANTRY_VISION_TOKEN" ]]; then
echo "Warning: PANTRY_VISION_TOKEN is empty. The kiosk window will load but every"
echo " call to pantry-vision will fail (401) until this matches the token in"
echo " pantry-vision/pantry-vision.env on the container host."
fi
echo
echo "=== Smart Home Kitchen-Display ISO Builder ==="
echo "Debian release : $DEBIAN_RELEASE"
echo "Kiosk user : $KIOSK_USERNAME"
echo "Image hostname : $IMAGE_HOSTNAME"
echo "MQTT broker : ${MQTT_BROKER_HOST}:${MQTT_BROKER_PORT}"
echo "pantry-web : $PANTRY_WEB_URL"
echo "pantry-vision : $PANTRY_VISION_URL"
echo "Keyboard layout : $KEYBOARD_LAYOUT"
echo
# ---------------------------------------------------------------------------
# 1. Regenerate includes.chroot from configs/ and agent/
# ---------------------------------------------------------------------------
echo "--- Regenerating $INCLUDES ---"
rm -rf "$INCLUDES"
mkdir -p \
"$INCLUDES/etc/greetd" \
"$INCLUDES/etc/kitchen-display-agent" \
"$INCLUDES/usr/local/bin" \
"$INCLUDES/opt/kitchen-display-agent" \
"$INCLUDES/home/${KIOSK_USERNAME}/.config/sway" \
"$INCLUDES/home/${KIOSK_USERNAME}/.ssh"
subst() {
sed -e "s/@KIOSK_USERNAME@/${KIOSK_USERNAME}/g" \
-e "s/@KEYBOARD_LAYOUT@/${KEYBOARD_LAYOUT}/g" "$1" > "$2"
}
subst "${CONFIGS_DIR}/greetd/config.toml" "$INCLUDES/etc/greetd/config.toml"
subst "${CONFIGS_DIR}/sway/config" "$INCLUDES/home/${KIOSK_USERNAME}/.config/sway/config"
subst "${AGENT_DIR}/kitchen-display-agent.service" "$INCLUDES/opt/kitchen-display-agent/kitchen-display-agent.service"
install -m 0755 "${CONFIGS_DIR}/greetd/kiosk-session" "$INCLUDES/usr/local/bin/kiosk-session"
install -m 0755 "${CONFIGS_DIR}/sway/pantry-kiosk" "$INCLUDES/usr/local/bin/pantry-kiosk"
mkdir -p "$INCLUDES/etc/default"
cat > "$INCLUDES/etc/default/keyboard" <<EOF
XKBMODEL="pc105"
XKBLAYOUT="${KEYBOARD_LAYOUT}"
XKBVARIANT=""
XKBOPTIONS=""
BACKSPACE="guess"
EOF
cp -r "${AGENT_DIR}/kitchen_display_agent" "$INCLUDES/opt/kitchen-display-agent/"
install -m 0644 "${AGENT_DIR}/requirements.txt" "$INCLUDES/opt/kitchen-display-agent/requirements.txt"
find "$INCLUDES/opt/kitchen-display-agent" -name '__pycache__' -type d -prune -exec rm -rf {} +
if [[ -n "$SSH_AUTHORIZED_KEY" ]]; then
echo "$SSH_AUTHORIZED_KEY" > "$INCLUDES/home/${KIOSK_USERNAME}/.ssh/authorized_keys"
chmod 600 "$INCLUDES/home/${KIOSK_USERNAME}/.ssh/authorized_keys"
echo " Baked an SSH authorized_keys entry for ${KIOSK_USERNAME}."
else
echo " No SSH_AUTHORIZED_KEY set — SSH admin access will not be possible on this image."
fi
# ---------------------------------------------------------------------------
# 2. Runtime config
# ---------------------------------------------------------------------------
echo "--- Writing /etc/kitchen-display-agent/config.env into includes.chroot ---"
cat > "$INCLUDES/etc/kitchen-display-agent/config.env" <<EOF
# Generated by hosts/kitchen-display/scripts/build-kitchen-display-iso.sh — do not
# hand-edit here; edit the CONFIGURATION block in that script and rebuild.
KIOSK_USERNAME=${KIOSK_USERNAME}
KITCHEN_DISPLAY_NAME=${KITCHEN_DISPLAY_NAME}
MQTT_BROKER_HOST=${MQTT_BROKER_HOST}
MQTT_BROKER_PORT=${MQTT_BROKER_PORT}
MQTT_USERNAME=${MQTT_USERNAME}
MQTT_PASSWORD=${MQTT_PASSWORD}
PANTRY_WEB_URL=${PANTRY_WEB_URL}
PANTRY_VISION_URL=${PANTRY_VISION_URL}
PANTRY_VISION_TOKEN=${PANTRY_VISION_TOKEN}
EOF
chmod 0644 "$INCLUDES/etc/kitchen-display-agent/config.env"
# ---------------------------------------------------------------------------
# 3. lb config
# ---------------------------------------------------------------------------
cd "$LIVE_BUILD_DIR"
chmod +x config/hooks/normal/*.hook.chroot
if [[ -e .build ]]; then
echo "--- Previous build found, running 'lb clean' (package cache is kept) ---"
lb clean
fi
INSTALLER_MODE="none"
if [[ "$ENABLE_INSTALLER" == "true" ]]; then
INSTALLER_MODE="live"
fi
echo "--- Running lb config ---"
lb config \
--distribution "$DEBIAN_RELEASE" \
--architectures amd64 \
--linux-flavours amd64 \
--archive-areas "main contrib non-free-firmware" \
--binary-images iso-hybrid \
--debian-installer "$INSTALLER_MODE" \
--iso-application "SmartestHome Kitchen Display" \
--iso-publisher "SmartestHome" \
--iso-volume "smarthome-kitchen-display" \
--memtest none \
--bootappend-live "boot=live components quiet splash noautologin username=${KIOSK_USERNAME} hostname=${IMAGE_HOSTNAME}"
# ---------------------------------------------------------------------------
# 4. lb build
# ---------------------------------------------------------------------------
echo "--- Running lb build (this takes a while and needs network) ---"
lb build
ISO_PATH="$(find "$LIVE_BUILD_DIR" -maxdepth 1 -name '*.iso' -print -quit)"
ISO_PATH="${ISO_PATH:-${LIVE_BUILD_DIR}/live-image-amd64.hybrid.iso}"
echo
echo "=== Done ==="
echo "ISO written to:"
echo " ${ISO_PATH}"
echo
echo "What's in it:"
echo " greetd : autologin as '${KIOSK_USERNAME}' straight into Sway on vt1"
echo " Sway : one workspace, one Chromium kiosk window"
echo " Kiosk page : ${PANTRY_WEB_URL}/index.html (Scan / Inventory / Recipes tabs)"
echo " Camera : getUserMedia in Chromium (--use-fake-ui-for-media-stream"
echo " auto-accepts the permission prompt), not a native capture app"
echo " kitchen-display-agent: system service, MQTT ${MQTT_BROKER_HOST}:${MQTT_BROKER_PORT}"
echo " Keyboard layout : ${KEYBOARD_LAYOUT} (console + Sway)"
echo " Maintenance shell : Super+Shift+Ctrl+M opens a floating terminal locally"
echo " Remote admin : SSH only — no wayvnc, same scope decision as hosts/touch-panel"
echo
echo "Next steps:"
echo " 1. Write the ISO to a USB stick:"
echo " sudo dd if=${ISO_PATH} of=/dev/sdX bs=4M status=progress oflag=sync"
echo " (double-check /dev/sdX with 'lsblk' first)"
echo " 2. Boot the target machine — a webcam pointed at the counter/fridge front is"
echo " required hardware, not built in; see the README for what's assumed about it."
echo " 3. Confirm greetd autologin lands in Sway with the Chromium kiosk window up."
echo " 4. Grant/verify camera access: --use-fake-ui-for-media-stream should skip the"
echo " prompt entirely; if the Scan tab shows a camera error, check"
echo " 'v4l2-ctl --list-devices' and that Chromium picked the right one (Chromium"
echo " defaults to the system default video input — untested on real hardware)."
echo " 5. Confirm kitchen-display-agent connected and registered:"
echo " systemctl status kitchen-display-agent"
echo " In Home Assistant, a '${KITCHEN_DISPLAY_NAME}' device should appear under the"
echo " MQTT integration with: Show scan / Show inventory / Show recipes (buttons)."
echo " 6. Hold a grocery item up to the camera, tap Capture, and confirm a proposal"
echo " comes back from pantry-vision within a few seconds — first checkpoint for"
echo " whether the chosen Ollama vision model's latency is workable at all."
echo
echo "Then pull the power on the container host and re-check: the kiosk must still"
echo "boot (Scan/Inventory/Recipes will show connection errors, which is the expected"
echo "degraded state — this device has no local fallback data source, same acceptance"
echo "already documented for hosts/touch-panel's Home workspace)."
echo
echo "Rebuilding later: edit hosts/kitchen-display/configs/* or agent/*, then re-run"
echo "this script — includes.chroot is regenerated from them every time."

203
hosts/touch-panel/README.md Normal file
View File

@ -0,0 +1,203 @@
# Sway touch panel
Phase 16 of `docs/project-plan.md`. Builds a Debian 12 live ISO for a touch-driven
wall/counter panel: autologin into Sway, three fixed apps (Spotify, Home Assistant,
a general web browser) switched with a finger via an always-on dock, and also
controllable by Home Assistant/the local LLM over MQTT.
**A different device from `../thin-client/`**, not a variant of it. The thin client is
a couch-distance media station whose primary control surface is HA/MQTT and wayvnc,
deliberately with no on-screen bars. This one is touched directly, so it needs the
opposite: a persistent, finger-sized on-screen dock, native touchscreen input, and no
gesture/remote-control machinery. It reuses the thin client's live-build toolchain and
`configs/`+`agent/` directory-split convention, not its live-build tree — same
relationship as `../audio-endpoint/`'s amd64 build to the thin client's.
What ends up on the image:
| | |
|---|---|
| Compositor | Sway, workspaces `1:spotify` / `2:home` / `3:web`, always-on touch dock |
| Autologin | greetd, `default_session` straight into `/usr/local/bin/kiosk-session` |
| Touch input | Native Wayland `wl_touch` (libinput), no pointer-emulation layer |
| Music | Full Spotify GUI client (Flathub `com.spotify.Client`) — not a headless Connect receiver |
| Home dashboard | Chromium in kiosk mode, `--app=$HA_URL`, auto-restart if it crashes |
| Web browser | Firefox, minimal chrome (back/forward/reload/address bar), general browsing |
| On-screen keyboard | wvkbd, toggled from the dock — see the caveat below, no auto-show |
| Remote control (HA/LLM) | `touchpanel-agent`, a systemd service publishing HA MQTT-discovery entities |
| Remote control (human) | SSH only — **no wayvnc on this image**, see below |
## Before you build
> **No touch-panel hardware has been chosen.** Nothing here has been booted on real
> metal or a real touchscreen. Touch-as-`wl_touch` (as opposed to some panels that only
> expose an emulated-mouse HID interface) is assumed, not confirmed.
Edit the `# CONFIGURATION` block at the top of
[`scripts/build-touch-panel-iso.sh`](scripts/build-touch-panel-iso.sh):
| Variable | What to put in it |
|---|---|
| `MQTT_BROKER_HOST` | LAN IP of the container host running Mosquitto (Phase 1) |
| `HA_URL` | Home Assistant URL, e.g. `http://192.168.1.10:8123` — this is what loads in the Home workspace |
| `KIOSK_USERNAME` | Autologin account name (`kiosk`) |
| `TOUCHPANEL_NAME` / `IMAGE_HOSTNAME` | Per-panel identity; each touch panel needs its own |
| `SSH_AUTHORIZED_KEY` | Optional. Password auth is disabled and there is no wayvnc, so without a key the only admin path is the local console. |
## Build
```sh
sudo ./scripts/build-touch-panel-iso.sh
```
Same directory-split convention as the thin client: `configs/` and `agent/` are the
human-edited, git-tracked source of truth; `live-build/config/includes.chroot/` is
**generated** — wiped and rebuilt on every run, gitignored, never hand-edited. The only
templated tokens are `@KIOSK_USERNAME@`/`@KEYBOARD_LAYOUT@`; everything else the
in-chroot hooks need comes from `/etc/touchpanel-agent/config.env`.
## Why there's no wayvnc on this image
The thin client's primary control surface is deliberately remote (HA/MQTT + wayvnc)
because nobody is standing at a couch-distance media station. This panel is the
opposite: it's mounted somewhere and touched directly, so remote *view/control* isn't
the primary need the way it is for the thin client — SSH already covers "something's
broken, let me fix it from a shell." If a real need for a remote-view channel shows up
once this is on real hardware, adding wayvnc back is a small, isolated change (copy
`../thin-client/configs/wayvnc/` and its two hooks) — deliberately left out for now
rather than built against a guess.
## Home Assistant login
The Chromium kiosk window's profile (`~/.config/touchpanel-chromium-ha`) is persistent
across restarts specifically so logging into Home Assistant once survives a reboot —
unlike the thin client's digest/admin kiosk Firefox profiles, which are stateless
pages re-copied on every launch. Log in on first boot (tap the dock's Home button if
the window doesn't already have focus) and it should stay logged in after that,
**unverified** — depends on HA's own session cookie lifetime and whether Chromium's
Wayland profile persistence behaves the way assumed.
## Spotify: full GUI, not a headless Connect receiver
`../thin-client/` and `../audio-endpoint/` both run `spotifyd`/`librespot` — a
Spotify Connect *receiver* with no interface of its own, controlled from the phone
app or HA. This panel runs the real, official Spotify Linux client instead
(`configs/spotify/spotify-launch`, Flathub `com.spotify.Client`), because the whole
point of a touch panel is a screen you interact with directly: browsing your library,
searching, picking a playlist with a finger. **Spotify Premium is required for
playback either way** — same requirement as the headless receivers.
Login is interactive, on the device, on first launch, and persists in the Flatpak's
own data directory (`~/.var/app/com.spotify.Client`) — nothing here wipes or reseeds
it.
`touchpanel_agent/mpris_bridge.py` bridges Spotify's MPRIS interface (via
`playerctl`) into an HA `media_player`-shaped set of entities, identical in shape to
the thin client's own bridge, just pointed at `spotify` instead of `mpv`/`spotifyd` as
the player name — **unverified that the Spotify Flatpak's MPRIS bus name is actually
`spotify`** (some Flatpak-sandboxed apps register under a `flatpak.<app-id>`-shaped
name instead). If Playback state never updates, check `playerctl -l` on the booted
panel and adjust `PLAYER_PRIORITY` in `mpris_bridge.py`.
## Touch dock
An always-visible bar reserved at the bottom of the screen (`configs/eww/`, layer-shell
`:exclusive true` so nothing ever tiles under or draws over it) with four buttons:
**Spotify**, **Home**, **Web**, and **Keyboard**. The first three call `swaymsg
workspace` directly — fixed constants in `eww.yuck`, nothing from MQTT/HA is ever
interpolated into them, same rule as the thin client's now-playing widget. This is the
touch-first equivalent of the thin client's HA **Screen** select entity — both exist,
one for a finger, one for the LLM.
Session-scoped (`exec_always` in `configs/sway/config`), not part of
`touchpanel-agent`, for the identical reason the thin client's now-playing widget
isn't: compositor UI should die with the compositor, not need a restart-independent
lifecycle or open a second inbound control channel into the security-sensitive agent.
**If `eww` isn't installed** (it's not in Debian bookworm main —
`0500-eww-dock.hook.chroot` tries apt and otherwise leaves a documented placeholder),
there is no on-screen app switcher at all. The panel still boots and shows Home; HA/
LLM-driven app switching via `touchpanel-agent`'s **Screen** select and **Show
\<app\>** buttons still works either way, since that's a separate control path.
## On-screen keyboard — no auto-show, by design
`configs/keyboard/toggle-keyboard` toggles `wvkbd` on and off. There is deliberately
**no automatic show-on-text-field-focus** — that needs the `text-input-v3`/
`virtual-keyboard-v1` Wayland protocols wired all the way through each app via a shell
component (the way Phosh does it for squeekboard), which this image doesn't run.
Instead: tap the dock's Keyboard button before typing, tap it again to dismiss. This
is a known, accepted limitation, not a bug to chase before the panel is otherwise
useful — see the verification list below if you want to pursue proper auto-show later.
`wvkbd`, not `squeekboard`, was picked specifically because it doesn't assume that
shell integration exists; see `0700-onscreen-keyboard.hook.chroot`'s comment. It is
also not in Debian bookworm main, so that hook has the same apt-first/documented-
placeholder shape as `eww` and the thin client's `spotifyd`/`librespot` install.
## Home Assistant entities
`touchpanel-agent` publishes MQTT-discovery configs on connect. Under the MQTT
integration you should get one device per touch panel with:
- **Show Spotify**, **Show Home**, **Show web browser** (buttons) — switch workspace
and, for Spotify/Home, focus the already-running app rather than relaunching it
(both are meant to stay open and stateful, unlike the thin client's stateless
digest/admin kiosk pages)
- **Screen** (select) — `1:spotify` / `2:home` / `3:web`
- **Playback state** (sensor, with track metadata as attributes), **Volume**
(number), and play/pause / next / previous / stop (buttons), bridged from
Spotify's own MPRIS interface
- A **media_player** discovery payload — see the thin client README's identical
caveat: stock Home Assistant's MQTT integration has no `media_player` platform: the
button/number/sensor entities above are what give playback control on a plain
install; the HACS *MQTT Media Player* integration is what picks up this one.
### Security boundary
Identical shape and reasoning to `../thin-client/README.md`'s section of the same
name: `touchpanel_agent/mqtt_discovery.py` is the **entire** inbound control surface
of this machine. The LLM never gets a direct network path here — the only chain is
*LLM tool call → HA service call → MQTT → touchpanel-agent*. No HTTP listener, no
websocket server, no exposed Sway IPC socket, no VNC (this image has none at all, see
above). New control features belong as additional MQTT entities, not as a second
listener.
## Manual verification still outstanding
None of this has been run on hardware. In rough order:
1. The ISO builds at all (`lb build` is network-heavy and can fail on mirror hiccups).
2. greetd lands in Sway with no login prompt, on vt1, with getty@tty1 masked.
3. **Touch input actually arrives as `wl_touch`**, not an emulated pointer — the whole
design (native touch in Chromium/Firefox/Spotify, no gesture-translation layer)
assumes a real Wayland touchscreen driver. If it only shows up as a mouse, tapping
will still mostly work by accident (a single point maps to a click) but pinch/
multi-touch in Chromium or Spotify won't.
4. `touchpanel-agent` connects to Mosquitto and the device appears in HA.
5. **Flathub app ID `com.spotify.Client` is correct** — flagged for verification in
`0300-flatpak-spotify.hook.chroot`, same as the thin client's Steam Link ID.
6. Spotify login persists in `~/.var/app/com.spotify.Client` across a reboot.
7. Spotify's real MPRIS bus name — assumed `spotify`, not confirmed (see the Spotify
section above); `playerctl -l` on the booted panel is the way to check.
8. **Chromium's Wayland `app_id` for a `--app=` kiosk window** — assumed to start with
`chromium` (`configs/sway/config`'s `for_window` rule and
`touchpanel_agent/main.py`'s `focus_criteria` both match on it). If focusing the
Home workspace's window from the dock/HA stops working, check the real app_id with
`swaymsg -t get_tree` and correct both places.
9. HA login persistence in the Chromium kiosk profile across a reboot (see above).
10. `eww` and `wvkbd` are not in Debian bookworm main — both hooks try apt and
otherwise leave a documented placeholder (dock absent / keyboard absent
respectively); neither has been checked against the real bookworm archive.
11. **The Firefox extension IDs in `configs/firefox/policies.json`** — same unverified
caveat as the thin client's copy; see that README's identical item.
12. **Phase 16's own "reactive path" check**: power off the container host and confirm
the panel still boots to Spotify + the browser (Home will show a connection
error, which is the expected/acceptable failure mode — unlike the thin client,
there's no local-media fallback to fall back to here since this device has no
local media source at all).
13. Keyboard layout defaults to German (`KEYBOARD_LAYOUT="de"`) — same per-image,
no-runtime-override handling as the thin client's identical setting.
14. Dock touch-target sizing (`configs/eww/eww.scss`, ~84px buttons) — a starting
guess, not measured against a real finger on real glass at the real screen DPI
chosen once hardware exists.

View File

@ -0,0 +1,8 @@
# The ISO installs this from apt (python3-paho-mqtt, 1.6.x on bookworm) rather than
# pip — see live-build/config/package-lists/touch-panel.list.chroot. This file is for
# running the agent outside the image (development, a venv on a test box). The code
# works against both the 1.x and 2.x callback APIs.
paho-mqtt>=1.6
# Config is read from /etc/touchpanel-agent/config.env by a small parser in main.py,
# so there is no python-dotenv dependency.

View File

@ -0,0 +1,31 @@
[Unit]
Description=Touch panel agent (Home Assistant MQTT control surface for Sway)
Documentation=file:///opt/touchpanel-agent
# A system unit, not a `systemctl --user` unit — see thinclient-agent.service's
# identical reasoning: this must be reachable over MQTT whether or not a graphical
# session is up, and must survive sway restarting. Session-scoped bits (SWAYSOCK,
# XDG_RUNTIME_DIR, DBUS_SESSION_BUS_ADDRESS) are derived at call time in
# sway_control.SwayControl.session_env().
#
# Deliberately NOT After=network-online.target: the panel must boot to a usable
# session (Spotify/HA/browser) with the container host powered off.
After=network.target
[Service]
Type=simple
User=@KIOSK_USERNAME@
Group=@KIOSK_USERNAME@
WorkingDirectory=/opt/touchpanel-agent
Environment=PYTHONPATH=/opt/touchpanel-agent
Environment=PYTHONUNBUFFERED=1
EnvironmentFile=-/etc/touchpanel-agent/config.env
ExecStart=/usr/bin/python3 -m touchpanel_agent.main
Restart=always
RestartSec=5
# Sandboxing stops here on purpose: this unit's whole job is to spawn GUI child
# processes (Chromium, the Spotify flatpak, Firefox) that need the real /tmp, the
# user's home, and the session bus.
NoNewPrivileges=true
[Install]
WantedBy=multi-user.target

View File

@ -0,0 +1,3 @@
"""touchpanel-agent — Home Assistant MQTT control surface for a Sway touch panel."""
__version__ = "0.1.0"

View File

@ -0,0 +1,211 @@
"""touchpanel-agent entrypoint."""
from __future__ import annotations
import logging
import os
import signal
import socket
import sys
import threading
from dataclasses import dataclass
import paho.mqtt.client as mqtt
from .mpris_bridge import MprisBridge
from .mqtt_discovery import Discovery
from .sway_control import WS_HOME, WS_SPOTIFY, WS_WEB, SwayControl
CONFIG_PATH = os.environ.get("TOUCHPANEL_AGENT_CONFIG", "/etc/touchpanel-agent/config.env")
CONFIG_KEYS = (
"MQTT_BROKER_HOST",
"MQTT_BROKER_PORT",
"MQTT_USERNAME",
"MQTT_PASSWORD",
"HA_URL",
"KIOSK_USERNAME",
"TOUCHPANEL_NAME",
)
WORKSPACES = (WS_SPOTIFY, WS_HOME, WS_WEB)
log = logging.getLogger("touchpanel-agent")
@dataclass(frozen=True)
class App:
name: str
command: list[str]
process_pattern: str | None = None
workspace: str | None = None
focus_criteria: str | None = None
icon: str = "mdi:application"
def load_config(path: str = CONFIG_PATH) -> dict[str, str]:
values: dict[str, str] = {}
try:
with open(path, encoding="utf-8") as handle:
for line in handle:
line = line.strip()
if not line or line.startswith("#") or "=" not in line:
continue
key, _, value = line.partition("=")
values[key.strip()] = value.strip().strip('"').strip("'")
except OSError as exc:
log.warning("could not read %s (%s); falling back to the environment", path, exc)
for key in CONFIG_KEYS:
if key in os.environ:
values[key] = os.environ[key]
return values
def build_apps(_config: dict[str, str]) -> dict[str, App]:
return {
"spotify": App(
name="Spotify",
command=["/usr/local/bin/spotify-launch"],
process_pattern="com.spotify.Client",
workspace=WS_SPOTIFY,
focus_criteria='app_id="spotify"',
icon="mdi:spotify",
),
"home": App(
name="Home",
# ha-kiosk is already running (started at session boot, see
# configs/sway/config) and supervises its own restart loop — this just
# switches workspace and focuses the existing window rather than
# relaunching it, since the HA dashboard is stateful. See ha-kiosk's own
# comment for why this is deliberately not a kill-and-relaunch like the
# thin client's digest canvas.
command=["/usr/local/bin/ha-kiosk"],
process_pattern="ha-kiosk",
workspace=WS_HOME,
focus_criteria='app_id="chromium.*"',
icon="mdi:home-assistant",
),
"web_browser": App(
name="web browser",
command=["/usr/local/bin/web-browser"],
workspace=WS_WEB,
icon="mdi:web",
),
}
def make_client(client_id: str) -> mqtt.Client:
# paho-mqtt 2.x requires an explicit callback API version; bookworm's
# python3-paho-mqtt is 1.6.x and has no such argument. VERSION1 is requested when
# available so the callback signatures below are identical under both.
callback_api = getattr(mqtt, "CallbackAPIVersion", None)
if callback_api is not None:
return mqtt.Client(callback_api.VERSION1, client_id=client_id)
return mqtt.Client(client_id=client_id)
def main() -> int:
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s %(levelname)s %(name)s: %(message)s",
stream=sys.stdout,
)
config = load_config()
hostname = socket.gethostname()
node_id = "".join(c if c.isalnum() else "_" for c in hostname).strip("_") or "touchpanel"
friendly_name = config.get("TOUCHPANEL_NAME") or f"Touch panel ({hostname})"
broker_host = config.get("MQTT_BROKER_HOST", "")
broker_port = int(config.get("MQTT_BROKER_PORT") or 1883)
sway = SwayControl()
apps = build_apps(config)
client = make_client(f"touchpanel-agent-{node_id}")
if config.get("MQTT_USERNAME"):
client.username_pw_set(config["MQTT_USERNAME"], config.get("MQTT_PASSWORD") or None)
discovery = Discovery(client, node_id, friendly_name)
mpris = MprisBridge(discovery.publish_media_state, sway.session_env)
def on_launch(key: str) -> None:
app = apps[key]
sway.launch_app(
app.command,
process_pattern=app.process_pattern,
workspace=app.workspace,
focus_criteria=app.focus_criteria,
)
if app.workspace:
discovery.publish_workspace(app.workspace)
def on_workspace(payload: str) -> None:
name = payload.strip()
# Enumerated, never passed through: see the security note in mqtt_discovery.py.
if name not in WORKSPACES:
log.warning("ignoring unknown workspace %r", name)
return
sway.switch_workspace(name)
discovery.publish_workspace(name)
def on_connect(_client, _userdata, _flags, rc):
if rc != 0:
log.error("MQTT connection refused (rc=%s)", rc)
return
log.info("connected to MQTT broker %s:%s", broker_host, broker_port)
discovery.register_media_player(mpris.handle_command, mpris.set_volume)
discovery.register_app_launchers(apps, on_launch)
discovery.register_workspace_select(WORKSPACES, on_workspace, WS_HOME)
discovery.subscribe_all()
discovery.publish_available(True)
def on_disconnect(_client, _userdata, rc):
log.warning("disconnected from MQTT broker (rc=%s); paho will retry", rc)
def on_message(_client, _userdata, message):
discovery.dispatch(message.topic, message.payload.decode("utf-8", "replace"))
client.on_connect = on_connect
client.on_disconnect = on_disconnect
client.on_message = on_message
client.will_set(discovery.availability_topic, "offline", qos=1, retain=True)
stop_event = threading.Event()
def handle_signal(_signum, _frame):
stop_event.set()
signal.signal(signal.SIGTERM, handle_signal)
signal.signal(signal.SIGINT, handle_signal)
if not broker_host:
log.error("MQTT_BROKER_HOST is not set in %s — running without HA control", CONFIG_PATH)
else:
# connect_async + loop_start, never a blocking connect(): the panel must come
# up and show Spotify/HA/the browser with the container host powered off,
# same "reactive path never depends on a remote service" rule as the thin
# client's Phase 11.10.
client.connect_async(broker_host, broker_port, keepalive=60)
client.loop_start()
log.info("touchpanel-agent %s started (node_id=%s)", node_id, node_id)
if not config.get("HA_URL"):
log.warning("HA_URL is not set — the Home workspace will have nothing to show")
try:
mpris.run_forever(stop_event)
finally:
log.info("shutting down")
if broker_host:
discovery.publish_available(False)
client.loop_stop()
client.disconnect()
return 0
if __name__ == "__main__":
raise SystemExit(main())

View File

@ -0,0 +1,127 @@
"""Bridges the Spotify client's MPRIS state to the HA media_player entity via playerctl.
Identical in shape and reasoning to
hosts/thin-client/agent/thinclient_agent/mpris_bridge.py, with PLAYER_PRIORITY pointed
at the Spotify GUI client's own MPRIS bus name instead of mpv/spotifyd — see that
file's docstring for why this polls via subprocess rather than holding a dbus
connection open.
"""
from __future__ import annotations
import logging
import subprocess
import threading
log = logging.getLogger(__name__)
PLAYER_PRIORITY = "spotify,%any"
_STATUS_TO_HA = {
"Playing": "playing",
"Paused": "paused",
"Stopped": "idle",
}
_METADATA_FORMAT = "{{title}}\x1f{{artist}}\x1f{{album}}\x1f{{mpris:length}}\x1f{{mpris:artUrl}}"
class MprisBridge:
def __init__(self, publish_state, env_provider, poll_interval: float = 2.0):
self._publish_state = publish_state
# playerctl needs DBUS_SESSION_BUS_ADDRESS, which this system service does not
# inherit; SwayControl.session_env() derives it from the kiosk user's runtime dir.
self._env_provider = env_provider
self._poll_interval = poll_interval
self._last_state: dict | None = None
def _playerctl(self, *args: str) -> str | None:
try:
result = subprocess.run(
["playerctl", "-p", PLAYER_PRIORITY, *args],
env=self._env_provider(),
capture_output=True,
text=True,
timeout=5,
check=False,
)
except (OSError, subprocess.TimeoutExpired) as exc:
log.debug("playerctl %s failed: %s", " ".join(args), exc)
return None
if result.returncode != 0:
return None
return result.stdout.strip()
def read_state(self) -> dict:
status = self._playerctl("status")
if status is None:
return {"state": "off"}
state = {"state": _STATUS_TO_HA.get(status, "idle")}
metadata = self._playerctl("metadata", "--format", _METADATA_FORMAT)
if metadata:
title, artist, album, length, art_url = (metadata.split("\x1f") + [""] * 5)[:5]
state["title"] = title
state["artist"] = artist
state["album"] = album
state["art_url"] = art_url
if length.isdigit():
state["duration"] = int(length) // 1_000_000
position = self._playerctl("position")
if position:
try:
state["position"] = int(float(position))
except ValueError:
pass
volume = self._playerctl("volume")
if volume:
try:
state["volume"] = round(float(volume), 3)
except ValueError:
pass
return state
def poll_once(self) -> None:
state = self.read_state()
if state != self._last_state:
self._last_state = state
self._publish_state(state)
def run_forever(self, stop_event: threading.Event) -> None:
while not stop_event.is_set():
try:
self.poll_once()
except Exception:
log.exception("MPRIS poll failed")
stop_event.wait(self._poll_interval)
# --- command side -------------------------------------------------------
def handle_command(self, command: str) -> None:
command = command.strip().upper()
action = {
"PLAY": ("play",),
"PAUSE": ("pause",),
"PLAY_PAUSE": ("play-pause",),
"TOGGLE": ("play-pause",),
"STOP": ("stop",),
"NEXT": ("next",),
"PREVIOUS": ("previous",),
"PREV": ("previous",),
}.get(command)
if action is None:
log.warning("ignoring unknown media command %r", command)
return
log.info("media command %s", command)
self._playerctl(*action)
self.poll_once()
def set_volume(self, level: float) -> None:
level = max(0.0, min(1.0, level))
self._playerctl("volume", f"{level:.3f}")
self.poll_once()

View File

@ -0,0 +1,193 @@
"""Home Assistant MQTT Discovery payloads and command dispatch.
SECURITY BOUNDARY this module is the entire remote-control API of the touch panel.
Same principle as hosts/thin-client/agent/thinclient_agent/mqtt_discovery.py (Phase
11.4), applied to this device: the local LLM never gets a network path to this
machine. The only chain is: LLM tool call -> Home Assistant service call -> MQTT ->
this dispatcher. That property holds only as long as this stays the sole inbound
control surface: no HTTP listener, no websocket server, no exposed Sway IPC socket, no
shell endpoint. A new feature belongs as another entity below, not as another
listener.
Every command handler here is a fixed, enumerated action. A payload never becomes an
argv element, a shell string, or a URL host see the launch table in main.py, which
builds every command from local constants and uses the payload only to pick between
known values.
"""
from __future__ import annotations
import json
import logging
from typing import Callable
from . import __version__
log = logging.getLogger(__name__)
DISCOVERY_PREFIX = "homeassistant"
class Discovery:
def __init__(self, client, node_id: str, friendly_name: str):
self.client = client
self.node_id = node_id
self.friendly_name = friendly_name
self.base = f"touchpanel/{node_id}"
self.availability_topic = f"{self.base}/availability"
self.media_state_topic = f"{self.base}/media/state"
self._handlers: dict[str, Callable[[str], None]] = {}
self.device = {
"identifiers": [f"touchpanel_{node_id}"],
"name": friendly_name,
"manufacturer": "SmartestHome",
"model": "Sway touch panel",
"sw_version": __version__,
}
# --- plumbing -----------------------------------------------------------
def _publish_config(self, component: str, object_id: str, payload: dict) -> None:
payload = {
"availability_topic": self.availability_topic,
"device": self.device,
"unique_id": f"{self.node_id}_{object_id}",
**payload,
}
topic = f"{DISCOVERY_PREFIX}/{component}/{self.node_id}/{object_id}/config"
self.client.publish(topic, json.dumps(payload), qos=1, retain=True)
def _command_topic(self, suffix: str, handler) -> str:
topic = f"{self.base}/{suffix}"
self._handlers[topic] = handler
return topic
def subscribe_all(self) -> None:
for topic in self._handlers:
self.client.subscribe(topic, qos=1)
def dispatch(self, topic: str, payload: str) -> None:
handler = self._handlers.get(topic)
if handler is None:
log.warning("no handler for %s", topic)
return
try:
handler(payload)
except Exception:
log.exception("handler for %s failed", topic)
def publish_available(self, available: bool = True) -> None:
self.client.publish(
self.availability_topic,
"online" if available else "offline",
qos=1,
retain=True,
)
def publish_media_state(self, state: dict) -> None:
self.client.publish(self.media_state_topic, json.dumps(state), qos=0, retain=True)
# --- entities -----------------------------------------------------------
def register_media_player(self, on_command, on_volume) -> None:
command_topic = self._command_topic("media/command", on_command)
volume_topic = self._command_topic("media/volume/set", lambda p: on_volume(float(p)))
# Core Home Assistant's MQTT integration has NO media_player platform — see
# hosts/thin-client/README.md's identical caveat. The button/number entities
# below give the same transport control with stock HA.
self._publish_config(
"media_player",
"media",
{
"name": "Media",
"state_topic": self.media_state_topic,
"state_template": "{{ value_json.state }}",
"command_topic": command_topic,
"volume_command_topic": volume_topic,
"volume_state_topic": self.media_state_topic,
"volume_template": "{{ value_json.volume }}",
"title_template": "{{ value_json.title }}",
"artist_template": "{{ value_json.artist }}",
"album_template": "{{ value_json.album }}",
},
)
for object_id, name, payload, icon in (
("media_play_pause", "Play/pause", "PLAY_PAUSE", "mdi:play-pause"),
("media_next", "Next track", "NEXT", "mdi:skip-next"),
("media_previous", "Previous track", "PREVIOUS", "mdi:skip-previous"),
("media_stop", "Stop", "STOP", "mdi:stop"),
):
self._publish_config(
"button",
object_id,
{
"name": name,
"command_topic": command_topic,
"payload_press": payload,
"icon": icon,
},
)
self._publish_config(
"sensor",
"media_state",
{
"name": "Playback state",
"state_topic": self.media_state_topic,
"value_template": "{{ value_json.state }}",
"json_attributes_topic": self.media_state_topic,
"icon": "mdi:spotify",
},
)
self._publish_config(
"number",
"media_volume",
{
"name": "Volume",
"command_topic": volume_topic,
"state_topic": self.media_state_topic,
"value_template": "{{ value_json.volume }}",
"min": 0,
"max": 1,
"step": 0.05,
"mode": "slider",
"icon": "mdi:volume-high",
},
)
def register_app_launchers(self, apps, on_launch) -> None:
for key, app in apps.items():
self._publish_config(
"button",
f"launch_{key}",
{
"name": f"Show {app.name}",
"command_topic": self._command_topic(
f"app/{key}/launch",
lambda _payload, key=key: on_launch(key),
),
"icon": app.icon,
},
)
def register_workspace_select(self, workspaces, on_workspace, state_topic_value) -> None:
self._publish_config(
"select",
"workspace",
{
"name": "Screen",
"command_topic": self._command_topic("workspace/set", on_workspace),
"state_topic": f"{self.base}/workspace/state",
"options": list(workspaces),
"icon": "mdi:view-dashboard",
},
)
self.client.publish(
f"{self.base}/workspace/state", state_topic_value, qos=1, retain=True
)
def publish_workspace(self, name: str) -> None:
self.client.publish(f"{self.base}/workspace/state", name, qos=1, retain=True)

View File

@ -0,0 +1,115 @@
"""Thin wrapper around swaymsg and local process launching.
Identical in shape to hosts/thin-client/agent/thinclient_agent/sway_control.py
duplicated rather than imported across hosts, same convention as the digest/admin
canvas SDK split in Phase 13.
"""
from __future__ import annotations
import glob
import logging
import os
import subprocess
log = logging.getLogger(__name__)
# Contract with configs/sway/config — these strings must match the `set $ws_*` lines.
WS_SPOTIFY = "1:spotify"
WS_HOME = "2:home"
WS_WEB = "3:web"
def runtime_dir() -> str:
return os.environ.get("XDG_RUNTIME_DIR") or f"/run/user/{os.getuid()}"
class SwayControl:
def _socket_path(self) -> str | None:
path = os.environ.get("SWAYSOCK")
if path and os.path.exists(path):
return path
# sway names the socket sway-ipc.<uid>.<pid>.sock, so the path changes every
# time sway restarts. touchpanel-agent is a system service that outlives the
# session, so the socket is re-resolved per call instead of cached at startup.
matches = sorted(glob.glob(os.path.join(runtime_dir(), "sway-ipc.*.sock")))
return matches[-1] if matches else None
def session_env(self) -> dict[str, str]:
env = dict(os.environ)
env["XDG_RUNTIME_DIR"] = runtime_dir()
env.setdefault("DBUS_SESSION_BUS_ADDRESS", f"unix:path={runtime_dir()}/bus")
env.setdefault("WAYLAND_DISPLAY", "wayland-1")
sock = self._socket_path()
if sock:
env["SWAYSOCK"] = sock
return env
def swaymsg(self, *args: str) -> str | None:
if self._socket_path() is None:
log.warning("no sway IPC socket found; dropping command %s", " ".join(args))
return None
try:
result = subprocess.run(
["swaymsg", *args],
env=self.session_env(),
capture_output=True,
text=True,
timeout=10,
check=False,
)
except (OSError, subprocess.TimeoutExpired) as exc:
log.warning("swaymsg %s failed: %s", " ".join(args), exc)
return None
if result.returncode != 0:
log.warning("swaymsg %s: %s", " ".join(args), result.stderr.strip())
return None
return result.stdout
def switch_workspace(self, name: str) -> None:
log.info("switching to workspace %s", name)
self.swaymsg("workspace", name)
def focus_window(self, criteria: str) -> None:
self.swaymsg(f"[{criteria}] focus")
def is_running(self, pattern: str) -> bool:
try:
result = subprocess.run(
["pgrep", "-u", str(os.getuid()), "-f", pattern],
capture_output=True,
timeout=5,
check=False,
)
except (OSError, subprocess.TimeoutExpired):
return False
return result.returncode == 0
def launch_app(
self,
command: list[str],
process_pattern: str | None = None,
workspace: str | None = None,
focus_criteria: str | None = None,
) -> None:
if workspace:
self.switch_workspace(workspace)
if process_pattern and self.is_running(process_pattern):
log.info("%s already running; focusing instead of launching", command[0])
if focus_criteria:
self.focus_window(focus_criteria)
return
log.info("launching %s", " ".join(command))
try:
subprocess.Popen(
command,
env=self.session_env(),
stdin=subprocess.DEVNULL,
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL,
start_new_session=True,
)
except OSError as exc:
log.error("could not launch %s: %s", " ".join(command), exc)

View File

@ -0,0 +1,50 @@
// Styling for the touch dock. Installed to
// /home/<KIOSK_USERNAME>/.config/eww/eww.scss by build-touch-panel-iso.sh.
//
// Touch targets are deliberately large (min ~72px tap area) this is a finger
// interface, not a mouse one.
$bg: rgba(16, 16, 20, 0.96);
$fg: #e8e8ec;
$muted: #9a9aa6;
$accent: #6ea8fe;
.dock-root {
background-color: $bg;
padding: 6px 12px;
}
.dock-btn {
min-width: 84px;
min-height: 68px;
background-color: transparent;
border: none;
border-radius: 12px;
padding: 4px 10px;
color: $muted;
&:active {
background-color: rgba(255, 255, 255, 0.14);
}
}
.dock-btn-active {
background-color: rgba(110, 168, 254, 0.16);
color: $fg;
}
.dock-icon {
font-size: 26px;
}
.dock-label {
font-size: 13px;
font-weight: 500;
color: inherit;
}
.dock-sep {
min-width: 1px;
background-color: rgba(255, 255, 255, 0.12);
margin: 8px 4px;
}

View File

@ -0,0 +1,44 @@
;; The touch panel's on-screen dock — the primary local navigation surface (this
;; device is touched directly, unlike hosts/thin-client, which deliberately has no
;; bars because its primary control is HA/MQTT and wayvnc). Installed to
;; /home/<KIOSK_USERNAME>/.config/eww/eww.yuck by build-touch-panel-iso.sh.
;;
;; Session-scoped, not part of touchpanel-agent, same reasoning as the thin client's
;; now-playing widget: this is compositor UI, not device control, and dies with sway
;; rather than needing a restart-independent lifecycle. The :onclick strings below are
;; fixed constants in this file — nothing from MQTT, HA, or player metadata is ever
;; interpolated into them.
(defpoll active_workspace :interval "1s" :initial "2:home"
"swaymsg -t get_workspaces | jq -r '.[] | select(.focused) | .name' 2>/dev/null || echo 2:home")
(defwidget dock-btn [ws icon label]
(button :class {active_workspace == ws ? "dock-btn dock-btn-active" : "dock-btn"}
:onclick "swaymsg workspace ${ws}"
(box :orientation "vertical" :space-evenly false :spacing 2
(label :class "dock-icon" :text icon)
(label :class "dock-label" :text label))))
(defwidget dock-root []
(box :class "dock-root" :orientation "horizontal" :space-evenly false :spacing 8
:halign "center" :valign "center"
(dock-btn :ws "1:spotify" :icon "🎵" :label "Spotify")
(dock-btn :ws "2:home" :icon "🏠" :label "Home")
(dock-btn :ws "3:web" :icon "🌐" :label "Web")
(box :class "dock-sep")
(button :class "dock-btn" :onclick "toggle-keyboard"
(box :orientation "vertical" :space-evenly false :spacing 2
(label :class "dock-icon" :text "⌨")
(label :class "dock-label" :text "Keyboard")))))
;; :exclusive true reserves this strip so it is never overlapped by, and never steals
;; area from behind, the app on screen — unlike the thin client's overlay widgets,
;; which deliberately float over fullscreen video. A dock a finger can miss because
;; something else drew over it is worse than a smaller app area.
(defwindow touch-dock
:monitor 0
:geometry (geometry :width "100%" :height "84px" :anchor "bottom center")
:stacking "overlay"
:exclusive true
:focusable false
(dock-root))

View File

@ -0,0 +1,63 @@
{
"_comment_path": "Installed to /etc/firefox/policies/policies.json, the documented Linux location for Firefox enterprise policy. 0600-firefox.hook.chroot also links it into firefox-esr's distribution/ directory, an older location some builds read instead; whichever the installed build honours wins and the other is ignored. Every key below lives OUTSIDE the 'policies' object on purpose — Firefox flags unrecognised keys *inside* it as invalid policies in about:policies.",
"_comment_extensions": "Same two extensions and the same unverified-ID caveat as hosts/thin-client/configs/firefox/policies.json — see that file's comment and the touch-panel README's verification list. Duplicated, not shared, per this project's established cross-host convention (Phase 13's admin-canvas/digest-canvas SDK split).",
"_comment_kiosk_ui": "The Disable*/UserMessaging/FirefoxHome blocks all remove a prompt, tour, or nag that would otherwise sit on an unattended panel with nobody there to dismiss it.",
"_comment_updates": "The browser is part of the image and is replaced by rebuilding and reflashing it, so in-browser updates would only produce version drift between panels plus a restart banner nobody is there to click.",
"policies": {
"ExtensionSettings": {
"uBlock0@raymondhill.net": {
"installation_mode": "force_installed",
"install_url": "https://addons.mozilla.org/firefox/downloads/latest/ublock-origin/latest.xpi",
"default_area": "menupanel"
},
"sponsorBlocker@ajay.app": {
"installation_mode": "force_installed",
"install_url": "https://addons.mozilla.org/firefox/downloads/latest/sponsorblock/latest.xpi",
"default_area": "menupanel"
}
},
"DisableProfileImport": true,
"DisableProfileRefresh": true,
"DisableFirefoxAccounts": true,
"DisableFirefoxStudies": true,
"DisableTelemetry": true,
"DisablePocket": true,
"DisableFeedbackCommands": true,
"DisableSetDesktopBackground": true,
"DontCheckDefaultBrowser": true,
"NoDefaultBookmarks": true,
"OfferToSaveLogins": false,
"PasswordManagerEnabled": false,
"PromptForDownloadLocation": false,
"AppAutoUpdate": false,
"DisableAppUpdate": true,
"UserMessaging": {
"WhatsNew": false,
"ExtensionRecommendations": false,
"FeatureRecommendations": false,
"UrlbarInterventions": false,
"SkipOnboarding": true,
"MoreFromMozilla": false
},
"FirefoxHome": {
"Search": true,
"TopSites": true,
"SponsoredTopSites": false,
"Highlights": false,
"Pocket": false,
"SponsoredPocket": false,
"Snippets": false
},
"OverrideFirstRunPage": "",
"OverridePostUpdatePage": ""
}
}

View File

@ -0,0 +1,45 @@
// Prefs for the touch panel's general-browsing Firefox profile. Installed by
// build-touch-panel-iso.sh into /home/<KIOSK_USERNAME>/.mozilla/firefox/web.
//
// Same shape as hosts/thin-client/configs/firefox/user.js — see that file's comment
// for why user.js (re-applied on every startup) rather than a one-shot prefs.js seed,
// and why anything expressible as enterprise policy lives in policies.json instead.
// Duplicated rather than shared between the two hosts on purpose — same call as
// admin-canvas/digest-canvas's SDK split in Phase 13: this profile is free to drift
// (touch-target padding, no ydotool-specific concerns) without touching the thin
// client's.
user_pref("toolkit.legacyUserProfileCustomizations.stylesheets", true);
// First-run / post-update interstitials. An unattended panel has nobody to close them.
user_pref("browser.startup.homepage_override.mstone", "ignore");
user_pref("browser.aboutwelcome.enabled", false);
user_pref("browser.messaging-system.whatsNewPanel.enabled", false);
user_pref("trailhead.firstrun.didSeeAboutWelcome", true);
user_pref("datareporting.policy.firstRunURL", "");
user_pref("datareporting.policy.dataSubmissionPolicyBypassNotification", true);
// Session restore prompts after a power cut would leave the panel stuck on a
// "restore your session?" page nobody is there to click through.
user_pref("browser.sessionstore.resume_from_crash", false);
user_pref("browser.tabs.warnOnClose", false);
user_pref("browser.tabs.warnOnCloseOtherTabs", false);
user_pref("browser.warnOnQuit", false);
// No permission doorhangers on a shared touch panel. Autoplay is left blocked
// (unlike the thin client's digest canvas) — general browsing has no reason to want
// arbitrary pages making sound on a household wall panel.
user_pref("permissions.default.desktop-notification", 2);
user_pref("permissions.default.geo", 2);
user_pref("gfx.webrender.all", true);
// Touch-scrolling tuning: Firefox's Wayland/GTK touch handling is on by default, but
// the overlay/kinetic scrollbar reads better at panel viewing distance than the
// default desktop one.
user_pref("widget.gtk.overlay-scrollbars.enabled", true);
user_pref("browser.urlbar.suggest.topsites", false);
user_pref("browser.urlbar.suggest.quicksuggest.sponsored", false);
user_pref("browser.newtabpage.activity-stream.showSponsored", false);
user_pref("browser.newtabpage.activity-stream.showSponsoredTopSites", false);

View File

@ -0,0 +1,64 @@
/* Minimal browser chrome for the touch panel's general-browsing Firefox window.
* Installed to <profile>/chrome/userChrome.css by build-touch-panel-iso.sh, and only
* read because user.js sets toolkit.legacyUserProfileCustomizations.stylesheets=true.
*
* What survives: back, forward, reload, and the address bar the four things a
* finger actually needs to drive this window, same reasoning as
* hosts/thin-client/configs/firefox/userChrome.css (adapted here for a touchscreen
* instead of ydotool/wayvnc remote input). Every extra toolbar button is one more
* small target to mis-tap.
*/
/* Tab strip. One page at a time; a tab bar on a shared panel just accumulates
* whatever the last person left open. */
#TabsToolbar,
#tabbrowser-tabs,
#alltabs-button {
visibility: collapse !important;
}
/* Menu bar and bookmarks bar. */
#toolbar-menubar,
#PersonalToolbar,
#PlacesToolbar {
visibility: collapse !important;
}
/* Everything on the nav bar except back / forward / reload / the address bar. */
#home-button,
#library-button,
#sidebar-button,
#fxa-toolbar-menu-button,
#unified-extensions-button,
#PanelUI-button,
#downloads-button,
#save-to-pocket-button,
#pageActionButton,
#star-button-box,
#reader-mode-button,
#customizableui-special-spring1,
#customizableui-special-spring2 {
display: none !important;
}
#urlbar-container {
min-width: 0 !important;
}
#nav-bar {
border: none !important;
box-shadow: none !important;
}
.titlebar-buttonbox-container,
.titlebar-spacer {
display: none !important;
}
/* Touch targets: the remaining nav-bar buttons default to a mouse-sized hit area.
* Bumping padding here is cheaper than a full touch-optimised theme. */
#nav-bar toolbarbutton {
padding: 8px !important;
}
/* Findbar and notification popups stay: they are transient and user-initiated. */

View File

@ -0,0 +1,34 @@
#!/bin/sh
# Opens the general-browsing Firefox window — minimal chrome, not kiosk mode.
# Installed to /usr/local/bin/web-browser. Its own profile, separate from any other
# Firefox instance on this machine (there is only this one on the touch panel, but the
# profile is still named explicitly for the same reason hosts/thin-client keeps three
# separate ones: a Firefox profile can only be open in one process).
set -eu
PROFILE_DIR="${HOME:-/home/$(id -un)}/.mozilla/firefox/web"
if command -v firefox-esr >/dev/null 2>&1; then
FIREFOX=firefox-esr
else
FIREFOX=firefox
fi
mkdir -p "$PROFILE_DIR/chrome"
# Re-copied on every launch rather than once at build time: this is a locked-down
# profile with no interactive customisation expected, so keeping it in lockstep with
# /etc/touchpanel-firefox/ (edited by rebuilding the image) is simpler than a one-shot
# seed that could drift after a userChrome.css update.
cp /etc/touchpanel-firefox/userChrome.css "$PROFILE_DIR/chrome/userChrome.css" 2>/dev/null || true
cp /etc/touchpanel-firefox/user.js "$PROFILE_DIR/user.js" 2>/dev/null || true
# Focus an already-open window rather than stacking a second one: this window holds
# state (history, a half-typed URL, a logged-in page) that a kill-and-relaunch would
# throw away.
if pgrep -u "$(id -u)" -f "$FIREFOX .*--profile $PROFILE_DIR" >/dev/null 2>&1; then
[ -n "${1:-}" ] && exec "$FIREFOX" --profile "$PROFILE_DIR" --new-tab "$1"
exit 0
fi
exec "$FIREFOX" --profile "$PROFILE_DIR" --new-instance --new-window "${1:-about:blank}"

View File

@ -0,0 +1,14 @@
# greetd — autologin straight into the kiosk Sway session, no greeter UI.
# Installed to /etc/greetd/config.toml by build-touch-panel-iso.sh, which substitutes
# @KIOSK_USERNAME@ with its own KIOSK_USERNAME variable on the way in.
#
# Same shape as hosts/thin-client/configs/greetd/config.toml — see that file's comment
# for why [default_session] (not [initial_session]) is what makes autologin permanent
# and self-healing after a crash or a `swaymsg exit`.
[terminal]
vt = 1
[default_session]
command = "/usr/local/bin/kiosk-session"
user = "@KIOSK_USERNAME@"

View File

@ -0,0 +1,25 @@
#!/bin/sh
# greetd's default_session command. Installed to /usr/local/bin/kiosk-session.
#
# Same purpose as the thin client's own kiosk-session: gets MQTT_BROKER_HOST / HA_URL /
# SPOTIFY_APP_ID / etc into Sway's environment, since sway's config file has no way to
# read an env file itself but every `exec` line it runs inherits this process's
# environment.
set -eu
[ -r /etc/touchpanel-agent/config.env ] && . /etc/touchpanel-agent/config.env
export MQTT_BROKER_HOST HA_URL
export XDG_CURRENT_DESKTOP=sway
export XDG_SESSION_TYPE=wayland
export XDG_SESSION_DESKTOP=sway
export MOZ_ENABLE_WAYLAND=1
# Chromium's Wayland/Ozone backend, needed for native touch events in the HA kiosk
# window (see configs/sway/ha-kiosk) — without it Chromium falls back to Xwayland,
# where touch input is translated through XTest and loses multi-touch gestures.
export OZONE_PLATFORM=wayland
: "${XDG_RUNTIME_DIR:=/run/user/$(id -u)}"
export XDG_RUNTIME_DIR
exec sway

View File

@ -0,0 +1,25 @@
#!/bin/sh
# Toggles the on-screen keyboard. Installed to /usr/local/bin/toggle-keyboard, run
# from the dock's Keyboard button (configs/eww/eww.yuck) and bindable to any other
# tap target later.
#
# wvkbd, not squeekboard: squeekboard's auto-show behaviour is wired through Phosh's
# session/DBus setup, which this image doesn't run, so under plain Sway it would need
# the same manual-toggle treatment anyway while pulling in more of the GNOME-mobile
# stack. wvkbd is a small layer-shell surface built for exactly this — a wlroots
# kiosk with no shell-level input-method integration.
#
# KNOWN LIMITATION, documented rather than hidden: there is no automatic show-on-
# text-field-focus here (that needs the text-input-v3/virtual-keyboard-v1 protocols
# wired all the way through each app, which is inconsistent across Chromium/Firefox/
# Spotify under plain wlroots). The dock's Keyboard button is therefore the only way
# to open it — tap before typing, tap again to dismiss.
set -eu
if pgrep -u "$(id -u)" -x wvkbd-mobintl >/dev/null 2>&1; then
pkill -u "$(id -u)" -x wvkbd-mobintl
else
# -L: keyboard layer height in px. 280 is a starting point for a touch-panel-sized
# screen, not measured against real hardware — see the README's verification list.
wvkbd-mobintl -L 280 &
fi

View File

@ -0,0 +1,20 @@
#!/bin/sh
# Full Spotify GUI client (docs/project-plan.md Phase 16). Installed to
# /usr/local/bin/spotify-launch.
#
# Deliberately NOT spotifyd/librespot — those are the thin client's and audio-endpoint's
# headless Spotify Connect *receivers*, with no GUI at all. This device is touched
# directly, so it runs the real, official Spotify Linux client: its own library
# browser, search, and playlists. Spotify ships no apt package for bookworm, so this
# comes from Flathub (0300-flatpak.hook.chroot installs it) — same distribution
# channel already used for the thin client's Steam Link, and the only maintained one
# for Spotify on Linux.
#
# Login is interactive, on-device, on first launch (Spotify account/Premium required
# for playback — Premium is also what the thin client's headless receivers need). The
# Flatpak's own persistent data dir (~/.var/app/com.spotify.Client) keeps that session
# across restarts; nothing here wipes or reseeds it the way the kiosk Firefox profiles
# are re-copied on every launch, since staying logged in is the whole point here.
set -eu
exec flatpak run com.spotify.Client "$@"

View File

@ -0,0 +1,126 @@
# Sway kiosk session for the touch panel (docs/project-plan.md Phase 16).
#
# Installed to /home/<KIOSK_USERNAME>/.config/sway/config by build-touch-panel-iso.sh.
# Every `exec` below inherits the environment set by /usr/local/bin/kiosk-session,
# which sources /etc/touchpanel-agent/config.env — that is how $HA_URL gets here
# without this file being templated.
#
# This is a DIFFERENT device from hosts/thin-client: that one is a couch-distance
# media station whose primary control surface is HA/MQTT and wayvnc, deliberately with
# "no bars" (see its config's own comment). This one is touched directly, so the whole
# point is an always-visible on-screen dock a finger can hit — see the eww bar below.
set $mod Mod4
set $ws_spotify 1:spotify
set $ws_home 2:home
set $ws_web 3:web
# Workspace names are a contract with touchpanel_agent/sway_control.py — changing one
# side means changing the other.
# ---------------------------------------------------------------------------
# Output / input
# ---------------------------------------------------------------------------
output * bg #101014 solid_color
input type:keyboard {
xkb_layout @KEYBOARD_LAYOUT@
}
# Real touchscreen hardware, not the thin client's couch-distance camera-gesture
# trick: libinput/Wayland surfaces (GTK, Chromium, Firefox) all speak wl_touch
# natively, so taps/scrolls/pinches just work without a pointer-emulation layer.
input type:touch {
map_to_output "*"
}
input type:tablet_tool {
map_to_output "*"
}
# ---------------------------------------------------------------------------
# Look
# ---------------------------------------------------------------------------
default_border none
default_floating_border none
hide_edge_borders both
gaps inner 0
gaps outer 0
# Reserve a strip at the bottom for the touch dock (bar/app-switcher), so it is
# never covered by, and never steals area from behind, whatever app is on screen.
# :exclusive is the eww window's own setting (configs/eww/eww.yuck) — sway just
# needs to know nothing else should tile into that space, which layer-shell
# exclusivity already guarantees without a `bar {}` block here.
# ---------------------------------------------------------------------------
# Session-scoped helpers
# ---------------------------------------------------------------------------
# touchpanel-agent is NOT started here — see thin-client's identical note. systemd
# owns it so it is reachable over MQTT whether or not a graphical session is up, and
# survives a sway restart. Starting it from sway too would give two competing MQTT
# clients.
# The touch dock (eww bar) and the on-screen keyboard toggle both live for the
# session, not the system, for the same reason the thin client's now-playing widget
# does: they are compositor UI, not device control, and should die with sway rather
# than need their own restart-independent lifecycle.
exec_always eww daemon
exec_always eww open touch-dock
# ---------------------------------------------------------------------------
# Applications
# ---------------------------------------------------------------------------
# No `assign` rules, same reasoning as the thin client: every launch path is explicit
# about its own workspace via touchpanel_agent.sway_control.launch_app(), so a
# separately-matched `assign` rule can't race it.
for_window [app_id="chromium.*"] fullscreen enable
for_window [app_id="spotify"] fullscreen enable
# HA kiosk window and Spotify both auto-launch at session start — this is a fixed
# 3-app panel, not an on-demand surface like the thin client's digest/admin
# workspaces, so there is no "nothing to show yet" state to guard against beyond an
# unset HA_URL.
exec sh -c '[ -n "$HA_URL" ] && { swaymsg workspace $ws_home; /usr/local/bin/ha-kiosk; }'
exec sh -c 'swaymsg workspace $ws_spotify; /usr/local/bin/spotify-launch'
exec swaymsg workspace $ws_home
# ---------------------------------------------------------------------------
# Idle
# ---------------------------------------------------------------------------
# Blank after 10 minutes to save the panel; any touch (or local input) resumes it.
# No lock screen, same reasoning as the thin client: a lock screen would make a wall
# panel unusable to anyone in the room who isn't holding a passcode.
exec swayidle -w \
timeout 600 'swaymsg "output * power off"' \
resume 'swaymsg "output * power on"'
for_window [app_id="chromium.*"] inhibit_idle fullscreen
for_window [app_id="spotify"] inhibit_idle fullscreen
# ---------------------------------------------------------------------------
# Local override keys — a fallback for standing in front of the machine (or a
# connected keyboard), not the primary control surface, which is touch + HA/MQTT.
# ---------------------------------------------------------------------------
bindsym $mod+Shift+Ctrl+m exec foot --title maintenance-shell
for_window [title="maintenance-shell"] floating enable, resize set width 800 height 500, move position center
bindsym $mod+Return exec foot
bindsym $mod+q kill
bindsym $mod+f fullscreen toggle
bindsym $mod+Shift+c reload
bindsym $mod+1 workspace $ws_spotify
bindsym $mod+2 workspace $ws_home
bindsym $mod+3 workspace $ws_web
bindsym XF86AudioPlay exec playerctl -p spotify play-pause
bindsym XF86AudioNext exec playerctl -p spotify next
bindsym XF86AudioPrev exec playerctl -p spotify previous
bindsym XF86AudioRaiseVolume exec wpctl set-volume @DEFAULT_AUDIO_SINK@ 5%+
bindsym XF86AudioLowerVolume exec wpctl set-volume @DEFAULT_AUDIO_SINK@ 5%-
bindsym XF86AudioMute exec wpctl set-mute @DEFAULT_AUDIO_SINK@ toggle
# Deliberately no exit binding — see the thin client's identical note: `swaymsg exit`
# would just drop to a black VT and greetd would autologin straight back in.
workspace $ws_home

View File

@ -0,0 +1,47 @@
#!/bin/sh
# Persistent Chromium kiosk window pointed at Home Assistant. Installed to
# /usr/local/bin/ha-kiosk. Launched once at session start (configs/sway/config) and
# supervised right here with a restart loop, not by sway (its `exec` has no
# restart-on-crash) or systemd (this needs the graphical session's env, and a system
# unit chasing a per-session Wayland socket is more trouble than a two-line loop).
#
# Unlike hosts/thin-client's digest-browser, this is never killed-and-relaunched on
# an MQTT command — touchpanel_agent.sway_control.launch_app()'s is_running/
# focus_criteria path just switches workspace and focuses the existing window, since
# the HA dashboard is stateful (login session, scroll position, which view you left
# it on) and reloading it on every "show home" tap would throw that away for nothing.
set -eu
PROFILE_DIR="${HOME:-/home/$(id -un)}/.config/touchpanel-chromium-ha"
URL="${HA_URL:-}"
[ -n "$URL" ] || { echo "ha-kiosk: HA_URL is unset" >&2; exit 1; }
if command -v chromium >/dev/null 2>&1; then
CHROMIUM=chromium
elif command -v chromium-browser >/dev/null 2>&1; then
CHROMIUM=chromium-browser
else
echo "ha-kiosk: no chromium/chromium-browser binary found" >&2
exit 1
fi
mkdir -p "$PROFILE_DIR"
# VERIFY: Chromium's Wayland app_id for a --app= kiosk window is assumed to start
# with "chromium" (matched by configs/sway/config's [app_id="chromium.*"] and by
# touchpanel_agent's focus_criteria) — not confirmed against a real build. If the
# fullscreen/for_window rule or the "Home" dock button's focus-instead-of-relaunch
# stops working, check the real app_id with `swaymsg -t get_tree`.
while true; do
"$CHROMIUM" \
--user-data-dir="$PROFILE_DIR" \
--ozone-platform=wayland \
--kiosk --app="$URL" \
--start-fullscreen \
--noerrdialogs --disable-infobars --disable-session-crashed-bubble \
--overscroll-history-navigation=0 \
--touch-events=enabled \
--check-for-update-interval=31536000 \
>/tmp/ha-kiosk.log 2>&1 || true
sleep 2
done

View File

@ -0,0 +1,50 @@
#!/bin/sh
# Creates the kiosk account the whole image is built around. Identical logic to
# hosts/thin-client's 0100-user-setup.hook.chroot — see that file's comments for the
# reasoning behind the locked password, passwordless sudo, and key-only SSH.
set -eu
. /etc/touchpanel-agent/config.env
if ! id "$KIOSK_USERNAME" >/dev/null 2>&1; then
useradd --create-home --shell /bin/bash --comment "Touch panel kiosk session" "$KIOSK_USERNAME"
fi
for grp in audio video input render dialout netdev plugdev seat _seatd; do
if getent group "$grp" >/dev/null 2>&1; then
adduser "$KIOSK_USERNAME" "$grp" >/dev/null
fi
done
# No password is baked in: the account is locked so it can never be used to log in
# remotely, while the physical console still autologins via greetd.
passwd --lock "$KIOSK_USERNAME" >/dev/null
adduser "$KIOSK_USERNAME" sudo >/dev/null
# Passwordless sudo: this image autologins to an unattended interactive Sway session
# at the physical console, so anyone standing in front of the panel already has the
# equivalent of a root shell. The boundary that actually matters is key-only SSH below
# — there is no wayvnc on this image (see the README for that scope decision).
cat > "/etc/sudoers.d/010-${KIOSK_USERNAME}" <<EOF
${KIOSK_USERNAME} ALL=(ALL) NOPASSWD: ALL
EOF
chmod 0440 "/etc/sudoers.d/010-${KIOSK_USERNAME}"
mkdir -p /etc/ssh/sshd_config.d
cat > /etc/ssh/sshd_config.d/10-touch-panel.conf <<'EOF'
PermitRootLogin no
PasswordAuthentication no
KbdInteractiveAuthentication no
PubkeyAuthentication yes
EOF
if [ -d "/home/${KIOSK_USERNAME}/.ssh" ]; then
chmod 700 "/home/${KIOSK_USERNAME}/.ssh"
[ -f "/home/${KIOSK_USERNAME}/.ssh/authorized_keys" ] && \
chmod 600 "/home/${KIOSK_USERNAME}/.ssh/authorized_keys"
fi
chown -R "${KIOSK_USERNAME}:${KIOSK_USERNAME}" "/home/${KIOSK_USERNAME}"
systemctl enable ssh >/dev/null 2>&1 || true

View File

@ -0,0 +1,23 @@
#!/bin/sh
# Makes greetd the boot target so the machine comes up straight in the kiosk Sway
# session. Identical logic to hosts/thin-client's 0200-greetd.hook.chroot.
set -eu
. /etc/touchpanel-agent/config.env
chmod 0755 /usr/local/bin/kiosk-session
if ! id greeter >/dev/null 2>&1; then
useradd --system --create-home --home-dir /var/lib/greetd --shell /usr/sbin/nologin greeter
fi
systemctl enable greetd
systemctl set-default graphical.target
# live-config would otherwise autologin its own account on tty1 and fight greetd for
# the VT. build-touch-panel-iso.sh passes `noautologin` on the kernel command line;
# masking the getty on vt1 makes that robust even if someone edits the boot args.
systemctl mask getty@tty1.service
mkdir -p "/home/${KIOSK_USERNAME}/.config/sway"
chown -R "${KIOSK_USERNAME}:${KIOSK_USERNAME}" "/home/${KIOSK_USERNAME}/.config"

View File

@ -0,0 +1,24 @@
#!/bin/sh
# Flathub remote + the full Spotify GUI client (docs/project-plan.md Phase 16).
#
# Spotify ships no apt package for bookworm, so this comes from Flathub — the same
# distribution channel already used for hosts/thin-client's Steam Link. No release
# version is pinned, same "don't hardcode a URL that will just rot" reasoning as that
# hook and as 0500-spotify-connect.hook.chroot's headless-receiver install.
set -eu
flatpak remote-add --if-not-exists flathub https://dl.flathub.org/repo/flathub.flatpakrepo
# VERIFY BEFORE THE FIRST REAL BUILD: confirm this application ID against the live
# Flathub listing (`flatpak search Spotify`). Believed correct (it's Spotify's own
# published app on Flathub) but not checked from this environment.
SPOTIFY_APP_ID="com.spotify.Client"
if flatpak install -y --noninteractive flathub "$SPOTIFY_APP_ID"; then
echo "0300-flatpak-spotify: installed ${SPOTIFY_APP_ID}."
else
echo "0300-flatpak-spotify: WARNING — could not install ${SPOTIFY_APP_ID} during the"
echo " build (no network in the chroot, or the app ID is wrong). Run this on the"
echo " booted image instead: flatpak install -y flathub ${SPOTIFY_APP_ID}"
echo " The Spotify workspace will show a blank window until this is done."
fi

View File

@ -0,0 +1,14 @@
#!/bin/sh
# Installs the touchpanel-agent systemd unit. Identical logic to hosts/thin-client's
# 0700-thinclient-agent.hook.chroot — the package arrives under /opt/touchpanel-agent
# via includes.chroot and runs on the system interpreter against apt's
# python3-paho-mqtt, so there is nothing to pip-install here.
set -eu
install -m 0644 /opt/touchpanel-agent/touchpanel-agent.service \
/etc/systemd/system/touchpanel-agent.service
chmod 0644 /etc/touchpanel-agent/config.env
chown -R root:root /opt/touchpanel-agent
systemctl enable touchpanel-agent

View File

@ -0,0 +1,20 @@
#!/bin/sh
# The touch dock widget (docs/project-plan.md Phase 16). Identical apt-first/
# documented-placeholder pattern as hosts/thin-client's 0800-eww-widget.hook.chroot —
# eww is not in Debian bookworm main (a Rust binary, normally cargo-built or fetched
# as a GitHub release).
set -eu
if apt-get install -y --no-install-recommends eww 2>/dev/null; then
echo "0500-eww-dock: installed eww from apt."
else
echo "0500-eww-dock: eww is not available from the configured apt sources."
echo " The touch dock (the on-screen Spotify/Home/Web/Keyboard bar) will not appear"
echo " on this image until eww is installed:"
echo " a) cargo install eww (needs a Rust toolchain in a build hook), or"
echo " b) fetch a release binary from https://github.com/elkowar/eww/releases"
echo " (check the current tag yourself) into /usr/local/bin/eww."
echo " Without it there is NO on-screen app switcher — the panel still boots and"
echo " shows Home, but touch navigation between apps needs the dock. Home Assistant/"
echo " MQTT app switching (touchpanel_agent) still works either way."
fi

View File

@ -0,0 +1,17 @@
#!/bin/sh
# Firefox enterprise policy plumbing for the general-browsing window. Identical logic
# to hosts/thin-client's 0900-firefox.hook.chroot.
set -eu
FIREFOX_LIB_DIR="/usr/lib/firefox-esr"
if [ -d "$FIREFOX_LIB_DIR" ]; then
mkdir -p "$FIREFOX_LIB_DIR/distribution"
ln -sf /etc/firefox/policies/policies.json "$FIREFOX_LIB_DIR/distribution/policies.json"
echo "0600-firefox: linked policies.json into $FIREFOX_LIB_DIR/distribution/"
else
echo "0600-firefox: $FIREFOX_LIB_DIR not found (firefox-esr not installed yet, or a"
echo " different path in this Debian release) — /etc/firefox/policies/policies.json"
echo " still applies if the package's install order runs after this hook; verify"
echo " about:policies shows uBlock Origin/SponsorBlock as force-installed on first boot."
fi

View File

@ -0,0 +1,22 @@
#!/bin/sh
# On-screen keyboard (docs/project-plan.md Phase 16), toggled from the touch dock's
# Keyboard button. Same apt-first/documented-placeholder pattern as eww and the
# headless Spotify Connect receiver: wvkbd is not in Debian bookworm main (it's a
# small C wlroots layer-shell client, normally built from source).
set -eu
if apt-get install -y --no-install-recommends wvkbd 2>/dev/null; then
echo "0700-onscreen-keyboard: installed wvkbd from apt."
elif command -v wvkbd-mobintl >/dev/null 2>&1; then
echo "0700-onscreen-keyboard: wvkbd-mobintl already present."
else
echo "0700-onscreen-keyboard: wvkbd is not available from the configured apt sources."
echo " Typing into Spotify search, the HA dashboard, or the web browser will have no"
echo " on-screen keyboard until it's installed:"
echo " a) build from source: https://github.com/jjsullivan5196/wvkbd"
echo " (needs libwayland-dev, wayland-protocols, cairo, pango — add a"
echo " build-dependencies block to this hook if you go this route), or"
echo " b) fetch a distro package once one exists for your release."
echo " configs/keyboard/toggle-keyboard already handles a missing binary as a no-op"
echo " (the dock button just does nothing), so the rest of the session is unaffected."
fi

View File

@ -0,0 +1,66 @@
# Touch-panel package list (live-build .list.chroot format: one package per line).
# Phase 16, docs/project-plan.md.
# --- Wayland compositor / session ---
sway
swayidle
swaybg
greetd
foot
# --- Browsers ---
firefox-esr
chromium
playerctl
# --- Audio ---
pipewire
pipewire-pulse
wireplumber
alsa-utils
# --- Spotify GUI client and Steam-Link-style apps come from Flathub, not apt
# (see 0300-flatpak-spotify.hook.chroot) ---
flatpak
# --- Agent runtime ---
python3
python3-venv
# Debian bookworm ships paho-mqtt 1.6.x. touchpanel_agent is written to work against
# both the 1.x and 2.x callback APIs, so this package is used as-is rather than pip'd.
python3-paho-mqtt
procps
# --- Networking ---
network-manager
# --- Out-of-band admin. This image has no wayvnc (unlike hosts/thin-client) — see
# the README's scope note — so SSH is the only remote admin path. ---
openssh-server
sudo
# --- Touch dock (0500-eww-dock.hook.chroot tries apt, falls back to a documented
# placeholder if not packaged) ---
eww
jq
# --- On-screen keyboard (0700-onscreen-keyboard.hook.chroot, same apt-first/
# documented-fallback pattern) ---
wvkbd
# --- Firmware for real hardware (needs contrib + non-free-firmware archive areas,
# which build-touch-panel-iso.sh passes to `lb config`) ---
firmware-linux
firmware-iwlwifi
firmware-realtek
firmware-misc-nonfree
# --- Misc ---
ca-certificates
curl
openssl
less
vim-tiny
# NOTE: Spotify (com.spotify.Client) is Flatpak-only — see
# 0300-flatpak-spotify.hook.chroot. There is no apt package.

View File

@ -0,0 +1,7 @@
# Only relevant when ENABLE_INSTALLER="true" in build-touch-panel-iso.sh (off by
# default — the normal path is live-boot straight into Sway, no disk install at all).
# Identical mechanism/caveat as hosts/thin-client/live-build/config/preseed.cfg.
#
# No "keyboard-configuration/xkb-keymap seen true" line on purpose: this preseeds the
# *default* answer, it does not skip the question.
d-i keyboard-configuration/xkb-keymap select de

View File

@ -0,0 +1,297 @@
#!/usr/bin/env bash
#
# Smart Home Touch-Panel ISO Builder
# Target: builds a Debian 12 (Bookworm) live ISO on a Debian/Ubuntu build machine
#
# Drives `lb config && lb build` over hosts/touch-panel/live-build/ to produce the
# Sway touch-panel image described in docs/project-plan.md Phase 16:
# - greetd autologin straight into a kiosk Sway session (no greeter UI)
# - touchpanel-agent (Python, systemd) — HA MQTT-discovery entities + swaymsg control
# - workspaces 1:spotify (full Spotify GUI, Flathub) / 2:home (Chromium kiosk
# pointed at Home Assistant) / 3:web (general Firefox browsing)
# - an always-visible eww touch dock for on-screen app switching
# - a toggleable on-screen keyboard (wvkbd)
#
# Reuses hosts/thin-client's own live-build toolchain and directory-split convention,
# NOT its live-build tree — this is a different device (touched directly, not
# remote-controlled from the couch) with a different app set, so it gets its own
# smaller config/, agent/, and live-build/, same relationship as
# hosts/audio-endpoint's amd64 build to the thin client's.
#
# This script is also the single point that keeps configs/ (the human-edited source of
# truth, reviewed in git) in sync with live-build/config/includes.chroot/ (the
# generated tree that actually gets baked into the image). Never hand-edit anything
# under includes.chroot — it is wiped and regenerated on every run.
#
# Run as: sudo ./build-touch-panel-iso.sh
#
# EDIT THE VARIABLES BELOW BEFORE RUNNING.
set -euo pipefail
# ---------------------------------------------------------------------------
# CONFIGURATION — edit these before running
# ---------------------------------------------------------------------------
DEBIAN_RELEASE="bookworm" # Matches the container host's OS
KIOSK_USERNAME="kiosk" # The autologin account the whole image is built around
IMAGE_HOSTNAME="touch-panel" # Hostname baked into the image
TOUCHPANEL_NAME="Kitchen touch panel" # Friendly name shown on the HA device
KEYBOARD_LAYOUT="de" # xkb layout name (`localectl list-x11-keymap-layouts`)
ENABLE_INSTALLER="false" # "true" adds a debian-installer to the ISO (install to disk)
# --- Where the touch panel talks to -----------------------------------------
MQTT_BROKER_HOST="192.168.1.10" # <-- EDIT: container-host IP running Mosquitto
MQTT_BROKER_PORT="1883"
MQTT_USERNAME="" # Leave empty while Mosquitto runs allow_anonymous
MQTT_PASSWORD="" # Never commit a real value here
HA_URL="http://192.168.1.10:8123" # <-- EDIT: Home Assistant URL — the "Home" workspace
# Optional: an SSH public key to bake into the kiosk account for out-of-band admin.
# The image ships with password auth disabled and no wayvnc (see README's scope note),
# so without this the only admin path is the local console.
SSH_AUTHORIZED_KEY=""
# ---------------------------------------------------------------------------
# Paths
# ---------------------------------------------------------------------------
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
TOUCH_PANEL_DIR="$(dirname "$SCRIPT_DIR")"
CONFIGS_DIR="${TOUCH_PANEL_DIR}/configs"
AGENT_DIR="${TOUCH_PANEL_DIR}/agent"
LIVE_BUILD_DIR="${TOUCH_PANEL_DIR}/live-build"
INCLUDES="${LIVE_BUILD_DIR}/config/includes.chroot"
PACKAGE_LIST="${LIVE_BUILD_DIR}/config/package-lists/touch-panel.list.chroot"
# ---------------------------------------------------------------------------
# Sanity checks
# ---------------------------------------------------------------------------
if [[ $EUID -ne 0 ]]; then
echo "Warning: not running as root. 'lb build' needs root to bootstrap and chroot,"
echo " and will fail partway through. Re-run with: sudo $0"
echo " Continuing anyway so you can at least regenerate includes.chroot..."
fi
if ! grep -qi "debian\|ubuntu" /etc/os-release; then
echo "Warning: live-build targets a Debian/Ubuntu build host. Proceeding anyway..."
fi
if ! command -v lb &> /dev/null; then
if [[ $EUID -eq 0 ]]; then
echo "--- Installing live-build ---"
apt-get update
apt-get install -y live-build
else
echo "live-build is not installed and this script is not running as root." >&2
echo " Install it first: sudo apt-get install live-build" >&2
exit 1
fi
else
echo "--- live-build already installed, skipping ---"
fi
if [[ ! -f "$PACKAGE_LIST" ]]; then
echo "Missing package list: $PACKAGE_LIST" >&2
exit 1
fi
if [[ "$MQTT_BROKER_HOST" == "192.168.1.10" ]]; then
echo "Warning: MQTT_BROKER_HOST is still the placeholder IP."
echo " Edit it at the top of this script to your container host's real LAN address,"
echo " or the touch panel won't show up as Home Assistant entities."
fi
if [[ "$HA_URL" == "http://192.168.1.10:8123" ]]; then
echo "Warning: HA_URL is still the placeholder."
echo " The image builds and boots fine, but the Home workspace will show a"
echo " connection error until this points at a real Home Assistant."
fi
echo
echo "=== Smart Home Touch-Panel ISO Builder ==="
echo "Debian release : $DEBIAN_RELEASE"
echo "Kiosk user : $KIOSK_USERNAME"
echo "Image hostname : $IMAGE_HOSTNAME"
echo "MQTT broker : ${MQTT_BROKER_HOST}:${MQTT_BROKER_PORT}"
echo "Home Assistant : $HA_URL"
echo "Keyboard layout : $KEYBOARD_LAYOUT"
echo
# ---------------------------------------------------------------------------
# 1. Regenerate includes.chroot from configs/ and agent/
# ---------------------------------------------------------------------------
echo "--- Regenerating $INCLUDES ---"
rm -rf "$INCLUDES"
mkdir -p \
"$INCLUDES/etc/greetd" \
"$INCLUDES/etc/touchpanel-agent" \
"$INCLUDES/etc/firefox/policies" \
"$INCLUDES/etc/touchpanel-firefox" \
"$INCLUDES/usr/local/bin" \
"$INCLUDES/opt/touchpanel-agent" \
"$INCLUDES/home/${KIOSK_USERNAME}/.config/sway" \
"$INCLUDES/home/${KIOSK_USERNAME}/.config/eww" \
"$INCLUDES/home/${KIOSK_USERNAME}/.ssh"
# @KIOSK_USERNAME@ and @KEYBOARD_LAYOUT@ are the only templated tokens in the configs.
# Everything else the hooks need is read at build time from
# /etc/touchpanel-agent/config.env (written below), which live-build copies in
# (chroot_local-includes) before it runs the hooks (chroot_local-hooks).
subst() {
sed -e "s/@KIOSK_USERNAME@/${KIOSK_USERNAME}/g" \
-e "s/@KEYBOARD_LAYOUT@/${KEYBOARD_LAYOUT}/g" "$1" > "$2"
}
subst "${CONFIGS_DIR}/greetd/config.toml" "$INCLUDES/etc/greetd/config.toml"
subst "${CONFIGS_DIR}/sway/config" "$INCLUDES/home/${KIOSK_USERNAME}/.config/sway/config"
subst "${AGENT_DIR}/touchpanel-agent.service" "$INCLUDES/opt/touchpanel-agent/touchpanel-agent.service"
install -m 0755 "${CONFIGS_DIR}/greetd/kiosk-session" "$INCLUDES/usr/local/bin/kiosk-session"
install -m 0755 "${CONFIGS_DIR}/sway/ha-kiosk" "$INCLUDES/usr/local/bin/ha-kiosk"
install -m 0755 "${CONFIGS_DIR}/spotify/spotify-launch" "$INCLUDES/usr/local/bin/spotify-launch"
install -m 0755 "${CONFIGS_DIR}/keyboard/toggle-keyboard" "$INCLUDES/usr/local/bin/toggle-keyboard"
# Console (VT/TTY) keymap — separate from Sway's own xkb_layout, since greetd briefly
# owns the console before Sway starts.
mkdir -p "$INCLUDES/etc/default"
cat > "$INCLUDES/etc/default/keyboard" <<EOF
XKBMODEL="pc105"
XKBLAYOUT="${KEYBOARD_LAYOUT}"
XKBVARIANT=""
XKBOPTIONS=""
BACKSPACE="guess"
EOF
# Touch dock (eww) — session-scoped files under the kiosk user's own config.
install -m 0644 "${CONFIGS_DIR}/eww/eww.yuck" "$INCLUDES/home/${KIOSK_USERNAME}/.config/eww/eww.yuck"
install -m 0644 "${CONFIGS_DIR}/eww/eww.scss" "$INCLUDES/home/${KIOSK_USERNAME}/.config/eww/eww.scss"
# Firefox: the general-browsing window launcher, plus the shared chrome/prefs template
# it copies into its own profile at launch. policies.json is global and goes straight
# into includes.chroot.
install -m 0755 "${CONFIGS_DIR}/firefox/web-browser" "$INCLUDES/usr/local/bin/web-browser"
install -m 0644 "${CONFIGS_DIR}/firefox/policies.json" "$INCLUDES/etc/firefox/policies/policies.json"
install -m 0644 "${CONFIGS_DIR}/firefox/userChrome.css" "$INCLUDES/etc/touchpanel-firefox/userChrome.css"
install -m 0644 "${CONFIGS_DIR}/firefox/user.js" "$INCLUDES/etc/touchpanel-firefox/user.js"
cp -r "${AGENT_DIR}/touchpanel_agent" "$INCLUDES/opt/touchpanel-agent/"
install -m 0644 "${AGENT_DIR}/requirements.txt" "$INCLUDES/opt/touchpanel-agent/requirements.txt"
find "$INCLUDES/opt/touchpanel-agent" -name '__pycache__' -type d -prune -exec rm -rf {} +
if [[ -n "$SSH_AUTHORIZED_KEY" ]]; then
echo "$SSH_AUTHORIZED_KEY" > "$INCLUDES/home/${KIOSK_USERNAME}/.ssh/authorized_keys"
chmod 600 "$INCLUDES/home/${KIOSK_USERNAME}/.ssh/authorized_keys"
echo " Baked an SSH authorized_keys entry for ${KIOSK_USERNAME}."
else
echo " No SSH_AUTHORIZED_KEY set — SSH admin access will not be possible on this image."
fi
# ---------------------------------------------------------------------------
# 2. Runtime config, read by touchpanel-agent, the sway session wrapper, and hooks
# ---------------------------------------------------------------------------
echo "--- Writing /etc/touchpanel-agent/config.env into includes.chroot ---"
cat > "$INCLUDES/etc/touchpanel-agent/config.env" <<EOF
# Generated by hosts/touch-panel/scripts/build-touch-panel-iso.sh — do not hand-edit
# here; edit the CONFIGURATION block in that script and rebuild.
KIOSK_USERNAME=${KIOSK_USERNAME}
TOUCHPANEL_NAME=${TOUCHPANEL_NAME}
MQTT_BROKER_HOST=${MQTT_BROKER_HOST}
MQTT_BROKER_PORT=${MQTT_BROKER_PORT}
MQTT_USERNAME=${MQTT_USERNAME}
MQTT_PASSWORD=${MQTT_PASSWORD}
HA_URL=${HA_URL}
EOF
chmod 0644 "$INCLUDES/etc/touchpanel-agent/config.env"
# ---------------------------------------------------------------------------
# 3. lb config
# ---------------------------------------------------------------------------
cd "$LIVE_BUILD_DIR"
chmod +x config/hooks/normal/*.hook.chroot
if [[ -e .build ]]; then
echo "--- Previous build found, running 'lb clean' (package cache is kept) ---"
lb clean
fi
INSTALLER_MODE="none"
if [[ "$ENABLE_INSTALLER" == "true" ]]; then
INSTALLER_MODE="live"
fi
echo "--- Running lb config ---"
lb config \
--distribution "$DEBIAN_RELEASE" \
--architectures amd64 \
--linux-flavours amd64 \
--archive-areas "main contrib non-free-firmware" \
--binary-images iso-hybrid \
--debian-installer "$INSTALLER_MODE" \
--iso-application "SmartestHome Touch Panel" \
--iso-publisher "SmartestHome" \
--iso-volume "smarthome-touch-panel" \
--memtest none \
--bootappend-live "boot=live components quiet splash noautologin username=${KIOSK_USERNAME} hostname=${IMAGE_HOSTNAME}"
# ---------------------------------------------------------------------------
# 4. lb build
# ---------------------------------------------------------------------------
echo "--- Running lb build (this takes a while and needs network) ---"
lb build
ISO_PATH="$(find "$LIVE_BUILD_DIR" -maxdepth 1 -name '*.iso' -print -quit)"
ISO_PATH="${ISO_PATH:-${LIVE_BUILD_DIR}/live-image-amd64.hybrid.iso}"
echo
echo "=== Done ==="
echo "ISO written to:"
echo " ${ISO_PATH}"
echo
echo "What's in it:"
echo " greetd : autologin as '${KIOSK_USERNAME}' straight into Sway on vt1"
echo " Sway : workspaces 1:spotify / 2:home / 3:web, always-on touch dock"
echo " touchpanel-agent : system service, MQTT ${MQTT_BROKER_HOST}:${MQTT_BROKER_PORT}"
echo " Spotify : full GUI client (Flathub com.spotify.Client), workspace 1"
echo " Home Assistant : Chromium kiosk, ${HA_URL}, workspace 2, auto-restart-on-crash"
echo " Web browser : Firefox, minimal chrome, uBlock Origin + SponsorBlock, workspace 3"
echo " Touch dock : bottom bar (eww) — Spotify / Home / Web / Keyboard buttons"
echo " On-screen keyboard: wvkbd, toggled from the dock's Keyboard button"
echo " Keyboard layout : ${KEYBOARD_LAYOUT} (console + Sway)"
echo " Maintenance shell : Super+Shift+Ctrl+M opens a floating terminal locally"
echo " Remote admin : SSH only — no wayvnc on this image, see README"
echo
echo "Next steps:"
echo " 1. Write the ISO to a USB stick:"
echo " sudo dd if=${ISO_PATH} of=/dev/sdX bs=4M status=progress oflag=sync"
echo " (double-check /dev/sdX with 'lsblk' first — dd will happily eat the wrong disk)"
echo " 2. Boot the target touchscreen machine from it. No touch-panel hardware has been"
echo " chosen yet (see the README) — confirm touch input is recognised as a Wayland"
echo " wl_touch device (not emulated as a pointer) before treating this as validated."
echo " 3. Confirm greetd autologin lands in Sway: dark background, the touch dock at"
echo " the bottom, and the Home Assistant kiosk window on workspace 2:home."
echo " 4. Tap through the dock: Spotify (log in on first launch — Premium required),"
echo " Home, Web."
echo " 5. Confirm touchpanel-agent connected and registered:"
echo " systemctl status touchpanel-agent"
echo " In Home Assistant, a '${TOUCHPANEL_NAME}' device should appear under the MQTT"
echo " integration with: Show Spotify / Show Home / Show web browser (buttons),"
echo " Screen (select), Volume (number), Playback state (sensor), and"
echo " play/pause/next/prev/stop buttons."
echo " 6. Start something in Spotify and confirm Playback state follows it (playerctl"
echo " -> touchpanel-agent's MPRIS bridge — needs Spotify's own MPRIS bus name to"
echo " actually be 'spotify', unverified against the real Flatpak build)."
echo " 7. Tap the dock's Keyboard button over a text field (Spotify search, the browser"
echo " address bar) and confirm wvkbd shows/hides. There is no auto-show — see README."
echo
echo "Then pull the power on the container host and re-check: the panel must still boot"
echo "and show Spotify/the browser (Home will show a connection error, which is"
echo "expected) with the container host powered off — same reactive-path philosophy as"
echo "hosts/thin-client's Phase 11.10."
echo
echo "Rebuilding later: edit hosts/touch-panel/configs/* or agent/*, then re-run this"
echo "script — includes.chroot is regenerated from them every time."

15
pantry-vision/Dockerfile Normal file
View File

@ -0,0 +1,15 @@
# pantry-vision — camera-driven grocery cataloguing API (docs/project-plan.md Phase
# 17). `restart: unless-stopped`, same as admin-canvas: it has to be up whenever the
# kitchen display might hold an item up to the camera.
FROM python:3.11-slim
ENV PYTHONUNBUFFERED=1 \
PYTHONDONTWRITEBYTECODE=1
WORKDIR /app
# stdlib only (urllib for the Ollama/Grocy calls) — no requirements.txt, same call as
# admin-canvas/server.py.
COPY server.py ./
CMD ["python", "server.py"]

125
pantry-vision/README.md Normal file
View File

@ -0,0 +1,125 @@
# pantry-vision
The kitchen display's backend, from [Phase 17 of the project plan](../docs/project-plan.md).
The workflow this exists for: **come home, put down the shopping bag, hold one item
up to the kitchen display's camera, the system proposes what it is and roughly how
long it keeps, you confirm (editing anything it got wrong), put it away.** The same
display then shows the resulting inventory ordered by what expires soonest, and
Grocy's recipes, on request.
- **`pantry-vision`** (this directory) — a small always-on Python HTTP service.
`POST /identify` (a photo → a proposal via an Ollama vision model), `POST /confirm`
(a human-reviewed proposal → written into Grocy stock), `GET /inventory` and
`GET /recipes` (proxy Grocy, reshaped for the frontend). All four endpoints are
bearer-token gated.
- **`frontend/`** — the static single-page app the kitchen display's kiosk browser
loads: Scan / Inventory / Recipes, vanilla JS, no build step, no framework — same
"vendored, dependency-free" choice as the digest/admin canvas SDKs. Served
read-only by a `pantry-web` nginx container (`setup-container-host.sh`), the same
role `digest-web`/`admin-web` already play for their own hosts.
- **`hosts/kitchen-display/`** — the touch kiosk image that runs the frontend. See
that directory's own README for the device side of this.
## A real network listener, unlike admin-canvas
`admin-canvas` deliberately has **no published port** — only Home Assistant, on the
same compose network, ever calls it. `pantry-vision` is different on purpose: the
kitchen display is a separate physical device on the LAN and has to reach this
service directly (there is no HA-mediation step between "hold item up to camera" and
"get an identification back" — that has to be fast and synchronous). So
`PANTRY_VISION_PORT` **is** published, and every request — including the two GETs —
requires the bearer token, as the actual boundary instead of network placement.
The same token has to be baked into the kitchen display's own build config
(`hosts/kitchen-display/scripts/build-kitchen-display-iso.sh`), not just Home
Assistant's — see that host's README.
## `/identify` never writes anything by itself
This is the one guardrail that matters most in this whole phase. The vision model's
guess — name, category, how many days until it likely goes bad — is a **proposal**,
shown on screen for the person to review and edit before anything is confirmed. Only
`/confirm`, a separate call the frontend makes after the person taps "Confirm & add,"
ever writes to Grocy. This is the same "propose, never auto-commit" rule this project
already applies to identity-merge confirmation (see the *Identity store* row in
`docs/project-plan.md` §2) — a wrong camera guess costs one tap to fix, not a wrong
fact silently written into the household's inventory.
## Configure
```sh
cp pantry-vision/pantry-vision.env.example /opt/smart-home/pantry-vision/pantry-vision.env
openssl rand -hex 32 # put the result in PANTRY_VISION_TOKEN
chmod 600 /opt/smart-home/pantry-vision/pantry-vision.env
$EDITOR /opt/smart-home/pantry-vision/pantry-vision.env
```
You also need, inside Grocy's own UI (it's already running as the always-on `grocy`
container regardless of this phase): **Settings → Manage API keys** for
`GROCY_API_KEY`, and to confirm **Settings → Locations / Quantity units** actually
match `GROCY_DEFAULT_LOCATION_ID`/`GROCY_DEFAULT_QU_ID` (fresh-install defaults, not
guaranteed to match a Grocy that's already been customised).
## Pick a vision model
`OLLAMA_VISION_MODEL` defaults to `llava`, but **nothing here has confirmed that name
against a real pull** — pick a vision-capable model (`llava`, `qwen2.5vl`, or whatever
your LLM host's GPU/CPU tier can run at acceptable latency for someone standing at
the counter holding a can of beans) and:
```sh
ollama pull llava # on the LLM host, or whichever model you picked
```
Plain text models (`qwen2.5:14b-instruct`, used elsewhere in this project for
digest/Assist) **cannot see images at all** — pointing `OLLAMA_VISION_MODEL` at one
of those will not error clearly, it will just produce a useless/hallucinated
response for every photo. Latency is unmeasured; a vision pass on a CPU-only LLM
host could easily be too slow for a "hold item up to camera" interaction to feel
responsive — this needs to be measured on real hardware, not assumed.
## Grocy API assumptions — unverified against a real instance
`server.py`'s Grocy calls (`_find_or_create_product`, `_add_to_stock`, the `/inventory`
and `/recipes` proxies) are written against Grocy's *documented* API shape, not
checked against a running instance. In particular:
- Whether `GET /api/stock` rows carry a nested `product` object with a `name` field
by default, or need an explicit embed/expand parameter — `_handle_inventory`
degrades to `Product #<id>` if not, rather than dropping the row, but that's a
fallback, not a fix.
- Whether `POST /api/objects/products` with just
`name`/`location_id`/`qu_id_purchase`/`qu_id_stock` is actually enough to create a
minimal product on your Grocy version, or whether it requires more fields.
- Whether the Recipes feature (`GET /api/objects/recipes`,
`GET /api/recipes/{id}/fulfillment`) needs to be explicitly enabled/populated
before it returns anything meaningful — `_handle_recipes` degrades to
`"fulfilled": null` per-recipe on any failure rather than breaking the whole list.
Grocy exposes a live OpenAPI spec at `http://<grocy-host>:9283/api/openapi/specification`
once it's running — read that against a real instance before trusting any of the
above, and adjust `server.py` if the shapes differ.
## Deploy
Wired into `hosts/container-host/scripts/setup-container-host.sh` behind
`ENABLE_PANTRY_VISION` (off by default) — see that script's `# CONFIGURATION` block
and its own README. It builds two containers: `pantry-vision` (this API) and
`pantry-web` (nginx, serves `frontend/` read-only).
## Manual verification still outstanding
1. All of the Grocy API assumptions above.
2. Real-world vision-model accuracy and latency for grocery items — untested with
any actual model or camera.
3. Whether Ollama's `/api/generate` `images` field is still the right call shape for
whichever vision model you pick — some multimodal models are only exposed through
Ollama's newer `/api/chat` with a `images` field per-message instead; this was
written against `/api/generate`'s documented multimodal support and not run
against a real model.
4. CORS: `_respond`'s blanket `Access-Control-Allow-Origin: *` is fine for a
LAN-only, bearer-token-gated service with no cookies, but hasn't been checked
against a real browser's preflight behavior for the raw-image-bytes `POST
/identify` call in particular (some browsers preflight non-simple `Content-Type`s
like `image/jpeg``do_OPTIONS` is written to handle that but is untested).

View File

@ -0,0 +1,227 @@
// pantry-vision frontend logic (docs/project-plan.md Phase 17). Vanilla JS, no
// framework, no build step — same "vendored, dependency-free" choice as the digest/
// admin canvas SDKs' render.js.
//
// Config comes from URL query params (?api=...&token=...), set by the
// kitchen-display kiosk's own launch command, NOT hardcoded here — this file is a
// generic static asset with no secret in it, served read-only by pantry-web to
// whatever device points a browser at it.
"use strict";
const params = new URLSearchParams(location.search);
const API = (params.get("api") || "").replace(/\/$/, "");
const TOKEN = params.get("token") || "";
if (!API || !TOKEN) {
document.body.innerHTML =
'<p class="error" style="padding:24px">pantry-vision not configured — missing ' +
"?api=&token= in the URL. See hosts/kitchen-display/configs/sway/pantry-kiosk.</p>";
throw new Error("pantry-vision: missing ?api=/&token= query params");
}
function api(path, options) {
options = options || {};
options.headers = Object.assign({ Authorization: `Bearer ${TOKEN}` }, options.headers || {});
return fetch(`${API}${path}`, options).then((res) => {
if (!res.ok) {
return res.json().catch(() => ({})).then((body) => {
throw new Error(body.error || `${res.status} ${res.statusText}`);
});
}
return res.json();
});
}
// --- Tabs --------------------------------------------------------------------
const tabs = document.querySelectorAll(".tab");
const panels = document.querySelectorAll(".panel");
function activateTab(name) {
tabs.forEach((t) => t.classList.toggle("active", t.dataset.tab === name));
panels.forEach((p) => p.classList.toggle("active", p.id === name));
if (name === "inventory") loadInventory();
if (name === "recipes") loadRecipes();
if (name === "scan") resetScan();
}
tabs.forEach((t) => t.addEventListener("click", () => activateTab(t.dataset.tab)));
// A "Show <tab>" MQTT command from touchpanel-style HA control arrives as a URL
// fragment reload (hosts/kitchen-display/agent kills and relaunches Chromium at
// index.html#recipes, same pattern as the thin client's digest-browser) — honour it
// on load, same as any manual tap.
if (location.hash) {
const initial = location.hash.slice(1);
if (["scan", "inventory", "recipes"].includes(initial)) activateTab(initial);
}
// --- Scan ----------------------------------------------------------------------
const video = document.getElementById("camera-preview");
const captureBtn = document.getElementById("capture-btn");
const cameraError = document.getElementById("camera-error");
const scanCamera = document.getElementById("scan-camera");
const scanResult = document.getElementById("scan-result");
const capturedFrame = document.getElementById("captured-frame");
const confirmForm = document.getElementById("confirm-form");
const confirmStatus = document.getElementById("confirm-status");
const retakeBtn = document.getElementById("retake-btn");
let stream = null;
function startCamera() {
if (stream) return;
navigator.mediaDevices
.getUserMedia({ video: { facingMode: "environment" }, audio: false })
.then((s) => {
stream = s;
video.srcObject = s;
cameraError.hidden = true;
})
.catch((err) => {
cameraError.textContent = `Camera unavailable: ${err.message}. See hosts/kitchen-display/README.md.`;
cameraError.hidden = false;
});
}
function resetScan() {
scanResult.hidden = true;
scanCamera.hidden = false;
confirmStatus.textContent = "";
startCamera();
}
captureBtn.addEventListener("click", () => {
if (!stream) return;
capturedFrame.width = video.videoWidth;
capturedFrame.height = video.videoHeight;
capturedFrame.getContext("2d").drawImage(video, 0, 0);
scanCamera.hidden = true;
scanResult.hidden = false;
confirmStatus.textContent = "Identifying…";
document.getElementById("f-confidence").textContent = "";
capturedFrame.toBlob(
(blob) => {
api("/identify", { method: "POST", body: blob, headers: { "Content-Type": "image/jpeg" } })
.then((proposal) => {
document.getElementById("f-name").value = proposal.name || "";
document.getElementById("f-category").value = proposal.category || "other";
const days = Number.isFinite(proposal.estimated_shelf_life_days)
? proposal.estimated_shelf_life_days
: 7;
const due = new Date();
due.setDate(due.getDate() + days);
document.getElementById("f-date").value = due.toISOString().slice(0, 10);
document.getElementById("f-confidence").textContent =
`Model confidence: ${proposal.confidence || "unknown"}` +
(proposal.note ? `${proposal.note}` : "") +
". Review before confirming.";
confirmStatus.textContent = "";
})
.catch((err) => {
confirmStatus.textContent = `Could not identify: ${err.message}. Fill in manually.`;
});
},
"image/jpeg",
0.85
);
});
retakeBtn.addEventListener("click", resetScan);
confirmForm.addEventListener("submit", (event) => {
event.preventDefault();
confirmStatus.textContent = "Adding…";
api("/confirm", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
name: document.getElementById("f-name").value.trim(),
category: document.getElementById("f-category").value,
best_before_date: document.getElementById("f-date").value,
quantity: Number(document.getElementById("f-quantity").value) || 1,
}),
})
.then(() => {
confirmStatus.textContent = "Added. Put it away!";
setTimeout(resetScan, 1500);
})
.catch((err) => {
confirmStatus.textContent = `Could not add: ${err.message}`;
});
});
// --- Inventory -------------------------------------------------------------------
function urgencyClass(daysLeft) {
if (daysLeft === null || daysLeft === undefined) return "";
if (daysLeft < 0) return "urgent-expired";
if (daysLeft <= 2) return "urgent-soon";
if (daysLeft <= 7) return "urgent-week";
return "";
}
function loadInventory() {
const el = document.getElementById("inventory-list");
api("/inventory")
.then((data) => {
const items = data.items || [];
if (!items.length) {
el.innerHTML = '<p class="hint">Nothing in stock yet — scan something!</p>';
return;
}
el.innerHTML = items
.map((item) => {
const days = item.days_left;
const label =
days === null || days === undefined
? "no date"
: days < 0
? `expired ${-days}d ago`
: days === 0
? "expires today"
: `${days}d left`;
return `<div class="card ${urgencyClass(days)}">
<span class="card-name">${escapeHtml(item.name)}</span>
<span class="card-amount">×${escapeHtml(String(item.amount ?? ""))}</span>
<span class="card-due">${escapeHtml(label)}</span>
</div>`;
})
.join("");
})
.catch((err) => {
el.innerHTML = `<p class="error">Could not load inventory: ${escapeHtml(err.message)}</p>`;
});
}
// --- Recipes -----------------------------------------------------------------
function loadRecipes() {
const el = document.getElementById("recipes-list");
api("/recipes")
.then((data) => {
const recipes = data.recipes || [];
if (!recipes.length) {
el.innerHTML = '<p class="hint">No recipes in Grocy yet.</p>';
return;
}
el.innerHTML = recipes
.map((r) => {
const badge = r.fulfilled === true ? "✓ can make now" : r.fulfilled === false ? "missing items" : "";
return `<div class="card">
<span class="card-name">${escapeHtml(r.name || "")}</span>
<span class="card-due">${escapeHtml(badge)}</span>
</div>`;
})
.join("");
})
.catch((err) => {
el.innerHTML = `<p class="error">Could not load recipes: ${escapeHtml(err.message)}</p>`;
});
}
function escapeHtml(s) {
return String(s).replace(/[&<>"']/g, (c) => ({ "&": "&amp;", "<": "&lt;", ">": "&gt;", '"': "&quot;", "'": "&#39;" }[c]));
}
// Start on whatever tab is active (default: scan).
resetScan();

View File

@ -0,0 +1,75 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, user-scalable=no">
<title>Pantry</title>
<link rel="stylesheet" href="style.css">
</head>
<body>
<!--
pantry-vision's frontend (docs/project-plan.md Phase 17). A single static page, no
build step, no framework — same "vendored, dependency-free" choice as the digest/
admin canvas SDKs. Config (the pantry-vision API URL and its bearer token) arrives
as URL query params from the kitchen-display kiosk's own launch command
(hosts/kitchen-display/configs/sway/pantry-kiosk), never baked into this file, so
this stays a plain static asset with no secret in git — see app.js's top comment.
-->
<nav id="tabs">
<button class="tab active" data-tab="scan">📷<span>Scan</span></button>
<button class="tab" data-tab="inventory">🧊<span>Inventory</span></button>
<button class="tab" data-tab="recipes">🍳<span>Recipes</span></button>
</nav>
<main>
<section id="scan" class="panel active">
<div id="scan-camera">
<video id="camera-preview" autoplay playsinline muted></video>
<button id="capture-btn" class="big-btn">Capture</button>
<p id="camera-error" class="error" hidden></p>
</div>
<div id="scan-result" hidden>
<canvas id="captured-frame"></canvas>
<form id="confirm-form">
<label>Name <input id="f-name" type="text" required></label>
<label>Category
<select id="f-category">
<option value="produce">Produce</option>
<option value="dairy">Dairy</option>
<option value="meat">Meat</option>
<option value="frozen">Frozen</option>
<option value="pantry">Pantry</option>
<option value="bakery">Bakery</option>
<option value="beverage">Beverage</option>
<option value="other">Other</option>
</select>
</label>
<label>Best before <input id="f-date" type="date" required></label>
<label>Quantity <input id="f-quantity" type="number" min="1" step="1" value="1"></label>
<p id="f-confidence" class="hint"></p>
<div class="form-actions">
<button type="button" id="retake-btn" class="big-btn secondary">Retake</button>
<button type="submit" id="confirm-btn" class="big-btn primary">Confirm &amp; add</button>
</div>
<p id="confirm-status" class="hint"></p>
</form>
</div>
</section>
<section id="inventory" class="panel">
<div id="inventory-list" class="card-list">
<p class="hint">Loading…</p>
</div>
</section>
<section id="recipes" class="panel">
<div id="recipes-list" class="card-list">
<p class="hint">Loading…</p>
</div>
</section>
</main>
<script src="app.js"></script>
</body>
</html>

View File

@ -0,0 +1,192 @@
/* pantry-vision frontend styling (docs/project-plan.md Phase 17). Touch targets are
* deliberately large this is a kitchen-counter finger interface. */
* {
box-sizing: border-box;
}
html, body {
margin: 0;
height: 100%;
background: #101014;
color: #e8e8ec;
font-family: sans-serif;
overflow: hidden;
}
#tabs {
display: flex;
height: 84px;
background: rgba(16, 16, 20, 0.96);
border-bottom: 1px solid rgba(255, 255, 255, 0.08);
}
.tab {
flex: 1;
display: flex;
flex-direction: column;
align-items: center;
justify-content: center;
gap: 2px;
background: transparent;
border: none;
color: #9a9aa6;
font-size: 26px;
}
.tab span {
font-size: 13px;
font-weight: 500;
}
.tab.active {
color: #e8e8ec;
background: rgba(110, 168, 254, 0.16);
}
main {
height: calc(100% - 84px);
overflow-y: auto;
}
.panel {
display: none;
padding: 16px;
}
.panel.active {
display: block;
}
/* --- Scan ------------------------------------------------------------------- */
#scan-camera, #scan-result {
display: flex;
flex-direction: column;
align-items: center;
gap: 16px;
}
#camera-preview, #captured-frame {
width: 100%;
max-width: 640px;
border-radius: 12px;
background: #000;
}
.big-btn {
min-height: 64px;
min-width: 200px;
font-size: 18px;
font-weight: 600;
border-radius: 12px;
border: none;
color: #101014;
background: #6ea8fe;
}
.big-btn.secondary {
background: rgba(255, 255, 255, 0.14);
color: #e8e8ec;
}
.big-btn.primary {
background: #6ea8fe;
}
#confirm-form {
width: 100%;
max-width: 640px;
display: flex;
flex-direction: column;
gap: 12px;
}
#confirm-form label {
display: flex;
flex-direction: column;
gap: 4px;
font-size: 14px;
color: #9a9aa6;
}
#confirm-form input, #confirm-form select {
min-height: 48px;
font-size: 16px;
padding: 8px 12px;
border-radius: 8px;
border: 1px solid rgba(255, 255, 255, 0.16);
background: #1a1a20;
color: #e8e8ec;
}
.form-actions {
display: flex;
gap: 12px;
justify-content: center;
}
.hint {
color: #9a9aa6;
font-size: 14px;
}
.error {
color: #ff8080;
font-size: 14px;
}
/* --- Cards (inventory + recipes) --------------------------------------------- */
.card-list {
display: flex;
flex-direction: column;
gap: 8px;
}
.card {
display: flex;
align-items: center;
gap: 12px;
min-height: 56px;
padding: 8px 16px;
border-radius: 10px;
background: rgba(255, 255, 255, 0.06);
}
.card-name {
flex: 1;
font-size: 16px;
font-weight: 500;
}
.card-amount {
color: #9a9aa6;
font-size: 14px;
}
.card-due {
font-size: 14px;
color: #9a9aa6;
min-width: 110px;
text-align: right;
}
.urgent-week {
background: rgba(255, 200, 80, 0.10);
}
.urgent-week .card-due {
color: #ffc850;
}
.urgent-soon {
background: rgba(255, 140, 80, 0.14);
}
.urgent-soon .card-due {
color: #ff8c50;
}
.urgent-expired {
background: rgba(255, 90, 90, 0.16);
}
.urgent-expired .card-due {
color: #ff5a5a;
}

View File

@ -0,0 +1,48 @@
# pantry-vision configuration template.
#
# Copy this to the container host as (for example)
# /opt/smart-home/pantry-vision/pantry-vision.env, fill in real values, and chmod 600
# it. Same never-commit handling as admin-canvas.env / digest-engine.env.
# ---------------------------------------------------------------------------
# Auth — required. pantry-vision fails closed (rejects every request) while this is
# empty. Unlike admin-canvas's token (which only ever needs to be known by Home
# Assistant), this one also has to be baked into the kitchen-display kiosk's own
# build config (hosts/kitchen-display/scripts/build-kitchen-display-iso.sh), since
# the kiosk's frontend calls this API directly, not through HA. Generate one with:
# openssl rand -hex 32
# ---------------------------------------------------------------------------
PANTRY_VISION_TOKEN=
# ---------------------------------------------------------------------------
# Vision LLM (Ollama, the same LLM host as digest-engine/Assist)
# ---------------------------------------------------------------------------
OLLAMA_HOST=http://llm-host:11434
# MUST be a vision-capable model, and MUST already be pulled on the LLM host
# (`ollama pull llava`, or whatever you pick) — plain text models like the
# qwen2.5:14b-instruct used elsewhere in this project cannot see images at all.
# Not verified against a real pull/run — see README.md.
OLLAMA_VISION_MODEL=llava
OLLAMA_TIMEOUT=120
# ---------------------------------------------------------------------------
# Grocy — already running as the always-on `grocy` container from
# hosts/container-host/scripts/setup-container-host.sh, reachable on the compose
# network at this internal name/port regardless of ENABLE_PANTRY_VISION.
# ---------------------------------------------------------------------------
GROCY_URL=http://grocy:80
# Settings -> Manage API keys, inside Grocy's own UI. Required for /confirm (writes);
# /inventory and /recipes (reads) use the same key.
GROCY_API_KEY=
# VERIFY against your own Grocy instance (Settings -> Locations / Quantity units) —
# these are fresh-install defaults ("Default" location, "Piece" unit), not guaranteed
# to match a Grocy that's already been customised.
GROCY_DEFAULT_LOCATION_ID=1
GROCY_DEFAULT_QU_ID=2
# ---------------------------------------------------------------------------
# Run behaviour
# ---------------------------------------------------------------------------
PANTRY_VISION_PORT=8095
PANTRY_VISION_MAX_IMAGE_MB=15
LOG_LEVEL=INFO

391
pantry-vision/server.py Normal file
View File

@ -0,0 +1,391 @@
"""pantry-vision — camera-driven grocery cataloguing for the kitchen display, from
docs/project-plan.md Phase 17.
The workflow this exists for: come home, put down the shopping bag, hold one item up
to the kitchen display's camera, the system proposes what it is and how long it keeps,
you confirm (editing anything it got wrong) before it's written anywhere, then put the
item away. The display also shows the resulting inventory sorted by what expires
soonest, and Grocy's recipes.
SECURITY BOUNDARY, same shape as admin-canvas/server.py's, with one deliberate
difference: this service DOES get a published port (see the README and
setup-container-host.sh's PANTRY_VISION_PORT), because unlike admin-canvas — which is
only ever called by Home Assistant on the compose network this one is called
directly by the kitchen-display kiosk, a separate physical device on the LAN. Every
request still requires the bearer token below; an unset token fails closed (rejects
everything), never "auth optional".
Four endpoints, all token-gated:
- POST /identify raw image bytes -> Ollama vision model -> a *proposal*
- POST /confirm JSON, human-reviewed/edited -> writes to Grocy stock
- GET /inventory proxies Grocy stock, sorted by soonest-expiring first
- GET /recipes proxies Grocy recipes (+ fulfillment, best-effort)
/identify NEVER writes anything by itself same "propose, never auto-commit" rule
this project already applies to identity-merge confirmation (see the Identity store
row in docs/project-plan.md §2): the vision model's guess is shown to the person
standing at the display, who can edit the name/category/date/location before /confirm
is ever called. A wrong guess costs a tap to fix, not a wrong fact silently written
into the household's inventory.
"""
from __future__ import annotations
import base64
import json
import logging
import os
import re
import sys
import urllib.error
import urllib.request
from datetime import date, datetime
from http import HTTPStatus
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from urllib.parse import urlsplit
LOG = logging.getLogger("pantry-vision")
TOKEN = os.environ.get("PANTRY_VISION_TOKEN", "")
OLLAMA_HOST = os.environ.get("OLLAMA_HOST", "http://llm-host:11434").rstrip("/")
# TBD/unverified — see README.md's "Pick a vision model" section. Not every Ollama
# model can see images; this must be one that can (llava, qwen2.5vl, etc.) and must
# actually be pulled on the LLM host (`ollama pull <name>`) before /identify works.
OLLAMA_VISION_MODEL = os.environ.get("OLLAMA_VISION_MODEL", "llava")
OLLAMA_TIMEOUT = float(os.environ.get("OLLAMA_TIMEOUT", "120"))
GROCY_URL = os.environ.get("GROCY_URL", "http://grocy:80").rstrip("/")
GROCY_API_KEY = os.environ.get("GROCY_API_KEY", "")
# Fresh-install defaults ("Default" location, "Piece" quantity unit) — VERIFY against
# your own Grocy instance (Settings -> Locations / Quantity units) and override if
# they differ. Same "don't build against a guess" flag as every other cross-service ID
# in this project (ADMIN_WEB_URL placeholders, weather_entity_id, etc.).
GROCY_DEFAULT_LOCATION_ID = os.environ.get("GROCY_DEFAULT_LOCATION_ID", "1")
GROCY_DEFAULT_QU_ID = os.environ.get("GROCY_DEFAULT_QU_ID", "2")
MAX_IMAGE_BYTES = int(os.environ.get("PANTRY_VISION_MAX_IMAGE_MB", "15")) * 1024 * 1024
MAX_JSON_BYTES = 64 * 1024
_IDENTIFY_PROMPT = """You are looking at a single grocery item held up to a kitchen \
camera, on a plain background. Identify it and answer ONLY with a JSON object, no \
other text, in exactly this shape:
{"name": "<short product name>", "category": "<produce|dairy|meat|frozen|pantry|bakery|beverage|other>", \
"estimated_shelf_life_days": <integer, typical days from today until it goes bad, \
assuming normal fridge/pantry storage for its category>, "confidence": "<high|medium|low>"}
If you cannot identify the item, still return the JSON shape with your best guess and \
"confidence": "low"."""
_JSON_OBJECT_RE = re.compile(r"\{.*\}", re.DOTALL)
def _http_json(method: str, url: str, payload: dict | None = None, headers: dict | None = None,
timeout: float = 30.0) -> dict:
data = json.dumps(payload).encode("utf-8") if payload is not None else None
req = urllib.request.Request(url, data=data, method=method)
req.add_header("Content-Type", "application/json")
for key, value in (headers or {}).items():
req.add_header(key, value)
with urllib.request.urlopen(req, timeout=timeout) as resp:
body = resp.read()
return json.loads(body) if body else {}
def _grocy_headers() -> dict:
return {"GROCY-API-KEY": GROCY_API_KEY}
def _grocy_get(path: str):
return _http_json("GET", f"{GROCY_URL}{path}", headers=_grocy_headers(), timeout=15)
def _grocy_post(path: str, payload: dict):
return _http_json("POST", f"{GROCY_URL}{path}", payload=payload, headers=_grocy_headers(), timeout=15)
def _identify_via_ollama(image_bytes: bytes) -> dict:
"""Calls the vision model and returns a best-effort proposal — degrades to a
low-confidence placeholder rather than raising, mirroring digest-engine's
llm_client._fallback_document: a broken/blank result is worse than one flagged
as unreliable that a human has to review anyway (which they always do see the
module docstring, /identify never writes anything by itself).
"""
payload = {
"model": OLLAMA_VISION_MODEL,
"prompt": _IDENTIFY_PROMPT,
"images": [base64.b64encode(image_bytes).decode("ascii")],
"stream": False,
"options": {"temperature": 0.2},
}
try:
result = _http_json("POST", f"{OLLAMA_HOST}/api/generate", payload=payload, timeout=OLLAMA_TIMEOUT)
text = result.get("response", "")
except Exception:
LOG.warning("pantry-vision: Ollama call failed", exc_info=True)
return {
"name": "Unknown item",
"category": "other",
"estimated_shelf_life_days": 7,
"confidence": "low",
"note": f"Could not reach the vision model at {OLLAMA_HOST} — filled in manually.",
}
match = _JSON_OBJECT_RE.search(text)
if match:
try:
proposal = json.loads(match.group(0))
proposal.setdefault("name", "Unknown item")
proposal.setdefault("category", "other")
proposal.setdefault("estimated_shelf_life_days", 7)
proposal.setdefault("confidence", "low")
return proposal
except ValueError:
pass
LOG.warning("pantry-vision: could not parse a JSON proposal out of the model's response")
return {
"name": "Unknown item",
"category": "other",
"estimated_shelf_life_days": 7,
"confidence": "low",
"note": "The vision model's response wasn't valid JSON — filled in manually.",
"raw_response": text[:500],
}
def _find_or_create_product(name: str) -> int:
"""Best-effort product lookup/create against Grocy's object API.
VERIFY against a real Grocy instance this assumes GET /api/objects/products
returns a flat list of {id, name, ...} and that POST to the same path with just
name/location_id/qu_id_purchase/qu_id_stock is enough to create a minimal product.
Grocy's actual required-field set depends on its own settings and has not been
checked against a live instance; see README.md's verification list.
"""
products = _grocy_get("/api/objects/products")
needle = name.strip().lower()
for product in products:
if str(product.get("name", "")).strip().lower() == needle:
return int(product["id"])
created = _grocy_post(
"/api/objects/products",
{
"name": name,
"location_id": int(GROCY_DEFAULT_LOCATION_ID),
"qu_id_purchase": int(GROCY_DEFAULT_QU_ID),
"qu_id_stock": int(GROCY_DEFAULT_QU_ID),
},
)
return int(created["created_object_id"])
def _add_to_stock(product_id: int, amount: float, best_before_date: str) -> dict:
return _grocy_post(
f"/api/stock/products/{product_id}/add",
{"amount": amount, "best_before_date": best_before_date, "transaction_type": "purchase"},
)
def _days_left(best_before: str | None) -> int | None:
if not best_before:
return None
try:
due = datetime.strptime(best_before, "%Y-%m-%d").date()
except ValueError:
return None
return (due - date.today()).days
class Handler(BaseHTTPRequestHandler):
server_version = "pantry-vision/1"
def log_message(self, format, *args): # noqa: A002
LOG.info("%s - %s", self.address_string(), format % args)
def _authorized(self) -> bool:
if not TOKEN:
return False
return self.headers.get("Authorization", "") == f"Bearer {TOKEN}"
def _respond(self, status: HTTPStatus, payload) -> None:
body = json.dumps(payload).encode("utf-8")
self.send_response(status)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(body)))
self.send_header("Access-Control-Allow-Origin", "*")
self.end_headers()
self.wfile.write(body)
def _read_body(self, max_bytes: int) -> bytes:
try:
length = int(self.headers.get("Content-Length") or 0)
except ValueError:
raise ValueError("missing or invalid Content-Length") from None
if length <= 0:
return b""
if length > max_bytes:
raise ValueError(f"body too large ({length} > {max_bytes} bytes)")
return self.rfile.read(length)
def do_OPTIONS(self): # noqa: N802
# The frontend (pantry-web, a separate origin/port) fetches this API directly
# from the browser, so a plain CORS preflight has to succeed.
self.send_response(HTTPStatus.NO_CONTENT)
self.send_header("Access-Control-Allow-Origin", "*")
self.send_header("Access-Control-Allow-Headers", "Authorization, Content-Type")
self.send_header("Access-Control-Allow-Methods", "GET, POST, OPTIONS")
self.end_headers()
def do_GET(self): # noqa: N802
if not self._authorized():
self._respond(HTTPStatus.UNAUTHORIZED, {"error": "missing or invalid bearer token"})
return
path = urlsplit(self.path).path
if path == "/inventory":
self._handle_inventory()
elif path == "/recipes":
self._handle_recipes()
else:
self._respond(HTTPStatus.NOT_FOUND, {"error": "no such endpoint"})
def do_POST(self): # noqa: N802
if not self._authorized():
self._respond(HTTPStatus.UNAUTHORIZED, {"error": "missing or invalid bearer token"})
return
path = urlsplit(self.path).path
if path == "/identify":
self._handle_identify()
elif path == "/confirm":
self._handle_confirm()
else:
self._respond(HTTPStatus.NOT_FOUND, {"error": "no such endpoint"})
def _handle_identify(self) -> None:
try:
image = self._read_body(MAX_IMAGE_BYTES)
except ValueError as exc:
self._respond(HTTPStatus.REQUEST_ENTITY_TOO_LARGE, {"error": str(exc)})
return
if not image:
self._respond(HTTPStatus.BAD_REQUEST, {"error": "empty image body"})
return
proposal = _identify_via_ollama(image)
self._respond(HTTPStatus.OK, proposal)
def _handle_confirm(self) -> None:
try:
raw = self._read_body(MAX_JSON_BYTES)
payload = json.loads(raw or b"{}")
except (ValueError, json.JSONDecodeError) as exc:
self._respond(HTTPStatus.BAD_REQUEST, {"error": f"bad request body: {exc}"})
return
name = str(payload.get("name", "")).strip()
best_before = str(payload.get("best_before_date", "")).strip()
amount = float(payload.get("quantity") or 1)
if not name:
self._respond(HTTPStatus.BAD_REQUEST, {"error": "'name' is required"})
return
if not re.match(r"^\d{4}-\d{2}-\d{2}$", best_before):
self._respond(HTTPStatus.BAD_REQUEST, {"error": "'best_before_date' must be YYYY-MM-DD"})
return
if not GROCY_API_KEY:
self._respond(HTTPStatus.SERVICE_UNAVAILABLE, {"error": "GROCY_API_KEY is not configured"})
return
try:
product_id = _find_or_create_product(name)
_add_to_stock(product_id, amount, best_before)
except (urllib.error.URLError, urllib.error.HTTPError, KeyError, ValueError) as exc:
LOG.warning("pantry-vision: Grocy write failed", exc_info=True)
self._respond(HTTPStatus.BAD_GATEWAY, {"error": f"Grocy write failed: {exc}"})
return
LOG.info("pantry-vision: added %s x%s (best before %s) to Grocy", name, amount, best_before)
self._respond(HTTPStatus.OK, {"ok": True, "product_id": product_id})
def _handle_inventory(self) -> None:
try:
stock = _grocy_get("/api/stock")
except (urllib.error.URLError, urllib.error.HTTPError) as exc:
LOG.warning("pantry-vision: Grocy /api/stock unreachable", exc_info=True)
self._respond(HTTPStatus.BAD_GATEWAY, {"error": f"Grocy unreachable: {exc}"})
return
# VERIFY: assumes each stock row carries a nested "product" object with at
# least a "name" (Grocy's documented default `?embed` behaviour varies by
# version) — falls back to "product_id" as the display name if not, rather
# than dropping the row, same "degrade, don't blank" rule as everywhere else
# in this project's renderers.
items = []
for row in stock if isinstance(stock, list) else []:
product = row.get("product") or {}
name = product.get("name") or f"Product #{row.get('product_id')}"
best_before = row.get("best_before_date")
items.append(
{
"name": name,
"amount": row.get("amount"),
"best_before_date": best_before,
"days_left": _days_left(best_before),
}
)
items.sort(key=lambda i: (i["days_left"] is None, i["days_left"]))
self._respond(HTTPStatus.OK, {"items": items})
def _handle_recipes(self) -> None:
try:
recipes = _grocy_get("/api/objects/recipes")
except (urllib.error.URLError, urllib.error.HTTPError) as exc:
LOG.warning("pantry-vision: Grocy recipes unreachable", exc_info=True)
self._respond(HTTPStatus.BAD_GATEWAY, {"error": f"Grocy unreachable: {exc}"})
return
results = []
for recipe in recipes if isinstance(recipes, list) else []:
entry = {"id": recipe.get("id"), "name": recipe.get("name"), "fulfilled": None}
# Best-effort only: Grocy's fulfillment endpoint needs the recipes feature
# actually set up (ingredients entered per recipe) to mean anything, and
# its exact response shape is unverified — see README.md. A failure here
# degrades to "fulfilled: null" (rendered as "unknown" client-side), never
# a broken page.
try:
fulfillment = _grocy_get(f"/api/recipes/{entry['id']}/fulfillment")
entry["fulfilled"] = bool(fulfillment.get("need_fulfilled"))
except Exception:
pass
results.append(entry)
self._respond(HTTPStatus.OK, {"recipes": results})
def main() -> int:
logging.basicConfig(
level=os.environ.get("LOG_LEVEL", "INFO").upper(),
format="%(asctime)s %(levelname)-7s %(name)s: %(message)s",
)
if not TOKEN:
LOG.error(
"PANTRY_VISION_TOKEN is not set — every request will be rejected until it is. "
"See pantry-vision.env.example."
)
if not GROCY_API_KEY:
LOG.warning("GROCY_API_KEY is not set — /confirm will fail until it is configured.")
port = int(os.environ.get("PANTRY_VISION_PORT", "8095"))
server = ThreadingHTTPServer(("0.0.0.0", port), Handler)
LOG.info("pantry-vision listening on :%d (Ollama: %s, Grocy: %s)", port, OLLAMA_HOST, GROCY_URL)
try:
server.serve_forever()
except KeyboardInterrupt:
pass
return 0
if __name__ == "__main__":
sys.exit(main())