126 lines
7.5 KiB
Markdown
126 lines
7.5 KiB
Markdown
# Sway kitchen display
|
|
|
|
Phase 17 of `docs/project-plan.md`. Builds a Debian 12 live ISO for a single-purpose
|
|
fridge/pantry display: hold a grocery item up to its camera, confirm what the vision
|
|
model thinks it is and when it'll go bad, put it away — then browse the resulting
|
|
inventory (soonest-to-expire first) and Grocy's recipes, all touch-driven.
|
|
|
|
**The camera-vision and Grocy read/write logic does not live in this directory** —
|
|
that's [`../../pantry-vision/`](../../pantry-vision/README.md), a container-host
|
|
service this device's kiosk browser calls directly. This host is deliberately thin:
|
|
a Sway session, one Chromium kiosk window, and `kitchen-display-agent`, which only
|
|
ever controls *which screen is showing* (Scan / Inventory / Recipes) over MQTT — the
|
|
same "the agent controls the surface, a separate write API owns the content" split
|
|
`hosts/thin-client`'s admin canvas already established.
|
|
|
|
**A third, simpler sibling of `../thin-client/` and `../touch-panel/`**, not a variant
|
|
of either — one workspace, one app, no dock, no music, no general browsing. It reuses
|
|
the thin client's live-build toolchain and `configs/`+`agent/` split.
|
|
|
|
| | |
|
|
|---|---|
|
|
| Compositor | Sway, one workspace, one Chromium kiosk window |
|
|
| Autologin | greetd, straight into `/usr/local/bin/kiosk-session` |
|
|
| Touch input | Native Wayland `wl_touch`, same assumption as `../touch-panel/` |
|
|
| Camera | Browser-native `getUserMedia()` inside the Chromium kiosk page — no separate native capture app on this device at all |
|
|
| Content | `pantry-vision`'s frontend (`../../pantry-vision/frontend/`), served by `pantry-web`; in-page tabs, not separate Sway workspaces |
|
|
| Remote control (HA/LLM) | `kitchen-display-agent` — **Show scan / Show inventory / Show recipes** buttons only |
|
|
| Remote control (human) | SSH only — no wayvnc, same scope decision as `../touch-panel/` |
|
|
|
|
## Hardware
|
|
|
|
**No specific unit has been chosen.** What's assumed: a touchscreen (native
|
|
`wl_touch`, same caveat as `../touch-panel/README.md`) mounted somewhere near the
|
|
fridge/pantry, with a USB webcam pointed at wherever items get held up — either
|
|
built into the panel or a separate camera on a short cable/gooseneck mount so the
|
|
angle can be adjusted independently of the screen. See §1.15 of
|
|
`docs/project-plan.md` for the (unverified, non-specific) hardware line item.
|
|
|
|
## Before you build
|
|
|
|
Deploy `pantry-vision`/`pantry-web` first (`ENABLE_PANTRY_VISION` in
|
|
`hosts/container-host/scripts/setup-container-host.sh`) — this image builds and boots
|
|
fine without it, but every tab will show a connection error until it exists. Then edit
|
|
the `# CONFIGURATION` block at the top of
|
|
[`scripts/build-kitchen-display-iso.sh`](scripts/build-kitchen-display-iso.sh):
|
|
|
|
| Variable | What to put in it |
|
|
|---|---|
|
|
| `MQTT_BROKER_HOST` | LAN IP of the container host running Mosquitto |
|
|
| `PANTRY_WEB_URL` | The `pantry-web` service's URL, e.g. `http://192.168.1.10:8096` |
|
|
| `PANTRY_VISION_URL` | The `pantry-vision` API's URL, e.g. `http://192.168.1.10:8095` — read by `app.js` from the kiosk window's own URL query params (see `pantry-kiosk`'s comment), never baked into the served frontend files |
|
|
| `PANTRY_VISION_TOKEN` | **Must exactly match** `PANTRY_VISION_TOKEN` in `pantry-vision/pantry-vision.env` on the container host — there's no way for this repo to push that value between the two hosts for you |
|
|
| `KIOSK_USERNAME` / `IMAGE_HOSTNAME` / `KITCHEN_DISPLAY_NAME` | Per-device identity |
|
|
| `SSH_AUTHORIZED_KEY` | Optional — password auth is disabled and there is no wayvnc |
|
|
|
|
## Build
|
|
|
|
```sh
|
|
sudo ./scripts/build-kitchen-display-iso.sh
|
|
```
|
|
|
|
Same directory-split convention as every other host in this project: `configs/` and
|
|
`agent/` are human-edited and git-tracked; `live-build/config/includes.chroot/` is
|
|
generated, gitignored, never hand-edited.
|
|
|
|
## Why the camera has no native capture app
|
|
|
|
`../touch-panel/` and `../thin-client/` both shell out to native processes for their
|
|
media (mpv, the Spotify Flatpak). This device doesn't need that: Chromium's own
|
|
`getUserMedia()` API, called from `pantry-vision/frontend/app.js`, talks to the
|
|
webcam directly and hands a captured frame straight to `POST /identify` as a blob —
|
|
no intermediate script, no temp file, no separate process to supervise. The one thing
|
|
this needs from the kiosk launch itself is `--use-fake-ui-for-media-stream`
|
|
(`configs/sway/pantry-kiosk`), which auto-accepts the camera permission prompt that
|
|
would otherwise sit unanswered on a screen nobody is there to click "Allow" on.
|
|
|
|
## Home Assistant entities
|
|
|
|
`kitchen-display-agent` publishes three buttons on connect: **Show scan**, **Show
|
|
inventory**, **Show recipes** — each kills and relaunches the kiosk Chromium window
|
|
at `PANTRY_WEB_URL/index.html?...#<tab>`. That's the entire MQTT surface; there is no
|
|
media_player, no workspace select (one workspace), no capture/audio/remote-desktop
|
|
entities the way the thin client has. Reading or writing the actual inventory (from
|
|
voice, e.g. "what's about to expire?" or "add three eggs") is a **separate** path —
|
|
Home Assistant/the LLM calling `pantry-vision`'s own published API directly, not
|
|
through this agent, exactly mirroring the split `hosts/thin-client`'s admin canvas
|
|
already established between "which screen is showing" (MQTT, this agent) and "what's
|
|
actually on it" (a dedicated write API).
|
|
|
|
### Security boundary
|
|
|
|
Same shape and reasoning as every other host's agent in this project:
|
|
`kitchen_display_agent/mqtt_discovery.py` is the entire inbound MQTT control surface
|
|
of this machine — LLM tool call → HA service call → MQTT → this agent, no HTTP
|
|
listener, no websocket, no exposed Sway IPC socket, no VNC. `pantry-vision` is a
|
|
**separate, deliberately published** service with its own bearer-token boundary — see
|
|
`../../pantry-vision/README.md`'s "A real network listener, unlike admin-canvas"
|
|
section for why that one's trust model is different on purpose.
|
|
|
|
## Manual verification still outstanding
|
|
|
|
None of this has been run on hardware. In rough order — on top of everything already
|
|
flagged as unverified in `../../pantry-vision/README.md` (the Grocy API assumptions,
|
|
the vision model's accuracy/latency, which this device's whole usefulness rides on):
|
|
|
|
1. The ISO builds at all.
|
|
2. greetd lands in Sway with no login prompt.
|
|
3. **Touch input as native `wl_touch`** — same open item as `../touch-panel/`.
|
|
4. **Camera enumeration and `--use-fake-ui-for-media-stream`** — whether Chromium
|
|
picks the right `/dev/videoN` when more than one video device is present (a
|
|
built-in laptop-style webcam plus a dedicated one, say), and whether that flag
|
|
actually suppresses the permission prompt on this Chromium/Wayland combination
|
|
specifically (it's a well-known flag for automated testing, not confirmed here
|
|
against a real kiosk session).
|
|
5. `kitchen-display-agent` connects to Mosquitto and the device appears in HA.
|
|
6. End-to-end scan latency: camera capture → `/identify` → a usable proposal on
|
|
screen, with a real vision model on real LLM-host hardware. This is the number
|
|
that decides whether the whole "hold it up, wait, confirm" workflow feels usable
|
|
or like standing at the fridge for 30 seconds waiting on an LLM.
|
|
7. Whether `?api=...&token=...` survives being passed through `--app=` URL escaping
|
|
correctly for every character that can appear in `PANTRY_VISION_URL` — the
|
|
`sed`-based percent-encoding in `configs/sway/pantry-kiosk` handles `/` and `:`
|
|
only, which is enough for a bare `http://host:port` but not for a URL with a path
|
|
or query string of its own.
|
|
8. Idle-blank timeout (20 minutes, `configs/sway/config`) and touch-resume — untested.
|