The_miro
/
SmartestHome
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
SmartestHome/hosts/kitchen-display/README.md

9.4 KiB
Raw Blame History

Sway kitchen display

Phase 17 of docs/project-plan.md. Builds a Debian 12 live ISO for a single-purpose fridge/pantry display: hold a grocery item up to its camera, confirm what the vision model thinks it is and when it'll go bad, put it away — then browse the resulting inventory (soonest-to-expire first) and Grocy's recipes, all touch-driven.

The camera-vision and Grocy read/write logic does not live in this directory — that's ../../pantry-vision/, a container-host service this device's kiosk browser calls directly. This host is deliberately thin: a Sway session, one Chromium kiosk window, and kitchen-display-agent, which only ever controls which screen is showing (Scan / Inventory / Recipes, plus Registration — see below) over MQTT — the same "the agent controls the surface, a separate write API owns the content" split hosts/thin-client's admin canvas already established.

This is also one of the camera-equipped endpoints ../../identity/'s person registration flow can run on — "register me as <name>" by voice (needs ENABLE_VOICE_SATELLITE=true and a real mic on this specific unit) or the touchscreen form either way. See identity's own README for the actual design (multi-phone support, anti-spoofing); this host only adds the mic/kiosk-page plumbing.

A third, simpler sibling of ../thin-client/ and ../touch-panel/, not a variant of either — one workspace, one app, no dock, no music, no general browsing. It reuses the thin client's live-build toolchain and configs/+agent/ split.
Compositor Sway, one workspace, one Chromium kiosk window
Autologin greetd, straight into /usr/local/bin/kiosk-session
Touch input Native Wayland wl_touch, same assumption as ../touch-panel/
Camera Browser-native getUserMedia() inside the Chromium kiosk page — no separate native capture app on this device at all
Content pantry-vision's frontend (../../pantry-vision/frontend/) by default; identity's registration page on demand
Voice wyoming-satellite + openWakeWord — opt-in, off by default, needed for "register me as <name>"
Remote control (HA/LLM) kitchen-display-agent — Show scan / Show inventory / Show recipes / Show registration buttons only
Remote control (human) SSH only — no wayvnc, same scope decision as ../touch-panel/

Hardware

No specific unit has been chosen. What's assumed: a touchscreen (native wl_touch, same caveat as ../touch-panel/README.md) mounted somewhere near the fridge/pantry, with a USB webcam pointed at wherever items get held up — either built into the panel or a separate camera on a short cable/gooseneck mount so the angle can be adjusted independently of the screen. See §1.15 of docs/project-plan.md for the (unverified, non-specific) hardware line item.

Before you build

Deploy pantry-vision/pantry-web first (ENABLE_PANTRY_VISION in hosts/container-host/scripts/setup-container-host.sh) — this image builds and boots fine without it, but every tab will show a connection error until it exists. Then edit the # CONFIGURATION block at the top of scripts/build-kitchen-display-iso.sh:

Variable What to put in it
MQTT_BROKER_HOST LAN IP of the container host running Mosquitto
PANTRY_WEB_URL The pantry-web service's URL, e.g. http://192.168.1.10:8096
PANTRY_VISION_URL The pantry-vision API's URL, e.g. http://192.168.1.10:8095 — read by app.js from the kiosk window's own URL query params (see pantry-kiosk's comment), never baked into the served frontend files
PANTRY_VISION_TOKEN Must exactly match PANTRY_VISION_TOKEN in pantry-vision/pantry-vision.env on the container host — there's no way for this repo to push that value between the two hosts for you
IDENTITY_WEB_URL / IDENTITY_URL / IDENTITY_TOKEN Same shape as the PANTRY_* ones above, for ../../identity/ (ENABLE_IDENTITY in setup-container-host.sh) — IDENTITY_TOKEN must match identity/identity.env's own token
ENABLE_VOICE_SATELLITE false unless this specific unit has a real microphone attached — see ../thin-client/README.md's identical flag for the same per-image reasoning
KIOSK_USERNAME / IMAGE_HOSTNAME / KITCHEN_DISPLAY_NAME Per-device identity
SSH_AUTHORIZED_KEY Optional — password auth is disabled and there is no wayvnc

Build

sudo ./scripts/build-kitchen-display-iso.sh

Same directory-split convention as every other host in this project: configs/ and agent/ are human-edited and git-tracked; live-build/config/includes.chroot/ is generated, gitignored, never hand-edited.

Why the camera has no native capture app

../touch-panel/ and ../thin-client/ both shell out to native processes for their media (mpv, the Spotify Flatpak). This device doesn't need that: Chromium's own getUserMedia() API, called from pantry-vision/frontend/app.js, talks to the webcam directly and hands a captured frame straight to POST /identify as a blob — no intermediate script, no temp file, no separate process to supervise. The one thing this needs from the kiosk launch itself is --use-fake-ui-for-media-stream (configs/sway/pantry-kiosk), which auto-accepts the camera permission prompt that would otherwise sit unanswered on a screen nobody is there to click "Allow" on.

Home Assistant entities

kitchen-display-agent publishes four buttons on connect: Show scan, Show inventory, Show recipes, Show registration — the first three kill and relaunch the kiosk Chromium window at PANTRY_WEB_URL/index.html?...#<tab>; the fourth relaunches it at IDENTITY_WEB_URL/register.html?... instead (a different backend, identity, behind the same "Show X" shape — see kitchen_display_agent/main.py's on_show()). That's the entire MQTT surface; there is no media_player, no workspace select (one workspace), no capture/audio/ remote-desktop entities the way the thin client has. Reading or writing the actual inventory, or registering a person, is a separate path — Home Assistant/the LLM calling pantry-vision's or identity's own published API directly, not through this agent, exactly mirroring the split hosts/thin-client's admin canvas already established between "which screen is showing" (MQTT, this agent) and "what's actually on it" (a dedicated write API). Voice registration in particular never touches this agent at all — see ../../identity/README.md's worked HA intent-script example.

Security boundary

Same shape and reasoning as every other host's agent in this project: kitchen_display_agent/mqtt_discovery.py is the entire inbound MQTT control surface of this machine — LLM tool call → HA service call → MQTT → this agent, no HTTP listener, no websocket, no exposed Sway IPC socket, no VNC. pantry-vision is a separate, deliberately published service with its own bearer-token boundary — see ../../pantry-vision/README.md's "A real network listener, unlike admin-canvas" section for why that one's trust model is different on purpose.

Manual verification still outstanding

None of this has been run on hardware. In rough order — on top of everything already flagged as unverified in ../../pantry-vision/README.md (the Grocy API assumptions, the vision model's accuracy/latency, which this device's whole usefulness rides on):

  1. The ISO builds at all.
  2. greetd lands in Sway with no login prompt.
  3. Touch input as native wl_touch — same open item as ../touch-panel/.
  4. Camera enumeration and --use-fake-ui-for-media-stream — whether Chromium picks the right /dev/videoN when more than one video device is present (a built-in laptop-style webcam plus a dedicated one, say), and whether that flag actually suppresses the permission prompt on this Chromium/Wayland combination specifically (it's a well-known flag for automated testing, not confirmed here against a real kiosk session).
  5. kitchen-display-agent connects to Mosquitto and the device appears in HA.
  6. End-to-end scan latency: camera capture → /identify → a usable proposal on screen, with a real vision model on real LLM-host hardware. This is the number that decides whether the whole "hold it up, wait, confirm" workflow feels usable or like standing at the fridge for 30 seconds waiting on an LLM.
  7. Whether ?api=...&token=... survives being passed through --app= URL escaping correctly for every character that can appear in PANTRY_VISION_URL — the sed-based percent-encoding in configs/sway/pantry-kiosk handles / and : only, which is enough for a bare http://host:port but not for a URL with a path or query string of its own.
  8. Idle-blank timeout (20 minutes, configs/sway/config) and touch-resume — untested.
  9. Switching between pantry-kiosk and identity-kiosk (both configs/sway/) actually replaces the window rather than leaving two Chromium instances up — both scripts kill any chromium process, not just their own profile, specifically because this device has only one workspace; never exercised against a real in-progress scan or registration getting interrupted mid-flow by the other.
  10. Everything in ../../identity/README.md's own verification list, especially TRUSTED_ENTITY_PREFIXES — registration will not find a candidate at all until that's set to match your real HA entity IDs.