12 KiB

Raw Blame History

Sway touch panel

Phase 16 of docs/project-plan.md. Builds a Debian 12 live ISO for a touch-driven wall/counter panel: autologin into Sway, three fixed apps (Spotify, Home Assistant, a general web browser) switched with a finger via an always-on dock, and also controllable by Home Assistant/the local LLM over MQTT.

A different device from ../thin-client/, not a variant of it. The thin client is a couch-distance media station whose primary control surface is HA/MQTT and wayvnc, deliberately with no on-screen bars. This one is touched directly, so it needs the opposite: a persistent, finger-sized on-screen dock, native touchscreen input, and no gesture/remote-control machinery. It reuses the thin client's live-build toolchain and configs/+agent/ directory-split convention, not its live-build tree — same relationship as ../audio-endpoint/'s amd64 build to the thin client's.

What ends up on the image:


Compositor	Sway, workspaces `1:spotify` / `2:home` / `3:web`, always-on touch dock
Autologin	greetd, `default_session` straight into `/usr/local/bin/kiosk-session`
Touch input	Native Wayland `wl_touch` (libinput), no pointer-emulation layer
Music	Full Spotify GUI client (Flathub `com.spotify.Client`) — not a headless Connect receiver
Home dashboard	Chromium in kiosk mode, `--app=$HA_URL`, auto-restart if it crashes
Web browser	Firefox, minimal chrome (back/forward/reload/address bar), general browsing
On-screen keyboard	wvkbd, toggled from the dock — see the caveat below, no auto-show
Remote control (HA/LLM)	`touchpanel-agent`, a systemd service publishing HA MQTT-discovery entities
Remote control (human)	SSH only — no wayvnc on this image, see below

Before you build

No touch-panel hardware has been chosen. Nothing here has been booted on real metal or a real touchscreen. Touch-as-wl_touch (as opposed to some panels that only expose an emulated-mouse HID interface) is assumed, not confirmed.

Edit the # CONFIGURATION block at the top of scripts/build-touch-panel-iso.sh:

Variable	What to put in it
`MQTT_BROKER_HOST`	LAN IP of the container host running Mosquitto (Phase 1)
`HA_URL`	Home Assistant URL, e.g. `http://192.168.1.10:8123` — this is what loads in the Home workspace
`KIOSK_USERNAME`	Autologin account name (`kiosk`)
`TOUCHPANEL_NAME` / `IMAGE_HOSTNAME`	Per-panel identity; each touch panel needs its own
`SSH_AUTHORIZED_KEY`	Optional. Password auth is disabled and there is no wayvnc, so without a key the only admin path is the local console.

Build

sudo ./scripts/build-touch-panel-iso.sh

Same directory-split convention as the thin client: configs/ and agent/ are the human-edited, git-tracked source of truth; live-build/config/includes.chroot/ is generated — wiped and rebuilt on every run, gitignored, never hand-edited. The only templated tokens are @KIOSK_USERNAME@/@KEYBOARD_LAYOUT@; everything else the in-chroot hooks need comes from /etc/touchpanel-agent/config.env.

Why there's no wayvnc on this image

The thin client's primary control surface is deliberately remote (HA/MQTT + wayvnc) because nobody is standing at a couch-distance media station. This panel is the opposite: it's mounted somewhere and touched directly, so remote view/control isn't the primary need the way it is for the thin client — SSH already covers "something's broken, let me fix it from a shell." If a real need for a remote-view channel shows up once this is on real hardware, adding wayvnc back is a small, isolated change (copy ../thin-client/configs/wayvnc/ and its two hooks) — deliberately left out for now rather than built against a guess.

The Chromium kiosk window's profile (~/.config/touchpanel-chromium-ha) is persistent across restarts specifically so logging into Home Assistant once survives a reboot — unlike the thin client's digest/admin kiosk Firefox profiles, which are stateless pages re-copied on every launch. Log in on first boot (tap the dock's Home button if the window doesn't already have focus) and it should stay logged in after that, unverified — depends on HA's own session cookie lifetime and whether Chromium's Wayland profile persistence behaves the way assumed.

Spotify: full GUI, not a headless Connect receiver

../thin-client/ and ../audio-endpoint/ both run spotifyd/librespot — a Spotify Connect receiver with no interface of its own, controlled from the phone app or HA. This panel runs the real, official Spotify Linux client instead (configs/spotify/spotify-launch, Flathub com.spotify.Client), because the whole point of a touch panel is a screen you interact with directly: browsing your library, searching, picking a playlist with a finger. Spotify Premium is required for playback either way — same requirement as the headless receivers.

Login is interactive, on the device, on first launch, and persists in the Flatpak's own data directory (~/.var/app/com.spotify.Client) — nothing here wipes or reseeds it.

touchpanel_agent/mpris_bridge.py bridges Spotify's MPRIS interface (via playerctl) into an HA media_player-shaped set of entities, identical in shape to the thin client's own bridge, just pointed at spotify instead of mpv/spotifyd as the player name — unverified that the Spotify Flatpak's MPRIS bus name is actually spotify (some Flatpak-sandboxed apps register under a flatpak.<app-id>-shaped name instead). If Playback state never updates, check playerctl -l on the booted panel and adjust PLAYER_PRIORITY in mpris_bridge.py.

Touch dock

An always-visible bar reserved at the bottom of the screen (configs/eww/, layer-shell :exclusive true so nothing ever tiles under or draws over it) with four buttons: Spotify, Home, Web, and Keyboard. The first three call swaymsg workspace directly — fixed constants in eww.yuck, nothing from MQTT/HA is ever interpolated into them, same rule as the thin client's now-playing widget. This is the touch-first equivalent of the thin client's HA Screen select entity — both exist, one for a finger, one for the LLM.

Session-scoped (exec_always in configs/sway/config), not part of touchpanel-agent, for the identical reason the thin client's now-playing widget isn't: compositor UI should die with the compositor, not need a restart-independent lifecycle or open a second inbound control channel into the security-sensitive agent.

If eww isn't installed (it's not in Debian bookworm main — 0500-eww-dock.hook.chroot tries apt and otherwise leaves a documented placeholder), there is no on-screen app switcher at all. The panel still boots and shows Home; HA/ LLM-driven app switching via touchpanel-agent's Screen select and Show <app> buttons still works either way, since that's a separate control path.

On-screen keyboard — no auto-show, by design

configs/keyboard/toggle-keyboard toggles wvkbd on and off. There is deliberately no automatic show-on-text-field-focus — that needs the text-input-v3/ virtual-keyboard-v1 Wayland protocols wired all the way through each app via a shell component (the way Phosh does it for squeekboard), which this image doesn't run. Instead: tap the dock's Keyboard button before typing, tap it again to dismiss. This is a known, accepted limitation, not a bug to chase before the panel is otherwise useful — see the verification list below if you want to pursue proper auto-show later.

wvkbd, not squeekboard, was picked specifically because it doesn't assume that shell integration exists; see 0700-onscreen-keyboard.hook.chroot's comment. It is also not in Debian bookworm main, so that hook has the same apt-first/documented- placeholder shape as eww and the thin client's spotifyd/librespot install.

Home Assistant entities

touchpanel-agent publishes MQTT-discovery configs on connect. Under the MQTT integration you should get one device per touch panel with:

Show Spotify, Show Home, Show web browser (buttons) — switch workspace and, for Spotify/Home, focus the already-running app rather than relaunching it (both are meant to stay open and stateful, unlike the thin client's stateless digest/admin kiosk pages)
Screen (select) — 1:spotify / 2:home / 3:web
Playback state (sensor, with track metadata as attributes), Volume (number), and play/pause / next / previous / stop (buttons), bridged from Spotify's own MPRIS interface
A media_player discovery payload — see the thin client README's identical caveat: stock Home Assistant's MQTT integration has no media_player platform: the button/number/sensor entities above are what give playback control on a plain install; the HACS MQTT Media Player integration is what picks up this one.

Security boundary

Identical shape and reasoning to ../thin-client/README.md's section of the same name: touchpanel_agent/mqtt_discovery.py is the entire inbound control surface of this machine. The LLM never gets a direct network path here — the only chain is LLM tool call → HA service call → MQTT → touchpanel-agent. No HTTP listener, no websocket server, no exposed Sway IPC socket, no VNC (this image has none at all, see above). New control features belong as additional MQTT entities, not as a second listener.

Manual verification still outstanding

None of this has been run on hardware. In rough order:

The ISO builds at all (lb build is network-heavy and can fail on mirror hiccups).
greetd lands in Sway with no login prompt, on vt1, with getty@tty1 masked.
Touch input actually arrives as wl_touch, not an emulated pointer — the whole design (native touch in Chromium/Firefox/Spotify, no gesture-translation layer) assumes a real Wayland touchscreen driver. If it only shows up as a mouse, tapping will still mostly work by accident (a single point maps to a click) but pinch/ multi-touch in Chromium or Spotify won't.
touchpanel-agent connects to Mosquitto and the device appears in HA.
Flathub app ID com.spotify.Client is correct — flagged for verification in 0300-flatpak-spotify.hook.chroot, same as the thin client's Steam Link ID.
Spotify login persists in ~/.var/app/com.spotify.Client across a reboot.
Spotify's real MPRIS bus name — assumed spotify, not confirmed (see the Spotify section above); playerctl -l on the booted panel is the way to check.
Chromium's Wayland app_id for a --app= kiosk window — assumed to start with chromium (configs/sway/config's for_window rule and touchpanel_agent/main.py's focus_criteria both match on it). If focusing the Home workspace's window from the dock/HA stops working, check the real app_id with swaymsg -t get_tree and correct both places.
HA login persistence in the Chromium kiosk profile across a reboot (see above).
eww and wvkbd are not in Debian bookworm main — both hooks try apt and otherwise leave a documented placeholder (dock absent / keyboard absent respectively); neither has been checked against the real bookworm archive.
The Firefox extension IDs in configs/firefox/policies.json — same unverified caveat as the thin client's copy; see that README's identical item.
Phase 16's own "reactive path" check: power off the container host and confirm the panel still boots to Spotify + the browser (Home will show a connection error, which is the expected/acceptable failure mode — unlike the thin client, there's no local-media fallback to fall back to here since this device has no local media source at all).
Keyboard layout defaults to German (KEYBOARD_LAYOUT="de") — same per-image, no-runtime-override handling as the thin client's identical setting.
Dock touch-target sizing (configs/eww/eww.scss, ~84px buttons) — a starting guess, not measured against a real finger on real glass at the real screen DPI chosen once hardware exists.

12 KiB Raw Blame History