18 KiB

Raw Blame History

Sway touch panel

Phase 16 of docs/project-plan.md. Builds a Debian 12 live ISO for a touch-driven wall/counter panel: autologin into Sway, three fixed apps (Spotify, Home Assistant, a general web browser) — four where the panel asks for Discord — switched with a finger via an always-on dock, and also controllable by Home Assistant/the local LLM over MQTT.

A different device from ../thin-client/, not a variant of it. The thin client is a couch-distance media station whose primary control surface is HA/MQTT and wayvnc, deliberately with no on-screen bars. This one is touched directly, so it needs the opposite: a persistent, finger-sized on-screen dock, native touchscreen input, and no gesture/remote-control machinery. It reuses the thin client's live-build toolchain and configs/+agent/ directory-split convention, not its live-build tree — same relationship as ../audio-endpoint/'s amd64 build to the thin client's.

What ends up on the image:


Compositor	Sway, workspaces `1:spotify` / `2:home` / `3:web` (+ `4:discord` where enabled), always-on touch dock
Autologin	greetd, `default_session` straight into `/usr/local/bin/kiosk-session`
Touch input	Native Wayland `wl_touch` preferred; degrades to a usable single-touch pointer if the hardware only exposes an emulated-mouse HID interface — see below
Music	Full Spotify GUI client (Flathub `com.spotify.Client`) — not a headless Connect receiver
Chat / voice	Discord (Flathub `com.discordapp.Discord`), only where `enable_discord` is set — see below
Home dashboard	Chromium in kiosk mode, `--app=$HA_URL`, auto-restart if it crashes
Web browser	Firefox, minimal chrome (back/forward/reload/address bar), general browsing
On-screen keyboard	wvkbd, toggled from the dock — see the caveat below, no auto-show
Remote control (HA/LLM)	`touchpanel-agent`, a systemd service publishing HA MQTT-discovery entities
Remote control (human)	SSH only — no wayvnc on this image, see below

Before you build

No touch-panel hardware has been chosen. Nothing here has been booted on real metal or a real touchscreen. See "Touch input: two tiers" below for how this image handles either way a real touchscreen might show up to Linux.

Edit the # CONFIGURATION block at the top of tools/build-touch-panel-iso.sh:

Variable	What to put in it
`MQTT_BROKER_HOST`	LAN IP of the container host running Mosquitto (Phase 1)
`HA_URL`	Home Assistant URL, e.g. `http://192.168.1.10:8123` — this is what loads in the Home workspace
`KIOSK_USERNAME`	Autologin account name (`kiosk`)
`TOUCHPANEL_NAME` / `IMAGE_HOSTNAME`	Per-panel identity; each touch panel needs its own
`SSH_AUTHORIZED_KEY`	Optional. Password auth is disabled and there is no wayvnc, so without a key the only admin path is the local console.

Build

sudo -E tools/build-touch-panel-iso.sh

Same directory-split convention as the thin client: configs/ and agent/ are the human-edited, git-tracked source of truth; live-build/config/includes.chroot/ is generated — wiped and rebuilt on every run, gitignored, never hand-edited. The only templated tokens are @KIOSK_USERNAME@/@KEYBOARD_LAYOUT@; everything else the in-chroot hooks need comes from /etc/touchpanel-agent/config.env.

Touch input: two tiers

Not every touchscreen tells Linux the truth about being a touchscreen. Some cheap USB controllers implement single-touch by emulating an absolute-position HID mouse (a click at wherever the finger is) instead of a proper HID digitizer. udev reads the device's capability bits to decide ID_INPUT_TOUCHSCREEN vs. ID_INPUT_MOUSE, gets it wrong for exactly this class of hardware, and libinput then hands sway a type:pointer device instead of type:touch — every tap becomes a mouse click, not a touch event. This image handles both cases rather than assuming the good one:

Native wl_touch (input type:touch in configs/sway/config) — full multi-touch, works with no extra setup. This is what you get if the hardware reports correctly.
Fallback: input type:pointer — active on this image regardless, so a misclassified touchscreen is still usable out of the box: accel_profile flat + pointer_accel 0 keep the absolute tap position from being distorted by mouse- acceleration curves meant for relative motion, map_to_output "*" keeps the coordinate space aligned to the screen, and seat seat0 hide_cursor 1 hides the cursor within ~1ms of it stopping so a tap doesn't leave a visible arrow sitting on screen. The ceiling here is single-touch tap/click-drag — no pinch, no true multi-touch gestures — which is enough for the dock and most kiosk UI, not enough for e.g. pinch-zooming a map in the web browser workspace.

The real fix, once real hardware is known: configs/udev/99-touchscreen-override.rules re-tags the specific device by its USB vendor/product ID so udev (and therefore libinput, and therefore sway) treats it as a genuine touchscreen — restoring full wl_touch semantics instead of settling for the pointer fallback's ceiling. It ships as an inert 0000:0000 template that matches no real hardware; that file's own header comment has the exact udevadm/libinput list-devices commands to find your device's real IDs and confirm the fix took. Fill it in and rebuild — or edit it directly on a booted image and udevadm control --reload-rules && udevadm trigger && swaymsg reload, no rebuild needed to test it.

Quick check on a booted image: libinput list-devices | grep -A2 Capabilities — if it lists touch, you're on tier 1 already and don't need the udev override at all.

Why there's no wayvnc on this image

The thin client's primary control surface is deliberately remote (HA/MQTT + wayvnc) because nobody is standing at a couch-distance media station. This panel is the opposite: it's mounted somewhere and touched directly, so remote view/control isn't the primary need the way it is for the thin client — SSH already covers "something's broken, let me fix it from a shell." If a real need for a remote-view channel shows up once this is on real hardware, adding wayvnc back is a small, isolated change (copy ../thin-client/configs/wayvnc/ and its two hooks) — deliberately left out for now rather than built against a guess.

The Chromium kiosk window's profile (~/.config/touchpanel-chromium-ha) is persistent across restarts specifically so logging into Home Assistant once survives a reboot — unlike the thin client's digest/admin kiosk Firefox profiles, which are stateless pages re-copied on every launch. Log in on first boot (tap the dock's Home button if the window doesn't already have focus) and it should stay logged in after that, unverified — depends on HA's own session cookie lifetime and whether Chromium's Wayland profile persistence behaves the way assumed.

Spotify: full GUI, not a headless Connect receiver

../thin-client/ and ../audio-endpoint/ both run spotifyd/librespot — a Spotify Connect receiver with no interface of its own, controlled from the phone app or HA. This panel runs the real, official Spotify Linux client instead (configs/spotify/spotify-launch, Flathub com.spotify.Client), because the whole point of a touch panel is a screen you interact with directly: browsing your library, searching, picking a playlist with a finger. Spotify Premium is required for playback either way — same requirement as the headless receivers.

Login is interactive, on the device, on first launch, and persists in the Flatpak's own data directory (~/.var/app/com.spotify.Client) — nothing here wipes or reseeds it.

touchpanel_agent/mpris_bridge.py bridges Spotify's MPRIS interface (via playerctl) into an HA media_player-shaped set of entities, identical in shape to the thin client's own bridge, just pointed at spotify instead of mpv/spotifyd as the player name — unverified that the Spotify Flatpak's MPRIS bus name is actually spotify (some Flatpak-sandboxed apps register under a flatpak.<app-id>-shaped name instead). If Playback state never updates, check playerctl -l on the booted panel and adjust PLAYER_PRIORITY in mpris_bridge.py.

Touch dock

An always-visible bar reserved at the bottom of the screen (configs/eww/, layer-shell :exclusive true so nothing ever tiles under or draws over it) with four buttons — five on a panel with Discord: Spotify, Home, Web, (Discord), and Keyboard. The workspace ones call swaymsg workspace directly — fixed constants in eww.yuck, nothing from MQTT/HA is ever interpolated into them, same rule as the thin client's now-playing widget. This is the touch-first equivalent of the thin client's HA Screen select entity — both exist, one for a finger, one for the LLM.

Session-scoped (exec_always in configs/sway/config), not part of touchpanel-agent, for the identical reason the thin client's now-playing widget isn't: compositor UI should die with the compositor, not need a restart-independent lifecycle or open a second inbound control channel into the security-sensitive agent.

If eww isn't installed (it's not in Debian bookworm main — 0500-eww-dock.hook.chroot tries apt and otherwise leaves a documented placeholder), there is no on-screen app switcher at all. The panel still boots and shows Home; HA/ LLM-driven app switching via touchpanel-agent's Screen select and Show <app> buttons still works either way, since that's a separate control path.

Discord — a per-panel fourth app

Off by default; turned on per panel with enable_discord on that kiosk in CoreSystemConfig.json. tools/validate-config.py rejects it on any other image type — no other build installs the Flatpak or has a workspace for it, so the flag would otherwise be silently ignored on a panel that boots looking fine.

What it is for. The Loggia panel is the balcony one, and the point is that a voice call survives stepping outside for a cigarette: you move the call onto this machine on the way out and back onto the desktop on the way in. That is a switch device action, which is why Discord starts at session boot and stays logged in rather than launching on demand — an app that needs fifteen seconds and a login is not one you use on the way past. It is also why the HA Show Discord button and the dock button both resolve to focus-the-running-app, exactly like Home and Spotify.

Three things follow from it being an Electron/XWayland app rather than a native Wayland one:

Sway matches it on class="discord", not app_id — an app_id rule would silently never fire.
Touch arrives as emulated pointer events, so it lands in the same tier-2 behaviour described under Touch input below: tap and drag work, two-finger scrolling in the message list may not. Fine for join / mute / leave, which is the job.
It deliberately does not get inhibit_idle. A call would otherwise hold the panel's display on for an hour in an empty room; the screen blanks and the call keeps running, because sway's idle timeout powers off the output, it does not suspend the machine or the audio stream.

Which audio device the call actually uses is Discord's own setting, on-device, once — this image does not manage it, and touchpanel-agent has no entity for it.

This panel as a remote microphone

The Loggia panel doubles as a microphone for a desktop in another room (../../mic-follow/): that machine opens an SSH session and runs pw-record here, so somebody on the balcony stays in a voice call running on their PC. Nothing listens for it — it is an ordinary SSH command over the admin path this image already has — and audio is captured only while that session is open.

Because it is a microphone in a shared flat that somebody elsewhere can open, the dock carries a red MIC LIVE badge, driven by mic-in-use (polled every 2s). It lights whenever anything is capturing this panel's microphone, not only when mic-follow is: an indicator whose job is to be believed should not depend on knowing who asked. Monitor streams — something recording what the panel is playing — do not light it.

The image ships pipewire-bin (for pw-record) and pulseaudio-utils (for pactl) for this. Neither is needed by anything else on the panel.

On-screen keyboard — no auto-show, by design

configs/keyboard/toggle-keyboard toggles wvkbd on and off. There is deliberately no automatic show-on-text-field-focus — that needs the text-input-v3/ virtual-keyboard-v1 Wayland protocols wired all the way through each app via a shell component (the way Phosh does it for squeekboard), which this image doesn't run. Instead: tap the dock's Keyboard button before typing, tap it again to dismiss. This is a known, accepted limitation, not a bug to chase before the panel is otherwise useful — see the verification list below if you want to pursue proper auto-show later.

wvkbd, not squeekboard, was picked specifically because it doesn't assume that shell integration exists; see 0700-onscreen-keyboard.hook.chroot's comment. It is also not in Debian bookworm main, so that hook has the same apt-first/documented- placeholder shape as eww and the thin client's spotifyd/librespot install.

Home Assistant entities

touchpanel-agent publishes MQTT-discovery configs on connect. Under the MQTT integration you should get one device per touch panel with:

Show Spotify, Show Home, Show web browser, and on a Discord panel Show Discord (buttons) — switch workspace and, for Spotify/Home, focus the already-running app rather than relaunching it (both are meant to stay open and stateful, unlike the thin client's stateless digest/admin kiosk pages)
Screen (select) — 1:spotify / 2:home / 3:web, plus 4:discord on a panel that has it. The options are derived from the app table rather than fixed, so a dropdown never offers a workspace this image has no app for
Playback state (sensor, with track metadata as attributes), Volume (number), and play/pause / next / previous / stop (buttons), bridged from Spotify's own MPRIS interface
A media_player discovery payload — see the thin client README's identical caveat: stock Home Assistant's MQTT integration has no media_player platform: the button/number/sensor entities above are what give playback control on a plain install; the HACS MQTT Media Player integration is what picks up this one.

Security boundary

Identical shape and reasoning to ../thin-client/README.md's section of the same name: touchpanel_agent/mqtt_discovery.py is the entire inbound control surface of this machine. The LLM never gets a direct network path here — the only chain is LLM tool call → HA service call → MQTT → touchpanel-agent. No HTTP listener, no websocket server, no exposed Sway IPC socket, no VNC (this image has none at all, see above). New control features belong as additional MQTT entities, not as a second listener.

Manual verification still outstanding

None of this has been run on hardware. In rough order:

The ISO builds at all (lb build is network-heavy and can fail on mirror hiccups).
greetd lands in Sway with no login prompt, on vt1, with getty@tty1 masked.
Which tier the real touchscreen lands in — see "Touch input: two tiers" above. Both input type:touch and the input type:pointer fallback are shipped and should both work without a rebuild; what's unverified is whether the fallback's accel_profile flat/pointer_accel 0/hide_cursor combination actually feels good on real absolute-position-emulated-as-mouse hardware (as opposed to just being technically functional), and whether 99-touchscreen-override.rules' ATTRS{idVendor}/ATTRS{idProduct} match actually walks up to the right USB parent on a given controller's specific kernel driver.
touchpanel-agent connects to Mosquitto and the device appears in HA.
Flathub app ID com.spotify.Client is correct — flagged for verification in 0300-flatpak-spotify.hook.chroot, same as the thin client's Steam Link ID. 5b. Flathub app ID com.discordapp.Discord is correct, on a Discord panel — flagged in 0350-flatpak-discord.hook.chroot for the same reason. 5c. Discord's real X11 class — assumed discord, which is what the sway fullscreen rule and the agent's focus criteria both match on. swaymsg -t get_tree on the booted panel is the way to check; if it is capitalised or suffixed, the app opens in a window that never goes fullscreen and Show Discord switches workspace without focusing anything.
Spotify login persists in ~/.var/app/com.spotify.Client across a reboot.
Spotify's real MPRIS bus name — assumed spotify, not confirmed (see the Spotify section above); playerctl -l on the booted panel is the way to check.
Chromium's Wayland app_id for a --app= kiosk window — assumed to start with chromium (configs/sway/config's for_window rule and touchpanel_agent/main.py's focus_criteria both match on it). If focusing the Home workspace's window from the dock/HA stops working, check the real app_id with swaymsg -t get_tree and correct both places.
HA login persistence in the Chromium kiosk profile across a reboot (see above).
eww and wvkbd are not in Debian bookworm main — both hooks try apt and otherwise leave a documented placeholder (dock absent / keyboard absent respectively); neither has been checked against the real bookworm archive.
The Firefox extension IDs in configs/firefox/policies.json — same unverified caveat as the thin client's copy; see that README's identical item.
Phase 16's own "reactive path" check: power off the container host and confirm the panel still boots to Spotify + the browser (Home will show a connection error, which is the expected/acceptable failure mode — unlike the thin client, there's no local-media fallback to fall back to here since this device has no local media source at all).
Keyboard layout defaults to German (KEYBOARD_LAYOUT="de") — same per-image, no-runtime-override handling as the thin client's identical setting.
Dock touch-target sizing (configs/eww/eww.scss, ~84px buttons) — a starting guess, not measured against a real finger on real glass at the real screen DPI chosen once hardware exists.

18 KiB Raw Blame History