SmartestHome/hosts/touch-panel

Sway touch panel

Phase 16 of docs/project-plan.md. Builds a Debian 12 live ISO for a touch-driven wall/counter panel: autologin into Sway, three fixed apps (Spotify, Home Assistant, a general web browser) switched with a finger via an always-on dock, and also controllable by Home Assistant/the local LLM over MQTT.

A different device from ../thin-client/, not a variant of it. The thin client is a couch-distance media station whose primary control surface is HA/MQTT and wayvnc, deliberately with no on-screen bars. This one is touched directly, so it needs the opposite: a persistent, finger-sized on-screen dock, native touchscreen input, and no gesture/remote-control machinery. It reuses the thin client's live-build toolchain and configs/+agent/ directory-split convention, not its live-build tree — same relationship as ../audio-endpoint/'s amd64 build to the thin client's.

What ends up on the image:

Compositor	Sway, workspaces 1:spotify / 2:home / 3:web, always-on touch dock
Autologin	greetd, default_session straight into /usr/local/bin/kiosk-session
Touch input	Native Wayland wl_touch preferred; degrades to a usable single-touch pointer if the hardware only exposes an emulated-mouse HID interface — see below
Music	Full Spotify GUI client (Flathub com.spotify.Client) — not a headless Connect receiver
Home dashboard	Chromium in kiosk mode, --app=$HA_URL, auto-restart if it crashes
Web browser	Firefox, minimal chrome (back/forward/reload/address bar), general browsing
On-screen keyboard	wvkbd, toggled from the dock — see the caveat below, no auto-show
Remote control (HA/LLM)	touchpanel-agent, a systemd service publishing HA MQTT-discovery entities
Remote control (human)	SSH only — no wayvnc on this image, see below

Before you build

No touch-panel hardware has been chosen. Nothing here has been booted on real metal or a real touchscreen. See "Touch input: two tiers" below for how this image handles either way a real touchscreen might show up to Linux.

Edit the # CONFIGURATION block at the top of tools/build-touch-panel-iso.sh:

Variable	What to put in it
MQTT_BROKER_HOST	LAN IP of the container host running Mosquitto (Phase 1)
HA_URL	Home Assistant URL, e.g. http://192.168.1.10:8123 — this is what loads in the Home workspace
KIOSK_USERNAME	Autologin account name (kiosk)
TOUCHPANEL_NAME / IMAGE_HOSTNAME	Per-panel identity; each touch panel needs its own
SSH_AUTHORIZED_KEY	Optional. Password auth is disabled and there is no wayvnc, so without a key the only admin path is the local console.

Build

sudo -E tools/build-touch-panel-iso.sh

Same directory-split convention as the thin client: configs/ and agent/ are the human-edited, git-tracked source of truth; live-build/config/includes.chroot/ is generated — wiped and rebuilt on every run, gitignored, never hand-edited. The only templated tokens are @KIOSK_USERNAME@/@KEYBOARD_LAYOUT@; everything else the in-chroot hooks need comes from /etc/touchpanel-agent/config.env.

Touch input: two tiers

Not every touchscreen tells Linux the truth about being a touchscreen. Some cheap USB controllers implement single-touch by emulating an absolute-position HID mouse (a click at wherever the finger is) instead of a proper HID digitizer. udev reads the device's capability bits to decide ID_INPUT_TOUCHSCREEN vs. ID_INPUT_MOUSE, gets it wrong for exactly this class of hardware, and libinput then hands sway a type:pointer device instead of type:touch — every tap becomes a mouse click, not a touch event. This image handles both cases rather than assuming the good one:

1. Native wl_touch (input type:touch in configs/sway/config) — full multi-touch, works with no extra setup. This is what you get if the hardware reports correctly.
2. Fallback: input type:pointer — active on this image regardless, so a misclassified touchscreen is still usable out of the box: accel_profile flat + pointer_accel 0 keep the absolute tap position from being distorted by mouse- acceleration curves meant for relative motion, map_to_output "*" keeps the coordinate space aligned to the screen, and seat seat0 hide_cursor 1 hides the cursor within ~1ms of it stopping so a tap doesn't leave a visible arrow sitting on screen. The ceiling here is single-touch tap/click-drag — no pinch, no true multi-touch gestures — which is enough for the dock and most kiosk UI, not enough for e.g. pinch-zooming a map in the web browser workspace.

The real fix, once real hardware is known: configs/udev/99-touchscreen-override.rules re-tags the specific device by its USB vendor/product ID so udev (and therefore libinput, and therefore sway) treats it as a genuine touchscreen — restoring full wl_touch semantics instead of settling for the pointer fallback's ceiling. It ships as an inert 0000:0000 template that matches no real hardware; that file's own header comment has the exact udevadm/libinput list-devices commands to find your device's real IDs and confirm the fix took. Fill it in and rebuild — or edit it directly on a booted image and udevadm control --reload-rules && udevadm trigger && swaymsg reload, no rebuild needed to test it.

Quick check on a booted image: libinput list-devices | grep -A2 Capabilities — if it lists touch, you're on tier 1 already and don't need the udev override at all.

Why there's no wayvnc on this image

The thin client's primary control surface is deliberately remote (HA/MQTT + wayvnc) because nobody is standing at a couch-distance media station. This panel is the opposite: it's mounted somewhere and touched directly, so remote view/control isn't the primary need the way it is for the thin client — SSH already covers "something's broken, let me fix it from a shell." If a real need for a remote-view channel shows up once this is on real hardware, adding wayvnc back is a small, isolated change (copy ../thin-client/configs/wayvnc/ and its two hooks) — deliberately left out for now rather than built against a guess.

Home Assistant login

The Chromium kiosk window's profile (~/.config/touchpanel-chromium-ha) is persistent across restarts specifically so logging into Home Assistant once survives a reboot — unlike the thin client's digest/admin kiosk Firefox profiles, which are stateless pages re-copied on every launch. Log in on first boot (tap the dock's Home button if the window doesn't already have focus) and it should stay logged in after that, unverified — depends on HA's own session cookie lifetime and whether Chromium's Wayland profile persistence behaves the way assumed.

Spotify: full GUI, not a headless Connect receiver

../thin-client/ and ../audio-endpoint/ both run spotifyd/librespot — a Spotify Connect receiver with no interface of its own, controlled from the phone app or HA. This panel runs the real, official Spotify Linux client instead (configs/spotify/spotify-launch, Flathub com.spotify.Client), because the whole point of a touch panel is a screen you interact with directly: browsing your library, searching, picking a playlist with a finger. Spotify Premium is required for playback either way — same requirement as the headless receivers.

Login is interactive, on the device, on first launch, and persists in the Flatpak's own data directory (~/.var/app/com.spotify.Client) — nothing here wipes or reseeds it.

touchpanel_agent/mpris_bridge.py bridges Spotify's MPRIS interface (via playerctl) into an HA media_player-shaped set of entities, identical in shape to the thin client's own bridge, just pointed at spotify instead of mpv/spotifyd as the player name — unverified that the Spotify Flatpak's MPRIS bus name is actually spotify (some Flatpak-sandboxed apps register under a flatpak.<app-id>-shaped name instead). If Playback state never updates, check playerctl -l on the booted panel and adjust PLAYER_PRIORITY in mpris_bridge.py.

Touch dock

An always-visible bar reserved at the bottom of the screen (configs/eww/, layer-shell :exclusive true so nothing ever tiles under or draws over it) with four buttons: Spotify, Home, Web, and Keyboard. The first three call swaymsg workspace directly — fixed constants in eww.yuck, nothing from MQTT/HA is ever interpolated into them, same rule as the thin client's now-playing widget. This is the touch-first equivalent of the thin client's HA Screen select entity — both exist, one for a finger, one for the LLM.

Session-scoped (exec_always in configs/sway/config), not part of touchpanel-agent, for the identical reason the thin client's now-playing widget isn't: compositor UI should die with the compositor, not need a restart-independent lifecycle or open a second inbound control channel into the security-sensitive agent.

If eww isn't installed (it's not in Debian bookworm main — 0500-eww-dock.hook.chroot tries apt and otherwise leaves a documented placeholder), there is no on-screen app switcher at all. The panel still boots and shows Home; HA/ LLM-driven app switching via touchpanel-agent's Screen select and Show <app> buttons still works either way, since that's a separate control path.

On-screen keyboard — no auto-show, by design

configs/keyboard/toggle-keyboard toggles wvkbd on and off. There is deliberately no automatic show-on-text-field-focus — that needs the text-input-v3/ virtual-keyboard-v1 Wayland protocols wired all the way through each app via a shell component (the way Phosh does it for squeekboard), which this image doesn't run. Instead: tap the dock's Keyboard button before typing, tap it again to dismiss. This is a known, accepted limitation, not a bug to chase before the panel is otherwise useful — see the verification list below if you want to pursue proper auto-show later.

wvkbd, not squeekboard, was picked specifically because it doesn't assume that shell integration exists; see 0700-onscreen-keyboard.hook.chroot's comment. It is also not in Debian bookworm main, so that hook has the same apt-first/documented- placeholder shape as eww and the thin client's spotifyd/librespot install.

Home Assistant entities

touchpanel-agent publishes MQTT-discovery configs on connect. Under the MQTT integration you should get one device per touch panel with:

• Show Spotify, Show Home, Show web browser (buttons) — switch workspace and, for Spotify/Home, focus the already-running app rather than relaunching it (both are meant to stay open and stateful, unlike the thin client's stateless digest/admin kiosk pages)
• Screen (select) — 1:spotify / 2:home / 3:web
• Playback state (sensor, with track metadata as attributes), Volume (number), and play/pause / next / previous / stop (buttons), bridged from Spotify's own MPRIS interface
• A media_player discovery payload — see the thin client README's identical caveat: stock Home Assistant's MQTT integration has no media_player platform: the button/number/sensor entities above are what give playback control on a plain install; the HACS MQTT Media Player integration is what picks up this one.

Security boundary

Identical shape and reasoning to ../thin-client/README.md's section of the same name: touchpanel_agent/mqtt_discovery.py is the entire inbound control surface of this machine. The LLM never gets a direct network path here — the only chain is LLM tool call → HA service call → MQTT → touchpanel-agent. No HTTP listener, no websocket server, no exposed Sway IPC socket, no VNC (this image has none at all, see above). New control features belong as additional MQTT entities, not as a second listener.

Manual verification still outstanding

None of this has been run on hardware. In rough order:

1. The ISO builds at all (lb build is network-heavy and can fail on mirror hiccups).
2. greetd lands in Sway with no login prompt, on vt1, with getty@tty1 masked.
3. Which tier the real touchscreen lands in — see "Touch input: two tiers" above. Both input type:touch and the input type:pointer fallback are shipped and should both work without a rebuild; what's unverified is whether the fallback's accel_profile flat/pointer_accel 0/hide_cursor combination actually feels good on real absolute-position-emulated-as-mouse hardware (as opposed to just being technically functional), and whether 99-touchscreen-override.rules' ATTRS{idVendor}/ATTRS{idProduct} match actually walks up to the right USB parent on a given controller's specific kernel driver.
4. touchpanel-agent connects to Mosquitto and the device appears in HA.
5. Flathub app ID com.spotify.Client is correct — flagged for verification in 0300-flatpak-spotify.hook.chroot, same as the thin client's Steam Link ID.
6. Spotify login persists in ~/.var/app/com.spotify.Client across a reboot.
7. Spotify's real MPRIS bus name — assumed spotify, not confirmed (see the Spotify section above); playerctl -l on the booted panel is the way to check.
8. Chromium's Wayland app_id for a --app= kiosk window — assumed to start with chromium (configs/sway/config's for_window rule and touchpanel_agent/main.py's focus_criteria both match on it). If focusing the Home workspace's window from the dock/HA stops working, check the real app_id with swaymsg -t get_tree and correct both places.
9. HA login persistence in the Chromium kiosk profile across a reboot (see above).
10. eww and wvkbd are not in Debian bookworm main — both hooks try apt and otherwise leave a documented placeholder (dock absent / keyboard absent respectively); neither has been checked against the real bookworm archive.
11. The Firefox extension IDs in configs/firefox/policies.json — same unverified caveat as the thin client's copy; see that README's identical item.
12. Phase 16's own "reactive path" check: power off the container host and confirm the panel still boots to Spotify + the browser (Home will show a connection error, which is the expected/acceptable failure mode — unlike the thin client, there's no local-media fallback to fall back to here since this device has no local media source at all).
13. Keyboard layout defaults to German (KEYBOARD_LAYOUT="de") — same per-image, no-runtime-override handling as the thin client's identical setting.
14. Dock touch-target sizing (configs/eww/eww.scss, ~84px buttons) — a starting guess, not measured against a real finger on real glass at the real screen DPI chosen once hardware exists.