305 lines
18 KiB
Markdown
305 lines
18 KiB
Markdown
# Sway touch panel
|
|
|
|
Phase 16 of `docs/project-plan.md`. Builds a Debian 12 live ISO for a touch-driven
|
|
wall/counter panel: autologin into Sway, three fixed apps (Spotify, Home Assistant,
|
|
a general web browser) — four where the panel asks for **Discord** — switched with a
|
|
finger via an always-on dock, and also controllable by Home Assistant/the local LLM
|
|
over MQTT.
|
|
|
|
**A different device from `../thin-client/`**, not a variant of it. The thin client is
|
|
a couch-distance media station whose primary control surface is HA/MQTT and wayvnc,
|
|
deliberately with no on-screen bars. This one is touched directly, so it needs the
|
|
opposite: a persistent, finger-sized on-screen dock, native touchscreen input, and no
|
|
gesture/remote-control machinery. It reuses the thin client's live-build toolchain and
|
|
`configs/`+`agent/` directory-split convention, not its live-build tree — same
|
|
relationship as `../audio-endpoint/`'s amd64 build to the thin client's.
|
|
|
|
What ends up on the image:
|
|
|
|
| | |
|
|
|---|---|
|
|
| Compositor | Sway, workspaces `1:spotify` / `2:home` / `3:web` (+ `4:discord` where enabled), always-on touch dock |
|
|
| Autologin | greetd, `default_session` straight into `/usr/local/bin/kiosk-session` |
|
|
| Touch input | Native Wayland `wl_touch` preferred; degrades to a usable single-touch pointer if the hardware only exposes an emulated-mouse HID interface — see below |
|
|
| Music | Full Spotify GUI client (Flathub `com.spotify.Client`) — not a headless Connect receiver |
|
|
| Chat / voice | Discord (Flathub `com.discordapp.Discord`), **only where `enable_discord` is set** — see below |
|
|
| Home dashboard | Chromium in kiosk mode, `--app=$HA_URL`, auto-restart if it crashes |
|
|
| Web browser | Firefox, minimal chrome (back/forward/reload/address bar), general browsing |
|
|
| On-screen keyboard | wvkbd, toggled from the dock — see the caveat below, no auto-show |
|
|
| Remote control (HA/LLM) | `touchpanel-agent`, a systemd service publishing HA MQTT-discovery entities |
|
|
| Remote control (human) | SSH only — **no wayvnc on this image**, see below |
|
|
|
|
## Before you build
|
|
|
|
> **No touch-panel hardware has been chosen.** Nothing here has been booted on real
|
|
> metal or a real touchscreen. See "Touch input: two tiers" below for how this image
|
|
> handles either way a real touchscreen might show up to Linux.
|
|
|
|
Edit the `# CONFIGURATION` block at the top of
|
|
[`tools/build-touch-panel-iso.sh`](../../tools/build-touch-panel-iso.sh):
|
|
|
|
| Variable | What to put in it |
|
|
|---|---|
|
|
| `MQTT_BROKER_HOST` | LAN IP of the container host running Mosquitto (Phase 1) |
|
|
| `HA_URL` | Home Assistant URL, e.g. `http://192.168.1.10:8123` — this is what loads in the Home workspace |
|
|
| `KIOSK_USERNAME` | Autologin account name (`kiosk`) |
|
|
| `TOUCHPANEL_NAME` / `IMAGE_HOSTNAME` | Per-panel identity; each touch panel needs its own |
|
|
| `SSH_AUTHORIZED_KEY` | Optional. Password auth is disabled and there is no wayvnc, so without a key the only admin path is the local console. |
|
|
|
|
## Build
|
|
|
|
```sh
|
|
sudo -E tools/build-touch-panel-iso.sh
|
|
```
|
|
|
|
Same directory-split convention as the thin client: `configs/` and `agent/` are the
|
|
human-edited, git-tracked source of truth; `live-build/config/includes.chroot/` is
|
|
**generated** — wiped and rebuilt on every run, gitignored, never hand-edited. The only
|
|
templated tokens are `@KIOSK_USERNAME@`/`@KEYBOARD_LAYOUT@`; everything else the
|
|
in-chroot hooks need comes from `/etc/touchpanel-agent/config.env`.
|
|
|
|
## Touch input: two tiers
|
|
|
|
Not every touchscreen tells Linux the truth about being a touchscreen. Some cheap USB
|
|
controllers implement single-touch by emulating an absolute-position HID **mouse**
|
|
(a click at wherever the finger is) instead of a proper HID **digitizer**. udev reads
|
|
the device's capability bits to decide `ID_INPUT_TOUCHSCREEN` vs. `ID_INPUT_MOUSE`,
|
|
gets it wrong for exactly this class of hardware, and libinput then hands sway a
|
|
`type:pointer` device instead of `type:touch` — every tap becomes a mouse click, not
|
|
a touch event. This image handles both cases rather than assuming the good one:
|
|
|
|
1. **Native `wl_touch`** (`input type:touch` in `configs/sway/config`) — full
|
|
multi-touch, works with no extra setup. This is what you get if the hardware
|
|
reports correctly.
|
|
2. **Fallback: `input type:pointer`** — active on this image regardless, so a
|
|
misclassified touchscreen is still usable out of the box: `accel_profile flat` +
|
|
`pointer_accel 0` keep the absolute tap position from being distorted by mouse-
|
|
acceleration curves meant for relative motion, `map_to_output "*"` keeps the
|
|
coordinate space aligned to the screen, and `seat seat0 hide_cursor 1` hides the
|
|
cursor within ~1ms of it stopping so a tap doesn't leave a visible arrow sitting on
|
|
screen. The ceiling here is single-touch tap/click-drag — no pinch, no true
|
|
multi-touch gestures — which is enough for the dock and most kiosk UI, not enough
|
|
for e.g. pinch-zooming a map in the web browser workspace.
|
|
|
|
**The real fix, once real hardware is known**:
|
|
`configs/udev/99-touchscreen-override.rules` re-tags the specific device by its USB
|
|
vendor/product ID so udev (and therefore libinput, and therefore sway) treats it as a
|
|
genuine touchscreen — restoring full `wl_touch` semantics instead of settling for the
|
|
pointer fallback's ceiling. It ships as an inert `0000:0000` template that matches no
|
|
real hardware; that file's own header comment has the exact `udevadm`/`libinput
|
|
list-devices` commands to find your device's real IDs and confirm the fix took. Fill
|
|
it in and rebuild — or edit it directly on a booted image and
|
|
`udevadm control --reload-rules && udevadm trigger && swaymsg reload`, no rebuild
|
|
needed to test it.
|
|
|
|
Quick check on a booted image: `libinput list-devices | grep -A2 Capabilities` — if it
|
|
lists `touch`, you're on tier 1 already and don't need the udev override at all.
|
|
|
|
## Why there's no wayvnc on this image
|
|
|
|
The thin client's primary control surface is deliberately remote (HA/MQTT + wayvnc)
|
|
because nobody is standing at a couch-distance media station. This panel is the
|
|
opposite: it's mounted somewhere and touched directly, so remote *view/control* isn't
|
|
the primary need the way it is for the thin client — SSH already covers "something's
|
|
broken, let me fix it from a shell." If a real need for a remote-view channel shows up
|
|
once this is on real hardware, adding wayvnc back is a small, isolated change (copy
|
|
`../thin-client/configs/wayvnc/` and its two hooks) — deliberately left out for now
|
|
rather than built against a guess.
|
|
|
|
## Home Assistant login
|
|
|
|
The Chromium kiosk window's profile (`~/.config/touchpanel-chromium-ha`) is persistent
|
|
across restarts specifically so logging into Home Assistant once survives a reboot —
|
|
unlike the thin client's digest/admin kiosk Firefox profiles, which are stateless
|
|
pages re-copied on every launch. Log in on first boot (tap the dock's Home button if
|
|
the window doesn't already have focus) and it should stay logged in after that,
|
|
**unverified** — depends on HA's own session cookie lifetime and whether Chromium's
|
|
Wayland profile persistence behaves the way assumed.
|
|
|
|
## Spotify: full GUI, not a headless Connect receiver
|
|
|
|
`../thin-client/` and `../audio-endpoint/` both run `spotifyd`/`librespot` — a
|
|
Spotify Connect *receiver* with no interface of its own, controlled from the phone
|
|
app or HA. This panel runs the real, official Spotify Linux client instead
|
|
(`configs/spotify/spotify-launch`, Flathub `com.spotify.Client`), because the whole
|
|
point of a touch panel is a screen you interact with directly: browsing your library,
|
|
searching, picking a playlist with a finger. **Spotify Premium is required for
|
|
playback either way** — same requirement as the headless receivers.
|
|
|
|
Login is interactive, on the device, on first launch, and persists in the Flatpak's
|
|
own data directory (`~/.var/app/com.spotify.Client`) — nothing here wipes or reseeds
|
|
it.
|
|
|
|
`touchpanel_agent/mpris_bridge.py` bridges Spotify's MPRIS interface (via
|
|
`playerctl`) into an HA `media_player`-shaped set of entities, identical in shape to
|
|
the thin client's own bridge, just pointed at `spotify` instead of `mpv`/`spotifyd` as
|
|
the player name — **unverified that the Spotify Flatpak's MPRIS bus name is actually
|
|
`spotify`** (some Flatpak-sandboxed apps register under a `flatpak.<app-id>`-shaped
|
|
name instead). If Playback state never updates, check `playerctl -l` on the booted
|
|
panel and adjust `PLAYER_PRIORITY` in `mpris_bridge.py`.
|
|
|
|
## Touch dock
|
|
|
|
An always-visible bar reserved at the bottom of the screen (`configs/eww/`, layer-shell
|
|
`:exclusive true` so nothing ever tiles under or draws over it) with four buttons —
|
|
five on a panel with Discord: **Spotify**, **Home**, **Web**, (**Discord**), and
|
|
**Keyboard**. The workspace ones call `swaymsg
|
|
workspace` directly — fixed constants in `eww.yuck`, nothing from MQTT/HA is ever
|
|
interpolated into them, same rule as the thin client's now-playing widget. This is the
|
|
touch-first equivalent of the thin client's HA **Screen** select entity — both exist,
|
|
one for a finger, one for the LLM.
|
|
|
|
Session-scoped (`exec_always` in `configs/sway/config`), not part of
|
|
`touchpanel-agent`, for the identical reason the thin client's now-playing widget
|
|
isn't: compositor UI should die with the compositor, not need a restart-independent
|
|
lifecycle or open a second inbound control channel into the security-sensitive agent.
|
|
|
|
**If `eww` isn't installed** (it's not in Debian bookworm main —
|
|
`0500-eww-dock.hook.chroot` tries apt and otherwise leaves a documented placeholder),
|
|
there is no on-screen app switcher at all. The panel still boots and shows Home; HA/
|
|
LLM-driven app switching via `touchpanel-agent`'s **Screen** select and **Show
|
|
\<app\>** buttons still works either way, since that's a separate control path.
|
|
|
|
## Discord — a per-panel fourth app
|
|
|
|
Off by default; turned on per panel with `enable_discord` on that kiosk in
|
|
`CoreSystemConfig.json`. `tools/validate-config.py` rejects it on any other image
|
|
type — no other build installs the Flatpak or has a workspace for it, so the flag
|
|
would otherwise be silently ignored on a panel that boots looking fine.
|
|
|
|
**What it is for.** The Loggia panel is the balcony one, and the point is that a voice
|
|
call survives stepping outside for a cigarette: you move the call onto this machine on
|
|
the way out and back onto the desktop on the way in. That is a *switch device* action,
|
|
which is why Discord starts at session boot and stays logged in rather than launching
|
|
on demand — an app that needs fifteen seconds and a login is not one you use on the
|
|
way past. It is also why the HA **Show Discord** button and the dock button both
|
|
resolve to focus-the-running-app, exactly like Home and Spotify.
|
|
|
|
Three things follow from it being an Electron/XWayland app rather than a native
|
|
Wayland one:
|
|
|
|
- Sway matches it on `class="discord"`, **not** `app_id` — an `app_id` rule would
|
|
silently never fire.
|
|
- Touch arrives as emulated pointer events, so it lands in the same tier-2 behaviour
|
|
described under *Touch input* below: tap and drag work, two-finger scrolling in the
|
|
message list may not. Fine for join / mute / leave, which is the job.
|
|
- It deliberately does **not** get `inhibit_idle`. A call would otherwise hold the
|
|
panel's display on for an hour in an empty room; the screen blanks and the call
|
|
keeps running, because sway's idle timeout powers off the output, it does not
|
|
suspend the machine or the audio stream.
|
|
|
|
Which audio device the call actually uses is Discord's own setting, on-device, once —
|
|
this image does not manage it, and `touchpanel-agent` has no entity for it.
|
|
|
|
## This panel as a remote microphone
|
|
|
|
The Loggia panel doubles as a microphone for a desktop in another room
|
|
(`../../mic-follow/`): that machine opens an SSH session and runs `pw-record` here, so
|
|
somebody on the balcony stays in a voice call running on their PC. Nothing listens for
|
|
it — it is an ordinary SSH command over the admin path this image already has — and
|
|
audio is captured only while that session is open.
|
|
|
|
Because it is a microphone in a shared flat that somebody elsewhere can open, the dock
|
|
carries a red **MIC LIVE** badge, driven by `mic-in-use` (polled every 2s). It lights
|
|
whenever *anything* is capturing this panel's microphone, not only when mic-follow is:
|
|
an indicator whose job is to be believed should not depend on knowing who asked.
|
|
Monitor streams — something recording what the panel is *playing* — do not light it.
|
|
|
|
The image ships `pipewire-bin` (for `pw-record`) and `pulseaudio-utils` (for `pactl`)
|
|
for this. Neither is needed by anything else on the panel.
|
|
|
|
## On-screen keyboard — no auto-show, by design
|
|
|
|
`configs/keyboard/toggle-keyboard` toggles `wvkbd` on and off. There is deliberately
|
|
**no automatic show-on-text-field-focus** — that needs the `text-input-v3`/
|
|
`virtual-keyboard-v1` Wayland protocols wired all the way through each app via a shell
|
|
component (the way Phosh does it for squeekboard), which this image doesn't run.
|
|
Instead: tap the dock's Keyboard button before typing, tap it again to dismiss. This
|
|
is a known, accepted limitation, not a bug to chase before the panel is otherwise
|
|
useful — see the verification list below if you want to pursue proper auto-show later.
|
|
|
|
`wvkbd`, not `squeekboard`, was picked specifically because it doesn't assume that
|
|
shell integration exists; see `0700-onscreen-keyboard.hook.chroot`'s comment. It is
|
|
also not in Debian bookworm main, so that hook has the same apt-first/documented-
|
|
placeholder shape as `eww` and the thin client's `spotifyd`/`librespot` install.
|
|
|
|
## Home Assistant entities
|
|
|
|
`touchpanel-agent` publishes MQTT-discovery configs on connect. Under the MQTT
|
|
integration you should get one device per touch panel with:
|
|
|
|
- **Show Spotify**, **Show Home**, **Show web browser**, and on a Discord panel
|
|
**Show Discord** (buttons) — switch workspace
|
|
and, for Spotify/Home, focus the already-running app rather than relaunching it
|
|
(both are meant to stay open and stateful, unlike the thin client's stateless
|
|
digest/admin kiosk pages)
|
|
- **Screen** (select) — `1:spotify` / `2:home` / `3:web`, plus `4:discord` on a panel
|
|
that has it. The options are derived from the app table rather than fixed, so a
|
|
dropdown never offers a workspace this image has no app for
|
|
- **Playback state** (sensor, with track metadata as attributes), **Volume**
|
|
(number), and play/pause / next / previous / stop (buttons), bridged from
|
|
Spotify's own MPRIS interface
|
|
- A **media_player** discovery payload — see the thin client README's identical
|
|
caveat: stock Home Assistant's MQTT integration has no `media_player` platform: the
|
|
button/number/sensor entities above are what give playback control on a plain
|
|
install; the HACS *MQTT Media Player* integration is what picks up this one.
|
|
|
|
### Security boundary
|
|
|
|
Identical shape and reasoning to `../thin-client/README.md`'s section of the same
|
|
name: `touchpanel_agent/mqtt_discovery.py` is the **entire** inbound control surface
|
|
of this machine. The LLM never gets a direct network path here — the only chain is
|
|
*LLM tool call → HA service call → MQTT → touchpanel-agent*. No HTTP listener, no
|
|
websocket server, no exposed Sway IPC socket, no VNC (this image has none at all, see
|
|
above). New control features belong as additional MQTT entities, not as a second
|
|
listener.
|
|
|
|
## Manual verification still outstanding
|
|
|
|
None of this has been run on hardware. In rough order:
|
|
|
|
1. The ISO builds at all (`lb build` is network-heavy and can fail on mirror hiccups).
|
|
2. greetd lands in Sway with no login prompt, on vt1, with getty@tty1 masked.
|
|
3. **Which tier the real touchscreen lands in** — see "Touch input: two tiers" above.
|
|
Both `input type:touch` and the `input type:pointer` fallback are shipped and
|
|
should both work without a rebuild; what's unverified is whether the fallback's
|
|
`accel_profile flat`/`pointer_accel 0`/`hide_cursor` combination actually feels
|
|
good on real absolute-position-emulated-as-mouse hardware (as opposed to just
|
|
being technically functional), and whether `99-touchscreen-override.rules`'
|
|
`ATTRS{idVendor}`/`ATTRS{idProduct}` match actually walks up to the right USB
|
|
parent on a given controller's specific kernel driver.
|
|
4. `touchpanel-agent` connects to Mosquitto and the device appears in HA.
|
|
5. **Flathub app ID `com.spotify.Client` is correct** — flagged for verification in
|
|
`0300-flatpak-spotify.hook.chroot`, same as the thin client's Steam Link ID.
|
|
5b. **Flathub app ID `com.discordapp.Discord` is correct**, on a Discord panel —
|
|
flagged in `0350-flatpak-discord.hook.chroot` for the same reason.
|
|
5c. **Discord's real X11 `class`** — assumed `discord`, which is what the sway
|
|
fullscreen rule and the agent's focus criteria both match on. `swaymsg -t
|
|
get_tree` on the booted panel is the way to check; if it is capitalised or
|
|
suffixed, the app opens in a window that never goes fullscreen and **Show
|
|
Discord** switches workspace without focusing anything.
|
|
6. Spotify login persists in `~/.var/app/com.spotify.Client` across a reboot.
|
|
7. Spotify's real MPRIS bus name — assumed `spotify`, not confirmed (see the Spotify
|
|
section above); `playerctl -l` on the booted panel is the way to check.
|
|
8. **Chromium's Wayland `app_id` for a `--app=` kiosk window** — assumed to start with
|
|
`chromium` (`configs/sway/config`'s `for_window` rule and
|
|
`touchpanel_agent/main.py`'s `focus_criteria` both match on it). If focusing the
|
|
Home workspace's window from the dock/HA stops working, check the real app_id with
|
|
`swaymsg -t get_tree` and correct both places.
|
|
9. HA login persistence in the Chromium kiosk profile across a reboot (see above).
|
|
10. `eww` and `wvkbd` are not in Debian bookworm main — both hooks try apt and
|
|
otherwise leave a documented placeholder (dock absent / keyboard absent
|
|
respectively); neither has been checked against the real bookworm archive.
|
|
11. **The Firefox extension IDs in `configs/firefox/policies.json`** — same unverified
|
|
caveat as the thin client's copy; see that README's identical item.
|
|
12. **Phase 16's own "reactive path" check**: power off the container host and confirm
|
|
the panel still boots to Spotify + the browser (Home will show a connection
|
|
error, which is the expected/acceptable failure mode — unlike the thin client,
|
|
there's no local-media fallback to fall back to here since this device has no
|
|
local media source at all).
|
|
13. Keyboard layout defaults to German (`KEYBOARD_LAYOUT="de"`) — same per-image,
|
|
no-runtime-override handling as the thin client's identical setting.
|
|
14. Dock touch-target sizing (`configs/eww/eww.scss`, ~84px buttons) — a starting
|
|
guess, not measured against a real finger on real glass at the real screen DPI
|
|
chosen once hardware exists.
|