SmartestHome/hosts/touch-panel/README.md

# Sway touch panel

Phase 16 of `docs/project-plan.md`. Builds a Debian 12 live ISO for a touch-driven
wall/counter panel: autologin into Sway, three fixed apps (Spotify, Home Assistant,
a general web browser) switched with a finger via an always-on dock, and also
controllable by Home Assistant/the local LLM over MQTT.

**A different device from `../thin-client/`**, not a variant of it. The thin client is
a couch-distance media station whose primary control surface is HA/MQTT and wayvnc,
deliberately with no on-screen bars. This one is touched directly, so it needs the
opposite: a persistent, finger-sized on-screen dock, native touchscreen input, and no
gesture/remote-control machinery. It reuses the thin client's live-build toolchain and
`configs/`+`agent/` directory-split convention, not its live-build tree — same
relationship as `../audio-endpoint/`'s amd64 build to the thin client's.

What ends up on the image:

| | |
|---|---|
| Compositor | Sway, workspaces `1:spotify` / `2:home` / `3:web`, always-on touch dock |
| Autologin | greetd, `default_session` straight into `/usr/local/bin/kiosk-session` |
| Touch input | Native Wayland `wl_touch` preferred; degrades to a usable single-touch pointer if the hardware only exposes an emulated-mouse HID interface — see below |
| Music | Full Spotify GUI client (Flathub `com.spotify.Client`) — not a headless Connect receiver |
| Home dashboard | Chromium in kiosk mode, `--app=$HA_URL`, auto-restart if it crashes |
| Web browser | Firefox, minimal chrome (back/forward/reload/address bar), general browsing |
| On-screen keyboard | wvkbd, toggled from the dock — see the caveat below, no auto-show |
| Remote control (HA/LLM) | `touchpanel-agent`, a systemd service publishing HA MQTT-discovery entities |
| Remote control (human) | SSH only — **no wayvnc on this image**, see below |

## Before you build

> **No touch-panel hardware has been chosen.** Nothing here has been booted on real
> metal or a real touchscreen. See "Touch input: two tiers" below for how this image
> handles either way a real touchscreen might show up to Linux.

Edit the `# CONFIGURATION` block at the top of
[`tools/build-touch-panel-iso.sh`](../../tools/build-touch-panel-iso.sh):

| Variable | What to put in it |
|---|---|
| `MQTT_BROKER_HOST` | LAN IP of the container host running Mosquitto (Phase 1) |
| `HA_URL` | Home Assistant URL, e.g. `http://192.168.1.10:8123` — this is what loads in the Home workspace |
| `KIOSK_USERNAME` | Autologin account name (`kiosk`) |
| `TOUCHPANEL_NAME` / `IMAGE_HOSTNAME` | Per-panel identity; each touch panel needs its own |
| `SSH_AUTHORIZED_KEY` | Optional. Password auth is disabled and there is no wayvnc, so without a key the only admin path is the local console. |

## Build

```sh
sudo -E tools/build-touch-panel-iso.sh
```

Same directory-split convention as the thin client: `configs/` and `agent/` are the
human-edited, git-tracked source of truth; `live-build/config/includes.chroot/` is
**generated** — wiped and rebuilt on every run, gitignored, never hand-edited. The only
templated tokens are `@KIOSK_USERNAME@`/`@KEYBOARD_LAYOUT@`; everything else the
in-chroot hooks need comes from `/etc/touchpanel-agent/config.env`.

## Touch input: two tiers

Not every touchscreen tells Linux the truth about being a touchscreen. Some cheap USB
controllers implement single-touch by emulating an absolute-position HID **mouse**
(a click at wherever the finger is) instead of a proper HID **digitizer**. udev reads
the device's capability bits to decide `ID_INPUT_TOUCHSCREEN` vs. `ID_INPUT_MOUSE`,
gets it wrong for exactly this class of hardware, and libinput then hands sway a
`type:pointer` device instead of `type:touch` — every tap becomes a mouse click, not
a touch event. This image handles both cases rather than assuming the good one:

1. **Native `wl_touch`** (`input type:touch` in `configs/sway/config`) — full
   multi-touch, works with no extra setup. This is what you get if the hardware
   reports correctly.
2. **Fallback: `input type:pointer`** — active on this image regardless, so a
   misclassified touchscreen is still usable out of the box: `accel_profile flat` +
   `pointer_accel 0` keep the absolute tap position from being distorted by mouse-
   acceleration curves meant for relative motion, `map_to_output "*"` keeps the
   coordinate space aligned to the screen, and `seat seat0 hide_cursor 1` hides the
   cursor within ~1ms of it stopping so a tap doesn't leave a visible arrow sitting on
   screen. The ceiling here is single-touch tap/click-drag — no pinch, no true
   multi-touch gestures — which is enough for the dock and most kiosk UI, not enough
   for e.g. pinch-zooming a map in the web browser workspace.

**The real fix, once real hardware is known**:
`configs/udev/99-touchscreen-override.rules` re-tags the specific device by its USB
vendor/product ID so udev (and therefore libinput, and therefore sway) treats it as a
genuine touchscreen — restoring full `wl_touch` semantics instead of settling for the
pointer fallback's ceiling. It ships as an inert `0000:0000` template that matches no
real hardware; that file's own header comment has the exact `udevadm`/`libinput
list-devices` commands to find your device's real IDs and confirm the fix took. Fill
it in and rebuild — or edit it directly on a booted image and
`udevadm control --reload-rules && udevadm trigger && swaymsg reload`, no rebuild
needed to test it.

Quick check on a booted image: `libinput list-devices | grep -A2 Capabilities` — if it
lists `touch`, you're on tier 1 already and don't need the udev override at all.

## Why there's no wayvnc on this image

The thin client's primary control surface is deliberately remote (HA/MQTT + wayvnc)
because nobody is standing at a couch-distance media station. This panel is the
opposite: it's mounted somewhere and touched directly, so remote *view/control* isn't
the primary need the way it is for the thin client — SSH already covers "something's
broken, let me fix it from a shell." If a real need for a remote-view channel shows up
once this is on real hardware, adding wayvnc back is a small, isolated change (copy
`../thin-client/configs/wayvnc/` and its two hooks) — deliberately left out for now
rather than built against a guess.

## Home Assistant login

The Chromium kiosk window's profile (`~/.config/touchpanel-chromium-ha`) is persistent
across restarts specifically so logging into Home Assistant once survives a reboot —
unlike the thin client's digest/admin kiosk Firefox profiles, which are stateless
pages re-copied on every launch. Log in on first boot (tap the dock's Home button if
the window doesn't already have focus) and it should stay logged in after that,
**unverified** — depends on HA's own session cookie lifetime and whether Chromium's
Wayland profile persistence behaves the way assumed.

## Spotify: full GUI, not a headless Connect receiver

`../thin-client/` and `../audio-endpoint/` both run `spotifyd`/`librespot` — a
Spotify Connect *receiver* with no interface of its own, controlled from the phone
app or HA. This panel runs the real, official Spotify Linux client instead
(`configs/spotify/spotify-launch`, Flathub `com.spotify.Client`), because the whole
point of a touch panel is a screen you interact with directly: browsing your library,
searching, picking a playlist with a finger. **Spotify Premium is required for
playback either way** — same requirement as the headless receivers.

Login is interactive, on the device, on first launch, and persists in the Flatpak's
own data directory (`~/.var/app/com.spotify.Client`) — nothing here wipes or reseeds
it.

`touchpanel_agent/mpris_bridge.py` bridges Spotify's MPRIS interface (via
`playerctl`) into an HA `media_player`-shaped set of entities, identical in shape to
the thin client's own bridge, just pointed at `spotify` instead of `mpv`/`spotifyd` as
the player name — **unverified that the Spotify Flatpak's MPRIS bus name is actually
`spotify`** (some Flatpak-sandboxed apps register under a `flatpak.<app-id>`-shaped
name instead). If Playback state never updates, check `playerctl -l` on the booted
panel and adjust `PLAYER_PRIORITY` in `mpris_bridge.py`.

## Touch dock

An always-visible bar reserved at the bottom of the screen (`configs/eww/`, layer-shell
`:exclusive true` so nothing ever tiles under or draws over it) with four buttons:
**Spotify**, **Home**, **Web**, and **Keyboard**. The first three call `swaymsg
workspace` directly — fixed constants in `eww.yuck`, nothing from MQTT/HA is ever
interpolated into them, same rule as the thin client's now-playing widget. This is the
touch-first equivalent of the thin client's HA **Screen** select entity — both exist,
one for a finger, one for the LLM.

Session-scoped (`exec_always` in `configs/sway/config`), not part of
`touchpanel-agent`, for the identical reason the thin client's now-playing widget
isn't: compositor UI should die with the compositor, not need a restart-independent
lifecycle or open a second inbound control channel into the security-sensitive agent.

**If `eww` isn't installed** (it's not in Debian bookworm main —
`0500-eww-dock.hook.chroot` tries apt and otherwise leaves a documented placeholder),
there is no on-screen app switcher at all. The panel still boots and shows Home; HA/
LLM-driven app switching via `touchpanel-agent`'s **Screen** select and **Show
\<app\>** buttons still works either way, since that's a separate control path.

## On-screen keyboard — no auto-show, by design

`configs/keyboard/toggle-keyboard` toggles `wvkbd` on and off. There is deliberately
**no automatic show-on-text-field-focus** — that needs the `text-input-v3`/
`virtual-keyboard-v1` Wayland protocols wired all the way through each app via a shell
component (the way Phosh does it for squeekboard), which this image doesn't run.
Instead: tap the dock's Keyboard button before typing, tap it again to dismiss. This
is a known, accepted limitation, not a bug to chase before the panel is otherwise
useful — see the verification list below if you want to pursue proper auto-show later.

`wvkbd`, not `squeekboard`, was picked specifically because it doesn't assume that
shell integration exists; see `0700-onscreen-keyboard.hook.chroot`'s comment. It is
also not in Debian bookworm main, so that hook has the same apt-first/documented-
placeholder shape as `eww` and the thin client's `spotifyd`/`librespot` install.

## Home Assistant entities

`touchpanel-agent` publishes MQTT-discovery configs on connect. Under the MQTT
integration you should get one device per touch panel with:

- **Show Spotify**, **Show Home**, **Show web browser** (buttons) — switch workspace
  and, for Spotify/Home, focus the already-running app rather than relaunching it
  (both are meant to stay open and stateful, unlike the thin client's stateless
  digest/admin kiosk pages)
- **Screen** (select) — `1:spotify` / `2:home` / `3:web`
- **Playback state** (sensor, with track metadata as attributes), **Volume**
  (number), and play/pause / next / previous / stop (buttons), bridged from
  Spotify's own MPRIS interface
- A **media_player** discovery payload — see the thin client README's identical
  caveat: stock Home Assistant's MQTT integration has no `media_player` platform: the
  button/number/sensor entities above are what give playback control on a plain
  install; the HACS *MQTT Media Player* integration is what picks up this one.

### Security boundary

Identical shape and reasoning to `../thin-client/README.md`'s section of the same
name: `touchpanel_agent/mqtt_discovery.py` is the **entire** inbound control surface
of this machine. The LLM never gets a direct network path here — the only chain is
*LLM tool call → HA service call → MQTT → touchpanel-agent*. No HTTP listener, no
websocket server, no exposed Sway IPC socket, no VNC (this image has none at all, see
above). New control features belong as additional MQTT entities, not as a second
listener.

## Manual verification still outstanding

None of this has been run on hardware. In rough order:

1. The ISO builds at all (`lb build` is network-heavy and can fail on mirror hiccups).
2. greetd lands in Sway with no login prompt, on vt1, with getty@tty1 masked.
3. **Which tier the real touchscreen lands in** — see "Touch input: two tiers" above.
   Both `input type:touch` and the `input type:pointer` fallback are shipped and
   should both work without a rebuild; what's unverified is whether the fallback's
   `accel_profile flat`/`pointer_accel 0`/`hide_cursor` combination actually feels
   good on real absolute-position-emulated-as-mouse hardware (as opposed to just
   being technically functional), and whether `99-touchscreen-override.rules`'
   `ATTRS{idVendor}`/`ATTRS{idProduct}` match actually walks up to the right USB
   parent on a given controller's specific kernel driver.
4. `touchpanel-agent` connects to Mosquitto and the device appears in HA.
5. **Flathub app ID `com.spotify.Client` is correct** — flagged for verification in
   `0300-flatpak-spotify.hook.chroot`, same as the thin client's Steam Link ID.
6. Spotify login persists in `~/.var/app/com.spotify.Client` across a reboot.
7. Spotify's real MPRIS bus name — assumed `spotify`, not confirmed (see the Spotify
   section above); `playerctl -l` on the booted panel is the way to check.
8. **Chromium's Wayland `app_id` for a `--app=` kiosk window** — assumed to start with
   `chromium` (`configs/sway/config`'s `for_window` rule and
   `touchpanel_agent/main.py`'s `focus_criteria` both match on it). If focusing the
   Home workspace's window from the dock/HA stops working, check the real app_id with
   `swaymsg -t get_tree` and correct both places.
9. HA login persistence in the Chromium kiosk profile across a reboot (see above).
10. `eww` and `wvkbd` are not in Debian bookworm main — both hooks try apt and
    otherwise leave a documented placeholder (dock absent / keyboard absent
    respectively); neither has been checked against the real bookworm archive.
11. **The Firefox extension IDs in `configs/firefox/policies.json`** — same unverified
    caveat as the thin client's copy; see that README's identical item.
12. **Phase 16's own "reactive path" check**: power off the container host and confirm
    the panel still boots to Spotify + the browser (Home will show a connection
    error, which is the expected/acceptable failure mode — unlike the thin client,
    there's no local-media fallback to fall back to here since this device has no
    local media source at all).
13. Keyboard layout defaults to German (`KEYBOARD_LAYOUT="de"`) — same per-image,
    no-runtime-override handling as the thin client's identical setting.
14. Dock touch-target sizing (`configs/eww/eww.scss`, ~84px buttons) — a starting
    guess, not measured against a real finger on real glass at the real screen DPI
    chosen once hardware exists.