SmartestHome/hosts/thin-client/README.md

Sway thin client

Phase 11 of docs/project-plan.md. Builds a Debian 12 live ISO for a kiosk media station: autologin into Sway, remote-controlled by Home Assistant over MQTT and by a human over wayvnc.

What ends up on the image:

Compositor	Sway, no bars, no lock screen, workspaces 1:web / 2:digest / 3:media / 4:admin / 5:capture
Autologin	greetd, default_session straight into /usr/local/bin/kiosk-session
Remote control (human)	wayvnc on 0.0.0.0:5900, authentication required
Remote control (HA/LLM)	thinclient-agent, a systemd service publishing HA MQTT-discovery entities
Browser	Firefox ESR in kiosk mode on the digest workspace, pointed at digest-web
Media	mpv + mpv-mpris, playerctl, PipeWire/WirePlumber
Capture-card input ("receiver box")	Any USB/PCIe HDMI capture card plugged into the box, selectable in HA, shown via mpv on 5:capture — see below
Game streaming	Steam Link (Flathub flatpak) forced onto Xwayland
Voice	wyoming-satellite + openWakeWord — opt-in, off by default
Gesture control	Camera hand tracking (MediaPipe) — opt-in, off by default, see below

Before you build

Two open decisions from docs/project-plan.md §4 are still unresolved and block a real build + flash:

• #7 — no thin-client hardware has been chosen. Nothing here has been booted on real metal. The GPU, audio device, and Steam Link/Xwayland decode performance are all unvalidated, and wyoming-satellite's --mic-command/--snd-command are still set to the ALSA default device because the microphone is unknown.
• #6 — no mic-enabled room list has been chosen. ENABLE_VOICE_SATELLITE is a per-image setting, not a global one: build one ISO with it false for the silent rooms and a separate ISO with it true for each room that actually has a mic. Do not turn it on until that list exists.

Then edit the # CONFIGURATION block at the top of scripts/build-thin-client-iso.sh:

Variable	What to put in it
MQTT_BROKER_HOST	LAN IP of the container host running Mosquitto (Phase 1)
HA_URL	Home Assistant URL, e.g. http://192.168.1.10:8123
DIGEST_WEB_URL	The Phase 12 digest-web service. Placeholder until that is deployed — the image builds and boots fine without it, the digest workspace just shows a connection error.
ADMIN_WEB_URL	The Phase 13 admin-web service (see ../../admin-canvas/README.md). Same placeholder handling as DIGEST_WEB_URL — and unlike the digest workspace, 4:admin is never auto-launched at session start anyway, so an unset value just means "Show admin canvas" has nothing to open yet.
KIOSK_USERNAME	Autologin account name (kiosk)
THINCLIENT_NAME / IMAGE_HOSTNAME	Per-room identity; each thin client needs its own
ENABLE_STEAM_LINK	true/false
ENABLE_VOICE_SATELLITE	false unless this specific image is for a mic-enabled room
ENABLE_GESTURE_CONTROL	false unless this specific image is for a camera-enabled room. Installs the software only — the camera still stays off. See the privacy section below.
SSH_AUTHORIZED_KEY	Optional. Password auth is disabled on the image, so without a key the only admin paths are the local console and wayvnc.

Build

sudo ./scripts/build-thin-client-iso.sh

It installs live-build if missing, regenerates live-build/config/includes.chroot/ from configs/ and agent/, writes /etc/thinclient-agent/config.env into the image, then runs lb config && lb build. The ISO lands in live-build/. The script ends with a numbered list of what to verify on first boot.

Directory split

configs/ and agent/ are the human-edited, git-tracked source of truth. live-build/config/includes.chroot/ is generated — it is wiped and rebuilt on every run and is gitignored. Never hand-edit anything under it; edits there are lost on the next build.

The only templated token in configs/ is @KIOSK_USERNAME@, substituted by the build script. Everything the in-chroot hooks need is read from /etc/thinclient-agent/config.env, which live-build copies in (chroot_local-includes) before it runs the hooks (chroot_local-hooks) — that ordering is what lets the hooks be plain scripts with no build-script variables of their own.

The wayvnc password is not in this repo — set it on first boot

configs/wayvnc/config deliberately has no password= line. wayvnc only accepts the password inline, so shipping one would put a live credential for a full remote-control channel into git. Instead:

• 0300-wayvnc.hook.chroot writes the sentinel CHANGEME-SET-ON-FIRST-BOOT into /etc/wayvnc/wayvnc-password (mode 0600).
• /usr/local/bin/start-wayvnc refuses to launch while that sentinel is there, so the failure mode is "no remote access" rather than "an unauthenticated VNC server listening on the LAN".

On the booted machine:

sudo sh -c 'openssl rand -base64 24 > /etc/wayvnc/wayvnc-password'
sudo chmod 600 /etc/wayvnc/wayvnc-password
sudo chown kiosk:kiosk /etc/wayvnc/wayvnc-password
swaymsg reload

Then connect to <thin-client-ip>:5900 with username kiosk.

Camera gesture control is a camera in a room — read this before enabling it

If you turn this on, a camera continuously captures and analyses video of the room for as long as the kiosk session is up. Not on a trigger, not on a wake word — every frame, all day. Frames are processed in memory and nothing is recorded or sent anywhere, but that is a property of this code today, not a guarantee the hardware gives you: the camera is physically pointed at the room whatever the software does.

It is off by default and must stay off unless the room has agreed to it. This is the same decision, made the same way, as the microphones: per docs/project-plan.md Phase 11.8 only the specific rooms chosen for voice interactivity get a mic, and ENABLE_VOICE_SATELLITE is a per-image build flag precisely so that a silent room's ISO physically cannot listen. Only specific rooms get a gesture-control camera, and ENABLE_GESTURE_CONTROL is per-image for exactly the same reason. A shared media station in a living room is not a place to switch a camera on opportunistically because the feature happened to be available.

There is no HA entity for this and there deliberately never will be — nothing reachable over MQTT can turn the camera on. Enabling it takes physical or SSH access to the machine.

Open hand moves the pointer, closed fist clicks. It is the same idea as the Pointer up/down/left/right / Left click buttons in the HA entity list below — a way to drive the kiosk without a VNC session — done from the sofa instead of from a phone.

Hardware: a USB webcam, per camera-enabled room. Not part of the base thin-client bill of materials; it is a config-gated addition to the rooms that opt in, exactly like the USB mic in the mic-enabled rooms. Any ordinary UVC webcam that shows up as a /dev/video* node works; nothing here needs a depth camera or an accelerator.

Two separate gates, both defaulting to off

Gate	Where	What it decides
ENABLE_GESTURE_CONTROL	build script, per image	Whether MediaPipe and its ~400 MB dependency tree are installed at all
"enabled"	gesture-config.json, per machine at runtime	Whether the camera is ever opened

The second one is the one that matters. It is false even in an image built with the first one on, so flashing a gesture-capable ISO is not the same act as switching a room's camera on. gesture_pointer.py checks it before it imports OpenCV or MediaPipe, so while it is false the video stack is never loaded and /dev/video0 is never opened — not opened and ignored.

To turn it on, on the booted machine:

v4l2-ctl --list-devices                                    # find the right /dev/videoN
sudo $EDITOR /var/lib/thinclient-agent/gesture-config.json # "enabled": true
swaymsg reload

gesture-config.json follows the same template/runtime-copy split as audio-config.json and rdp-vnc.json: the committed file is baked in read-only at /etc/thinclient-agent/, and runtime_state.ensure_runtime_copy() seeds the writable one under /var/lib/thinclient-agent/ on first run. Edit the runtime copy — editing the /etc one does nothing, and it is inside a squashfs anyway.

Where it runs, and why it is not part of thinclient-agent

configs/gesture-control/gesture_pointer.py is a separate process, started as a sway exec_always via /usr/local/bin/gesture-control — the same shape as the eww now-playing widget, not a new module inside the thinclient-agent daemon. Three reasons, all pointing the same way:

• Privacy. A systemd system unit would hold the camera open from boot to shutdown, including while no session exists and there is nothing to point at. Tying the camera's lifetime to the session's means "the screen is up" and "the camera is open" cannot drift apart.
• Blast radius. thinclient-agent is the machine's sole MQTT control surface (see the security-boundary note below) and has to stay reliable. A continuous camera read plus ML inference is a completely different failure and CPU profile, and a crash in it must not be able to take HA control of the room down with it. This is the same call configs/eww/fullscreen-watcher.sh already makes.
• Interpreter. MediaPipe is PyPI-only and lives in its own venv under /opt/gesture-control (bookworm's system Python is PEP 668 externally-managed — the same situation as wyoming-satellite). thinclient-agent runs on the system interpreter, which cannot import it.

It does not reimplement input injection: it imports thinclient_agent.input_control.InputControl (stdlib-only, so it loads fine from inside the venv) and calls click("LEFT") and move_relative(). move_relative() is the one thing added to that module — continuous pointer control genuinely needs proportional deltas rather than the four fixed 20px directions the HA buttons use, and putting it there keeps a single ydotool call path with a single dialect detection for the whole image. It is clamped to ±200px per step and is not wired to any MQTT topic; the inbound control surface is unchanged.

Idle photo slideshow

After 15 minutes idle, the thin client mounts a read-only SMB share (gallery-smb on the container host, ENABLE_GALLERY_SMB in setup-container-host.sh) and cycles through whatever images are in it, instead of just blanking the panel. It falls all the way back to that original blank-the-panel behaviour — not a broken/blank slideshow window — if any of the following is true: /etc/thinclient-agent/ gallery-credentials hasn't been created yet, the share is unreachable, or it's empty. A freshly built or offline thin client behaves exactly as it did before this feature.

Nothing here is an HA entity — it's a local swayidle timeout/resume pair (configs/sway/config) calling /usr/local/bin/idle-gallery, which mounts the share with the kiosk user's already-unrestricted sudo (see the wayvnc-password section above for the same reasoning about physical-access trust) and hands a shuffled playlist to mpv.

Set this up: create /etc/thinclient-agent/gallery-credentials (chmod 600) from the .example next to it, with the same username/password as GALLERY_SMB_USERNAME/GALLERY_SMB_PASSWORD in setup-container-host.sh — this project has no way to push a secret from one machine to the other, so both sides are set by hand from the same value. Also fill in GALLERY_SMB_HOST in build-thin-client-iso.sh (the container host's LAN IP) before rebuilding.

Capture-card / receiver-box viewing

Any USB or PCIe HDMI/AV capture card plugged into this machine can be picked as a video source from Home Assistant — the intent is "use this box like a TV/AVR with selectable inputs" for a console, cable box, or anything else that only speaks HDMI/AV out. Selecting one switches to 5:capture and shows it full-screen via mpv (mpv av://v4l2:/dev/videoN); picking "none" clears it. See thinclient_agent/capture_control.py.

Detection is dynamic and periodic, not just at boot: the agent re-scans v4l2-ctl --list-devices roughly every 20 seconds and republishes the HA select's option list when it changes, so a capture card plugged in mid-session shows up on its own — no reconnect or restart needed. Each physical device's label includes its USB bus path (what v4l2-ctl itself reports), which is what keeps two identical dongles distinguishable without needing extra disambiguation logic.

The gesture-control camera is never offered as a capture source. Enumeration reads gesture-config.json's camera_device and excludes it — but only while "enabled": true, the same gate gesture_pointer.py itself uses to decide whether the camera is ever opened at all. camera_device defaults to /dev/video0 on every image whether or not gesture control was even built in, so the exclusion is deliberately conditional: excluding it unconditionally would silently hide a real capture card that happens to enumerate as /dev/video0 on the (default, common) image where gesture control is off, for no privacy benefit — there's nothing to protect while that camera is never opened in the first place. See "Camera gesture control" above for the invariant this preserves: no HA entity can turn that specific camera on, full stop.

Audio is best-effort. Many cheap USB HDMI-capture dongles carry their embedded HDMI audio over a separate USB Audio Class interface rather than in the V4L2 stream itself — capture_control.find_audio_card() tries to match a capture device to a sibling ALSA card by shared USB device topology (sysfs, not vendor/product IDs) and passes it to mpv as --external-file=alsa://hw:X,0 if found. Video-only playback (not a crash) if nothing matches. Unverified against real hardware — see the checklist below.

Home Assistant entities

thinclient-agent publishes MQTT-discovery configs on connect. Under the MQTT integration you should get one device per thin client with:

• Show digest canvas (button) — switches to 2:digest and reloads Firefox
• Digest detail level (select) — compact / full
• Show admin canvas (button) — switches to 4:admin and reloads Firefox at admin-web's canvas.html (Phase 13, see ../../admin-canvas/README.md). No detail level or any other option — everything it shows is populated out of band by admin-canvas's write API, not by this agent. This is the sys-admin-llm's surface for on-demand stats/graphics/media (e.g. "show me the kitchen outlet's power draw"), as opposed to the digest's scheduled 4x/day synthesis.
• Capture source (select) — "none" plus whatever capture cards are currently plugged in (dynamic, re-scanned periodically — see "Capture-card / receiver-box viewing" above); switches to 5:capture and shows the picked one full-screen via mpv.
• Workspace (select) — 1:web / 2:digest / 3:media / 4:admin / 5:capture
• Launch Firefox, Launch web browser, Launch Steam Link (buttons)
• Playback state (sensor, with track metadata as attributes), Volume (number), and play/pause / next / previous / stop (buttons)
• A media_player discovery payload — see the caveat below
• Audio output (select) — WirePlumber sinks by human-readable description; the choice is written to audio-config.json and survives a reboot
• Remote desktop target (select) plus Remote desktop connect / disconnect (buttons) — outbound RDP/VNC to another machine via Remmina, configured in configs/remote-desktop/rdp-vnc.json. Distinct from wayvnc, which is inbound.
• Type text (text field with a submit action) and Pointer up/down/left/right, Left click, Right click (buttons) — types into, and moves/clicks in, whatever window has focus, via ydotool. Meant for the HA mobile app when a full VNC session is overkill (searching in the kiosk Firefox, dismissing a dialog).

A manually-authored Lovelace card grouping the last two groups (audio output, remote desktop, and the text/pointer controls) under one "Thin client remote" section reads far better in the mobile app than the auto-generated entity list — nothing in this repo builds that card, since it is a few lines of YAML per household and not something a generic image should assume.

A now-playing widget also appears on-screen (not an HA entity — it is local to the kiosk display) whenever audio-only media is playing: cover art, track/artist, and prev/play-pause/next, via eww. It hides completely the moment anything visual is on screen — fullscreen video, Steam Link, or an mpv window with a video track — so it never draws over a film. See configs/eww/fullscreen-watcher.sh.

Caveat: core Home Assistant's MQTT integration has no media_player platform. The plan calls for a media_player entity and the agent publishes that payload, but stock HA ignores it; it is only picked up with the HACS MQTT Media Player custom integration installed. The sensor/number/button entities above are published alongside it precisely so transport control works on a plain HA install without that add-on. Verify which you want before wiring up automations.

Security boundary

thinclient_agent/mqtt_discovery.py is the entire inbound control surface of this machine. Per Phase 11.4 the LLM never gets a direct network path here — the only chain is LLM tool call → HA service call → MQTT → thinclient-agent. There is no HTTP listener, no websocket server, no exposed Sway IPC socket. New control features belong as additional MQTT entities, not as a second listener.

Related: thinclient_agent/digest_canvas.py does no presence, person, or room resolution. Home Assistant resolves who and where (including the "whose digest?" disambiguation when several people are in the room) and sends an already-resolved request; this agent only shows what it is told to show.

thinclient_agent/admin_canvas.py (Phase 13) follows the identical pattern, taken one step further: its MQTT payload isn't even inspected, since there is nothing content-specific for this agent to decide — "Show admin canvas" always switches to 4:admin and opens the same fixed, locally-configured ADMIN_WEB_URL/canvas.html. All of that page's actual content (stats, charts, images, short clips) is populated by a completely separate path — the sys-admin-llm, via an HA tool call, calling admin-canvas's own token-gated write API on the container host — that never touches this agent or this machine's MQTT surface at all. See ../../admin-canvas/README.md for that API and its own security notes.

thinclient_agent/capture_control.py follows the same "payload only ever selects among already-enumerated values" rule as audio_control.py/ remote_desktop.py: the HA select's payload is looked up in the device list list_devices() already built server-side, and it's that lookup's .path — never the payload — that reaches capture-view as an argv element. An unknown or since-unplugged selection resolves to "no source," never to acting on whatever string HA sent.

Manual verification still outstanding

None of this has been run on hardware. In rough order:

1. The ISO builds at all (lb build is network-heavy and can fail on mirror hiccups).
2. greetd lands in Sway with no login prompt, on vt1, with getty@tty1 masked.
3. wayvnc refuses to start with the sentinel password, and works once one is set.
4. thinclient-agent connects to Mosquitto and the device appears in HA.
5. mpv playback drives the Playback state sensor via mpv-mpris → playerctl.
6. Steam Link launches under Xwayland without the wlroots black-screen bug — and that the flatpak app ID com.valvesoftware.SteamLink is correct; it is flagged for verification against the live Flathub listing in 0400-flatpak-steamlink.hook.chroot.
7. Spotify Connect: neither spotifyd nor librespot is in bookworm main, so 0500-spotify-connect.hook.chroot currently stops at a documented placeholder rather than hardcoding a release URL that would rot. Pick an install route and fill it in.
8. wyoming-satellite / wyoming-openwakeword install from PyPI — 0600-voice-satellite.hook.chroot installs them into a venv under /opt (bookworm's system Python is PEP 668 externally-managed, and these pull a large onnxruntime/numpy tree that has no business overwriting apt-managed versions). If the PyPI names turn out to be wrong, the hook comments point at upstream's git-clone + script/setup install instead.
9. Phase 11.10: power off the container host and confirm the kiosk still boots to a usable session and plays local media. thinclient-agent uses connect_async and its unit is deliberately not After=network-online.target, so it should never be able to stall the session.
10. eww is not in Debian bookworm main — 0800-eww-widget.hook.chroot tries apt and otherwise leaves the now-playing widget absent (it degrades cleanly; nothing else depends on it). Pick an install route (cargo, or a release binary) if you want it.
11. The Firefox extension IDs in configs/firefox/policies.json are unverified. The AMO download-URL slugs (ublock-origin, sponsorblock) were checked against the live listings, but the ExtensionSettings keys (each extension's internal ID, e.g. uBlock0@raymondhill.net) are widely-published values reproduced from memory, not confirmed against AMO directly. If an extension silently fails to force-install, check about:policies and correct the key from about:debugging#/runtime/this-firefox on a manually-installed copy.
12. remmina-plugin-rdp/remmina-plugin-vnc package names are assumed, not confirmed against bookworm's actual archive — if apt-get install for them fails, check apt-cache search remmina-plugin on the build host and adjust the package list.
13. ydotool: whether ydotoold exists (and therefore which command-line dialect applies) depends on the exact bookworm package version — 1000-ydotool.hook.chroot detects this at build time and input_control.py detects it again at runtime, but neither has been checked against the real packaged version yet. If the HA text/ pointer controls do nothing, start here: systemctl status ydotoold and journalctl -u thinclient-agent.
14. Outbound RDP/VNC: fill in real host/port entries in configs/remote-desktop/rdp-vnc.json and create /etc/thinclient-agent/remote-desktop-credentials.env on the booted machine from the shipped .example (never baked into the image, same handling as the wayvnc password) before the Remote desktop connect button will reach anything real.
15. Fill in a real preferred_sink in configs/audio/audio-config.json (or leave it empty and pick one later from the Audio output select in HA) — the shipped template has no sink configured, which is a supported/safe default, not a gap.
16. Keyboard layout defaults to German (KEYBOARD_LAYOUT="de" in build-thin-client-iso.sh) — applies to both the console (/etc/default/keyboard) and Sway (input type:keyboard { xkb_layout ... }). Build a separate image with a different value for a non-German room; there is no per-room override at runtime. If ENABLE_INSTALLER="true", live-build/config/preseed.cfg also preseeds debian-installer's keyboard step to German (as a default, not a skipped question) — unverified against a real installer run, since the normal path never uses it.
17. Camera gesture control — nothing about it has been measured on real hardware, and the numbers below are estimates, not results. It is off by default so an unverified feature cannot surprise anyone; enable it on a test box before a room.
    • CPU load. Google publishes 17.12 ms per frame for the float16 hand-landmarker on a Pixel 6 CPU, and a desktop x86 report puts the full Python pipeline at 20–30 ms/frame and ~16 % of CPU. An Intel N100 is four Gracemont E-cores with AVX2 and no AVX-512, so expect the slower end of that — plan on the inference loop eating a meaningful fraction of one core continuously. That is why inference_fps defaults to 12 rather than the camera's 30: pointer control does not need 30 fps and the cost scales almost linearly with it. Measure with top -p "$(pgrep -f gesture_pointer.py)" and turn inference_fps down until it is acceptable. Check it specifically while Steam Link is streaming — that is the one workload on this image that already wants the whole CPU, and if the two cannot coexist, that is an argument for gesture control being a per-room feature rather than a fix in this code.
    • Accuracy and false-positive clicks. The open/fist thresholds (FIST_MIN_CURLED / OPEN_MAX_CURLED in gesture_pointer.py) and the fist_hold_seconds / click_cooldown_seconds defaults are reasoned guesses, never tested against a real hand at real room distance in real lighting. The failure mode that matters is a spurious click, so if it fires by itself, raise fist_hold_seconds first.
    • Pointer feel. dead_zone and pointer_speed set how twitchy it is. Untuned.
    • Camera latency. The loop reads every frame and discards the ones it does not analyse, specifically so a V4L2 backlog cannot make the pointer lag the hand. Not verified that CAP_PROP_BUFFERSIZE/CAP_PROP_FPS are honoured by any given webcam — many ignore them.
    • MediaPipe API. Written against the current Tasks API (mediapipe.tasks.python.vision.HandLandmarker). The old mediapipe.solutions.hands is not merely deprecated — the mediapipe.solutions package was removed from the wheel around 0.10.31 and is gone in 1.0.0, which is also why 1100-gesture-control.hook.chroot has to download hand_landmarker.task separately instead of it coming inside the wheel.
    • Install. pip install mediapipe is unrun here. 1.0.0 ships py3-none-manylinux_2_28_x86_64; bookworm is glibc 2.36 / Python 3.11, which fits, but confirm rather than assume. If it fails the hook stops at a documented placeholder and the rest of the image is unaffected.
18. Maintenance shell: Super+Shift+Ctrl+M opens a floating foot terminal locally. This is a physical-access escape hatch, deliberately outside the MQTT/HA control surface described above — see the comment above the keybind in configs/sway/config for why that is correct rather than an inconsistency.
19. Idle photo slideshow — none of idle-gallery.sh's CIFS mount options have been tried against a real Samba server. If mount.cifs rejects vers=3.0 (an older NAS, say) or the soft,retry=0 combination doesn't fail as fast as intended on a truly dead host, adjust the -o string there. The ghcr.io/servercontainers/samba ACCOUNT_<user>/SAMBA_VOLUME_CONFIG_<name> env-var syntax in setup-container-host.sh is confirmed against that project's documentation, but the container itself was never actually run.
20. Capture-card viewing — nothing here has been tried against a real capture device. In particular:
    • mpv av://v4l2:/dev/videoN itself: latency, whether --profile=low-latency --untimed --no-cache is actually the right flag combination for a given card, and whether any card needs an explicit --demuxer-lavf-format/pixel format hint v4l2-ctl doesn't surface.
    • The ALSA-audio-matching heuristic in find_audio_card() — whether a given dongle's video and audio interfaces actually share a sysfs USB device ancestor the way assumed, and whether hw:X,0 is always the right subdevice index (some cards expose audio on a non-zero one).
    • capture-view's --title=thinclient-capture-view pkill/pgrep scoping has only been checked to not match unrelated mpv invocations by string inspection, not exercised against a real running mpv process tree.
    • Whether v4l2-ctl --info's "Device Caps" parsing correctly picks the real capture node (and skips a metadata/extra node) for capture cards other than whatever specific chipset a future tester's dongle happens to use — the logic is chipset-agnostic in design but only reasoned about, not run against real v4l2-ctl output from more than one card model.