595 lines
35 KiB
Markdown
595 lines
35 KiB
Markdown
# Sway thin client
|
||
|
||
Phase 11 of `docs/project-plan.md`. Builds a Debian 12 live ISO for a kiosk media
|
||
station: autologin into Sway, remote-controlled by Home Assistant over MQTT and by a
|
||
human over wayvnc.
|
||
|
||
What ends up on the image:
|
||
|
||
| | |
|
||
|---|---|
|
||
| Compositor | Sway, no bars, no lock screen, workspaces `1:web` / `2:digest` / `3:media` / `4:admin` / `5:capture` |
|
||
| Autologin | greetd, `default_session` straight into `/usr/local/bin/kiosk-session` |
|
||
| Remote control (human) | wayvnc on `0.0.0.0:5900`, authentication required |
|
||
| Remote control (HA/LLM) | `thinclient-agent`, a systemd service publishing HA MQTT-discovery entities |
|
||
| Browser | Firefox ESR in kiosk mode on the digest workspace, pointed at `digest-web` |
|
||
| Media | mpv + mpv-mpris, playerctl, PipeWire/WirePlumber |
|
||
| Capture-card input ("receiver box") | Any USB/PCIe HDMI capture card plugged into the box, selectable in HA, shown via mpv on `5:capture` — see below |
|
||
| Idle-gallery overlay | Clock/date always, weather once `smarthome/weather/current` is published — see below |
|
||
| Game streaming | Steam Link (Flathub flatpak) forced onto Xwayland |
|
||
| Voice | wyoming-satellite + openWakeWord — **opt-in, off by default** |
|
||
| Gesture control | Camera hand tracking (MediaPipe) — **opt-in, off by default, see below** |
|
||
|
||
## Before you build
|
||
|
||
> **Two open decisions from `docs/project-plan.md` §4 are still unresolved and block a
|
||
> real build + flash:**
|
||
>
|
||
> - **#7 — no thin-client hardware has been chosen.** Nothing here has been booted on
|
||
> real metal. The GPU, audio device, and Steam Link/Xwayland decode performance are
|
||
> all unvalidated, and `wyoming-satellite`'s `--mic-command`/`--snd-command` are still
|
||
> set to the ALSA `default` device because the microphone is unknown.
|
||
> - **#6 — no mic-enabled room list has been chosen.** `ENABLE_VOICE_SATELLITE` is a
|
||
> **per-image** setting, not a global one: build one ISO with it `false` for the silent
|
||
> rooms and a separate ISO with it `true` for each room that actually has a mic. Do
|
||
> not turn it on until that list exists.
|
||
|
||
Then edit the `# CONFIGURATION` block at the top of
|
||
[`tools/build-thin-client-iso.sh`](../../tools/build-thin-client-iso.sh):
|
||
|
||
| Variable | What to put in it |
|
||
|---|---|
|
||
| `MQTT_BROKER_HOST` | LAN IP of the container host running Mosquitto (Phase 1) |
|
||
| `HA_URL` | Home Assistant URL, e.g. `http://192.168.1.10:8123` |
|
||
| `DIGEST_WEB_URL` | The Phase 12 `digest-web` service. Placeholder until that is deployed — the image builds and boots fine without it, the digest workspace just shows a connection error. |
|
||
| `ADMIN_WEB_URL` | The Phase 13 `admin-web` service (see `../../admin-canvas/README.md`). Same placeholder handling as `DIGEST_WEB_URL` — and unlike the digest workspace, `4:admin` is never auto-launched at session start anyway, so an unset value just means "Show admin canvas" has nothing to open yet. |
|
||
| `KIOSK_USERNAME` | Autologin account name (`kiosk`) |
|
||
| `THINCLIENT_NAME` / `IMAGE_HOSTNAME` | Per-room identity; each thin client needs its own |
|
||
| `ENABLE_STEAM_LINK` | `true`/`false` |
|
||
| `ENABLE_VOICE_SATELLITE` | `false` unless this specific image is for a mic-enabled room |
|
||
| `ENABLE_GESTURE_CONTROL` | `false` unless this specific image is for a camera-enabled room. Installs the software only — the camera still stays off. See the privacy section below. |
|
||
| `SSH_AUTHORIZED_KEY` | Optional. Password auth is disabled on the image, so without a key the only admin paths are the local console and wayvnc. |
|
||
|
||
## Build
|
||
|
||
```sh
|
||
sudo -E tools/build-thin-client-iso.sh
|
||
```
|
||
|
||
It installs `live-build` if missing, regenerates
|
||
`live-build/config/includes.chroot/` from `configs/` and `agent/`, writes
|
||
`/etc/thinclient-agent/config.env` into the image, then runs `lb config && lb build`.
|
||
The ISO lands in `live-build/`. The script ends with a numbered list of what to verify
|
||
on first boot.
|
||
|
||
### Directory split
|
||
|
||
`configs/` and `agent/` are the human-edited, git-tracked source of truth.
|
||
`live-build/config/includes.chroot/` is **generated** — it is wiped and rebuilt on every
|
||
run and is gitignored. Never hand-edit anything under it; edits there are lost on the
|
||
next build.
|
||
|
||
The only templated token in `configs/` is `@KIOSK_USERNAME@`, substituted by the build
|
||
script. Everything the in-chroot hooks need is read from
|
||
`/etc/thinclient-agent/config.env`, which live-build copies in (`chroot_local-includes`)
|
||
before it runs the hooks (`chroot_local-hooks`) — that ordering is what lets the hooks
|
||
be plain scripts with no build-script variables of their own.
|
||
|
||
## The wayvnc password is not in this repo — set it on first boot
|
||
|
||
`configs/wayvnc/config` deliberately has no `password=` line. wayvnc only accepts the
|
||
password inline, so shipping one would put a live credential for a full remote-control
|
||
channel into git. Instead:
|
||
|
||
- `0300-wayvnc.hook.chroot` writes the sentinel `CHANGEME-SET-ON-FIRST-BOOT` into
|
||
`/etc/wayvnc/wayvnc-password` (mode 0600).
|
||
- `/usr/local/bin/start-wayvnc` **refuses to launch** while that sentinel is there, so
|
||
the failure mode is "no remote access" rather than "an unauthenticated VNC server
|
||
listening on the LAN".
|
||
|
||
On the booted machine:
|
||
|
||
```sh
|
||
sudo sh -c 'openssl rand -base64 24 > /etc/wayvnc/wayvnc-password'
|
||
sudo chmod 600 /etc/wayvnc/wayvnc-password
|
||
sudo chown kiosk:kiosk /etc/wayvnc/wayvnc-password
|
||
swaymsg reload
|
||
```
|
||
|
||
Then connect to `<thin-client-ip>:5900` with username `kiosk`.
|
||
|
||
## Camera gesture control is a camera in a room — read this before enabling it
|
||
|
||
> **If you turn this on, a camera continuously captures and analyses video of the room
|
||
> for as long as the kiosk session is up.** Not on a trigger, not on a wake word — every
|
||
> frame, all day. Frames are processed in memory and nothing is recorded or sent
|
||
> anywhere, but that is a property of this code today, not a guarantee the hardware
|
||
> gives you: the camera is physically pointed at the room whatever the software does.
|
||
>
|
||
> **It is off by default and must stay off unless the room has agreed to it.** This is
|
||
> the same decision, made the same way, as the microphones: per `docs/project-plan.md`
|
||
> Phase 11.8 only the specific rooms chosen for voice interactivity get a mic, and
|
||
> `ENABLE_VOICE_SATELLITE` is a per-image build flag precisely so that a silent room's
|
||
> ISO physically cannot listen. **Only specific rooms get a gesture-control camera**, and
|
||
> `ENABLE_GESTURE_CONTROL` is per-image for exactly the same reason. A shared media
|
||
> station in a living room is not a place to switch a camera on opportunistically because
|
||
> the feature happened to be available.
|
||
>
|
||
> There is no HA entity for this and there deliberately never will be — nothing reachable
|
||
> over MQTT can turn the camera on. Enabling it takes physical or SSH access to the
|
||
> machine.
|
||
|
||
Open hand moves the pointer, closed fist clicks. It is the same idea as the **Pointer
|
||
up/down/left/right** / **Left click** buttons in the HA entity list below — a way to
|
||
drive the kiosk without a VNC session — done from the sofa instead of from a phone.
|
||
|
||
**Hardware: a USB webcam, per camera-enabled room.** Not part of the base thin-client
|
||
bill of materials; it is a config-gated addition to the rooms that opt in, exactly like
|
||
the USB mic in the mic-enabled rooms. Any ordinary UVC webcam that shows up as a
|
||
`/dev/video*` node works; nothing here needs a depth camera or an accelerator.
|
||
|
||
### Two separate gates, both defaulting to off
|
||
|
||
| Gate | Where | What it decides |
|
||
|---|---|---|
|
||
| `ENABLE_GESTURE_CONTROL` | build script, per image | Whether MediaPipe and its ~400 MB dependency tree are installed at all |
|
||
| `"enabled"` | `gesture-config.json`, per machine at runtime | Whether the camera is ever **opened** |
|
||
|
||
The second one is the one that matters. It is `false` even in an image built with the
|
||
first one on, so flashing a gesture-capable ISO is not the same act as switching a room's
|
||
camera on. `gesture_pointer.py` checks it *before it imports OpenCV or MediaPipe*, so
|
||
while it is false the video stack is never loaded and `/dev/video0` is never opened — not
|
||
opened and ignored.
|
||
|
||
To turn it on, on the booted machine:
|
||
|
||
```sh
|
||
v4l2-ctl --list-devices # find the right /dev/videoN
|
||
sudo $EDITOR /var/lib/thinclient-agent/gesture-config.json # "enabled": true
|
||
swaymsg reload
|
||
```
|
||
|
||
`gesture-config.json` follows the same template/runtime-copy split as `audio-config.json`
|
||
and `rdp-vnc.json`: the committed file is baked in read-only at
|
||
`/etc/thinclient-agent/`, and `runtime_state.ensure_runtime_copy()` seeds the writable
|
||
one under `/var/lib/thinclient-agent/` on first run. Edit the runtime copy — editing the
|
||
`/etc` one does nothing, and it is inside a squashfs anyway.
|
||
|
||
### Where it runs, and why it is not part of `thinclient-agent`
|
||
|
||
`configs/gesture-control/gesture_pointer.py` is a **separate process**, started as a sway
|
||
`exec_always` via `/usr/local/bin/gesture-control` — the same shape as the eww
|
||
now-playing widget, not a new module inside the `thinclient-agent` daemon. Three reasons,
|
||
all pointing the same way:
|
||
|
||
- **Privacy.** A systemd system unit would hold the camera open from boot to shutdown,
|
||
including while no session exists and there is nothing to point at. Tying the camera's
|
||
lifetime to the session's means "the screen is up" and "the camera is open" cannot
|
||
drift apart.
|
||
- **Blast radius.** `thinclient-agent` is the machine's sole MQTT control surface (see
|
||
the security-boundary note below) and has to stay reliable. A continuous camera read
|
||
plus ML inference is a completely different failure and CPU profile, and a crash in it
|
||
must not be able to take HA control of the room down with it. This is the same call
|
||
`configs/eww/fullscreen-watcher.sh` already makes.
|
||
- **Interpreter.** MediaPipe is PyPI-only and lives in its own venv under
|
||
`/opt/gesture-control` (bookworm's system Python is PEP 668 externally-managed — the
|
||
same situation as `wyoming-satellite`). `thinclient-agent` runs on the system
|
||
interpreter, which cannot import it.
|
||
|
||
It does *not* reimplement input injection: it imports
|
||
`thinclient_agent.input_control.InputControl` (stdlib-only, so it loads fine from inside
|
||
the venv) and calls `click("LEFT")` and `move_relative()`. `move_relative()` is the one
|
||
thing added to that module — continuous pointer control genuinely needs proportional
|
||
deltas rather than the four fixed 20px directions the HA buttons use, and putting it
|
||
there keeps a single ydotool call path with a single dialect detection for the whole
|
||
image. It is clamped to ±200px per step and is **not** wired to any MQTT topic; the
|
||
inbound control surface is unchanged.
|
||
|
||
## Idle photo slideshow
|
||
|
||
After 15 minutes idle, the thin client mounts a read-only SMB share (`gallery-smb` on
|
||
the container host, `ENABLE_GALLERY_SMB` in `setup-container-host.sh`) and cycles
|
||
through whatever images are in it, instead of just blanking the panel. It falls all
|
||
the way back to that original blank-the-panel behaviour — not a broken/blank
|
||
slideshow window — if any of the following is true: `/etc/thinclient-agent/
|
||
gallery-credentials` hasn't been created yet, the share is unreachable, or it's empty.
|
||
A freshly built or offline thin client behaves exactly as it did before this feature.
|
||
|
||
Nothing here is an HA entity — it's a local `swayidle` timeout/resume pair
|
||
(`configs/sway/config`) calling `/usr/local/bin/idle-gallery`, which mounts the share
|
||
with the kiosk user's already-unrestricted `sudo` (see the wayvnc-password section
|
||
above for the same reasoning about physical-access trust) and hands a shuffled
|
||
playlist to `mpv`.
|
||
|
||
**Set this up**: create `/etc/thinclient-agent/gallery-credentials` (`chmod 600`) from
|
||
the `.example` next to it, with the same username/password as
|
||
`GALLERY_SMB_USERNAME`/`GALLERY_SMB_PASSWORD` in `setup-container-host.sh` — this
|
||
project has no way to push a secret from one machine to the other, so both sides are
|
||
set by hand from the same value. Also fill in `GALLERY_SMB_HOST` in
|
||
`build-thin-client-iso.sh` (the container host's LAN IP) before rebuilding.
|
||
|
||
### Weather/clock overlay
|
||
|
||
The slideshow gets a text overlay in the top-right corner: time and date always (no
|
||
network needed — it's just the box's own clock), plus temperature/condition/location
|
||
once that data exists (see below). It never covers the whole panel — a solid card
|
||
over someone's photo would defeat the point of a slideshow — just on-screen text with
|
||
a shadow, photo-frame style. `idle-gallery.sh` owns showing and hiding it, exactly
|
||
when its own slideshow is running; it never appears anywhere else (Steam Link, the
|
||
digest/admin canvases, capture-card viewing, or an ordinary video).
|
||
|
||
Time/date come from `date`(1) via eww's `defpoll` — nothing to configure. Weather
|
||
does **not exist yet on a stock image** — nothing in this repo publishes to it, same
|
||
"needs a real decision, not built against a guess" rule as the digest's household/
|
||
calendar sourcing and the ESP32 voice-display firmware's `weather_entity_id`. It
|
||
reads one retained MQTT topic, `smarthome/weather/current`, a small JSON object:
|
||
|
||
```json
|
||
{"temperature": "8°C", "condition": "Partly cloudy", "location": "Vienna"}
|
||
```
|
||
|
||
Household-wide, not per-thin-client (every thin client's overlay shows the same
|
||
reading) — same reasoning as `smarthome/digest/viewed` in
|
||
`thinclient_agent/main.py`: weather is one household-wide fact, not something scoped
|
||
to whichever room asked. Wire it up with a small Home Assistant automation (nothing
|
||
under this repo builds the HA side, same convention as everywhere else):
|
||
|
||
```yaml
|
||
automation:
|
||
- alias: "Publish weather to thin clients"
|
||
trigger:
|
||
- platform: state
|
||
entity_id: weather.home # your real weather entity
|
||
action:
|
||
- service: mqtt.publish
|
||
data:
|
||
topic: smarthome/weather/current
|
||
retain: true
|
||
payload: >-
|
||
{
|
||
"temperature": "{{ state_attr('weather.home', 'temperature') }}°C",
|
||
"condition": "{{ states('weather.home') }}",
|
||
"location": "Vienna"
|
||
}
|
||
```
|
||
|
||
Until that automation exists (or while MQTT is unreachable), the overlay just shows
|
||
the clock — `configs/eww/weather-json` degrades to `{"available": false}` rather than
|
||
blocking or showing stale data, and the weather line simply doesn't render (see
|
||
`eww.yuck`'s `:visible {weatherinfo.available}`).
|
||
|
||
## Capture-card / receiver-box viewing
|
||
|
||
Any USB or PCIe HDMI/AV capture card plugged into this machine can be picked as a
|
||
video source from Home Assistant — the intent is "use this box like a TV/AVR with
|
||
selectable inputs" for a console, cable box, or anything else that only speaks
|
||
HDMI/AV out. Selecting one switches to `5:capture` and shows it full-screen via
|
||
mpv (`mpv av://v4l2:/dev/videoN`); picking "none" clears it. See
|
||
`thinclient_agent/capture_control.py`.
|
||
|
||
**Detection is dynamic and periodic**, not just at boot: the agent re-scans
|
||
`v4l2-ctl --list-devices` roughly every 20 seconds and republishes the HA select's
|
||
option list when it changes, so a capture card plugged in mid-session shows up on
|
||
its own — no reconnect or restart needed. Each physical device's label includes
|
||
its USB bus path (what `v4l2-ctl` itself reports), which is what keeps two
|
||
identical dongles distinguishable without needing extra disambiguation logic.
|
||
|
||
**The gesture-control camera is never offered as a capture source.** Enumeration
|
||
reads `gesture-config.json`'s `camera_device` and excludes it — but only while
|
||
`"enabled": true`, the same gate `gesture_pointer.py` itself uses to decide
|
||
whether the camera is ever opened at all. `camera_device` defaults to
|
||
`/dev/video0` on every image whether or not gesture control was even built in, so
|
||
the exclusion is deliberately conditional: excluding it unconditionally would
|
||
silently hide a real capture card that happens to enumerate as `/dev/video0` on
|
||
the (default, common) image where gesture control is off, for no privacy benefit
|
||
— there's nothing to protect while that camera is never opened in the first
|
||
place. See "Camera gesture control" above for the invariant this preserves: no HA
|
||
entity can turn that specific camera on, full stop.
|
||
|
||
**Audio is best-effort.** Many cheap USB HDMI-capture dongles carry their
|
||
embedded HDMI audio over a *separate* USB Audio Class interface rather than in
|
||
the V4L2 stream itself — `capture_control.find_audio_card()` tries to match a
|
||
capture device to a sibling ALSA card by shared USB device topology (sysfs, not
|
||
vendor/product IDs) and passes it to mpv as `--external-file=alsa://hw:X,0` if
|
||
found. Video-only playback (not a crash) if nothing matches. **Unverified against
|
||
real hardware** — see the checklist below.
|
||
|
||
## Home Assistant entities
|
||
|
||
`thinclient-agent` publishes MQTT-discovery configs on connect. Under the MQTT
|
||
integration you should get one device per thin client with:
|
||
|
||
- **Show digest canvas** (button) — switches to `2:digest` and reloads Firefox
|
||
- **Digest detail level** (select) — `compact` / `full`
|
||
- **Show admin canvas** (button) — switches to `4:admin` and reloads Firefox at
|
||
`admin-web`'s `canvas.html` (Phase 13, see `../../admin-canvas/README.md`). No
|
||
detail level or any other option — everything it shows is populated out of band by
|
||
`admin-canvas`'s write API, not by this agent. This is the sys-admin-llm's surface
|
||
for on-demand stats/graphics/media (e.g. "show me the kitchen outlet's power
|
||
draw"), as opposed to the digest's scheduled 4x/day synthesis.
|
||
- **Capture source** (select) — "none" plus whatever capture cards are currently
|
||
plugged in (dynamic, re-scanned periodically — see "Capture-card /
|
||
receiver-box viewing" above); switches to `5:capture` and shows the picked one
|
||
full-screen via mpv.
|
||
- **Display** (switch) — the TV's own power, over HDMI-CEC with a Sway DPMS
|
||
fallback. Meant to be driven by room presence; see "Turning the TV off when the
|
||
room is empty" below.
|
||
- **Workspace** (select) — `1:web` / `2:digest` / `3:media` / `4:admin` / `5:capture`
|
||
- **Launch Firefox**, **Launch web browser**, **Launch Steam Link** (buttons)
|
||
- **Playback state** (sensor, with track metadata as attributes), **Volume** (number),
|
||
and play/pause / next / previous / stop (buttons)
|
||
- A **media_player** discovery payload — see the caveat below
|
||
- **Audio output** (select) — WirePlumber sinks by human-readable description; the
|
||
choice is written to `audio-config.json` and survives a reboot
|
||
- **Remote desktop target** (select) plus **Remote desktop connect** / **disconnect**
|
||
(buttons) — outbound RDP/VNC to another machine via Remmina, configured in
|
||
`configs/remote-desktop/rdp-vnc.json`. Distinct from wayvnc, which is inbound.
|
||
- **Type text** (text field with a submit action) and **Pointer up/down/left/right**,
|
||
**Left click**, **Right click** (buttons) — types into, and moves/clicks in,
|
||
whatever window has focus, via `ydotool`. Meant for the HA mobile app when a full
|
||
VNC session is overkill (searching in the kiosk Firefox, dismissing a dialog).
|
||
|
||
A manually-authored Lovelace card grouping the last two groups (audio output, remote
|
||
desktop, and the text/pointer controls) under one "Thin client remote" section reads
|
||
far better in the mobile app than the auto-generated entity list — nothing in this repo
|
||
builds that card, since it is a few lines of YAML per household and not something a
|
||
generic image should assume.
|
||
|
||
A now-playing widget also appears on-screen (not an HA entity — it is local to the
|
||
kiosk display) whenever audio-only media is playing: cover art, track/artist, and
|
||
prev/play-pause/next, via `eww`. It hides completely the moment anything visual is on
|
||
screen — fullscreen video, Steam Link, or an mpv window with a video track — so it
|
||
never draws over a film. See `configs/eww/fullscreen-watcher.sh`.
|
||
|
||
> **Caveat: core Home Assistant's MQTT integration has no `media_player` platform.**
|
||
> The plan calls for a `media_player` entity and the agent publishes that payload, but
|
||
> stock HA ignores it; it is only picked up with the HACS *MQTT Media Player* custom
|
||
> integration installed. The sensor/number/button entities above are published
|
||
> alongside it precisely so transport control works on a plain HA install without that
|
||
> add-on. Verify which you want before wiring up automations.
|
||
|
||
### Security boundary
|
||
|
||
`thinclient_agent/mqtt_discovery.py` is the **entire** inbound control surface of this
|
||
machine. Per Phase 11.4 the LLM never gets a direct network path here — the only chain
|
||
is *LLM tool call → HA service call → MQTT → thinclient-agent*. There is no HTTP
|
||
listener, no websocket server, no exposed Sway IPC socket. New control features belong
|
||
as additional MQTT entities, not as a second listener.
|
||
|
||
Related: `thinclient_agent/digest_canvas.py` does **no** presence, person, or room
|
||
resolution. Home Assistant resolves *who* and *where* (including the "whose digest?"
|
||
disambiguation when several people are in the room) and sends an already-resolved
|
||
request; this agent only shows what it is told to show.
|
||
|
||
`thinclient_agent/admin_canvas.py` (Phase 13) follows the identical pattern, taken
|
||
one step further: its MQTT payload isn't even inspected, since there is nothing
|
||
content-specific for this agent to decide — "Show admin canvas" always switches to
|
||
`4:admin` and opens the same fixed, locally-configured `ADMIN_WEB_URL/canvas.html`.
|
||
All of that page's actual content (stats, charts, images, short clips) is populated
|
||
by a completely separate path — the sys-admin-llm, via an HA tool call, calling
|
||
`admin-canvas`'s own token-gated write API on the container host — that never
|
||
touches this agent or this machine's MQTT surface at all. See
|
||
`../../admin-canvas/README.md` for that API and its own security notes.
|
||
|
||
`thinclient_agent/capture_control.py` follows the same "payload only ever
|
||
selects among already-enumerated values" rule as `audio_control.py`/
|
||
`remote_desktop.py`: the HA select's payload is looked up in the device list
|
||
`list_devices()` already built server-side, and it's *that* lookup's `.path` —
|
||
never the payload — that reaches `capture-view` as an argv element. An unknown
|
||
or since-unplugged selection resolves to "no source," never to acting on
|
||
whatever string HA sent.
|
||
|
||
## Turning the TV off when the room is empty
|
||
|
||
A wall-mounted TV showing a canvas to an empty room is the largest power draw
|
||
this machine is attached to — 60–150 W of lit panel against the thin client's own
|
||
handful of watts. The **Display** switch turns it off on demand, and the intended
|
||
driver is room presence.
|
||
|
||
**How the agent does it** (`display_power.py`): HDMI-CEC first, over the same
|
||
cable that carries the picture — `cec-ctl --to 0 --standby` to sleep the panel,
|
||
`--image-view-on` plus `--active-source` to wake it and claim the input back. No
|
||
network path to the TV, no pairing, no account, and it keeps working with the LAN
|
||
down. Then, always, `swaymsg output <name> power off`, which stops the compositor
|
||
driving pixels — that is the fallback for a set whose CEC is broken or switched
|
||
off, and belt-and-braces on one where it works.
|
||
|
||
`cec-ctl` comes from `v4l-utils`, already in the image's package list. Most TVs
|
||
ship CEC **disabled**; enable it once in the TV's settings, where it will be called
|
||
HDMI-CEC, Bravia Sync, Anynet+, SimpLink, Viera Link or similar. `CEC_DEVICE`,
|
||
`DISPLAY_OUTPUTS` and `DISPLAY_USE_CEC` in the agent's config cover the machine
|
||
with two adapters, several screens, or a panel whose CEC you want left alone.
|
||
|
||
**"Off" means standby, honestly.** A TV in CEC standby still draws roughly half a
|
||
watt — that is what lets it hear the wake. This turns 60–150 W into ~0.5 W; it is
|
||
not a smart plug and does not claim to be.
|
||
|
||
**The presence decision stays in Home Assistant**, where presence already lives —
|
||
this agent only does what it is told, same as every other entity here. A worked
|
||
example, unverified against a running HA like every other HA snippet in this repo:
|
||
|
||
```yaml
|
||
# automations.yaml (excerpt). Replace the entity ids with your own.
|
||
- alias: "Living room TV on when the room is occupied"
|
||
trigger:
|
||
- platform: state
|
||
entity_id: binary_sensor.living_room_occupancy
|
||
to: "on"
|
||
action:
|
||
- service: switch.turn_on
|
||
target:
|
||
entity_id: switch.thinclient_living_room_display
|
||
|
||
- alias: "Living room TV off when the room empties"
|
||
trigger:
|
||
- platform: state
|
||
entity_id: binary_sensor.living_room_occupancy
|
||
to: "off"
|
||
# Long enough that walking to the kitchen for a glass of water does not
|
||
# cycle the panel. A TV that flickers off behind you is worse than one
|
||
# left on, and CEC wake takes a second or two.
|
||
for: "00:05:00"
|
||
condition:
|
||
# Don't black out a film. media_player state comes from the agent's own
|
||
# playback sensor — see "Home Assistant entities" above.
|
||
- condition: not
|
||
conditions:
|
||
- condition: state
|
||
entity_id: sensor.thinclient_living_room_playback_state
|
||
state: "playing"
|
||
action:
|
||
- service: switch.turn_off
|
||
target:
|
||
entity_id: switch.thinclient_living_room_display
|
||
```
|
||
|
||
**An Android TV with no thin client attached** is a Home Assistant question
|
||
rather than one for this repo: pair it with the **Android TV Remote** integration
|
||
and swap the `switch.turn_on`/`turn_off` calls above for
|
||
`media_player.turn_on`/`turn_off` on that entity. Waking one over the network
|
||
needs the TV's own "network standby"/"wake on cast" setting enabled — off by
|
||
default on most sets, and the reason a TV that sleeps fine refuses to wake.
|
||
|
||
## Manual verification still outstanding
|
||
|
||
None of this has been run on hardware. In rough order:
|
||
|
||
1. The ISO builds at all (`lb build` is network-heavy and can fail on mirror hiccups).
|
||
2. greetd lands in Sway with no login prompt, on vt1, with getty@tty1 masked.
|
||
3. wayvnc refuses to start with the sentinel password, and works once one is set.
|
||
4. `thinclient-agent` connects to Mosquitto and the device appears in HA.
|
||
5. mpv playback drives the Playback state sensor via mpv-mpris → playerctl.
|
||
6. Steam Link launches under Xwayland without the wlroots black-screen bug —
|
||
**and that the flatpak app ID `com.valvesoftware.SteamLink` is correct**; it is
|
||
flagged for verification against the live Flathub listing in
|
||
`0400-flatpak-steamlink.hook.chroot`.
|
||
7. Spotify Connect: neither `spotifyd` nor `librespot` is in bookworm main, so
|
||
`0500-spotify-connect.hook.chroot` currently stops at a documented placeholder
|
||
rather than hardcoding a release URL that would rot. Pick an install route and fill
|
||
it in.
|
||
8. `wyoming-satellite` / `wyoming-openwakeword` install from PyPI —
|
||
`0600-voice-satellite.hook.chroot` installs them into a venv under `/opt`
|
||
(bookworm's system Python is PEP 668 externally-managed, and these pull a large
|
||
onnxruntime/numpy tree that has no business overwriting apt-managed versions). If
|
||
the PyPI names turn out to be wrong, the hook comments point at upstream's
|
||
git-clone + `script/setup` install instead.
|
||
9. **Phase 11.10**: power off the container host and confirm the kiosk still boots to a
|
||
usable session and plays local media. `thinclient-agent` uses `connect_async` and
|
||
its unit is deliberately not `After=network-online.target`, so it should never be
|
||
able to stall the session.
|
||
10. `eww` is not in Debian bookworm main — `0800-eww-widget.hook.chroot` tries apt and
|
||
otherwise leaves the now-playing widget absent (it degrades cleanly; nothing else
|
||
depends on it). Pick an install route (cargo, or a release binary) if you want it.
|
||
11. **The Firefox extension IDs in `configs/firefox/policies.json` are unverified.**
|
||
The AMO download-URL slugs (`ublock-origin`, `sponsorblock`) were checked against
|
||
the live listings, but the `ExtensionSettings` *keys* (each extension's internal
|
||
ID, e.g. `uBlock0@raymondhill.net`) are widely-published values reproduced from
|
||
memory, not confirmed against AMO directly. If an extension silently fails to
|
||
force-install, check `about:policies` and correct the key from
|
||
`about:debugging#/runtime/this-firefox` on a manually-installed copy.
|
||
12. `remmina-plugin-rdp`/`remmina-plugin-vnc` package names are assumed, not confirmed
|
||
against bookworm's actual archive — if `apt-get install` for them fails, check
|
||
`apt-cache search remmina-plugin` on the build host and adjust the package list.
|
||
13. ydotool: whether `ydotoold` exists (and therefore which command-line dialect
|
||
applies) depends on the exact bookworm package version — `1000-ydotool.hook.chroot`
|
||
detects this at build time and `input_control.py` detects it again at runtime, but
|
||
neither has been checked against the real packaged version yet. If the HA text/
|
||
pointer controls do nothing, start here: `systemctl status ydotoold` and
|
||
`journalctl -u thinclient-agent`.
|
||
14. Outbound RDP/VNC: fill in real `host`/`port` entries in
|
||
`configs/remote-desktop/rdp-vnc.json` and create
|
||
`/etc/thinclient-agent/remote-desktop-credentials.env` on the booted machine from
|
||
the shipped `.example` (never baked into the image, same handling as the wayvnc
|
||
password) before the **Remote desktop connect** button will reach anything real.
|
||
15. Fill in a real `preferred_sink` in `configs/audio/audio-config.json` (or leave it
|
||
empty and pick one later from the **Audio output** select in HA) — the shipped
|
||
template has no sink configured, which is a supported/safe default, not a gap.
|
||
16. Keyboard layout defaults to German (`KEYBOARD_LAYOUT="de"` in
|
||
`build-thin-client-iso.sh`) — applies to both the console (`/etc/default/keyboard`)
|
||
and Sway (`input type:keyboard { xkb_layout ... }`). Build a separate image with a
|
||
different value for a non-German room; there is no per-room override at runtime.
|
||
If `ENABLE_INSTALLER="true"`, `live-build/config/preseed.cfg` also preseeds
|
||
debian-installer's keyboard step to German (as a default, not a skipped question)
|
||
— unverified against a real installer run, since the normal path never uses it.
|
||
17. **Camera gesture control — nothing about it has been measured on real hardware, and
|
||
the numbers below are estimates, not results.** It is off by default so an
|
||
unverified feature cannot surprise anyone; enable it on a test box before a room.
|
||
- **CPU load.** Google publishes 17.12 ms per frame for the float16 hand-landmarker
|
||
on a Pixel 6 CPU, and a desktop x86 report puts the full Python pipeline at
|
||
20–30 ms/frame and ~16 % of CPU. An Intel N100 is four Gracemont E-cores with AVX2
|
||
and no AVX-512, so expect the *slower* end of that — plan on the inference loop
|
||
eating a meaningful fraction of one core continuously. That is why
|
||
`inference_fps` defaults to `12` rather than the camera's 30: pointer control does
|
||
not need 30 fps and the cost scales almost linearly with it. Measure with
|
||
`top -p "$(pgrep -f gesture_pointer.py)"` and turn `inference_fps` down until it is
|
||
acceptable. **Check it specifically while Steam Link is streaming** — that is the
|
||
one workload on this image that already wants the whole CPU, and if the two cannot
|
||
coexist, that is an argument for gesture control being a per-room feature rather
|
||
than a fix in this code.
|
||
- **Accuracy and false-positive clicks.** The open/fist thresholds
|
||
(`FIST_MIN_CURLED` / `OPEN_MAX_CURLED` in `gesture_pointer.py`) and the
|
||
`fist_hold_seconds` / `click_cooldown_seconds` defaults are reasoned guesses, never
|
||
tested against a real hand at real room distance in real lighting. The failure mode
|
||
that matters is a spurious click, so if it fires by itself, raise
|
||
`fist_hold_seconds` first.
|
||
- **Pointer feel.** `dead_zone` and `pointer_speed` set how twitchy it is. Untuned.
|
||
- **Camera latency.** The loop reads every frame and discards the ones it does not
|
||
analyse, specifically so a V4L2 backlog cannot make the pointer lag the hand. Not
|
||
verified that `CAP_PROP_BUFFERSIZE`/`CAP_PROP_FPS` are honoured by any given
|
||
webcam — many ignore them.
|
||
- **MediaPipe API.** Written against the current **Tasks** API
|
||
(`mediapipe.tasks.python.vision.HandLandmarker`). The old
|
||
`mediapipe.solutions.hands` is not merely deprecated — the `mediapipe.solutions`
|
||
package was *removed* from the wheel around 0.10.31 and is gone in 1.0.0, which is
|
||
also why `1100-gesture-control.hook.chroot` has to download
|
||
`hand_landmarker.task` separately instead of it coming inside the wheel.
|
||
- **Install.** `pip install mediapipe` is unrun here. 1.0.0 ships
|
||
`py3-none-manylinux_2_28_x86_64`; bookworm is glibc 2.36 / Python 3.11, which fits,
|
||
but confirm rather than assume. If it fails the hook stops at a documented
|
||
placeholder and the rest of the image is unaffected.
|
||
18. Maintenance shell: `Super+Shift+Ctrl+M` opens a floating `foot` terminal locally.
|
||
This is a physical-access escape hatch, deliberately outside the MQTT/HA control
|
||
surface described above — see the comment above the keybind in `configs/sway/config`
|
||
for why that is correct rather than an inconsistency.
|
||
19. Idle photo slideshow — none of `idle-gallery.sh`'s CIFS mount options have been
|
||
tried against a real Samba server. If `mount.cifs` rejects `vers=3.0` (an older
|
||
NAS, say) or the `soft,retry=0` combination doesn't fail as fast as intended on a
|
||
truly dead host, adjust the `-o` string there. The `ghcr.io/servercontainers/samba`
|
||
`ACCOUNT_<user>`/`SAMBA_VOLUME_CONFIG_<name>` env-var syntax in
|
||
`setup-container-host.sh` is confirmed against that project's documentation, but
|
||
the container itself was never actually run.
|
||
20. **Capture-card viewing — nothing here has been tried against a real capture
|
||
device.** In particular:
|
||
- `mpv av://v4l2:/dev/videoN` itself: latency, whether `--profile=low-latency
|
||
--untimed --no-cache` is actually the right flag combination for a given
|
||
card, and whether any card needs an explicit `--demuxer-lavf-format`/pixel
|
||
format hint `v4l2-ctl` doesn't surface.
|
||
- The ALSA-audio-matching heuristic in `find_audio_card()` — whether a given
|
||
dongle's video and audio interfaces actually share a sysfs USB device
|
||
ancestor the way assumed, and whether `hw:X,0` is always the right
|
||
subdevice index (some cards expose audio on a non-zero one).
|
||
- `capture-view`'s `--title=thinclient-capture-view` pkill/pgrep scoping has
|
||
only been checked to not match unrelated `mpv` invocations by string
|
||
inspection, not exercised against a real running mpv process tree.
|
||
- Whether `v4l2-ctl --info`'s "Device Caps" parsing correctly picks the real
|
||
capture node (and skips a metadata/extra node) for capture cards other than
|
||
whatever specific chipset a future tester's dongle happens to use — the
|
||
logic is chipset-agnostic in design but only reasoned about, not run
|
||
against real `v4l2-ctl` output from more than one card model.
|
||
21. Weather/clock overlay — the `date`(1)-driven clock/date needs no verification
|
||
beyond confirming the box's own timezone is right, but the MQTT half is
|
||
untested against a real Home Assistant automation: whether `weather-json`
|
||
reconnects cleanly after the container host (and therefore Mosquitto) comes
|
||
back up from being powered off, and whether the overlay's text is legible
|
||
against a bright/high-contrast real photo rather than the dark backgrounds
|
||
assumed while choosing the text-shadow-only styling in `eww.scss`.
|
||
22. **Display power over CEC has never been run against a real TV.** The command
|
||
shapes are from `cec-ctl`'s own documentation, not from a session with a
|
||
panel on the other end. Three things to check on the first set it meets:
|
||
that CEC standby actually darkens it rather than just blanking the picture
|
||
(compare the mains draw, not the screen); that waking it comes back to *this*
|
||
input rather than to whatever it was on before; and that the Sway DPMS half
|
||
does not leave a "no signal" banner glowing on a set that ignored the CEC
|
||
standby. `DISPLAY_USE_CEC=false` is the escape hatch if a TV reacts badly to
|
||
being addressed at all.
|