The_miro
/
SmartestHome
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
SmartestHome/hosts/thin-client/README.md

641 lines
38 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters!

This file contains ambiguous Unicode characters that may be confused with others in your current locale. If your use case is intentional and legitimate, you can safely ignore this warning. Use the Escape button to highlight these characters.

# Sway thin client
Phase 11 of `docs/project-plan.md`. Builds a Debian 12 live ISO for a kiosk media
station: autologin into Sway, remote-controlled by Home Assistant over MQTT and by a
human over wayvnc.
What ends up on the image:
| | |
|---|---|
| Compositor | Sway, no bars, no lock screen, workspaces `1:web` / `2:digest` / `3:media` / `4:admin` / `5:capture` |
| Autologin | greetd, `default_session` straight into `/usr/local/bin/kiosk-session` |
| Remote control (human) | wayvnc on `0.0.0.0:5900`, authentication required |
| Remote control (HA/LLM) | `thinclient-agent`, a systemd service publishing HA MQTT-discovery entities |
| Browser | Firefox ESR in kiosk mode on the digest workspace, pointed at `digest-web` |
| Media | mpv + mpv-mpris, playerctl, PipeWire/WirePlumber |
| Capture-card input ("receiver box") | Any USB/PCIe HDMI capture card plugged into the box, selectable in HA, shown via mpv on `5:capture` — see below |
| Idle-gallery overlay | Clock/date always, weather once `smarthome/weather/current` is published — see below |
| Game streaming | Steam Link (Flathub flatpak) forced onto Xwayland |
| Voice | wyoming-satellite + openWakeWord — **opt-in, off by default** |
| Gesture control | Camera hand tracking (MediaPipe) — **opt-in, off by default, see below** |
## Before you build
> **Two open decisions from `docs/project-plan.md` §4 are still unresolved and block a
> real build + flash:**
>
> - **#7 — no thin-client hardware has been chosen.** Nothing here has been booted on
> real metal. The GPU, audio device, and Steam Link/Xwayland decode performance are
> all unvalidated, and `wyoming-satellite`'s `--mic-command`/`--snd-command` are still
> set to the ALSA `default` device because the microphone is unknown.
> - **#6 — no mic-enabled room list has been chosen.** `ENABLE_VOICE_SATELLITE` is a
> **per-image** setting, not a global one: build one ISO with it `false` for the silent
> rooms and a separate ISO with it `true` for each room that actually has a mic. Do
> not turn it on until that list exists.
Then edit the `# CONFIGURATION` block at the top of
[`tools/build-thin-client-iso.sh`](../../tools/build-thin-client-iso.sh):
| Variable | What to put in it |
|---|---|
| `MQTT_BROKER_HOST` | LAN IP of the container host running Mosquitto (Phase 1) |
| `HA_URL` | Home Assistant URL, e.g. `http://192.168.1.10:8123` |
| `DIGEST_WEB_URL` | The Phase 12 `digest-web` service. Placeholder until that is deployed — the image builds and boots fine without it, the digest workspace just shows a connection error. |
| `ADMIN_WEB_URL` | The Phase 13 `admin-web` service (see `../../admin-canvas/README.md`). Same placeholder handling as `DIGEST_WEB_URL` — and unlike the digest workspace, `4:admin` is never auto-launched at session start anyway, so an unset value just means "Show admin canvas" has nothing to open yet. |
| `KIOSK_USERNAME` | Autologin account name (`kiosk`) |
| `THINCLIENT_NAME` / `IMAGE_HOSTNAME` | Per-room identity; each thin client needs its own |
| `ENABLE_STEAM_LINK` | `true`/`false` |
| `ENABLE_VOICE_SATELLITE` | `false` unless this specific image is for a mic-enabled room |
| `ENABLE_GESTURE_CONTROL` | `false` unless this specific image is for a camera-enabled room. Installs the software only — the camera still stays off. See the privacy section below. |
| `SSH_AUTHORIZED_KEY` | Optional. Password auth is disabled on the image, so without a key the only admin paths are the local console and wayvnc. |
## Build
```sh
sudo -E tools/build-thin-client-iso.sh
```
It installs `live-build` if missing, regenerates
`live-build/config/includes.chroot/` from `configs/` and `agent/`, writes
`/etc/thinclient-agent/config.env` into the image, then runs `lb config && lb build`.
The ISO lands in `live-build/`. The script ends with a numbered list of what to verify
on first boot.
### Directory split
`configs/` and `agent/` are the human-edited, git-tracked source of truth.
`live-build/config/includes.chroot/` is **generated** — it is wiped and rebuilt on every
run and is gitignored. Never hand-edit anything under it; edits there are lost on the
next build.
The only templated token in `configs/` is `@KIOSK_USERNAME@`, substituted by the build
script. Everything the in-chroot hooks need is read from
`/etc/thinclient-agent/config.env`, which live-build copies in (`chroot_local-includes`)
before it runs the hooks (`chroot_local-hooks`) — that ordering is what lets the hooks
be plain scripts with no build-script variables of their own.
## The wayvnc password is not in this repo — set it on first boot
`configs/wayvnc/config` deliberately has no `password=` line. wayvnc only accepts the
password inline, so shipping one would put a live credential for a full remote-control
channel into git. Instead:
- `0300-wayvnc.hook.chroot` writes the sentinel `CHANGEME-SET-ON-FIRST-BOOT` into
`/etc/wayvnc/wayvnc-password` (mode 0600).
- `/usr/local/bin/start-wayvnc` **refuses to launch** while that sentinel is there, so
the failure mode is "no remote access" rather than "an unauthenticated VNC server
listening on the LAN".
On the booted machine:
```sh
sudo sh -c 'openssl rand -base64 24 > /etc/wayvnc/wayvnc-password'
sudo chmod 600 /etc/wayvnc/wayvnc-password
sudo chown kiosk:kiosk /etc/wayvnc/wayvnc-password
swaymsg reload
```
Then connect to `<thin-client-ip>:5900` with username `kiosk`.
## Camera gesture control is a camera in a room — read this before enabling it
> **If you turn this on, a camera continuously captures and analyses video of the room
> for as long as the kiosk session is up.** Not on a trigger, not on a wake word — every
> frame, all day. Frames are processed in memory and nothing is recorded or sent
> anywhere, but that is a property of this code today, not a guarantee the hardware
> gives you: the camera is physically pointed at the room whatever the software does.
>
> **It is off by default and must stay off unless the room has agreed to it.** This is
> the same decision, made the same way, as the microphones: per `docs/project-plan.md`
> Phase 11.8 only the specific rooms chosen for voice interactivity get a mic, and
> `ENABLE_VOICE_SATELLITE` is a per-image build flag precisely so that a silent room's
> ISO physically cannot listen. **Only specific rooms get a gesture-control camera**, and
> `ENABLE_GESTURE_CONTROL` is per-image for exactly the same reason. A shared media
> station in a living room is not a place to switch a camera on opportunistically because
> the feature happened to be available.
>
> There is no HA entity for this and there deliberately never will be — nothing reachable
> over MQTT can turn the camera on. Enabling it takes physical or SSH access to the
> machine.
Open hand moves the pointer, closed fist clicks. It is the same idea as the **Pointer
up/down/left/right** / **Left click** buttons in the HA entity list below — a way to
drive the kiosk without a VNC session — done from the sofa instead of from a phone.
**Hardware: a USB webcam, per camera-enabled room.** Not part of the base thin-client
bill of materials; it is a config-gated addition to the rooms that opt in, exactly like
the USB mic in the mic-enabled rooms. Any ordinary UVC webcam that shows up as a
`/dev/video*` node works; nothing here needs a depth camera or an accelerator.
### Two separate gates, both defaulting to off
| Gate | Where | What it decides |
|---|---|---|
| `ENABLE_GESTURE_CONTROL` | build script, per image | Whether MediaPipe and its ~400 MB dependency tree are installed at all |
| `"enabled"` | `gesture-config.json`, per machine at runtime | Whether the camera is ever **opened** |
The second one is the one that matters. It is `false` even in an image built with the
first one on, so flashing a gesture-capable ISO is not the same act as switching a room's
camera on. `gesture_pointer.py` checks it *before it imports OpenCV or MediaPipe*, so
while it is false the video stack is never loaded and `/dev/video0` is never opened — not
opened and ignored.
To turn it on, on the booted machine:
```sh
v4l2-ctl --list-devices # find the right /dev/videoN
sudo $EDITOR /var/lib/thinclient-agent/gesture-config.json # "enabled": true
swaymsg reload
```
`gesture-config.json` follows the same template/runtime-copy split as `audio-config.json`
and `rdp-vnc.json`: the committed file is baked in read-only at
`/etc/thinclient-agent/`, and `runtime_state.ensure_runtime_copy()` seeds the writable
one under `/var/lib/thinclient-agent/` on first run. Edit the runtime copy — editing the
`/etc` one does nothing, and it is inside a squashfs anyway.
### Where it runs, and why it is not part of `thinclient-agent`
`configs/gesture-control/gesture_pointer.py` is a **separate process**, started as a sway
`exec_always` via `/usr/local/bin/gesture-control` — the same shape as the eww
now-playing widget, not a new module inside the `thinclient-agent` daemon. Three reasons,
all pointing the same way:
- **Privacy.** A systemd system unit would hold the camera open from boot to shutdown,
including while no session exists and there is nothing to point at. Tying the camera's
lifetime to the session's means "the screen is up" and "the camera is open" cannot
drift apart.
- **Blast radius.** `thinclient-agent` is the machine's sole MQTT control surface (see
the security-boundary note below) and has to stay reliable. A continuous camera read
plus ML inference is a completely different failure and CPU profile, and a crash in it
must not be able to take HA control of the room down with it. This is the same call
`configs/eww/fullscreen-watcher.sh` already makes.
- **Interpreter.** MediaPipe is PyPI-only and lives in its own venv under
`/opt/gesture-control` (bookworm's system Python is PEP 668 externally-managed — the
same situation as `wyoming-satellite`). `thinclient-agent` runs on the system
interpreter, which cannot import it.
It does *not* reimplement input injection: it imports
`thinclient_agent.input_control.InputControl` (stdlib-only, so it loads fine from inside
the venv) and calls `click("LEFT")` and `move_relative()`. `move_relative()` is the one
thing added to that module — continuous pointer control genuinely needs proportional
deltas rather than the four fixed 20px directions the HA buttons use, and putting it
there keeps a single ydotool call path with a single dialect detection for the whole
image. It is clamped to ±200px per step and is **not** wired to any MQTT topic; the
inbound control surface is unchanged.
## Idle photo slideshow
After 15 minutes idle, the thin client mounts a read-only SMB share (`gallery-smb` on
the container host, `ENABLE_GALLERY_SMB` in `setup-container-host.sh`) and cycles
through whatever images are in it, instead of just blanking the panel. It falls all
the way back to that original blank-the-panel behaviour — not a broken/blank
slideshow window — if any of the following is true: `/etc/thinclient-agent/
gallery-credentials` hasn't been created yet, the share is unreachable, or it's empty.
A freshly built or offline thin client behaves exactly as it did before this feature.
Nothing here is an HA entity — it's a local `swayidle` timeout/resume pair
(`configs/sway/config`) calling `/usr/local/bin/idle-gallery`, which mounts the share
with the kiosk user's already-unrestricted `sudo` (see the wayvnc-password section
above for the same reasoning about physical-access trust) and hands a shuffled
playlist to `mpv`.
**Set this up**: create `/etc/thinclient-agent/gallery-credentials` (`chmod 600`) from
the `.example` next to it, with the same username/password as
`GALLERY_SMB_USERNAME`/`GALLERY_SMB_PASSWORD` in `setup-container-host.sh` — this
project has no way to push a secret from one machine to the other, so both sides are
set by hand from the same value. Also fill in `GALLERY_SMB_HOST` in
`build-thin-client-iso.sh` (the container host's LAN IP) before rebuilding.
### Weather/clock overlay
The slideshow gets a text overlay in the top-right corner: time and date always (no
network needed — it's just the box's own clock), plus temperature/condition/location
once that data exists (see below). It never covers the whole panel — a solid card
over someone's photo would defeat the point of a slideshow — just on-screen text with
a shadow, photo-frame style. `idle-gallery.sh` owns showing and hiding it, exactly
when its own slideshow is running; it never appears anywhere else (Steam Link, the
digest/admin canvases, capture-card viewing, or an ordinary video).
Time/date come from `date`(1) via eww's `defpoll` — nothing to configure. Weather
does **not exist yet on a stock image** — nothing in this repo publishes to it, same
"needs a real decision, not built against a guess" rule as the digest's household/
calendar sourcing and the ESP32 voice-display firmware's `weather_entity_id`. It
reads one retained MQTT topic, `smarthome/weather/current`, a small JSON object:
```json
{"temperature": "8°C", "condition": "Partly cloudy", "location": "Vienna"}
```
Household-wide, not per-thin-client (every thin client's overlay shows the same
reading) — same reasoning as `smarthome/digest/viewed` in
`thinclient_agent/main.py`: weather is one household-wide fact, not something scoped
to whichever room asked. Wire it up with a small Home Assistant automation (nothing
under this repo builds the HA side, same convention as everywhere else):
```yaml
automation:
- alias: "Publish weather to thin clients"
trigger:
- platform: state
entity_id: weather.home # your real weather entity
action:
- service: mqtt.publish
data:
topic: smarthome/weather/current
retain: true
payload: >-
{
"temperature": "{{ state_attr('weather.home', 'temperature') }}°C",
"condition": "{{ states('weather.home') }}",
"location": "Vienna"
}
```
Until that automation exists (or while MQTT is unreachable), the overlay just shows
the clock — `configs/eww/weather-json` degrades to `{"available": false}` rather than
blocking or showing stale data, and the weather line simply doesn't render (see
`eww.yuck`'s `:visible {weatherinfo.available}`).
## Capture-card / receiver-box viewing
Any USB or PCIe HDMI/AV capture card plugged into this machine can be picked as a
video source from Home Assistant — the intent is "use this box like a TV/AVR with
selectable inputs" for a console, cable box, or anything else that only speaks
HDMI/AV out. Selecting one switches to `5:capture` and shows it full-screen via
mpv (`mpv av://v4l2:/dev/videoN`); picking "none" clears it. See
`thinclient_agent/capture_control.py`.
**Detection is dynamic and periodic**, not just at boot: the agent re-scans
`v4l2-ctl --list-devices` roughly every 20 seconds and republishes the HA select's
option list when it changes, so a capture card plugged in mid-session shows up on
its own — no reconnect or restart needed. Each physical device's label includes
its USB bus path (what `v4l2-ctl` itself reports), which is what keeps two
identical dongles distinguishable without needing extra disambiguation logic.
**The gesture-control camera is never offered as a capture source.** Enumeration
reads `gesture-config.json`'s `camera_device` and excludes it — but only while
`"enabled": true`, the same gate `gesture_pointer.py` itself uses to decide
whether the camera is ever opened at all. `camera_device` defaults to
`/dev/video0` on every image whether or not gesture control was even built in, so
the exclusion is deliberately conditional: excluding it unconditionally would
silently hide a real capture card that happens to enumerate as `/dev/video0` on
the (default, common) image where gesture control is off, for no privacy benefit
— there's nothing to protect while that camera is never opened in the first
place. See "Camera gesture control" above for the invariant this preserves: no HA
entity can turn that specific camera on, full stop.
**Audio is best-effort.** Many cheap USB HDMI-capture dongles carry their
embedded HDMI audio over a *separate* USB Audio Class interface rather than in
the V4L2 stream itself — `capture_control.find_audio_card()` tries to match a
capture device to a sibling ALSA card by shared USB device topology (sysfs, not
vendor/product IDs) and passes it to mpv as `--external-file=alsa://hw:X,0` if
found. Video-only playback (not a crash) if nothing matches. **Unverified against
real hardware** — see the checklist below.
## A wireless USB remote works out of the box
The kind with TV buttons on the front and a small keyboard on the back. To Linux that
dongle is two ordinary HID keyboards — a normal one and a "consumer control" one — so
there is nothing to configure per device: the front buttons arrive as `XF86*` keysyms
and the back keyboard arrives as keys. `configs/sway/config`'s remote-control section
binds the full standardised set:
| Buttons | What they do |
|---|---|
| ▶ ⏸ ⏹ ⏭ ⏮ | `playerctl -p mpv,spotifyd` — play/pause, pause, stop, next, previous |
| ⏪ ⏩ | seek 10s back / 30s forward. Skip, not scan: remotes get *pressed*, not held, and the podcast convention is already in people's fingers |
| Volume, mute, mic mute | the **sink**, not the player — the rocker should move the room's volume whatever is making the noise |
| Channel ± | next/previous workspace. A media station's "channels" are its workspaces, which is the closest honest analogy |
| Home / Back | the media workspace / `back_and_forth` |
| Power, Sleep | **the display, not the machine** — see below |
**Two deliberate choices worth knowing before you remap anything.**
*Plain arrows and Return are not bound, on purpose.* A remote's D-pad and OK send
exactly those, unmodified, and Chromium, mpv and every kiosk page need them — binding
them at the compositor would break scrolling a web page with the remote, which is most
of what the remote is for. Window focus stays on `$mod`+arrows.
*The power button turns the screen off, not the computer.* On a TV that button turns
the picture off; on a thin client that autologins into a kiosk, `poweroff` takes the
room's screen away until somebody walks over to press a physical button. So it runs
`display-toggle`, which is a **toggle** rather than two bindings because with the
outputs dark there is no other way back — Sway is still running and still receiving
keys, so the next press wakes it. A stuck button therefore cannot leave the screen
dark either.
**Remotes vary more than their marketing does.** Run `wev` (or
`sudo libinput debug-events`) from the maintenance shell (`$mod+Shift+Ctrl+m`), press
every button, and add any that comes back with an unbound keysym. A button that reports
*no keysym at all* is one the kernel has no mapping for — that is a udev hwdb entry,
not a Sway binding, and is worth knowing before blaming the config.
## Home Assistant entities
`thinclient-agent` publishes MQTT-discovery configs on connect. Under the MQTT
integration you should get one device per thin client with:
- **Show digest canvas** (button) — switches to `2:digest` and reloads Firefox
- **Digest detail level** (select) — `compact` / `full`
- **Show admin canvas** (button) — switches to `4:admin` and reloads Firefox at
`admin-web`'s `canvas.html` (Phase 13, see `../../admin-canvas/README.md`). No
detail level or any other option — everything it shows is populated out of band by
`admin-canvas`'s write API, not by this agent. This is the sys-admin-llm's surface
for on-demand stats/graphics/media (e.g. "show me the kitchen outlet's power
draw"), as opposed to the digest's scheduled 4x/day synthesis.
- **Capture source** (select) — "none" plus whatever capture cards are currently
plugged in (dynamic, re-scanned periodically — see "Capture-card /
receiver-box viewing" above); switches to `5:capture` and shows the picked one
full-screen via mpv.
- **Display** (switch) — the TV's own power, over HDMI-CEC with a Sway DPMS
fallback. Meant to be driven by room presence; see "Turning the TV off when the
room is empty" below.
- **Workspace** (select) — `1:web` / `2:digest` / `3:media` / `4:admin` / `5:capture`
- **Launch Firefox**, **Launch web browser**, **Launch Steam Link** (buttons)
- **Playback state** (sensor, with track metadata as attributes), **Volume** (number),
and play/pause / next / previous / stop (buttons)
- A **media_player** discovery payload — see the caveat below
- **Audio output** (select) — WirePlumber sinks by human-readable description; the
choice is written to `audio-config.json` and survives a reboot
- **Remote desktop target** (select) plus **Remote desktop connect** / **disconnect**
(buttons) — outbound RDP/VNC to another machine via Remmina, configured in
`configs/remote-desktop/rdp-vnc.json`. Distinct from wayvnc, which is inbound.
- **Type text** (text field with a submit action) and **Pointer up/down/left/right**,
**Left click**, **Right click** (buttons) — types into, and moves/clicks in,
whatever window has focus, via `ydotool`. Meant for the HA mobile app when a full
VNC session is overkill (searching in the kiosk Firefox, dismissing a dialog).
A manually-authored Lovelace card grouping the last two groups (audio output, remote
desktop, and the text/pointer controls) under one "Thin client remote" section reads
far better in the mobile app than the auto-generated entity list — nothing in this repo
builds that card, since it is a few lines of YAML per household and not something a
generic image should assume.
A now-playing widget also appears on-screen (not an HA entity — it is local to the
kiosk display) whenever audio-only media is playing: cover art, track/artist, and
prev/play-pause/next, via `eww`. It hides completely the moment anything visual is on
screen — fullscreen video, Steam Link, or an mpv window with a video track — so it
never draws over a film. See `configs/eww/fullscreen-watcher.sh`.
> **Caveat: core Home Assistant's MQTT integration has no `media_player` platform.**
> The plan calls for a `media_player` entity and the agent publishes that payload, but
> stock HA ignores it; it is only picked up with the HACS *MQTT Media Player* custom
> integration installed. The sensor/number/button entities above are published
> alongside it precisely so transport control works on a plain HA install without that
> add-on. Verify which you want before wiring up automations.
### Security boundary
`thinclient_agent/mqtt_discovery.py` is the **entire** inbound control surface of this
machine. Per Phase 11.4 the LLM never gets a direct network path here — the only chain
is *LLM tool call → HA service call → MQTT → thinclient-agent*. There is no HTTP
listener, no websocket server, no exposed Sway IPC socket. New control features belong
as additional MQTT entities, not as a second listener.
Related: `thinclient_agent/digest_canvas.py` does **no** presence, person, or room
resolution. Home Assistant resolves *who* and *where* (including the "whose digest?"
disambiguation when several people are in the room) and sends an already-resolved
request; this agent only shows what it is told to show.
`thinclient_agent/admin_canvas.py` (Phase 13) follows the identical pattern, taken
one step further: its MQTT payload isn't even inspected, since there is nothing
content-specific for this agent to decide — "Show admin canvas" always switches to
`4:admin` and opens the same fixed, locally-configured `ADMIN_WEB_URL/canvas.html`.
All of that page's actual content (stats, charts, images, short clips) is populated
by a completely separate path — the sys-admin-llm, via an HA tool call, calling
`admin-canvas`'s own token-gated write API on the container host — that never
touches this agent or this machine's MQTT surface at all. See
`../../admin-canvas/README.md` for that API and its own security notes.
`thinclient_agent/capture_control.py` follows the same "payload only ever
selects among already-enumerated values" rule as `audio_control.py`/
`remote_desktop.py`: the HA select's payload is looked up in the device list
`list_devices()` already built server-side, and it's *that* lookup's `.path` —
never the payload — that reaches `capture-view` as an argv element. An unknown
or since-unplugged selection resolves to "no source," never to acting on
whatever string HA sent.
## Turning the TV off when the room is empty
A wall-mounted TV showing a canvas to an empty room is the largest power draw
this machine is attached to — 60–150 W of lit panel against the thin client's own
handful of watts. The **Display** switch turns it off on demand, and the intended
driver is room presence.
**How the agent does it** (`display_power.py`): HDMI-CEC first, over the same
cable that carries the picture — `cec-ctl --to 0 --standby` to sleep the panel,
`--image-view-on` plus `--active-source` to wake it and claim the input back. No
network path to the TV, no pairing, no account, and it keeps working with the LAN
down. Then, always, `swaymsg output <name> power off`, which stops the compositor
driving pixels — that is the fallback for a set whose CEC is broken or switched
off, and belt-and-braces on one where it works.
`cec-ctl` comes from `v4l-utils`, already in the image's package list. Most TVs
ship CEC **disabled**; enable it once in the TV's settings, where it will be called
HDMI-CEC, Bravia Sync, Anynet+, SimpLink, Viera Link or similar. `CEC_DEVICE`,
`DISPLAY_OUTPUTS` and `DISPLAY_USE_CEC` in the agent's config cover the machine
with two adapters, several screens, or a panel whose CEC you want left alone.
**"Off" means standby, honestly.** A TV in CEC standby still draws roughly half a
watt — that is what lets it hear the wake. This turns 60–150 W into ~0.5 W; it is
not a smart plug and does not claim to be.
**The presence decision stays in Home Assistant**, where presence already lives —
this agent only does what it is told, same as every other entity here. A worked
example, unverified against a running HA like every other HA snippet in this repo:
```yaml
# automations.yaml (excerpt). Replace the entity ids with your own.
- alias: "Living room TV on when the room is occupied"
trigger:
- platform: state
entity_id: binary_sensor.living_room_occupancy
to: "on"
action:
- service: switch.turn_on
target:
entity_id: switch.thinclient_living_room_display
- alias: "Living room TV off when the room empties"
trigger:
- platform: state
entity_id: binary_sensor.living_room_occupancy
to: "off"
# Long enough that walking to the kitchen for a glass of water does not
# cycle the panel. A TV that flickers off behind you is worse than one
# left on, and CEC wake takes a second or two.
for: "00:05:00"
condition:
# Don't black out a film. media_player state comes from the agent's own
# playback sensor — see "Home Assistant entities" above.
- condition: not
conditions:
- condition: state
entity_id: sensor.thinclient_living_room_playback_state
state: "playing"
action:
- service: switch.turn_off
target:
entity_id: switch.thinclient_living_room_display
```
**An Android TV with no thin client attached** is a Home Assistant question
rather than one for this repo: pair it with the **Android TV Remote** integration
and swap the `switch.turn_on`/`turn_off` calls above for
`media_player.turn_on`/`turn_off` on that entity. Waking one over the network
needs the TV's own "network standby"/"wake on cast" setting enabled — off by
default on most sets, and the reason a TV that sleeps fine refuses to wake.
## Manual verification still outstanding
0. **The remote.** No remote has been plugged into anything — the keysym list above is
the standardised set, not one read off a specific device. Expect one or two buttons
on any given remote to report something unbound (or nothing at all); `wev` from the
maintenance shell is the two-minute check, and `display-toggle`'s grep for
`'"dpms": true'` in `swaymsg -t get_outputs` is worth confirming against the Sway
version this image actually ships, since that field's spelling is the one thing
that would make the power button silently do nothing.
None of this has been run on hardware. In rough order:
1. The ISO builds at all (`lb build` is network-heavy and can fail on mirror hiccups).
2. greetd lands in Sway with no login prompt, on vt1, with getty@tty1 masked.
3. wayvnc refuses to start with the sentinel password, and works once one is set.
4. `thinclient-agent` connects to Mosquitto and the device appears in HA.
5. mpv playback drives the Playback state sensor via mpv-mpris → playerctl.
6. Steam Link launches under Xwayland without the wlroots black-screen bug —
**and that the flatpak app ID `com.valvesoftware.SteamLink` is correct**; it is
flagged for verification against the live Flathub listing in
`0400-flatpak-steamlink.hook.chroot`.
7. Spotify Connect: neither `spotifyd` nor `librespot` is in bookworm main, so
`0500-spotify-connect.hook.chroot` currently stops at a documented placeholder
rather than hardcoding a release URL that would rot. Pick an install route and fill
it in.
8. `wyoming-satellite` / `wyoming-openwakeword` install from PyPI —
`0600-voice-satellite.hook.chroot` installs them into a venv under `/opt`
(bookworm's system Python is PEP 668 externally-managed, and these pull a large
onnxruntime/numpy tree that has no business overwriting apt-managed versions). If
the PyPI names turn out to be wrong, the hook comments point at upstream's
git-clone + `script/setup` install instead.
9. **Phase 11.10**: power off the container host and confirm the kiosk still boots to a
usable session and plays local media. `thinclient-agent` uses `connect_async` and
its unit is deliberately not `After=network-online.target`, so it should never be
able to stall the session.
10. `eww` is not in Debian bookworm main — `0800-eww-widget.hook.chroot` tries apt and
otherwise leaves the now-playing widget absent (it degrades cleanly; nothing else
depends on it). Pick an install route (cargo, or a release binary) if you want it.
11. **The Firefox extension IDs in `configs/firefox/policies.json` are unverified.**
The AMO download-URL slugs (`ublock-origin`, `sponsorblock`) were checked against
the live listings, but the `ExtensionSettings` *keys* (each extension's internal
ID, e.g. `uBlock0@raymondhill.net`) are widely-published values reproduced from
memory, not confirmed against AMO directly. If an extension silently fails to
force-install, check `about:policies` and correct the key from
`about:debugging#/runtime/this-firefox` on a manually-installed copy.
12. `remmina-plugin-rdp`/`remmina-plugin-vnc` package names are assumed, not confirmed
against bookworm's actual archive — if `apt-get install` for them fails, check
`apt-cache search remmina-plugin` on the build host and adjust the package list.
13. ydotool: whether `ydotoold` exists (and therefore which command-line dialect
applies) depends on the exact bookworm package version — `1000-ydotool.hook.chroot`
detects this at build time and `input_control.py` detects it again at runtime, but
neither has been checked against the real packaged version yet. If the HA text/
pointer controls do nothing, start here: `systemctl status ydotoold` and
`journalctl -u thinclient-agent`.
14. Outbound RDP/VNC: fill in real `host`/`port` entries in
`configs/remote-desktop/rdp-vnc.json` and create
`/etc/thinclient-agent/remote-desktop-credentials.env` on the booted machine from
the shipped `.example` (never baked into the image, same handling as the wayvnc
password) before the **Remote desktop connect** button will reach anything real.
15. Fill in a real `preferred_sink` in `configs/audio/audio-config.json` (or leave it
empty and pick one later from the **Audio output** select in HA) — the shipped
template has no sink configured, which is a supported/safe default, not a gap.
16. Keyboard layout defaults to German (`KEYBOARD_LAYOUT="de"` in
`build-thin-client-iso.sh`) — applies to both the console (`/etc/default/keyboard`)
and Sway (`input type:keyboard { xkb_layout ... }`). Build a separate image with a
different value for a non-German room; there is no per-room override at runtime.
If `ENABLE_INSTALLER="true"`, `live-build/config/preseed.cfg` also preseeds
debian-installer's keyboard step to German (as a default, not a skipped question)
— unverified against a real installer run, since the normal path never uses it.
17. **Camera gesture control — nothing about it has been measured on real hardware, and
the numbers below are estimates, not results.** It is off by default so an
unverified feature cannot surprise anyone; enable it on a test box before a room.
- **CPU load.** Google publishes 17.12 ms per frame for the float16 hand-landmarker
on a Pixel 6 CPU, and a desktop x86 report puts the full Python pipeline at
20–30 ms/frame and ~16 % of CPU. An Intel N100 is four Gracemont E-cores with AVX2
and no AVX-512, so expect the *slower* end of that — plan on the inference loop
eating a meaningful fraction of one core continuously. That is why
`inference_fps` defaults to `12` rather than the camera's 30: pointer control does
not need 30 fps and the cost scales almost linearly with it. Measure with
`top -p "$(pgrep -f gesture_pointer.py)"` and turn `inference_fps` down until it is
acceptable. **Check it specifically while Steam Link is streaming** — that is the
one workload on this image that already wants the whole CPU, and if the two cannot
coexist, that is an argument for gesture control being a per-room feature rather
than a fix in this code.
- **Accuracy and false-positive clicks.** The open/fist thresholds
(`FIST_MIN_CURLED` / `OPEN_MAX_CURLED` in `gesture_pointer.py`) and the
`fist_hold_seconds` / `click_cooldown_seconds` defaults are reasoned guesses, never
tested against a real hand at real room distance in real lighting. The failure mode
that matters is a spurious click, so if it fires by itself, raise
`fist_hold_seconds` first.
- **Pointer feel.** `dead_zone` and `pointer_speed` set how twitchy it is. Untuned.
- **Camera latency.** The loop reads every frame and discards the ones it does not
analyse, specifically so a V4L2 backlog cannot make the pointer lag the hand. Not
verified that `CAP_PROP_BUFFERSIZE`/`CAP_PROP_FPS` are honoured by any given
webcam — many ignore them.
- **MediaPipe API.** Written against the current **Tasks** API
(`mediapipe.tasks.python.vision.HandLandmarker`). The old
`mediapipe.solutions.hands` is not merely deprecated — the `mediapipe.solutions`
package was *removed* from the wheel around 0.10.31 and is gone in 1.0.0, which is
also why `1100-gesture-control.hook.chroot` has to download
`hand_landmarker.task` separately instead of it coming inside the wheel.
- **Install.** `pip install mediapipe` is unrun here. 1.0.0 ships
`py3-none-manylinux_2_28_x86_64`; bookworm is glibc 2.36 / Python 3.11, which fits,
but confirm rather than assume. If it fails the hook stops at a documented
placeholder and the rest of the image is unaffected.
18. Maintenance shell: `Super+Shift+Ctrl+M` opens a floating `foot` terminal locally.
This is a physical-access escape hatch, deliberately outside the MQTT/HA control
surface described above — see the comment above the keybind in `configs/sway/config`
for why that is correct rather than an inconsistency.
19. Idle photo slideshow — none of `idle-gallery.sh`'s CIFS mount options have been
tried against a real Samba server. If `mount.cifs` rejects `vers=3.0` (an older
NAS, say) or the `soft,retry=0` combination doesn't fail as fast as intended on a
truly dead host, adjust the `-o` string there. The `ghcr.io/servercontainers/samba`
`ACCOUNT_<user>`/`SAMBA_VOLUME_CONFIG_<name>` env-var syntax in
`setup-container-host.sh` is confirmed against that project's documentation, but
the container itself was never actually run.
20. **Capture-card viewing — nothing here has been tried against a real capture
device.** In particular:
- `mpv av://v4l2:/dev/videoN` itself: latency, whether `--profile=low-latency
--untimed --no-cache` is actually the right flag combination for a given
card, and whether any card needs an explicit `--demuxer-lavf-format`/pixel
format hint `v4l2-ctl` doesn't surface.
- The ALSA-audio-matching heuristic in `find_audio_card()` — whether a given
dongle's video and audio interfaces actually share a sysfs USB device
ancestor the way assumed, and whether `hw:X,0` is always the right
subdevice index (some cards expose audio on a non-zero one).
- `capture-view`'s `--title=thinclient-capture-view` pkill/pgrep scoping has
only been checked to not match unrelated `mpv` invocations by string
inspection, not exercised against a real running mpv process tree.
- Whether `v4l2-ctl --info`'s "Device Caps" parsing correctly picks the real
capture node (and skips a metadata/extra node) for capture cards other than
whatever specific chipset a future tester's dongle happens to use — the
logic is chipset-agnostic in design but only reasoned about, not run
against real `v4l2-ctl` output from more than one card model.
21. Weather/clock overlay — the `date`(1)-driven clock/date needs no verification
beyond confirming the box's own timezone is right, but the MQTT half is
untested against a real Home Assistant automation: whether `weather-json`
reconnects cleanly after the container host (and therefore Mosquitto) comes
back up from being powered off, and whether the overlay's text is legible
against a bright/high-contrast real photo rather than the dark backgrounds
assumed while choosing the text-shadow-only styling in `eww.scss`.
22. **Display power over CEC has never been run against a real TV.** The command
shapes are from `cec-ctl`'s own documentation, not from a session with a
panel on the other end. Three things to check on the first set it meets:
that CEC standby actually darkens it rather than just blanking the picture
(compare the mains draw, not the screen); that waking it comes back to *this*
input rather than to whatever it was on before; and that the Sway DPMS half
does not leave a "no signal" banner glowing on a set that ignored the CEC
standby. `DISPLAY_USE_CEC=false` is the escape hatch if a TV reacts badly to
being addressed at all.
Reference in New Issue View Git Blame Copy Permalink