The_miro
/
SmartestHome
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Compare commits

base: The_miro:main
Branches Tags
The_miro:main
The_miro:digest-per-person-and-agendas
..
compare: The_miro:digest-per-person-and-agendas
Branches Tags
The_miro:main
The_miro:digest-per-person-and-agendas

No commits in common. "main" and "digest-per-person-and-agendas" have entirely different histories.
main ... digest-per

153 changed files with 415 additions and 21656 deletions
Show Stats Expand all files Collapse all files

8
.gitignore vendored
View File

@ -121,11 +121,3 @@ hosts/audio-endpoint/rpi-image-gen/config/generated-*.yaml
tokens.txt
# Caddy's exported root CA (public, but not something to publish casually).
proxy/ca/
# stream-dock/generate.py's output. led-sync.env holds the Home Assistant token, and
# bindings.md names every light entity in the room — regenerate it, never commit it.
stream-dock/generated/
# mic-follow/generate.py's output. Each client.json holds the MQTT password, and the
# identity-toggles env line names every switch — regenerate it, never commit it.
mic-follow/generated/

192
CoreSystemConfig.json.template
View File

@ -66,8 +66,6 @@
"portainer": true,
"mealie": false,
"gallery_smb": false,
"photos_web": false,
"workshop": false,
"music_assistant": false,
"backups": false
}
@ -111,9 +109,6 @@
"transit": 8099,
"otp": 8100,
"music_assistant": 8101,
"workshop": 8102,
"workshop_web": 8103,
"photos_web": 2283,
"ollama": 11434,
"proxy_http": 80,
"proxy_https": 443
@ -127,77 +122,22 @@
"mqtt_username": "",
"mqtt_password": "",
"ha_token": "",
"_opnsense_keys": "One entry per firewall in the opnsense list above, keyed by its `name`. A single-firewall household writes {\"main\": {...}} and is done. Keyed rather than positional because a list that has to line up with another list by index is a bug waiting for the day somebody reorders one of them.",
"opnsense_keys": {
"main": { "api_key": "", "api_secret": "" }
},
"_checkmk": "A CheckMK automation user with the read-only Guest role, and its AUTOMATION SECRET (Setup -> Users -> the user -> Automation secret) \u2014 not its login password.",
"checkmk_username": "",
"checkmk_secret": "",
"_freeipa_bind_password": "The password for freeipa.bind_dn. Prefer a dedicated service account with READ-ONLY access to the household subtree \u2014 the mirror never writes to the directory, so a bind that can is a grant with no matching use.",
"freeipa_bind_password": "",
"_workshop_token": "Required when container_host.enable.workshop is true. openssl rand -hex 32",
"workshop_token": "",
"_gitea": "Optional, and only for the workshop assistant's own repositories. Gitea -> Settings -> Applications -> Generate New Token, scope write:repository. STRONGLY PREFER a dedicated 'workshop-bot' user scoped to one organisation over a token on your own account: this token can create repositories, and the blast radius of a leaked env file should be one org of generated repos rather than everything you own. The assistant commits and pushes but never rewrites history, and repo deletion is not implemented at all \u2014 see workshop/README.md.",
"gitea_url": "",
"gitea_token": "",
"gitea_owner": "",
"opnsense_api_key": "",
"opnsense_api_secret": "",
"ssh_authorized_key": "",
"kiosk_password": "",
"admin_password_hash": ""
},
"opnsense": [
{
"_comment": "EVERY OPNsense firewall in the household, as a list. It was a single object until more than one firewall became plausible; a list is the shape that does not need changing again, and a one-firewall household just has one entry. `name` distinguishes them everywhere downstream — in the digest's network section, in the workshop health table, and in the alert text itself — so 'the IDS is quiet' can never silently mean 'one of the two is quiet'. Used by digest-engine's network digest (GET /api/ids/service/status and POST /api/ids/service/query_alerts \u2014 two read endpoints, nothing else, ever) and by workshop's health poller. A build writes this block plus the secrets into IDSconf.json; long-form field docs live in digest-engine/IDSconf.json.example. An empty list skips that file entirely. TWO THINGS THIS DOES NOT DO: it does not enable Suricata (Services -> Intrusion Detection on the firewall, then download a ruleset), and it does not enable the ingest (ENABLE_OPNSENSE_IDS_INGEST=true in digest-engine.env). SCOPE EACH API KEY: its own OPNsense user with only the 'Services: Intrusion Detection' privilege \u2014 which still covers api/ids/* including start/stop, because OPNsense ACLs are page-level, so the read-only guarantee comes from this project calling exactly two endpoints and not from the firewall enforcing it.",
"name": "main",
"base_url": "",
"verify_tls": true,
"interfaces": [],
"max_alerts_scanned": 5000,
"top_signatures": 8,
"top_hosts": 5,
"packet_capture_reference": ""
}
],
"checkmk": {
"_comment": "An existing CheckMK server, polled read-only for host/service state. Feeds two places: digest-engine's network section (so a failing disk shows up in the quarter-daily digest) and workshop's own infra_status table (so the workshop display can overlay system health on the gallery). Nothing here ever acknowledges, downtimes or reschedules anything \u2014 the API user only needs to read. site is the CheckMK site name, which is the path segment in every URL: http://<host>/<site>/check_mk/api/1.0/...",
"_credentials": "CheckMK: Setup -> Users -> add a user, Roles = 'Guest' (read-only), then its 'Automation secret' \u2014 NOT its login password. Put the username and secret in secrets.checkmk_username / secrets.checkmk_secret. A Guest-role automation user cannot change anything, which is the actual boundary here rather than a promise about which endpoints get called.",
"opnsense": {
"_comment": "The household's existing OPNsense firewall. Used by digest-engine's network digest, which pulls a Suricata intrusion-detection summary from it (GET /api/ids/service/status and POST /api/ids/service/query_alerts — two read endpoints, nothing else, ever). A build writes this block plus the secrets above into the container host's IDSconf.json; the long-form documentation for every field lives in digest-engine/IDSconf.json.example. Leave base_url empty to skip that file entirely. TWO THINGS THIS DOES NOT DO: it does not enable Suricata (do that at Services -> Intrusion Detection on the firewall, then download a ruleset), and it does not enable the ingest (set ENABLE_OPNSENSE_IDS_INGEST=true in digest-engine.env). SCOPE THE API KEY: give it its own OPNsense user with only the 'Services: Intrusion Detection' privilege — that privilege still covers api/ids/* including start/stop, because OPNsense ACLs are page-level, so the read-only guarantee comes from digest-engine calling exactly two endpoints and not from the firewall enforcing it.",
"base_url": "",
"site": "cmk",
"verify_tls": true,
"_only_problems": "true keeps the payload to hosts and services that are not OK, which is what both consumers want; false pulls everything and is mostly useful once, to see what the site knows about.",
"only_problems": true,
"max_rows": 200
},
"identity_provider": {
"_comment": "An EXTERNAL identity provider (Keycloak) in front of the household's web surfaces. NOTHING IN THIS REPO IMPLEMENTS SSO YET \u2014 this block exists so the decision and its values are recorded in one place before the work happens, and so the proxy config can be generated from it when it does. Leave issuer_url empty to skip it entirely, which is the current default and the tested path.",
"_pairs_with_freeipa": "KEYCLOAK AND FREEIPA ARE ONE DECISION, NEVER TWO. Keycloak federates FreeIPA as its user store; it is not a place people are created. Configuring Keycloak without FreeIPA would mean a second, parallel set of household accounts \u2014 which is the specific outcome having a directory exists to prevent \u2014 so the validator refuses either one alone. Group membership comes from FreeIPA and rides through Keycloak as a claim; roles are never assigned in Keycloak itself, or the directory stops being the answer to 'who is in this household'. WHAT IT WOULD PROTECT: the proxy-fronted web UIs (Home Assistant, the photo frontend, workshop, identity's admin panel). WHAT IT MUST NEVER PROTECT: the kiosk-to-service APIs \u2014 pantry-vision, identity's /presence, workshop's own API \u2014 which are bearer-token gated because a wall panel cannot complete an interactive login. Putting an OIDC redirect in front of those turns every kiosk into a dead screen.",
"issuer_url": "",
"_realm": "Keycloak realm name; part of the issuer URL too, kept separately because the proxy config and any client library both want it on its own.",
"realm": "smarthome",
"client_id": "smarthome-proxy",
"_protected_hosts": "Which proxied hostnames would sit behind SSO. Advisory until the work is done \u2014 see the _comment above.",
"protected_hosts": []
},
"freeipa": {
"_comment": "An existing FreeIPA domain, mirrored INTO identity as a source of people and group memberships. NOT IMPLEMENTED YET \u2014 this block records the decision and its values ahead of the work, like identity_provider above. Leave server empty to skip it, which is the current default and the only tested path.",
"_direction": "ONE WAY, FreeIPA -> identity, and it must stay that way. identity holds household facts FreeIPA has no opinion about (BLE identifiers, chore reminder style, digest preferences, a colour) and writing any of that back would make a directory serving real logins into a store of smart-home preferences. A mirrored person is matched on uid and their name/group memberships are refreshed; everything else identity knows about them is left alone.",
"_groups": "Group membership maps to roles here. `chore_exempt_group` members are dropped from the chore rotation exactly as the per-person flag does today \u2014 which means a FreeIPA group can grant an exemption but must never remove one somebody set by hand, or a directory sync would silently re-enrol a guest. `household_group` is who gets mirrored at all: without it every service account in the directory becomes a household member.",
"_naming": "EVERY group this project reads is named USR_HA_<parameter>, matching the field name after the _group suffix is dropped: household_group -> USR_HA_household, chore_exempt_group -> USR_HA_chore_exempt, admin_group -> USR_HA_admins. The prefix is what makes a directory shared with other systems auditable \u2014 'which groups does the smart home read?' is answerable with one filter instead of by reading this file. The validator warns on anything that does not follow it rather than erroring, since an existing directory may already have its own convention and renaming groups in FreeIPA is not a thing a config file should force.",
"server": "",
"domain": "",
"base_dn": "",
"bind_dn": "",
"verify_tls": true,
"household_group": "USR_HA_household",
"chore_exempt_group": "USR_HA_chore_exempt",
"admin_group": "USR_HA_admins",
"_sync_interval_minutes": "How often the mirror runs, once it exists. Directory changes are not urgent \u2014 somebody joining the household is a thing you also tell the door panel about.",
"sync_interval_minutes": 60
"interfaces": [],
"max_alerts_scanned": 5000,
"top_signatures": 8,
"top_hosts": 5,
"packet_capture_reference": ""
},
"proxy": {
@ -214,89 +154,11 @@
"wake_word": "ok_nabu"
},
"mic_follow": {
"_comment": "Follow-me microphone switching. A person walks out of the room their PC is in; the locator says which room they are in now; that machine's live microphone switches to one that can hear them, and back again when they return. Built for a voice call that has to survive a smoke break, and written per CLIENT so a second person with a second desktop is a second entry here and nothing else.",
"_how_it_decides": "Home Assistant owns the decision and the desktop agent owns the mechanism. The generated automations watch presence_entity and, ONLY while that client's Follow-me switch is on, set its Mic input select. Switch off is not 'ignore me' — it is an active guarantee that the machine is on its own desk microphone.",
"_locator_honesty": "presence_entity must be an entity whose STATE is an HA area_id. The only identity-bearing room-level source in this household is BLE (Bermuda, surfaced by identity/); RuView tells you a room is occupied but not by whom, and Frigate recognises faces at the door, not per room. So RuView/Frigate can corroborate but cannot drive this on their own — see mic-follow/README.md section 3 before trusting the room this switches on.",
"enabled": false,
"clients": [
{
"_comment": "One desktop machine and the one person it follows. node_id must be unique, lowercase, and is what every generated entity_id contains.",
"node_id": "amir_desktop",
"friendly_name": "Amir's desktop",
"person": "amir",
"room": "amirs_room",
"presence_entity": "sensor.amir_ble_area",
"_desk_source": "Any part of a PipeWire source name or description — run `mic-follow/desktop_agent.py --list-sources` on that machine to see the real strings. This is the microphone the client returns to whenever follow-me is off.",
"desk_source": "MV6",
"_move_streams": "Applications whose ALREADY-RUNNING capture stream gets moved as well as the default being changed. Anyone with a studio mic has picked it explicitly in Discord, and an explicitly-picked device does not follow the default.",
"move_streams": ["Discord"],
"_dwell": "Seconds the person has to be in a room before the microphone follows them. Leaving is slower than returning on purpose: a walk past the door should not move your microphone, but sitting back down should give you your good one back quickly.",
"dwell_seconds": 20,
"return_dwell_seconds": 5,
"_on_unknown_room": "hold | desk. What happens in a room with no microphone configured. 'hold' keeps the last one (you might be walking through); 'desk' is the honest one (nothing in that room can hear you).",
"on_unknown_room": "hold",
"reconcile_seconds": 10,
"_sources": "One per room this client can be heard in. A source is either LOCAL (a microphone plugged into this machine — a wireless clip-on receiver, say: give `source` a pattern and nothing else) or REMOTE (a microphone on another machine, streamed here — give it a `remote` block and mic-follow generates the rest). start_command/stop_command are the escape hatch for anything neither of those covers; the STOP hook is the important one, because a room microphone still streaming after the switch left it is a hot mic in somebody's flat.",
"_remote": "A microphone on another machine in the household — for the Loggia, the touch panel that is already in there. Audio travels over SSH (that panel already runs sshd as its admin path and already trusts this desktop's key), so there is no new listening service, no ACL to maintain, and audio exists ONLY while the stream is running — a property of the transport rather than a promise. host/user are that machine; mic_source is a pattern for ITS microphone, empty meaning its own default; latency_ms buys smoothness over wifi at the cost of delay. See mic-follow/README.md section 6.",
"sources": [
{
"room": "loggia",
"source": "micfollow_loggia",
"description": "Loggia panel mic",
"remote": {
"host": "touch-panel-loggia",
"user": "kiosk",
"mic_source": "",
"latency_ms": 40,
"rate": 48000,
"channels": 1,
"ssh_key": ""
}
}
]
}
]
},
"stream_dock": {
"_comment": "A desk-side Stream Dock (MiraBox N4 Pro / Ajazz AKP05 family: 10 LCD keys, 4 RGB-lit rotary encoders) driving ONE room's colour lamps through OpenDeck. This block generates the dock's key/dial bindings, its knob-LED colours and the LED sync service's environment. It does not build an image and nothing here runs on the container host — the dock hangs off a desktop machine. See stream-dock/README.md.",
"_room": "An HA area_id, the same vocabulary as every kiosk (docs/rooms-and-endpoints.md). It labels the generated bindings and names the systemd unit; `lights` is what actually gets controlled, because a dial has to name entities, not an area.",
"enabled": false,
"room": "living_room",
"lights": ["light.living_room_lamp"],
"_steps": "How far one encoder detent moves things. rgb_step is in 0-255 channel units (8 = 32 detents end to end); brightness_step_pct is percentage points. tick_bucket_ms is the plugin's own aggregation window — ticks inside it are summed into one service call, so spinning the dial fast does not queue fifty calls at Home Assistant.",
"rgb_step": 8,
"brightness_step_pct": 5,
"tick_bucket_ms": 120,
"_knob_leds": "The four encoder rings. r/g/b each glow their own channel's current value in their own colour; the fourth glows in the colour the room is actually emitting (rgb scaled by brightness). THIS NEEDS THE DEVICE PLUGIN TO RE-READ leds.toml AT RUNTIME, WHICH IT DOES NOT DO TODAY — the sync service writes the file correctly and then runs apply_command, which is the hook where that gap gets closed. Read stream-dock/README.md section 5 before relying on it.",
"knob_leds": {
"enabled": true,
"brightness": 100,
"min_channel_led": 0,
"poll_seconds": 2.0,
"debounce_ms": 400,
"config_path": "",
"_apply": "How a fresh leds.toml gets in front of the device plugin, which reads that file at startup and (as far as anyone has checked) not again. apply_strategy is what stream-dock/apply-leds.sh does after each write: 'none' writes the file and stops there; 'signal' sends SIGHUP to the plugin process; 'restart-plugin' kills it and lets OpenDeck respawn it; 'restart-opendeck' restarts the whole app. RUN `stream-dock/apply-leds.sh --probe` WITH THE DOCK PLUGGED IN AND SET WHAT IT TELLS YOU — its first test is whether the plugin already picks the file up on its own, in which case 'none' is the right answer and the rings are live for free. apply_command overrides all of it with a command of your own; leave it empty to use the strategy.",
"apply_strategy": "none",
"apply_command": "",
"_apply_pacing": "apply_min_interval_seconds rate-limits the strategy: a spun dial changes the colour on every service call, and 'restart-plugin' at that rate would re-initialise the device continuously. The last state is never dropped — it is applied once the interval elapses. plugin_process_pattern is what pgrep matches to find the device plugin; the probe prints the candidates on your machine.",
"apply_min_interval_seconds": 2.0,
"plugin_process_pattern": "akp05",
"_layer_gate": "The dock's lighting controls live on their own OpenDeck layer, and the rings belong to whatever layer is showing. layer_gate_command is run before every LED update: exit 0 means the lighting layer is up and the rings are ours to drive, exit 1 means it is not, and any other outcome (or a command that cannot run) is treated as 'not ours' — a service that cannot tell which layer is showing must not paint over another one's rings. Empty means no gate: drive the rings always. stream-dock/layer-active.sh is a starting implementation, and it needs verifying against a real OpenDeck install — see stream-dock/README.md section 6.",
"layer_gate_command": "",
"_idle": "What the rings do when the lighting layer is NOT showing. Defaults to the CyberQueer palette from ~/Dotfiles/colors.conf — COLOR_HIGHLIGHT, COLOR_DARK, COLOR_RED — chasing across the four rings, so an idle dock matches the rest of the desktop instead of sitting on the last lamp colour it happened to see. Bare 6-digit hex, same format as colors.conf. An empty list leaves the rings untouched instead. idle_cycle_seconds is how long each step holds; 0 holds the first colour forever, which is what you want when apply_command is expensive, because every step of the chase costs one apply.",
"idle_colors": ["E40046", "5018DD", "F50505"],
"idle_cycle_seconds": 3.0
}
},
"kiosks": [
{
"_comment": "type must be one of: thin-client, touch-panel, door-panel, kitchen-display, steam-tv-box. hostname must be unique and a valid DNS label — it is what the HA device shows up as.",
"_comment": "type must be one of: thin-client, touch-panel, door-panel, kitchen-display. hostname must be unique and a valid DNS label — it is what the HA device shows up as.",
"type": "door-panel",
"hostname": "door-panel",
"room": "hallway",
"friendly_name": "Door panel",
"kiosk_username": "kiosk",
"voice_satellite": true,
@ -305,7 +167,6 @@
{
"type": "kitchen-display",
"hostname": "kitchen-display",
"room": "kitchen",
"friendly_name": "Kitchen fridge display",
"kiosk_username": "kiosk",
"voice_satellite": false,
@ -314,7 +175,6 @@
{
"type": "thin-client",
"hostname": "thin-client-living",
"room": "living_room",
"friendly_name": "Living room thin client",
"kiosk_username": "kiosk",
"voice_satellite": false,
@ -325,40 +185,10 @@
{
"type": "touch-panel",
"hostname": "touch-panel-kitchen",
"room": "kitchen",
"friendly_name": "Kitchen touch panel",
"kiosk_username": "kiosk",
"voice_satellite": false,
"enable_installer": false
},
{
"_comment": "The Loggia's Lenovo all-in-one (docs/components.md #Have). Same touch-panel image as the kitchen's, plus Discord: this is the balcony panel, and the point of the fourth app is that a voice call survives stepping outside — you switch the call onto this machine on the way out and back onto the desktop on the way in.",
"type": "touch-panel",
"hostname": "touch-panel-loggia",
"room": "loggia",
"friendly_name": "Loggia touch panel",
"kiosk_username": "kiosk",
"voice_satellite": false,
"enable_installer": false,
"_comment_enable_discord": "touch-panel only. Installs the Discord Flatpak, adds a 4:discord workspace, a Discord dock button and an HA app-launch button, and starts it logged-in at session boot. Off everywhere else — a kitchen panel does not want a chat client.",
"enable_discord": true
},
{
"_comment": "The living-room gaming box. Games run ON this machine (real GPU/CPU) — it is not the thin client's Steam Link streaming. Boots straight into Steam Big Picture; the media apps (Firefox+uBlock Origin, Spotify, mpv) are launched lazily the first time somebody leaves Big Picture. Prism Launcher is installed and registered as a non-Steam game so it runs through Steam with Steam Input active.",
"type": "steam-tv-box",
"hostname": "steam-tv-box-living",
"room": "living_room",
"friendly_name": "Living room Steam TV box",
"kiosk_username": "kiosk",
"voice_satellite": false,
"_comment_installer": "true, unlike every other kiosk here: a live-booted system keeps its writable layer in RAM, so a Steam library would vanish on reboot. This box wants a real install on a real disk.",
"enable_installer": true,
"_comment_debian_release": "Overrides household.debian_release for this image only. bookworm ships Mesa 22.3, which is too old to run current titles well, and the graphics stack is the one thing on this machine that cannot be fixed later by editing a config file.",
"debian_release": "trixie",
"_comment_gpu_vendor": "amd | intel | nvidia. Not detected — the build host cannot see this machine's hardware, and installing the NVIDIA driver on an AMD box actively breaks it. amd and intel need nothing beyond Mesa; nvidia pulls the non-free driver.",
"gpu_vendor": "amd",
"_comment_enable_cec": "Turn the television itself on and off over the HDMI cable (HDMI-CEC), driven by Home Assistant presence. Set false if the set ignores CEC or is on a receiver that misbehaves; the compositor-side blanking still works either way.",
"enable_cec": true
}
],
@ -366,13 +196,11 @@
{
"_comment": "Headless Spotify Connect appliances for rooms with no thin client. arch picks the toolchain — and they are genuinely different toolchains producing different artifacts, not one image for both: 'amd64' is a mini PC + USB DAC built with live-build (an .iso), 'arm64' is a Raspberry Pi + HiFiBerry Amp2 built with rpi-image-gen (an .img). build-all.sh builds every entry here, so listing both architectures gets you both. hostname doubles as the Spotify Connect device name and must be unique across kiosks too — they're all devices on one network.",
"hostname": "audio-endpoint-livingroom",
"room": "living_room",
"friendly_name": "Living room",
"arch": "amd64"
},
{
"hostname": "audio-endpoint-kitchen",
"room": "kitchen",
"friendly_name": "Kitchen",
"arch": "arm64"
}

47
README.md
View File

@ -26,15 +26,8 @@ hosts/
browser, switched via an always-on touch dock or by HA/
the local LLM over MQTT
kitchen-display/ Single-purpose Sway kiosk for the fridge/pantry: one
Chromium window showing pantry-vision's unload/consume/
expired/edit frontend, camera capture via the browser
itself
steam-tv-box/ Living-room gaming box: boots into Steam Big Picture and
runs games LOCALLY (real GPU — not thin-client's Steam
Link streaming). Prism Launcher runs through Steam so
Steam Input is live for Minecraft. The media apps
(Firefox+uBlock Origin, Spotify, mpv) are launched lazily,
the first time somebody leaves Big Picture
Chromium window showing pantry-vision's Scan/Inventory/
Recipes frontend, camera capture via the browser itself
door-panel/ Single-purpose Sway kiosk by the door/wardrobe:
identity's weather+clothing/who's-home/groceries-low
dashboard by default, registration on demand — this
@ -67,10 +60,7 @@ admin-canvas/ On-demand sys-admin-llm display surface for the thin
static serving)
pantry-vision/ Kitchen-display backend: a photo held up to the camera
-> an Ollama vision-model proposal -> human-confirmed
write into Grocy stock. All four stock movements are
camera-driven (unload / consume / bin what expired /
correct by hand), counted in individual units and folded
brand-free; also proxies Grocy's inventory
write into Grocy stock; also proxies Grocy's inventory
(soonest-expiring first) and recipes to the kiosk
frontend (write API + frontend/ static serving)
trash-calendar/ Reads Kennelbach's personal trash-collection ICS feed,
@ -83,22 +73,11 @@ chores/ Presence/calendar-driven household chore nudging +
passive fairness tally + camera-verified trash-bin/
dishes/litter checks (systemd-timed oneshot, no
long-lived service, no LLM-picked assignment)
mic-follow/ Follow-me microphone switching, per client desktop: HA
decides which room the person is in, a small agent swaps
that machine's live input (and moves running capture
streams), toggled from the dock, the watch or HA
stream-dock/ Desk-side MiraBox N4 Pro (10 keys, 4 encoders) driving
one room's colour lamps through OpenDeck: an HA script
package for the relative-colour maths, generated dial/key
bindings, and a knob-ring colour service gated on the
dock's lighting layer
```
## Status
- [x] Project plan drafted
- [ ] **Follow-me microphone** (`mic-follow/`) — a voice call that survives walking out of the room: the person leaves their desk, the locator says which room they are in, and their machine's live microphone switches to one that can hear them there, then back to the studio mic when they sit down. **Written per client**, so a second person with a second desktop is one more entry in `CoreSystemConfig.json`. Home Assistant decides *where the person is*; the desktop agent only knows *how to change the input* — and it both sets the default source and **moves the already-running capture streams** of the configured applications, because anyone who owns a studio mic has picked it explicitly in Discord and an explicitly-picked device does not follow the default. **One rule makes it predictable: off means the desk mic** — the switch being off is an active guarantee, reachable from the dock, the watch, or MQTT, and honoured on shutdown too. Three surfaces, one entity to read: `sensor.mic_follow_<node>_status`, whose state is the *name of the live microphone*, shown in large type on a **Stream Dock key** (websocket-driven, nothing polls) and on a new **Pebble toggles screen** (long-press Select), which reaches Home Assistant through a narrow **allowlist** in `identity` (`/toggles`) rather than putting an HA admin token on a phone. Tested: 17 fixture cases over source selection (a monitor source can never be picked — that would transmit what the desktop is *playing*), a JS↔C round trip for the new watch message, and the generated HA package parsing for one client and several. **The Loggia case is a remote microphone**: the touch panel already in that room becomes an audio source on the desktop over **SSH** (`pw-record` there, `pw-cat` into a null sink and `module-remap-source` here) — chosen over RTP or a pulse-tunnel because that panel already runs sshd as its admin path, so the transport arrives with auth and encryption solved and **nothing new listens on the network**; audio exists only while the one SSH session does, which is a property of the transport rather than a promise. The panel now shows a red **MIC LIVE** badge on its dock whenever anything is capturing its microphone, because a room mic somebody elsewhere can open has to be visible to the person standing in front of it. **Untested:** the SSH audio pipe end to end, every command that changes state, and the thing most likely to bite — whether BLE presence reports rooms fast and accurately enough to be worth wiring to a microphone at all, which has never been measured in this flat (`mic-follow/README.md` §3, §6, §7)
- [ ] **Stream Dock lighting controls** (`stream-dock/`) — four rotary encoders on a desk-side **MiraBox N4 Pro** (the xVSDinside-branded one; Ajazz AKP05 family) as R / G / B / brightness for one room, through **OpenDeck**. **No new plugin was written, deliberately**: [streamdeck-homeassistant](https://github.com/cgiesche/streamdeck-homeassistant) already does HA-over-websocket with encoder actions, and [opendeck-akp05](https://github.com/aroaxinping/opendeck-akp05) already teaches OpenDeck this non-Elgato hardware — what was missing was the configuration between them, which is what this directory is. The one thing neither plugin can do is **relative colour**: an encoder emits "three ticks clockwise" and HA has `brightness_step_pct` but no equivalent for a colour channel, so the dock sends only *which channel, how many ticks* and `ha-package/stream_dock.yaml` does the read-clamp-write against the lamp's current `rgb_color`. Rings 1–3 glow their own channel's value in their own colour and ring 4 glows in **what the room is actually emitting** (rgb scaled by brightness); off the dock's lighting layer they fall back to the desktop's own `#E40046`/`#5018DD`/`#F50505` chase and Home Assistant stops being polled at all. **Nothing has touched hardware.** The ring colours are computed, debounced and written correctly — verified against a stub HA — but the akp05 plugin reads `leds.toml` only at startup and holds the USB device open, so the last hop is an `apply_command` hook that ships empty; the real fix is a file-watch upstream. The layer gate ships answering *"I cannot tell"* rather than guessing OpenDeck's undocumented profile-state schema, and `{{ticks}}` is the first thing to test before binding four dials — see `stream-dock/README.md` §5–7
- [x] Container host setup script v1 (HA, Mosquitto, Zigbee2MQTT USB, Frigate, Grocy)
- [x] Node-RED + monitoring (Netdata) + dashboard (Homepage) + ntfy + Portainer added to compose stack
- [x] Backup (restic) setup — scripted, off by default until a backup target is picked (`ENABLE_BACKUPS`)
@ -114,27 +93,13 @@ stream-dock/ Desk-side MiraBox N4 Pro (10 keys, 4 encoders) driv
- [x] admin-canvas + admin-web (sys-admin-llm on-demand display surface for the thin clients) — built and wired into `setup-container-host.sh` (`ENABLE_ADMIN_CANVAS`, off by default); the HA-side tool/rest_command wiring and the specific entities it surfaces (e.g. power-monitoring) are still undecided, see `docs/project-plan.md` §4
- [ ] ESP32-S3-Touch-LCD-1.85C-V2 voice satellite + status display (`firmware/esp32-s3-touch-lcd-1.85c/`) — ESPHome config written and passes `esphome config`, not yet flashed to real hardware; `media_player`/`weather` entity IDs still need to be chosen, see `docs/project-plan.md` §4
- [ ] Headless audio endpoint (`hosts/audio-endpoint/`) — per-room independent Spotify Connect appliance for rooms without a thin client, arm64 (Raspberry Pi + HiFiBerry Amp2, rpi-image-gen) and amd64 (mini PC + USB DAC/amp, live-build) build pipelines written, **neither built/flashed/booted on real hardware** — rpi-image-gen's exact config schema in particular is unverified, see `hosts/audio-endpoint/README.md`
- [ ] Sway touch panel (`hosts/touch-panel/`) — touch-driven Sway image: full Spotify GUI (Flathub), a dedicated Home Assistant Chromium kiosk window, a general web browser, an always-on touch dock for app switching, an on-screen keyboard (toggled manually, no auto-show), and `touchpanel-agent` (HA MQTT control, same LLM-mediated-through-HA security model as the thin client) — built, **no touch-panel hardware chosen and nothing booted on real metal**, see `hosts/touch-panel/README.md`. **Discord is now an optional fourth app** (`kiosks[].enable_discord`, validator-restricted to this image type), which is what the Loggia panel — the Lenovo all-in-one already owned — is for: the point is a voice call that survives stepping outside for a cigarette, so it starts logged-in at session boot and every control path (dock button, HA **Show Discord**, the **Screen** select, whose options are now derived from the app table rather than fixed) focuses the running app instead of launching it. It is an XWayland window, so sway matches `class` rather than `app_id`, touch is the emulated-pointer tier, and it deliberately does not inhibit idle — a call must not hold the panel's screen on in an empty room
- [ ] Steam TV box (`hosts/steam-tv-box/`) — living-room machine that boots straight into **Steam Big Picture** and plays games **locally** (native `steam-installer` + i386 multiarch + full Mesa/Vulkan stack, `steam-devices`, gamemode, gamescope when the release has it) — the opposite end of `hosts/thin-client/`'s Steam Link streaming, and the only host here pinned to **trixie** (per-kiosk `debian_release` override; bookworm's Mesa 22.3 is too old for a machine that renders). **Prism Launcher** is installed from Flathub and registered as a *non-Steam game* so Minecraft runs through Steam with Steam Input and the Steam Controller API live. Leaving Big Picture drops into the same media set as the other monitor clients — Firefox with uBlock Origin/SponsorBlock, Spotify, mpv — **launched lazily**, on two independent triggers (the Steam process exiting, and a sway workspace-focus IPC watcher), both idempotent, and never torn down when you go back into a game. `steamtv-agent` adds a session-mode sensor, mode/screen/audio-output selects, app-launch buttons and a CEC display switch over the same MQTT-only control surface as the other kiosks. Written and validated end-to-end through `validate-config.py`/`config-export.py`, **never built, never flashed, never booted, no hardware chosen** — the `shortcuts.vdf` binary format, the Flathub app IDs, gamescope's availability and the NVIDIA package names are all reasoned rather than verified, see `hosts/steam-tv-box/README.md`
- [ ] Kitchen/fridge display + `pantry-vision` (`hosts/kitchen-display/`, `pantry-vision/`) — hold a grocery item up to the camera, an Ollama vision model proposes what it is and roughly how long it keeps, a human confirms (never auto-committed) before it's written into Grocy stock; the display then shows inventory sorted by soonest-to-expire, groceries running low, and Grocy's recipes. All four stock movements now run off the camera — **unload** (a scan loop, one confirm per item, with pack size and where to put it away), **consume** (asks which brand and how many), **list expired** (cleared by scanning what you're binning, booked out as spoiled), and **edit inventory** (the deliberately camera-free correction screen) — with two invariants: stock counts *individual units* (a twelve-pack of eggs is twelve) and folds *brand-free* via Grocy product groups (12 of brand X + 10 of brand Y = 22 eggs, expandable per brand). Built and wired into `setup-container-host.sh` (`ENABLE_PANTRY_VISION`, off by default), **nothing run against a real camera, vision model, or Grocy instance** — the Grocy API call shapes in particular are written from documentation only, see `pantry-vision/README.md` and `hosts/kitchen-display/README.md`
- [x] Where-is-it-actually, for multiple fridges — `docs/fridge-item-location.md`: separates "which appliance" (a software-only change: Grocy locations + a transfer action) from "which shelf" (a *hint* at best) and "exact position" (occlusion makes it unbuildable), and rules out interior cameras on power/condensation/−18 °C grounds — which is exactly the compartment the question starts from. **Built**: appliances as Grocy locations, `POST /transfer` with a "Move to…" picker on the edit screen, and the door-sensor→camera→hint path (`pantry-vision/doorway.py`, `POST /doorway-event` — an HA automation on a Zigbee contact sensor, answered 202 with the camera burst running off-thread; what it recognises is stored in its own SQLite file with a timestamp and a confidence and is **never** written to stock). **Not bought, not tested**: no door sensor, no doorway camera, and the assumption the camera half rests on — that a local vision model can identify an item in a moving hand at ~1.5 m — has never been checked (open decision #40). An appliance can be configured with a sensor and no camera, which is still the recommended way to start
- [x] Per-person **colour** and a settable **profile picture** in `identity` — eight colours assigned automatically at registration, avoiding any colour already worn by somebody with the same initial (an Anna and an Amir are two identical "A"s on a wall panel, and the colour is what makes that readable), then least-used overall; editable in the admin panel, backfilled oldest-first for existing people so nobody's colour reshuffles on restart. `color` + `initial` now ride on `/people`, `/presence` and every `/floorplan/presence` occupant, so no consumer derives an initial or invents a palette. New `POST /people/<id>/photo` sets a picture without a walk to the door panel — the registration capture was the only source before, which left a device-less household member unable to have a face at all. **The palette values sit on the 2-bits-per-channel lattice a colour Pebble renders natively**, so the colour on a watch is the colour in the panel, see below
- [x] **Rooms are one vocabulary, and devices declare theirs** — `docs/rooms-and-endpoints.md`: the room id is an HA `area_id` everywhere (`CoreSystemConfig.json` → `config-export.py` → the ISO builder → the agent → `suggested_area` in MQTT discovery), which is the same string `identity`'s floorplan rooms join on. Every kiosk and audio endpoint now carries `room`, `tools/validate-config.py` rejects anything that isn't already an area_id (helpfully slugifying it is how you end up with two rooms) and warns rather than errors when it's missing. **The honest limit: `suggested_area` is only honoured at first discovery** — move a device and you move it in HA once, by hand
- [x] **`workshop/`** — built: project notebook (`workshop.db`), a **never-pruned knowledge store** (`workshop-knowledge.db`: workflow instructions by activity, keyword facts, project learnings, and the household's **one** hardware inventory), `GET /context` to hand an assistant everything that applies at once, a **web inventory editor**, **Gitea repos with append-only history** (commit/push/branch yes; force-push/rebase/reset/filter-repo never, enforced by branch protection server-side, with a print-only scrub-request for the token-in-history case), a **health poller** for CheckMK + every OPNsense firewall, a **cameras** tab over go2rtc, and a **fleet-scripts admin surface**: one monitoring-agent script per kind of machine (Debian x86, arm64 Pi, the Docker host, the GPU host, plus slots for the ESP32s and network appliances that *can't* run a script and whose CheckMK-server-side config goes there instead), fetched by each endpoint's `fleet-bootstrap` timer. It's remote code execution by design, so the constraints are the design: **upload is a draft, publishing is a separate click**, the service never executes anything, scripts live in SQLite rather than on the writable share (a share credential must not be a whole-fleet root-execution credential), every version is kept, and the endpoint verifies the checksum, runs a version once, and **reports back pass or fail** — a script that was served is not a script that succeeded. Off by default (`ENABLE_WORKSHOP`), **nothing run against a real deployment** — the CheckMK API shape and Gitea's branch-protection payload are both written from documentation and have version-sensitive field names
- [x] **Multi-firewall + CheckMK, and a config that says so** — `opnsense` is a **list** now (each with its own name and key pair, keyed by name rather than by index), `checkmk` is a first-class block, and both feed the digest (`ingest/infra_health.py`, which reads the *poller* rather than polling, so the digest can say "critical since Tuesday" instead of quoting a six-hour-old snapshot) **and** the workshop's `infra_status` table. Three states — `ok`/`problem`/**`unreachable`** — because "I could not ask" is not "nothing is wrong". Validator rejects duplicate firewall names and half-configured credentials for both
- [x] **Photo web frontend (Immich)** alongside the SMB share — `ENABLE_PHOTOS_WEB`, search/albums/faces over the **same** directory, mounted **read-only**: two writers to one photo tree with different ideas of the layout is how a collection gets quietly reorganised. The share stays right for bulk copy and for keeping photos openable with no software at all
- [x] **Keycloak + FreeIPA config blocks** — `identity_provider` and `freeipa`, validated and exported, **declaration-only**: nothing implements SSO or the directory mirror yet, and the validator says so out loud rather than letting a filled-in block imply it works. **They are one decision, not two** — Keycloak federates FreeIPA rather than being its own user store, so the validator errors on either configured alone; a Keycloak without the directory behind it is a second parallel set of household accounts, which is the thing a directory exists to prevent. Groups follow **`USR_HA_<parameter>`** (`USR_HA_household`, `USR_HA_chore_exempt`, `USR_HA_admins`), warned-not-errored so an existing directory's convention isn't overridden by a config file. Two rules recorded before the work: the mirror is **one-way** (a directory serving real logins must not become a store of chore-reminder preferences), and a group may **grant** a chore exemption but never **remove** one somebody set by hand
- [ ] Workshop/office assistant — `docs/workshop-assistant.md`: **read the label, don't recognise the object** (a closed T480 and T490 are the same black rectangle; the identity is in the service tag and the PCB silkscreen, so the pipeline is OCR/barcode-first with the VLM only locating the label). Specs must be **quoted from a fetched document with its URL, never generated** — a hallucinated pinout destroys hardware — which also makes this the first component here that deliberately reaches the open internet (outbound-only, allowlisted, cached). Per-room scoping = the room selects the toolset, riding the `room` plumbing above. An SMB **workspace share** for everything it produces (reusing the existing Samba container — a second one would collide on 445, which the gallery already holds — with its own volume, own account, and read-write where the gallery is read-only). Display widgets collapse to **one `svg` window kind plus server-side renderers** (Graphviz for code-flow/data-structure, netlistsvg for schematics, KiCad export for board plans), because a megabyte of JS on a kiosk buys a picture the server could render once — **IEC/EU notation is a symbol-library decision**, so "can I supply my own symbols?" disqualifies a tool before output quality does. Purple/magenta holo theme is one CSS variable override over the existing `glow.css`, with the drawings left untinted on purpose. Analysis only, nothing built
- [x] **Now-playing visualiser + 3D floorplan** — built, in `render/` (shared, vendored, dependency-free, config from `?query=params`). `media-visualiser/`: circular spectrum, album-art palette, LRC lyrics — **two-tier by necessity**, since most endpoints have no local audio (a kitchen panel showing what the living room plays cannot analyse anything), so it is real FFT where audio is local and a tempo-driven ring elsewhere, **which says on screen that it is not an analysis**. Palette rejects near-greys/near-blacks before ranking and lifts each colour until it clears the background — the step whose absence makes art-coloured visualisers invisible on dark covers. Plain lyrics are shown but never auto-scrolled at a guessed rate. `floorplan-3d/`: the same `/floorplan/presence` payload as the Pebble app, extruded — **canvas 2D, not three.js**, a change from the plan made while building it: the scene is prisms on a plane, which an isometric projection with painter's sorting draws in ~200 lines, predictably on weak panels, with the frontend still at zero dependencies. Lit/dark rooms plus the dashed third state for rooms HA never reports on, photos-or-initials in the person's colour ring, and a visible shelf for people who are home but unlocatable. Geometry unit-tested headlessly; **never opened in a real browser**
- [ ] ~~Now-playing visualiser + 3D floorplan (design)~~ — `docs/endpoint-surfaces.md`: a circular CAVA-style ring behind every playing screen, coloured from the album art, lyrics under the cover when they exist; and the Pebble app's presence view in 3D on any endpoint. Two findings shape both: **most endpoints have no local audio** (a kitchen panel showing what the living room plays cannot analyse anything), so the visualiser is explicitly two-tier — real FFT where audio is local, tempo-driven "mood light" elsewhere, and the synthetic tier must never claim to be the real one; and the 3D plan is an **extrusion of the existing 2D polygons**, not a hand-authored model, so it can't go stale when a room is redrawn. Three.js is ~1MB and a deliberate break with the dependency-free SDK rule — make it once, explicitly. Analysis only, nothing built
- [x] **Pebble watchapp built** — `pebble-presence/`: the floorplan and who's in each room, plus Up/Down room cycling into a plain-text list (with "Somewhere in the house" as the last entry in the cycle — the people it can't place are exactly who you picked the watch up to find). Lit/dark/dotted room states, positions when fused, hollow rings for radar targets nobody can name. The last plan is persisted and drawn **with its age** on launch, because the app's JS only lives while it's open so every launch starts with a fetch. **The wire format is round-trip tested** between the JS writer and the C reader against a stub SDK — two implementations of one format in two languages that never share a process, where drift decodes as plausible garbage rather than an error; it already caught one real disagreement. Unverified: the Round 2's platform identifier and its round-screen detection (nothing hardcodes a resolution, so both are one-liners once the SDK names them), and it has never run on a watch
- [ ] ~~Pebble presence watchapp (design)~~ — `docs/pebble-presence-watchface.md`: the floorplan drawn as quantised polygons (~200–250 bytes for a whole floor, against a ~2 KB AppMessage budget), occupants as initial-on-colour rather than photos (a face gets ~20×20px in 64 colours — four skin-toned blobs, and three of them exceed the message budget the entire floor fits in). Targets the **Pebble Round 2** that's actually owned: the plan is inscribed in the circle (`w = D·a/√(a²+1)`, so ~170×170 for a squarish plan at a 240px usable diameter — comparable to a Time 2 after chrome), the projection change is one line of phone-side JS, and the only real loss is corner furniture like a status footer. Occupancy reads as **dark room / lit room** in lightness rather than hue, since colour is already spoken for by *who* — with a third dashed state for rooms HA never reports on, because drawing "no data" as "empty" is a quiet lie. **Button-cycling through rooms forces a watchapp, not a watchface** — Pebble gives watchfaces no button events and restricts touch to apps — and the watchapp is the decision: Up/Down cycles rooms into a plain-text list of who is in each (plus a final "somewhere in the house" entry for people who are home but unlocatable, who are exactly who you picked the watch up to find). The cost is that it's a menu entry rather than your default screen, and its data only lives while it's open; a watchface variant later is a second `main()`, not a second project. Analysis only, nothing built. The finding that actually decides it is upstream: room-level presence has never been measured in this house, and the admin panel's floorplan tab with **Live** ticked tests it today for free
- [ ] "Put the groceries away" chore — `chores`' new `groceries_out_of_place` watch point: `litter`'s twin (same "whoever the camera just saw" attribution, same exemption-proof, same unassignable) with a **30-minute** neglect fuse instead of four hours, because the failure is spoiled food rather than untidiness. Its prompt names what to *ignore* (fruit bowls, bread bins, coffee, cookware) as well as what to look for — the general question gets a YES for any normal kitchen, and a false chore every two hours is how a household learns to mute the channel (open decision #41). Knows food is out, never what the food is; nothing here touches Grocy. Untested against a real camera or model, like every other watch point
- [ ] Sway touch panel (`hosts/touch-panel/`) — touch-driven Sway image: full Spotify GUI (Flathub), a dedicated Home Assistant Chromium kiosk window, a general web browser, an always-on touch dock for app switching, an on-screen keyboard (toggled manually, no auto-show), and `touchpanel-agent` (HA MQTT control, same LLM-mediated-through-HA security model as the thin client) — built, **no touch-panel hardware chosen and nothing booted on real metal**, see `hosts/touch-panel/README.md`
- [ ] Kitchen/fridge display + `pantry-vision` (`hosts/kitchen-display/`, `pantry-vision/`) — hold a grocery item up to the camera, an Ollama vision model proposes what it is and roughly how long it keeps, a human confirms (never auto-committed) before it's written into Grocy stock; the display then shows inventory sorted by soonest-to-expire, groceries running low, and Grocy's recipes — built and wired into `setup-container-host.sh` (`ENABLE_PANTRY_VISION`, off by default), **nothing run against a real camera, vision model, or Grocy instance** — the Grocy API call shapes in particular are written from documentation only, see `pantry-vision/README.md` and `hosts/kitchen-display/README.md`
- [ ] `identity` + door panel (`identity/`, `hosts/door-panel/`) — the person <-> BLE-identifier registry: "register me as `<name>`" by voice or touchscreen, multi-phone support (multiple identifiers per person), anti-spoofing (only allowlisted IRK-resolved/fixed-tag entities are ever accepted as candidates, never a raw MAC), device-less people (a "no device" flag plus a hand-operated Home/Away toggle — the concrete case: a grandmother without a smartphone), and an anonymous "Guest" path. Backs `hosts/door-panel/`'s weather+clothing/who's-home/groceries-running-low dashboard and `hosts/kitchen-display/`'s "Show registration" screen — built and wired into `setup-container-host.sh` (`ENABLE_IDENTITY`, off by default), **nothing run against a real HA instance, real Private BLE Device entities, or a real voice pipeline** — `TRUSTED_ENTITY_PREFIXES` above all needs checking against Developer Tools -> States, see `identity/README.md` and `hosts/door-panel/README.md`
- [ ] `identity` also corroborates presence from Frigate face recognition (Phase 20, Tapo pan/tilt cameras) — an OR-ed-in second signal only, **never** a registration signal; and owns the per-person chore-system settings (`chore_exempt`, `chore_reminder_style`, plus chore assignment) consumed by `chores/`, see `identity/README.md`
- [ ] `identity`'s admin panel (`identity/frontend/admin.html`, Phase 6b) — managing people/guests: edit every field, **nicknames** (an input alias only — `/resolve` accepts them, but the assistant always speaks the real `speak_name`), **visit history** sampled from `/presence` plus a derived "who was home with whom" view, **"select all that last visited before `<date>`"** pruning (the filter selects, a human confirms the exact list, the filter is never re-run at delete time), **per-device rights** for self-entry via a smart lock (`identity` only ever *answers* `GET /device-access` — HA asks and HA acts, deny is the default), chore assignment, a **floorplan editor** (draw levels and rooms as polygons, map each to an HA area, and watch occupied rooms light up — resolves the long-deferred open decision #22), and **opt-in arrival push notifications** ("tell me when someone gets home", via the self-hosted ntfy this stack already runs — `identity` itself never touches the WAN; ntfy stays LAN-only and remote delivery rides a WireGuard split tunnel — see `docs/network-integration.md` §2.2 for why a DMZ/port-forward was weighed and rejected). Deliberately **not** a kiosk page and not linked from any wall panel. Covered by API-level tests; **never opened in a real browser**, and `DEPARTURE_GRACE_SECONDS` is an untuned guess — see `identity/README.md`
- [ ] `trash-calendar` + `transit` (Phase 19, Kennelbach AT trash pickup + Vorarlberg public transit) — built and wired into `setup-container-host.sh` (`ENABLE_TRASH_CALENDAR`/`ENABLE_TRANSIT`/`ENABLE_TRIP_PLANNING`, all off by default), **nothing run against a live ICS feed, a live GTFS feed, or a real OpenTripPlanner instance** — trip planning also needs a manually-built OTP graph this repo does not build for you, see `trash-calendar/README.md` and `transit/README.md`'s "Route planning scope"
- [ ] `chores` (Phase 20) — presence/calendar-driven household chore nudging: "I don't care who does it, as long as it gets done" — prefers whoever's been assigned a chore in `identity`'s admin panel but falls through to whoever's actually home rather than waiting (`CHORE_ASSIGNMENT_STRICT` flips that), redirects to someone else if a chore goes neglected, keeps a passive fairness tally that never feeds back into who gets nudged, and camera-checks trash bins/dishes/litter/groceries-left-out via Frigate + an Ollama vision model. **Litter and groceries-left-out remain the exception to everything** — both ignore chore-exemption and assignment, because cleaning up what you left out was never a task anyone could be assigned; groceries additionally get a 30-minute fuse instead of four hours, since that failure spoils food rather than merely looking untidy. Built and wired into `setup-container-host.sh` (`ENABLE_CHORES`, off by default, every-2-hours systemd timer), **no Tapo camera hardware chosen and nothing run against real hardware**, see `chores/README.md`
- [ ] `chores` (Phase 20) — presence/calendar-driven household chore nudging: "I don't care who does it, as long as it gets done" — prefers whoever's been assigned a chore in `identity`'s admin panel but falls through to whoever's actually home rather than waiting (`CHORE_ASSIGNMENT_STRICT` flips that), redirects to someone else if a chore goes neglected, keeps a passive fairness tally that never feeds back into who gets nudged, and camera-checks trash bins/dishes/litter via Frigate + an Ollama vision model. **Litter remains the exception to everything** — it ignores both chore-exemption and assignment, because cleaning up what you left out was never a task anyone could be assigned. Built and wired into `setup-container-host.sh` (`ENABLE_CHORES`, off by default, every-2-hours systemd timer), **no Tapo camera hardware chosen and nothing run against real hardware**, see `chores/README.md`
- [ ] Music Assistant (optional, additive multi-room audio) — wired into `setup-container-host.sh` (`ENABLE_MUSIC_ASSISTANT`, off by default), **its default port is an unverified guess that collides with `PANTRY_VISION_PORT`** if both are enabled together, see `docs/project-plan.md` open decision #31
- [ ] `docs/network-integration.md` (OPNsense VLAN segmentation, the WireGuard split tunnel that carries arrival notifications, and why nothing here — ntfy included — gets port-forwarded to the WAN) — written, not run against a real OPNsense instance
- [ ] `tools/` + `CoreSystemConfig.json` — every build and setup script in one place, reading one config. The container host and LLM host build as a **twinned pair**: you set two last octets and the container host's `OLLAMA_HOST` is *derived* from the LLM host's, so the two ISOs cannot be built disagreeing about where the other one is; every kiosk's service URLs derive from the container host's address the same way. `build-all.sh` builds the set, `validate-config.py` refuses a build on duplicate ports (the `music_assistant`/`pantry_vision` 8095 clash, open decision #31), placeholder or padded tokens, duplicate hostnames, or a kiosk pointed at a disabled service. All secrets are burnt into the images so installs are unattended — **which makes every ISO a credential**; the filled-in config and `iso-out/` are gitignored. **No ISO has ever been built with this** (`lb build` needs live-build, root and a long fetch) — what is tested is config validation/derivation and every generated artifact, with `lb` stubbed. See `tools/README.md`

65
chores/README.md
View File

@ -21,43 +21,18 @@ module docstring for the full reasoning on each:
Frigate snapshot per configured watch point (optionally moving a PTZ camera to a
preset first), asks an Ollama vision model a one-word question ("is this bin
FULL/PARTIAL/EMPTY", "is this counter DIRTY/CLEAN", "is there litter left out
here, YES/NO", "are there groceries standing out that belong in a fridge/freezer/
cupboard, YES/NO"), opens a chore on "needs attention," auto-closes one on "clear."
here, YES/NO"), opens a chore on "needs attention," auto-closes one on "clear."
3. **Nudging** — ASAP, not on a fixed schedule: the first run after a chore opens
nudges whoever `identity` reports home right now — preferring anyone **assigned**
that chore type, minus anyone `chore_exempt`, both below. If the chore is still open `NEGLECT_THRESHOLD_HOURS` after the last
nudge (and the household calendar isn't showing a busy window), the nudge goes
to **someone different from who was last asked** — "the next person that walks
by" — rather than re-nagging the same person. `litter` and
`groceries_out_of_place` chores are special-cased to prefer whoever the camera
most recently recognized nearby (a best-effort "who left this" guess), since the
point there is telling the actual person, not just whoever's around — and both
also ignore `chore_exempt` entirely, see below. Each nudge's wording is a plain
template unless the target has a `chore_reminder_style` set, see below.
## Putting the groceries away is litter with a shorter fuse
`groceries_out_of_place` is built as `litter`'s twin on purpose — same "whoever the
camera just saw" attribution, same exemption-proofing, same unassignability, because
it is the same situation: somebody carried something in and put it down. Its one
deliberate difference is **`GROCERIES_NEGLECT_THRESHOLD_HOURS`, 30 minutes instead of
the usual four**, and that is the whole reason it isn't simply another watch point on
the `litter` prompt: litter left for an afternoon is untidiness, and a tub of ice
cream left for an afternoon is a bin bag. It is the only chore type in `check.py`
treated as more urgent than the others, because it is the only one where being late
costs something other than tidiness.
Two things worth knowing before pointing a camera at this:
- **The prompt names what to ignore, not just what to look for** — fruit bowls, bread
bins, coffee, oil, spices, appliances, cookware. Ask a vision model the general
question ("is anything out of place?") and it will say YES to a kitchen that is
simply a kitchen, every two hours, until somebody mutes the topic.
- **It is a chore, not an inventory update.** It notices that food is standing out; it
does not know *what* the food is and never books anything into Grocy. That is
`pantry-vision`'s job, on a display someone is actually standing at. Two systems
looking at the same counter for two different reasons is the intended shape, not a
duplication to collapse.
by" — rather than re-nagging the same person. `litter` chores are special-cased
to prefer whoever the camera most recently recognized nearby (a best-effort
"who left this" guess), since the point there is telling the actual person, not
just whoever's around — and `litter` also ignores `chore_exempt` entirely, see
below. Each nudge's wording is a plain template unless the target has a
`chore_reminder_style` set, see below.
## Assignment is a preference, not a lock
@ -77,22 +52,20 @@ legitimate and this file can't pick for you, so it's one env var rather than a
hard-coded opinion. Strict mode still never stalls a chore that was assigned to
*nobody* — that falls through regardless.
**Litter and groceries-left-out can't be assigned** (`_ASSIGNMENTS_DONT_APPLY` in
`check.py`), for the same reason they ignore exemptions: they go to whoever left the
mess, and cleaning up after yourself was never a task anyone could be handed.
**Litter can't be assigned** (`_ASSIGNMENTS_DONT_APPLY` in `check.py`), for the same
reason it ignores exemptions: it goes to whoever left the mess, and cleaning up after
yourself was never a task anyone could be handed.
## Chore-exempt people — everyone except litter and groceries
## Chore-exempt people — everyone except litter
Set in the same place (`identity`'s admin panel, or `POST /people/<id>/chore-settings`
directly). A `chore_exempt` person is
dropped from the nudge rotation entirely — the "cousin visits often but doesn't owe
me chores" case. **Litter and groceries-left-out are the deliberate exceptions**
(`_EXEMPTIONS_DONT_APPLY` in `check.py`): an exempt person still gets told to put
trash they left out into the bin and to put the milk away, because neither is "doing
a chore," both are cleaning up after yourself — and a guest who helped unpack the
shopping is exactly as able to finish the job as anyone else. If everyone currently
home is chore_exempt for one of the other chore types, that run just logs and skips —
the chore stays open until someone eligible is around.
me chores" case. **Litter is the deliberate exception** (`_EXEMPTIONS_DONT_APPLY`
in `check.py`): an exempt person still gets told to put trash they left out into
the bin, because that isn't "doing a chore," it's cleaning up after yourself. If
everyone currently home is chore_exempt for a non-litter chore, that run just logs
and skips — the chore stays open until someone eligible is around.
## Reminder tone is per-person and LLM-phrased, but never LLM-decided
@ -190,11 +163,7 @@ when left unconfigured.
2. Whether the vision model's one-word FULL/PARTIAL/EMPTY/DIRTY/CLEAN/YES/NO
answers are actually reliable for a real bin/sink/hallway from a real camera
angle — completely unmeasured, same caveat as pantry-vision's own vision-model
accuracy note. **`groceries_out_of_place` is the one to check first**: its whole
design rests on a model reliably telling a carton of milk from a fruit bowl, and
its failure mode is not a missed chore but a false one every two hours, which is
how a household learns to ignore the notification channel. Point it at a *clean*
counter for a day before trusting a YES.
accuracy note.
3. `_likely_culprit()`'s reliance on `identity`'s `face_seen_recently` field
assumes Frigate face-recognition presence corroboration is actually wired up
and working (`identity/README.md`'s own "Camera face recognition" section is

149
chores/check.py
View File

@ -18,9 +18,8 @@ in case something else is running" jitter) and does three things, in order:
camera hardware has been chosen yet, see docs/project-plan.md §1.18): for each
configured watch point (a Frigate camera + optional PTZ preset), grabs a
snapshot via Frigate's own API and asks an Ollama vision model whether it shows
a full bin, dirty dishes, litter left out, or **groceries still standing out that
belong in a fridge/freezer/cupboard**. A "needs attention" result opens a chore if
one isn't already open; a "clear" result auto-closes one if it was.
a full bin / dirty dishes. A "needs attention" result opens a chore if one isn't
already open; a "clear" result auto-closes one if it was.
3. **Nudging**: for every open chore, ASAP, not "wait for a schedule" — the first
run after a chore opens nudges whoever `identity` reports home right now (using
`identity`'s own room field to prefer someone actually near the relevant spot,
@ -36,20 +35,10 @@ in case something else is running" jitter) and does three things, in order:
`identity`'s `chore_exempt` flag (set via `POST /people/<id>/chore-settings`, see
identity/README.md) takes a person out of the nudge rotation entirely — a frequent
guest who isn't a household member doesn't owe chores. **`litter` and
`groceries_out_of_place` are the exceptions** (`_EXEMPTIONS_DONT_APPLY` below):
everyone, exempt or not, still gets told to put trash they left out into the bin and
to put the milk away — neither is "doing a chore," both are cleaning up after
yourself, and a guest who unpacked the shopping is exactly as able to finish the job
as anyone else.
`groceries_out_of_place` is deliberately built as litter's twin — same
culprit-attribution, same exemption-proofing, same unassignability — with **one
difference that matters: a much shorter fuse** (`_NEGLECT_HOURS_OVERRIDE`, 30 minutes
against the usual four hours). Litter left for an afternoon is untidiness; a tub of
ice cream left for an afternoon is a bin bag. It is the only place in this file where
one chore type is treated as more urgent than another, and the reason is that the cost
of being late differs in kind, not degree.
guest who isn't a household member doesn't owe chores. **`litter` is the one
exception** (`_EXEMPTIONS_DONT_APPLY` below): everyone, exempt or not, still gets
told to put trash they left out into the bin — that isn't "doing a chore," it's
cleaning up after yourself.
## Assignment is a preference, not a lock
@ -65,10 +54,9 @@ waiting. Set `CHORE_ASSIGNMENT_STRICT=true` if you'd rather it wait for the assi
that's the honest opposite reading of the same feature, and which one a household
wants isn't something this file can decide for it.
**`litter` and `groceries_out_of_place` ignore assignment entirely**
(`_ASSIGNMENTS_DONT_APPLY`), for the same reason they ignore exemptions: they go to
whoever left the mess, and "cleaning up after yourself" was never a task anyone could
be assigned in the first place.
**`litter` ignores assignment entirely** (`_ASSIGNMENTS_DONT_APPLY`), for the same
reason it ignores exemptions: it goes to whoever left the mess, and "cleaning up after
yourself" was never a task anyone could be assigned in the first place.
`identity`'s `chore_reminder_style` free-text field (same endpoint) is passed to an
LLM that **phrases** the ntfy message in that person's preferred tone ("be
@ -151,65 +139,26 @@ _CHORE_PROMPTS = {
"that does not belong there (not properly disposed of in a bin)? Answer with exactly "
"one word: YES or NO."
),
# Groceries left standing out — the same shape as litter (somebody put something
# down and walked away) with a much shorter fuse, because the failure here is not
# untidiness, it is food going off. The prompt names the perishable cases
# explicitly rather than asking the general "is anything out of place" question,
# which a vision model will happily answer YES to for a fruit bowl, a bread bin,
# or a kettle — and a chore that fires every two hours about the fruit bowl is one
# the household will mute within a week.
"groceries_out_of_place": (
"Look at this photo of a kitchen counter, table or worktop. Are there GROCERIES sitting "
"out that belong in a fridge, freezer or cupboard — for example milk, yoghurt, cheese, "
"meat, fish, eggs, opened jars, frozen food, or a shopping bag that has not been put "
"away? Ignore things that normally live on a worktop: fruit bowls, bread bins, coffee, "
"spices, oil, salt, appliances, cookware and dishes. Answer with exactly one word: "
"YES or NO."
),
}
# Which watch points get "who was just seen here" culprit-attribution treatment
# (see _likely_culprit()) instead of the general "whoever's around" nudge — litter
# is specifically about telling whoever left it, not just whoever's nearby now.
# Groceries left out is the same situation: somebody carried it there.
_ATTRIBUTE_TO_RECENT_VIEWER = {"litter", "groceries_out_of_place"}
_ATTRIBUTE_TO_RECENT_VIEWER = {"litter"}
# Chore types where identity's chore_exempt flag does NOT apply — everyone still
# gets told to clean up litter they left out, exempt household member or not (see
# module docstring). Currently the same set as _ATTRIBUTE_TO_RECENT_VIEWER, but
# they mean different things — one is about attribution, this is about eligibility
# — so they're kept as separate names rather than reusing one for both purposes.
_EXEMPTIONS_DONT_APPLY = {"litter", "groceries_out_of_place"}
_EXEMPTIONS_DONT_APPLY = {"litter"}
# Chore types that can't be assigned to anyone — see the module docstring's
# "Assignment is a preference, not a lock". Third set with the same members as the
# "Assignment is a preference, not a lock". Third set with the same one member as the
# two above, and kept separate for the third distinct reason: attribution, then
# eligibility, now assignability. If they ever diverge (a chore that's assignable but
# exempt-proof, say) collapsing them now would be the thing that made that painful.
_ASSIGNMENTS_DONT_APPLY = {"litter", "groceries_out_of_place"}
# How the one-word camera answer is read, per chore type. Extracted from an inline
# expression once there were four of them: adding a chore type should be a line in a
# table, not a new clause in a boolean nobody can read.
_ATTENTION_ANSWERS = {
# "PARTIAL" contains neither FULL nor a separate word, but a model answering
# "PARTIALLY FULL" must not be read as full — hence the exclusion, kept from the
# original inline check.
"bin_full": lambda a: "FULL" in a and "PARTIAL" not in a,
"dishes": lambda a: "DIRTY" in a,
"litter": lambda a: "YES" in a,
"groceries_out_of_place": lambda a: "YES" in a,
}
# Per-type override for how long a chore may sit before it is redirected to somebody
# else. A dropped crisp packet can wait the default four hours; a tub of ice cream on
# the counter cannot, and neither can the shopping nobody unpacked. This is the one
# place in this file where a chore type is treated as more urgent than another, and it
# is here because the cost of being late is different in kind — spoiled food, not an
# untidy room.
_NEGLECT_HOURS_OVERRIDE = {
"groceries_out_of_place": float(os.environ.get("GROCERIES_NEGLECT_THRESHOLD_HOURS", "0.5")),
}
_ASSIGNMENTS_DONT_APPLY = {"litter"}
# Whether an assigned person who ISN'T home blocks the chore from falling through to
# whoever is. Default false — "as long as it gets done" is the house rule; true makes
@ -312,8 +261,7 @@ def check_trash_day(conn) -> None:
# --- 2. Camera checks --------------------------------------------------------------
def _watchpoints() -> list[tuple[str, str, str | None]]:
"""CAMERA_WATCHPOINTS format: "type:camera[:preset],type:camera[:preset],...".
type must be a key in _CHORE_PROMPTS other than "trash" — i.e. bin_full, dishes,
litter, or groceries_out_of_place.
type must be a key in _CHORE_PROMPTS other than "trash" (bin_full, dishes).
"""
raw = os.environ.get("CAMERA_WATCHPOINTS", "").strip()
if not raw:
@ -321,10 +269,7 @@ def _watchpoints() -> list[tuple[str, str, str | None]]:
points = []
for entry in raw.split(","):
parts = [p.strip() for p in entry.split(":")]
# _ATTENTION_ANSWERS as well as _CHORE_PROMPTS: a chore type with a prompt but
# no way to read its answer is a KeyError in check_cameras(), i.e. a crash on
# the timer rather than here. Adding a type means adding both.
if len(parts) < 2 or parts[0] not in _CHORE_PROMPTS or parts[0] not in _ATTENTION_ANSWERS:
if len(parts) < 2 or parts[0] not in _CHORE_PROMPTS or parts[0] == "trash":
LOG.warning("chores: ignoring malformed CAMERA_WATCHPOINTS entry %r", entry)
continue
points.append((parts[0], parts[1], parts[2] if len(parts) > 2 else None))
@ -389,7 +334,9 @@ def check_cameras(conn) -> None:
if answer is None:
continue
needs_attention = _ATTENTION_ANSWERS[chore_type](answer)
needs_attention = ("FULL" in answer and "PARTIAL" not in answer) or "DIRTY" in answer or (
chore_type == "litter" and "YES" in answer
)
existing = _open_chore(conn, chore_type)
if needs_attention and existing is None:
@ -511,8 +458,7 @@ def nudge_open_chores(conn) -> None:
if last_nudged_at is not None:
elapsed_hours = (datetime.now(timezone.utc) - datetime.fromisoformat(last_nudged_at.replace("Z", "+00:00"))).total_seconds() / 3600
threshold = _NEGLECT_HOURS_OVERRIDE.get(chore["type"], NEGLECT_THRESHOLD_HOURS)
if elapsed_hours < threshold:
if elapsed_hours < NEGLECT_THRESHOLD_HOURS:
continue # not neglected yet — leave whoever was last nudged alone for now
# chore_exempt people are out of the rotation entirely, EXCEPT litter — see
@ -579,54 +525,12 @@ def nudge_open_chores(conn) -> None:
)
# The culprit-framed wording, per type. Both are "you probably left this, please deal
# with it" — but "put it in the bin" is exactly wrong for a tub of ice cream, and a
# reminder that tells you to do the wrong thing is one you stop reading.
_CULPRIT_MESSAGES = {
"litter": "{name}, looks like something was left out — could you put it in the bin?",
"groceries_out_of_place": (
"{name}, there are groceries still standing out — could you put them away before they spoil?"
),
}
# What the LLM phrasing pass is told the situation is, when there is a model
# configured. Same split, same reason.
_CULPRIT_SITUATIONS = {
"litter": (
"Something was left out and may belong to them specifically — ask them to put it away, "
"don't accuse them outright. "
),
"groceries_out_of_place": (
"Groceries have been left standing out instead of being put in the fridge, freezer or "
"cupboard, and they may be the person who left them — ask them to put the food away "
"before it spoils, don't accuse them outright. "
),
}
# Chore types are snake_case identifiers; people are not. Everything user-facing goes
# through here — a notification that says "could you take care of:
# groceries_out_of_place" is a notification that reads like a stack trace.
_CHORE_LABELS = {
"trash": "putting the bins out",
"bin_full": "emptying the bin",
"dishes": "the dishes",
"litter": "clearing up what was left out",
"groceries_out_of_place": "putting the groceries away",
}
def _label(chore_type: str) -> str:
return _CHORE_LABELS.get(chore_type, chore_type.replace("_", " "))
def _default_message(name: str, chore_type: str, redirected: bool, is_culprit: bool) -> str:
if is_culprit:
template = _CULPRIT_MESSAGES.get(chore_type, "{name}, looks like something was left out — could you deal with it?")
return template.format(name=name)
return f"{name}, looks like something was left out — could you put it in the bin?"
if redirected:
return f"{name}, this one's still open — could you take care of: {_label(chore_type)}?"
return f"{name}, could you take care of: {_label(chore_type)}?"
return f"{name}, this one's still open — could you take care of: {chore_type}?"
return f"{name}, could you take care of: {chore_type}?"
def _compose_message(name: str, chore_type: str, redirected: bool, is_culprit: bool, reminder_style: str | None) -> str:
@ -641,13 +545,10 @@ def _compose_message(name: str, chore_type: str, redirected: bool, is_culprit: b
prompt = (
f"Write ONE short household chore reminder (max 2 sentences) addressed to {name}. "
f"The chore is: {_label(chore_type)}. "
f"The chore is: {chore_type}. "
f"Follow {name}'s own stated preference for how they like to be reminded: \"{reminder_style}\". "
+ (_CULPRIT_SITUATIONS.get(
chore_type,
"Something was left out and may belong to them specifically — ask them to put it away, "
"don't accuse them outright. ",
) if is_culprit else "")
+ ("Something was left out and may belong to them specifically — ask them to put it away, don't accuse them outright. "
if is_culprit else "")
+ ("They were already asked about this once before and it's still not done. " if redirected else "")
+ "Reply with ONLY the message text itself — no preamble, no quotation marks."
)

23
chores/chores.env.example
View File

@ -48,28 +48,15 @@ CALDAV_QUIET_KEYWORDS=busy,meeting,call,movie,sleep
# sources.
#
# CAMERA_WATCHPOINTS format: "type:frigate_camera_name[:ptz_preset_name],...".
# type must be one of:
# bin_full — is the bin full
# dishes — is the sink/counter dirty
# litter — has someone left trash out somewhere it doesn't belong
# groceries_out_of_place — is there food still standing out that belongs in a
# fridge/freezer/cupboard
# The last two are told to whoever the camera most recently recognized nearby, not
# just whoever's home in general, and neither can be assigned or exempted away —
# see README.md. groceries_out_of_place also has a much shorter neglect fuse than
# everything else (below), because the failure mode is spoiled food, not untidiness.
# preset is optional — omit it if the camera doesn't need to move. Example:
# CAMERA_WATCHPOINTS=bin_full:driveway_cam:trash_preset,dishes:kitchen_cam,litter:hallway_cam,groceries_out_of_place:kitchen_cam
# type must be "bin_full", "dishes", or "litter" ("has someone left trash out
# somewhere it doesn't belong" — gets told to whoever the camera most recently
# recognized nearby, not just whoever's home in general, see README.md). preset
# is optional — omit it if the camera doesn't need to move. Example:
# CAMERA_WATCHPOINTS=bin_full:driveway_cam:trash_preset,dishes:kitchen_cam,litter:hallway_cam
# ---------------------------------------------------------------------------
FRIGATE_URL=
CAMERA_WATCHPOINTS=
# How long groceries may stand out before the nudge is redirected to somebody else,
# overriding NEGLECT_THRESHOLD_HOURS for that one chore type. 30 minutes by default:
# the default four hours is a sensible wait for a crisp packet and far too long for a
# tub of ice cream.
GROCERIES_NEGLECT_THRESHOLD_HOURS=0.5
# Same LLM host as digest-engine/pantry-vision. OLLAMA_VISION_MODEL must be a
# vision-capable model (see pantry-vision/README.md's identical caveat — plain text
# models cannot see images at all).

23
digest-engine/README.md
View File

@ -224,15 +224,11 @@ trusting a scheduled run, verify by hand:
half is verified — `ingest/rci_social.py` was run live against the committed
`feeds/rci-social.json` on 2026-08-06 and returned real YouTube and podcast
entries with durations — but no local model has yet been asked to produce the
five question windows, per-marker summaries and `sources` arrays in one JSON
four question windows, per-marker summaries and `sources` arrays in one JSON
document. Check on the first real run that a 14B model actually fills
`sources` rather than dropping the field, that it does not put episodes in
the analysis, and that the compact pass still fits the HA card now that the
section has more to say. The roll-call (`political-struggles`) is the window
to read most sceptically on that first run: confirm every line traces to an
entry that is actually in the context rather than to the model's own memory
of a famous strike, and that a long-running dispute stays on the list across
consecutive runs instead of being dropped as stale.
section has more to say.
14. **The Telegram channel entry in `feeds/rci-social.json`.** The channel name
comes from marxist.com's own footer but was never fetched — reading it needs
the Telethon session, which only exists on the real deployment. Also confirm
@ -268,9 +264,9 @@ trusting a scheduled run, verify by hand:
a Times of Israel claim about Palestinians never reaches the digest in the
section's own voice.
## The political section — sources, ownership, and the five questions
## The political section — sources, ownership, and the four questions
The political digest answers five questions, in this order, one window each
The political digest answers four questions, in this order, one window each
(`synth/prompts/political.md` is written around them):
1. What is relevant for the communist and class struggle **globally** right now.
@ -282,17 +278,6 @@ The political digest answers five questions, in this order, one window each
sources are the `theory` feeds, the organisation's own social output, and the
reports that arrive in the user's own **mail** — the one input here no feed
can supply.
5. What class struggles are **currently going on around the world** — the
`political-struggles` roll-call. This one is an inventory, not a curation:
one line per live strike, occupation or mass movement, kept on the list while
it runs even in a quarter that carried no fresh news of it, which is what the
archive's `times_seen`/`first_seen_at` annotations are for. It is deliberately
the one window where a story being old is not a reason to drop it, and it
survives the compact pass (shortened, and saying it is shortened) because a
roll-call folded into a summary line stops being a roll-call. Everything on it
still has to come from this run's context — the prompt says in as many words
that the model's own knowledge of the world is not a source, since an
inventory question is exactly the shape of prompt that invites one.
Plus a **watch-later** window: new videos and podcast episodes from the
organisation's channels, kept out of the analysis entirely because they are

96
digest-engine/ingest/infra_health.py
View File

@ -1,96 +0,0 @@
"""Infrastructure health for the network digest — CheckMK and every firewall.
READS THE POLLER, DOES NOT POLL. `workshop/health.py` already asks CheckMK and each
OPNsense box every few minutes and keeps a month of samples; this module fetches that
service's `GET /health` and reshapes it for the network section.
That indirection is the whole design and it is worth one paragraph. The digest runs
four times a day. "Is the NAS disk failing right now" is not a question with a
six-hour answer, so polling from here would have produced a snapshot taken at 06:00
and quoted at 12:00. Reading the poller instead gives the digest something a snapshot
cannot have: **how long the state has held**. "Critical since Tuesday" is a different
sentence from "critical", and it is the one that tells you whether to get up.
It also means one set of credentials in one place. digest-engine never learns the
CheckMK secret or any firewall's API key for this — those live in workshop's env file,
and this module needs only workshop's own bearer token.
WHY THIS IS NOT IN opnsense_ids.py
-----------------------------------
That module reads Suricata's *alert log*, with paging and a time window, from one
firewall. This one reads *service state* from many. They answer different questions
("what fired" vs "is it running and is anything broken"), and the alert query is
deliberately left where it already works rather than reimplemented here.
DEGRADES TO NOTHING, LOUDLY
----------------------------
An unreachable workshop service returns `None` and the run continues — same rule as
every other ingestion module here. But when the poller *is* reachable and reports a
target as `unreachable`, that is passed through as a finding rather than dropped: a
firewall nobody can reach is exactly the thing a network digest exists to mention.
"""
from __future__ import annotations
import json
import logging
import os
import urllib.error
import urllib.request
LOG = logging.getLogger(__name__)
WORKSHOP_URL = os.environ.get("WORKSHOP_URL", "").rstrip("/")
WORKSHOP_TOKEN = os.environ.get("WORKSHOP_TOKEN", "")
TIMEOUT = float(os.environ.get("WORKSHOP_TIMEOUT", "10"))
def fetch() -> list[dict]:
"""Health entries for the digest context, or [] when unavailable.
Every entry carries `target` — the firewall's name or the CheckMK site — because a
household with two firewalls must never be handed "the IDS is running" as if there
were one of them. The prompt is told to name the target in anything it says.
"""
if not (WORKSHOP_URL and WORKSHOP_TOKEN):
LOG.info("infra_health: WORKSHOP_URL/WORKSHOP_TOKEN unset, skipping")
return []
try:
req = urllib.request.Request(f"{WORKSHOP_URL}/health")
req.add_header("Authorization", f"Bearer {WORKSHOP_TOKEN}")
with urllib.request.urlopen(req, timeout=TIMEOUT) as resp:
payload = json.loads(resp.read() or b"{}")
except (urllib.error.URLError, urllib.error.HTTPError, TimeoutError, OSError, ValueError):
LOG.warning("infra_health: could not reach the workshop health poller", exc_info=True)
return []
if not payload.get("configured"):
# Nothing is being watched. Distinct from "everything is fine", and the prompt
# must never render the second when the truth is the first.
return []
entries = []
for row in payload.get("results", []) or []:
problems = row.get("detail") or []
entries.append({
"category": "infra_health",
"source": row.get("source"),
"target": row.get("target"),
"state": row.get("state"),
"title": f"{row.get('target')}: {row.get('state')}",
"summary": row.get("summary", ""),
"problem_count": row.get("problem_count", 0),
# When this state started. None means it has held for the whole retention
# window, which the prompt should read as "long-standing", not "just now".
"since": row.get("since"),
"checked_at": row.get("checked_at"),
# Capped: a site with 200 failing services is a real state, and putting all
# 200 in an LLM context is not how you say so.
"problems": problems[:12],
"problems_truncated": max(0, len(problems) - 12),
})
if entries:
LOG.info("infra_health: %d target(s), overall %s", len(entries), payload.get("overall"))
return entries

9
digest-engine/run.py
View File

@ -61,7 +61,6 @@ from ingest import (
financial,
flight_traffic,
grocy,
infra_health,
naval_traffic,
news_rss,
opnsense_ids,
@ -92,9 +91,6 @@ SOURCES = (
("ENABLE_FLIGHT_TRAFFIC_INGEST", "flight_traffic", flight_traffic),
("ENABLE_NAVAL_TRAFFIC_INGEST", "naval_traffic", naval_traffic),
("ENABLE_OPNSENSE_IDS_INGEST", "opnsense_ids", opnsense_ids),
# Reads workshop's health poller rather than polling CheckMK/the firewalls itself —
# see ingest/infra_health.py for why the digest must not be the thing that polls.
("ENABLE_INFRA_HEALTH_INGEST", "infra_health", infra_health),
("ENABLE_CALDAV_INGEST", "calendar", caldav_ingest),
("ENABLE_GROCY_INGEST", "grocy", grocy),
)
@ -113,7 +109,7 @@ SECTION_SOURCES = {
"political": ("news", "rci_social", "financial", "email", "flight_traffic",
"naval_traffic", "calendar"),
"household": ("calendar", "grocy"),
"network": ("opnsense_ids", "infra_health"),
"network": ("opnsense_ids",),
}
DEFAULT_SCHEDULE = "00,06,12,18"
@ -327,9 +323,6 @@ def build_section_contexts(collected, lookback_hours, is_evening_run, previous_r
contexts["network"] = {
"lookback_hours": lookback_hours,
"network_security": collected.get("opnsense_ids", []),
# Host/service state from CheckMK and every firewall's IDS status, each row
# naming its own target. See ingest/infra_health.py and synth/prompts/network.md.
"infra_health": collected.get("infra_health", []),
}
# A section nobody asked for is dropped here, before its context is ever built into

25
digest-engine/synth/prompts/network.md
View File

@ -31,31 +31,6 @@ section to pad.
no network data was collected this run. Do not infer that the network was
quiet, and do not invent an alert, a device, or a signature.
## Machine health: CheckMK, and every firewall by name
The context may carry entries tagged `"category": "infra_health"` — one per monitored
target, from the always-on poller rather than from this run (see
`ingest/infra_health.py`). Each has a `target`, a `state`, a `summary`, a `since`, and
up to twelve `problems`.
- **Always name the target.** There may be more than one firewall, and `main` being
healthy says nothing about `dmz`. "The IDS is running" is a sentence you must never
write when the context has two firewalls in it; write "Suricata is running on main;
dmz has not answered since 14:20."
- **`state: "unreachable"` is a finding, not an absence.** It means the poller asked
and got nothing — a machine that is off, a credential that expired, a cable. Report
it as prominently as a real failure, because it is one, and never as "no problems".
- **`since` is what makes this worth reading.** A critical service that went critical
four minutes ago and one that has been critical since Tuesday call for different
reactions. Say which, using the timestamp, whenever `since` is present. When it is
null the state has held for the whole retention window — say "long-standing", never
"just started".
- **`problems_truncated` is a count of what you were not shown.** If it is non-zero,
say so plainly ("12 of 47 shown"). Never summarise 47 failures from 12 of them.
- If there are no `infra_health` entries at all, say nothing about machine health.
Nothing being monitored and everything being fine look identical from here, and only
one of them is good news.
## History: one alert is noise, the same alert every night is a fact
The context may carry a `history` block from the digest's own archive of past

76
digest-engine/synth/prompts/political.md
View File

@ -15,9 +15,9 @@ Vorarlberg. Write for someone who already holds this politics and needs to be
oriented in the world this week — not for someone who needs to be convinced of
it.
## The five questions
## The four questions
Everything below serves five questions. They are the structure of the section,
Everything below serves four questions. They are the structure of the section,
not a checklist to append: decide what goes in the digest by asking which
question an item answers, and drop it if it answers none.
@ -46,59 +46,10 @@ question an item answers, and drop it if it answers none.
sections arrive there, and they are the one source in this context that no
feed can supply. Treat a comrade's report as a report, not as an anonymous
claim: say which section or comrade it came from where the mail says so.
5. **What class struggles are currently going on around the world?** Strikes,
occupations, factory takeovers, mass movements and general strikes that are
*live right now* — an inventory, not a curation. This is the one question that
does not ask what is most relevant or what moved this run: a dispute that has
been running for six weeks with nothing new to report still belongs on the
list, because the reader needs to know what is going on, not only what
happened since breakfast. See "The roll-call" below for how to build it.
Each question gets its own window (see "Output"). A question with nothing worth
saying this run gets one honest line, never filler.
Questions 1 and 5 look at the same material and are not the same job. Question 1
asks what a communist needs to understand — the featured, quoted, analysed items,
few of them. Question 5 asks what is happening — the full standing list, one line
each, no analysis. An item can be in both: featured in `political-global` with
its quote and its theory, and named again in the roll-call as one of the
struggles currently running. That is not duplication, it is the difference
between the briefing and the map.
## The roll-call
Question 5's window is a list of the class struggles this run's context shows to
be under way. It has its own rules, because a standing inventory fails in
different ways from a curated section.
- **Everything on it still comes from this run's context.** "No speculation"
below is not relaxed here: a struggle goes on the list because entries in this
run's material describe it, never because you know of it from elsewhere or
remember it from a previous digest. Your own knowledge of the world is not a
source.
- **`times_seen` and `first_seen_at` are what keep it standing.** An entry that
has appeared in six consecutive runs is exactly the long strike this question
exists to hold on to — it stays on the roll-call at full standing, and the
"a story featured for three runs needs a reason to be featured again" rule in
"History" governs questions 1–4, not this one.
- **One line each**: what, where, who is out or occupying, and since when if the
context says. "Rail workers, France — national strike over pensions, running
since 4 Aug (first seen 5 Aug, in every run since)." No analysis, no impact
paragraph; that belongs to question 1 if the item earns it.
- **Say when a struggle's state is unknown rather than assuming it continues.**
If the last thing the context said about a strike was a fortnight ago and
nothing since, the line says that: "no entry in this run's material since 27
Jul — outcome unknown here." A roll-call that quietly implies everything on it
is still live is worse than a short one.
- **Ended is a finding.** A strike settled, defeated, sold out or won this run
stays on the list once, marked with how it ended and by whose account, then
drops off. Defeats are not omitted to keep the list encouraging.
- Sources attach the same way as everywhere else — see "Curating and quoting".
A line with no entry behind it does not go on the list.
- Cap it at roughly a dozen entries. If more clear the bar, keep the largest and
those a reader could act on or be asked about, and say plainly that the list
is cut ("12 of ~20 disputes in this run's material").
## What this section is, and is not
This is a curation-and-correlation exercise, not a running commentary on
@ -198,7 +149,7 @@ the reader might choose to watch or listen to later.
says** — never guess at the content of a video from its title alone.
- Order newest first, cap it at about six entries, and omit the window entirely
when there are no new episodes this run. Never carry an episode over into one
of the five question windows or onto the globe.
of the four question windows or onto the globe.
- A `"theory"` article whose title is marked as a podcast (marxist.com prefixes
these with `[Podcast]`) belongs in this window too, not in the analysis.
@ -377,9 +328,7 @@ context — a specific entry, figure, or quote. This is not a style preference:
## Curating and quoting
For each entry that passes the relevance filter above — i.e. that answers one of
the five questions — and inside the window belonging to that question (question
5's window is the exception: it is one line per struggle with its sources, and
the quoting and analysis rules below do not apply to it, see "The roll-call"):
the four questions — and inside the window belonging to that question:
- Feature it explicitly with a short excerpt or quotation taken verbatim from
the entry's own text (its title/description/body field, not your
@ -574,14 +523,6 @@ Rules:
Omit the window entirely if the run genuinely has none.
- `political-rci` — question 4, the International and comrades' reports. Omit
if there is nothing; never manufacture organisational news.
- `political-struggles`, titled something like **"Struggles under way"** —
question 5, the roll-call. `kind: "list"`, one line per live struggle, per
"The roll-call" above. Most of the amber `star` and red `hammer-sickle`
markers on the globe should correspond to a line here; keep the two saying
the same thing. Omit the window only when this run's material describes no
live struggle anywhere — say so in one line in `political-global` when that
happens, because it is a surprising claim about the world rather than a quiet
run.
- `political-agenda` — the upcoming meetings and their agenda points, per "The
agendas" above. Omit when no agenda arrived.
- `political-todo`, titled **"Political todos"** — the tasks those agendas
@ -592,14 +533,11 @@ Rules:
You may add further windows beyond these when an item needs its own space (a
long piece of analysis, the financial indicators as their own `kind: "list"`
with each line stating the move *and* what it means for working people). Do not
drop or rename the five question windows to make room.
- At `detail_level: "compact"` keep the globe and fold questions 1–4 into a
drop or rename the four question windows to make room.
- At `detail_level: "compact"` keep the globe and fold the questions into a
single `kind: "list"` window — one or two lines each for questions 1 and 2, one
line each for 3 and 4 if they have anything. Sources still attach; they cost no
space when folded. **Keep `political-struggles` as its own window even here**,
shortened to about six lines and to the largest and nearest struggles, saying
it is cut: a roll-call folded into a summary line stops being a roll-call.
**Keep `political-todo` as its own window even here** too: it is
space when folded. **Keep `political-todo` as its own window even here**: it is
the one part of this section a person acts on rather than reads, and it is the
first thing they will look for on a phone.
- `narration` is spoken aloud by a TTS voice, so no markdown, no URLs, no emoji.

179
docs/components.md
View File

@ -44,45 +44,6 @@ Components
1xSound System
1xRuView Presence Node (ESP32-S3 CSI board)
#Workshop / Office
The room the workshop assistant runs in (`workshop/`, docs/workshop-assistant.md).
Structurally the kitchen display again — a Sway kiosk, a screen, a camera — with two
differences that matter for buying: the screen wants to be BIG (schematics, board
plans and a camera grid at the same time), and the camera is a MACRO problem, not a
wide-angle one.
1xTiny PC (have — "near infinite Tiny Pcs")
1xLarge monitor or TV, wall-mounted above the bench
1xUSB inspection/macro camera -> reading a service tag or PCB silkscreen at 10-20cm
1xVoice Reciever - Round screen thingy
1xRuView Presence Node (ESP32-S3 CSI board)
#Network cameras
Distinct from every USB camera in this project, which are all fixed-purpose at angles
useless for anything else (an item held to the kitchen lens; an appliance door). These
are the ones on the network, ingested by the Frigate/go2rtc this stack already runs,
and they are what the workshop display can actually show.
2-4xPoE or WiFi camera (RTSP/ONVIF) -> count depends on which rooms; see #Open items
#Appliance monitoring (fridges + freezers)
The household runs more than one cold appliance — the main fridge's freezer
compartment is too small, so there is at least a second freezer elsewhere. That is
what makes "which one is it in?" a real question; see
[`fridge-item-location.md`](fridge-item-location.md) for why the answer is *door
sensors plus an outward-facing camera*, and why nothing goes inside the appliance.
Per appliance (assume 3 to start — kitchen fridge, kitchen freezer compartment,
second freezer):
1xZigbee door contact sensor (mounted on the OUTSIDE face of the door, never inside)
1xDoorway camera (looking at the door opening from outside, ~1.8-2m up)
The appliances themselves are already owned and are not bought by this project. The
room the second freezer lives in is not decided yet — see #Open items.
##Lighting
~16x RGB smart bulb (exact count/room split TBD — see #Open items). Zigbee, not
@ -101,17 +62,13 @@ ecosystem. See #Need for the specific pick and why.
2x ThinClient (tiny)
1x Miniscreen System (Lenovo all-in-one, built-in touchscreen — already owned)
1x Beamer
5x Voice Reciever - round screen thingy (Living Room, Loggia, Linus Room, Kitchen, Workshop)
4x Voice Reciever - round screen thingy (Living Room, Loggia, Linus Room, Kitchen)
1x HA Voice PE
5x Sound System (all confirmed: Living Room, Loggia, Linus Room, Amirs Room, Kitchen)
1x Tiny PC with Mini Touchscreen (Kitchen)
1x Camera (Kitchen)
2x Spare webcam (destination TBD)
6x RuView Presence Node (ESP32-S3 CSI board — one per room: Living Room, Loggia, Linus Room, Amirs Room, Kitchen, Workshop)
3x Zigbee door contact sensor (one per cold appliance door — count depends on the final appliance list)
2x Doorway camera (network/RTSP, one per appliance that isn't already covered by a camera pointing the right way)
1x Tiny PC + 1x large monitor/TV + 1x USB inspection/macro camera (Workshop/Office)
2-4x Network camera (RTSP/ONVIF — rooms TBD, see #Open items)
5x RuView Presence Node (ESP32-S3 CSI board — one per room: Living Room, Loggia, Linus Room, Amirs Room, Kitchen)
#Lighting
~16x RGB smart bulb (Zigbee) — count/room split not finalized
@ -121,30 +78,12 @@ Living Room: 1x ThinClient, 1x Voice Reciever, 1x Sound System (have), 1x RuVie
Loggia: 1x Miniscreen System (have), 1x Sound System, 1x Voice Reciever, 1x RuView node
Linus Room: 1x Voice Reciever, 1x Sound System, 1x RuView node, +1 unresolved (???)
Amirs Room: 1x Beamer, 1x ThinClient, 1x HA Voice PE, 1x Sound System, 1x RuView node
Kitchen: 1x Tiny PC w/ Touchscreen, 1x Camera, 1x Voice Reciever, 1x Sound System, 1x RuView node,
2x door contact sensor (fridge + freezer compartment), 1x doorway camera
Workshop: 1x Tiny PC (have), 1x large monitor/TV, 1x inspection camera, 1x Voice Reciever, 1x RuView node
Wherever the second freezer is: 1x door contact sensor, 1x doorway camera
Kitchen: 1x Tiny PC w/ Touchscreen, 1x Camera, 1x Voice Reciever, 1x Sound System, 1x RuView node
#Open items
- Linus Room's "???" line is still undecided.
- Bulb count/room split is a guess ("like 16") — not yet mapped to specific rooms
or fixture counts per room.
- **Which room the second freezer is in**, and therefore whether its doorway camera
needs its own PoE/power run or can share the kitchen's. Also whether the kitchen
fridge and its freezer compartment are one door or two — that decides whether the
contact-sensor count is 2 or 3.
- **How many network cameras, and in which rooms.** Priced at 3 as a placeholder. The
workshop display can show any of them; which ones are worth watching is a decision
nobody has made, and unlike the bulbs this one has a privacy dimension — a camera in
a room is a camera in a room, the same caveat `hosts/thin-client`'s README already
spends a section on for gesture control.
- **Whether the workshop monitor needs to be new at all.** It is the only display here
with no touch requirement; almost any HDMI panel or old TV does the job.
- **Whether the kitchen's existing C920 can double as the kitchen fridge's doorway
camera.** It is already there and already on a host, so it is free if the display
happens to face the fridge — and useless if it doesn't. Nobody has measured the
angle; do that before buying a second camera for that room.
#Have
1xSound System (earmarked for Living Room)
@ -153,7 +92,6 @@ near infinite ATX Pcs
1xHA Voice PE
1xZigbee adapter
1xLenovo all-in-one PC, large built-in touchscreen (Loggia's Miniscreen System — free, already have it)
1xMiraBox N4 Pro stream dock, xVSDinside-branded (10 LCD keys, 4 RGB-lit rotary encoders) — sits on a desk, not in a room's build; drives one room's colour lamps via OpenDeck, see stream-dock/
##Need
@ -242,92 +180,6 @@ the identity registration flow, not just kitchen-display/door-panel).
pantry-vision's food-photo use case that specifically wants the C920's autofocus,
so there's no reason to pay for that here.
#Appliance door sensors — need ~3
One per cold-appliance door, feeding the Zigbee2MQTT mesh this project already runs
(§1.3) — the same reasoning as the bulbs: one local mesh, not a manufacturer cloud.
These are what trigger the doorway cameras below (`pantry-vision`'s
`POST /doorway-event`, see `pantry-vision/README.md`), and they are worth buying
**even if no camera is ever installed**: on their own they already answer "which
appliance was opened, when" and "has the freezer been standing open for four
minutes", which prevents more spoilage per euro than any amount of item-locating.
- [Aqara Door and Window Sensor (Zigbee)](https://www.amazon.com/Aqara-Window-Sensor-Wireless-Security/dp/B07D37FKGX)
(~€10–15 each) — plain reed-switch contact sensor, long-standing Zigbee2MQTT
support, coin-cell powered.
- **Mount the sensor body on the OUTSIDE face of the door and the magnet on the
outside of the frame** (or vice versa). Not inside the compartment: a coin cell at
−18 °C loses a large part of its usable capacity, and every door opening condenses
humid room air onto whatever is in there. The reed switch does not care which side
of the door it is on, so there is no reason to pay the cold penalty.
- Battery life in this application is **unverified** — a fridge door that opens forty
times a day is a much higher duty cycle than the window these are sold for. Budget
for spare CR1632/CR2032s and check one after a month.
#Doorway cameras (appliances) — need ~2
Outward-facing, above the door on the hinge side, ~1.8–2 m, angled so the field of
view is the door aperture and the ~40 cm in front of it. **Never inside the
appliance** — `docs/fridge-item-location.md` has the full argument (no interior
power, condensation on every opening, and −18 °C being outside every consumer camera
module's rating, which rules out exactly the compartment the question came from).
- [TP-Link Tapo C120 (2K, RTSP/ONVIF)](https://www.amazon.com/TP-Link-Tapo-Security-Detection-C120/dp/B0CL5RRZ2M)
(~€30–40 each) — needs to expose a plain **RTSP** stream so Frigate can ingest it,
the same way `chores`' watch points already work. **Unverified**: whether this
specific model serves RTSP without go2rtc as a bridge — the same open question
Phase 20 already carries for its own camera pick (project-plan open decision #23),
and worth resolving once for both rather than twice.
- One of the two **spare webcams** already on this list is a legitimate substitute
for whichever appliance sits next to an existing host — a USB webcam needs a
machine, and the kitchen has one; the second freezer's room probably doesn't.
- Buy **one** first, not both. `docs/fridge-item-location.md` recommends running door
sensors alone for a month before deciding whether item-level localisation is a
thing the household actually wanted, and the test that decides whether the camera
works at all — can a local vision model identify an item in a moving hand at
doorway distance? — can be run today with the kitchen's existing webcam.
#Workshop display — need 1 screen + 1 macro camera
The Tiny PC is covered by "Have". These two are not:
- Large monitor or TV (32-43", 1080p is enough; 4K only if you want two schematics
side by side). **Any HDMI display works** — this is the one screen in the project
with no touch requirement, because the workshop interaction is voice and keyboard
with dirty hands, not fingers on glass. Reuse anything you have before buying.
- [USB digital microscope / inspection camera, 1080p with adjustable stand](https://www.amazon.com/Microscope-Compatible-Magnification-Adjustable-Compatible/dp/B08BC7GDVL)
(~€30-60) — **not a webcam.** The job is reading 2mm text on a service tag or a PCB
silkscreen, so autofocus at 10-20cm and enough resolution to resolve small print
matter far more than field of view. The C920's autofocus does not go near enough.
This is the single most important pick for whether the OCR-first identification in
`docs/workshop-assistant.md` works at all, and it is cheap enough to test the
premise before committing to the rest.
#Network cameras — need 2-4
For the workshop display's camera view, and to give Frigate something to watch beyond
the peephole cam. Must expose **RTSP** — that is the ingest path Frigate and go2rtc
use, and it is the one thing worth checking on the listing before buying.
- [TP-Link Tapo C120 (2K, RTSP/ONVIF)](https://www.amazon.com/TP-Link-Tapo-Security-Detection-C120/dp/B0CL5RRZ2M)
(~€30-40 each) — same model already listed for the appliance doorways, deliberately:
one camera model across the house means one set of quirks to learn, one stream
configuration to get right, and spares that fit anywhere. **Unverified**: whether
this model serves RTSP without go2rtc as a bridge (project-plan open decision #23).
- Buy **one first** and get it into Frigate before ordering the rest. Every camera
after the first is a repeat of a solved problem; the first one is where you find out
whether the model was the right pick.
#mmWave presence radar — need 1 to start, then 1 per room you care about
For **exact positions inside a room** on the 3D floorplan. BLE answers "which room";
this answers "where in it", and the two are fused in `identity` — see
`docs/endpoint-surfaces.md` for the rule and for why it is deliberately timid.
- [HLK-LD2450 24GHz mmWave radar module](https://www.amazon.com/Waveshare-Millimeter-Detection-Multi-Target-Trajectory/dp/B0CX5QNXH2)
(~€15–25 each) — tracks up to **three moving targets** and reports each one's x/y,
which is the specific capability that makes this worth doing at all. Native ESPHome
support (`ld2450`), so it rides the same firmware pipeline as the other ESP nodes and
needs an ESP32 per sensor (the RuView boards in this list are a separate job — do not
double up firmware on one chip).
- **It cannot tell you who.** A radar sees a moving blob. Identity still comes from BLE,
and with two people in one room neither blob gets a name — by design, because guessing
is worse than not knowing.
- **Buy one, mount it, and check the rotation before buying more.** Getting the facing
wrong mirrors every position it reports, and the failure looks plausible rather than
broken. One room proves the whole chain (sensor → ESPHome → HA → identity's fusion →
the 3D plan) for ~€20.
#Fixed BLE tags — need ~4 to start
For household members who don't (or shouldn't have to) carry a phone for presence to
work — a grandmother without a smartphone is the concrete case, but this is also just
@ -403,8 +255,8 @@ local pricing/VAT/shipping.
| TMY 1080P mini projector | 1 | €40–65 | €40–65 |
| HDMI cable | 1 | €6–10 | €6–10 |
| Ceiling/shelf mount *(optional)* | 1 | €15–25 | €15–25 |
| Waveshare ESP32-S3-Touch-LCD-1.85C board | 5 | €35–45 | €175–225 |
| USB-C 5V/3A wall adapter | 5 | €7–12 | €35–60 |
| Waveshare ESP32-S3-Touch-LCD-1.85C board | 4 | €35–45 | €140–180 |
| USB-C 5V/3A wall adapter | 4 | €7–12 | €28–48 |
| Fosi Audio V3 amp | 4 | €130 | €520 |
| Micca MB42X G2 speakers, pair | 4 | €100 | €400 |
| 16AWG speaker wire, 100ft spool *(one-time)* | 1 | €15–22 | €15–22 |
@ -413,25 +265,10 @@ local pricing/VAT/shipping.
| Logitech C270 spare webcam | 2 | €25–35 | €50–70 |
| innr RB 285 C Zigbee RGB bulb | ~16 (TBD) | €15–20 | €240–320 |
| iTag-Tiny fixed BLE tag | 4 | €3–8 | €12–32 |
| Espressif ESP32-S3-DevKitC-1-N8R2 (RuView) | 6 | €8–12 | €48–72 |
| HLK-LD2450 mmWave radar *(buy 1 first — see that section)* | ~3 | €15–25 | €45–75 |
| Aqara Zigbee door contact sensor (appliance doors) | ~3 | €10–15 | €30–45 |
| Tapo C120 doorway camera *(buy 1 first — see that section)* | ~2 | €30–40 | €60–80 |
| Tapo C120 network camera (rooms/workshop view) | ~3 | €30–40 | €90–120 |
| Workshop monitor/TV *(reuse one first if you have it)* | 1 | €120–250 | €120–250 |
| USB inspection/macro camera (workshop) | 1 | €30–60 | €30–60 |
| Espressif ESP32-S3-DevKitC-1-N8R2 (RuView) | 5 | €8–12 | €40–60 |
**Subtotal (excludes the optional beamer mount): ~€2,289–2,921**
**Subtotal, including the optional beamer mount: ~€2,304–2,946**
The workshop line items are the ones to sequence rather than buy at once: the macro
camera is cheap and decides whether the OCR-first identification works at all, the
monitor can be anything you already own, and the network cameras are one-then-more.
The two appliance-monitoring lines are the only ones on this list bought in a
deliberate order rather than all at once: the door sensors are worth having on their
own merits, the cameras are not worth having until the sensors have proved the
question is real. Buying one camera instead of two costs €30–40 to find out.
**Subtotal (excludes the optional beamer mount): ~€1,831–2,182**
**Subtotal, including the optional beamer mount: ~€1,846–2,207**
All 5 Sound Systems (Living Room, Loggia, Linus Room, Amirs Room, Kitchen) are
confirmed — see `#Open items` above — so all 4 needed amp+speaker kits are in the

219
docs/endpoint-surfaces.md
View File

@ -1,219 +0,0 @@
# Two shared endpoint surfaces: the now-playing visualiser and the 3D floorplan
Design notes for two things that must look the same on every screen in the house — a
thin client on a TV, the Loggia all-in-one, the kitchen panel, the door panel — and
which therefore belong in a **shared, vendored SDK** rather than being written four
times.
> **Status: both are built** — `render/media-visualiser/` and `render/floorplan-3d/`.
> One thing changed in the building, and it is called out in section 2: the 3D view is
> **canvas 2D, not three.js**. The scene turned out to be prisms on a plane, which an
> isometric projection draws in ~200 lines without the megabyte, without a build step,
> and with predictable performance on the small panels. Everything else below was
> implemented as specified.
---
## 1. The now-playing visualiser
Wanted: a circular CAVA-style spectrum behind every playing-status screen, coloured
from the album art, with lyrics scrolling under the cover when they exist.
### The audio problem, which is the whole problem
CAVA reads an audio stream. The endpoints do not all have one:
| Endpoint | Is the audio local? | Can it get a spectrum? |
|---|---|---|
| Thin client running mpv/spotifyd | **yes** | Yes — real FFT of the actual output |
| Audio endpoint (Spotify Connect box) | yes, but it is headless | Yes, but there is no screen to draw on |
| Kitchen / door panel showing *what the living room is playing* | **no** | **No. There is no audio here at all** |
So a design that requires real audio analysis works on one endpoint and silently
degrades to a dead circle on the others — the worst outcome, because the screen looks
broken rather than looking deliberate.
**Two-tier, declared up front:**
- **Reactive tier** — where audio is local, CAVA (or a WebAudio `AnalyserNode` when the
player is in the browser) drives the bars. `cava` has a `raw` output mode writing
plain numbers to a FIFO, which a small agent can publish over MQTT; that is the least
fragile path on a machine already running an agent.
- **Synthetic tier** — everywhere else, the ring is animated from **track position and
tempo**, not from silence. It breathes with the beat rather than pretending to
analyse it. Nobody watching a kitchen panel from across the room can tell, and it
never looks broken.
Do not let the synthetic tier claim to be the reactive one anywhere in the UI. It is a
mood light, and the moment somebody believes it is a spectrum they will report it as a
bug every time it does not match a bass drop.
### Colours from the album art
Pull 3–5 dominant colours from the cover, client-side, at load:
1. Draw the cover into a small offscreen canvas (64×64 is plenty — this is a palette,
not a photograph).
2. Bucket pixels in RGB space, take the top buckets by population.
3. **Reject near-greys and near-blacks** before ranking. Album art is full of them, and
a palette extracted naively from a dark cover gives you four indistinguishable dark
greys and a visualiser that looks switched off.
4. **Force a minimum contrast against the background.** Lift the chosen colours in
lightness until they clear the panel behind them; a visualiser you cannot see is
the same as no visualiser, and this is the step that gets skipped.
No library needed — that is about forty lines of canvas work, and it keeps the
"vendored, dependency-free" property the canvas SDKs already have.
### Lyrics
- **Source**: whatever the player exposes. Music Assistant and MPD both surface
synced-lyrics fields when the provider has them; `.lrc` sidecar files are the other
common case for local libraries.
- **Synced (LRC) vs plain**: with timestamps, highlight the current line and scroll it
to centre. Without them, do not fake it — scroll slowly, or just show the text. A
plain-lyrics block auto-scrolled at a guessed rate is wrong within ten seconds and
stays wrong.
- **Absent is the normal case.** Most tracks in most libraries have no lyrics. The
layout must be designed for "no lyrics" as the default state, with lyrics as the
addition — not a gap where they would go.
- Never fetch lyrics from the internet at render time. If lyrics are worth having,
they are worth caching where the track is.
### Where it lives
A new `render/media-visualiser/` in the shared SDK style: one JS file, one CSS file,
vendored into each host that needs it, exactly as `digest-canvas-sdk` and
`canvas-sdk` already are. Inputs are a normalised now-playing object
(`{title, artist, album, art_url, position_ms, duration_ms, tempo?, lyrics?}`) and an
optional spectrum feed. Every endpoint already receives now-playing over MQTT; nothing
new has to be plumbed for the synthetic tier.
---
## 2. The 3D floorplan
Wanted: the Pebble app's presence view, in 3D, on any endpoint — rooms, and who is in
them, with profile pictures.
### It is the same data, and that is the point
`GET /floorplan/presence` already returns everything: room polygons normalised 0–1,
each room's occupants, each occupant's `color`, `initial` and `has_photo`, plus
`unplaced` for people who are home but not locatable. The Pebble app and this render
the same JSON at different fidelities — which is the reason to build the second one at
all. If they diverge, one of them is lying.
### Extrusion, not modelling
Do not author a 3D model. Take the existing 2D polygons and **extrude them to a wall
height**, which is one `THREE.ExtrudeGeometry` per room and needs no new data beyond a
single `wall_height_m` per level. A hand-authored model would be prettier, immediately
stale the first time a room is redrawn, and unmaintainable by the person who drew the
plan in a 2D editor.
Camera: fixed isometric by default, drag to orbit, no free-fly. An orbit camera on a
wall panel is something people knock out of alignment and cannot get back.
### The occupant markers
This is where 3D earns itself, because the Pebble's constraint — 18×18 px, no room for
a face — is gone:
- A **billboarded disc** above each occupied room, always facing the camera (never a
flat sprite lying on the floor, which is unreadable at a glance).
- **Profile picture inside the disc**, ringed in the person's colour, falling back to
the initial on the colour when `has_photo` is false. The ring matters even with a
photo: it is what ties this marker to the same person's marker on the watch and in
the admin panel.
- **Occupied rooms lit, empty rooms dark**, the same rule the watchapp uses, and for
the same reason: state should read before detail. In 3D that is an emissive floor
material rather than a fill colour, and the third state — *drawn but never reported
by HA* — stays visually distinct, because rendering "no data" as "empty" is a quiet
lie on any screen.
- **`unplaced` people get a shelf**, not a hidden list: a strip along the bottom
showing everyone who is home but unlocatable. They are the people you are most often
looking for.
### Exact positions within a room
Room-level presence answers *which room*. Exact positions need a sensor that reports
**coordinates**, and BLE is not that sensor: RSSI-to-distance is noisy enough that
trilateration in a house lands in the wrong room often enough to be useless.
The realistic source is **mmWave radar** (LD2450-class, ~€15–25 per room, ESPHome
native), which tracks up to three moving targets and publishes each one's x/y in
millimetres. What it cannot do is say **who** — it sees a moving blob.
So position and identity come from different sensors, and the whole design is in how
they are combined:
| Signal | Source | Gives |
|---|---|---|
| who | BLE / Bermuda | a person, resolved to a room |
| where | mmWave | a coordinate, with no name |
**The fusion rule is deliberately timid** (`floorplan_presence()` in `identity`):
> exactly one occupant in the room **and** exactly one target in the room → that target
> is that person. Anything else → the targets stay anonymous and the occupants stay
> unpositioned.
Two people in a room produce two blobs that cannot be told apart. Guessing which is
which would put a name on the wrong person, and a presence display that does that
occasionally is worse than one that never tries — you cannot tell the wrong answers
from the right ones. `position_ambiguous` says out loud when this has happened.
What the plan needs before any of it works, both drawn by a human because nothing can
infer them:
- **`metres_wide` per level** — the real-world width of the 0–1 extent. Without it a
reading in metres cannot become a point on the plan, and the API reports that rather
than guessing a scale.
- **A sensor placement per radar** — where it is on the plan and *which way it faces*.
A wrong rotation mirrors every target it reports, which is the single most likely way
to get positions that look plausible and are wrong.
Rendering follows the same honesty: a named person at a fused position gets their
marker with a footprint dot on the floor (without it, a marker floated above the floor
reads as being further back in the room); an unattributed target gets a **hollow dashed
puck with a question mark** — no colour, no initial, because every visual language this
view has for a person is reserved for people it can name. A radar target lights the
room even with nobody named, because somebody *is* in there.
### The honest constraint — and how it was resolved
The plan was to vendor three.js and call it a deliberate break with the
dependency-free rule. Building it made the cheaper answer obvious and it was taken
instead: **canvas 2D with an isometric projection and painter's-algorithm sorting.**
The scene is prisms standing on a plane — no lighting model worth the name, no
textures, no physics, no camera motion beyond an orbit. That is ~200 lines of canvas
2D, it redraws in well under a millisecond on the small panels (where a WebGL context
on integrated graphics is a much less predictable proposition), and the frontend stays
at zero dependencies with nothing to keep patched.
This is not a compromise version of the three.js plan; it is the smaller correct tool
for this specific scene. If the view ever grows real lighting or an imported model,
three.js becomes right and `floorplan3d.js` becomes the fallback for the weak panels.
And the same caveat as the watchapp, which no amount of rendering fixes: **this is only
as right as room-level presence is**, which has never been measured in this house. A
beautiful 3D house with everybody sitting in the `unplaced` shelf is a beautiful 3D
house that tells you nothing. Test that first — the admin panel's floorplan tab with
**Live** ticked answers it for free.
---
## Build order
1. **The visualiser's synthetic tier**, on one endpoint. It needs no new data and no
audio plumbing, and it is what tells you whether the effect is worth the reactive
tier's complexity.
2. **Album-art palette extraction**, which is where the look actually comes from.
3. **Lyrics**, synced-only at first.
4. **The reactive tier**, on the thin clients that have local audio.
5. **The 3D floorplan** — built. It turned out not to be a dependency addition at all
(see above), so the sequencing argument that put it last no longer applies. The
*other* reason still does, in full: **it is only as right as room-level presence,
which has never been measured here.** Test that before believing the picture.

223
docs/fridge-item-location.md
View File

@ -1,223 +0,0 @@
# Knowing *where* a thing is: multiple fridges, shelves, and what it would cost
A feasibility note that has since grown a build. `pantry-vision` (Phase 17) knows
**what** the household has and **when** it expires. This asks what it would take to
also know **where** — which of several fridges, and how far inside one.
The recommendation at the bottom is deliberately smaller than the question, and the
reason is in the middle.
> **Status, 2026-08-10.** Steps 1 and 4 of the recommendation are now written:
> appliances as Grocy locations with a `POST /transfer` action and a "Move to…"
> control on the edit screen, and the door-sensor→camera→hint path
> (`pantry-vision/doorway.py`, `POST /doorway-event`). **Step 2 is hardware nobody has
> bought and step 3 is a month nobody has spent** — which means the camera half is
> written against an assumption that has never been tested: that a local vision model
> can identify an item in a moving hand at doorway distance. An appliance can be
> configured with a door sensor and no camera, and that remains the recommended way to
> start. Nothing about writing the code changes the order in which it is worth
> switching on.
## First, decide which question is actually being asked
These are three different problems with three different price tags, and they get
conflated constantly:
| Level | The question | Honest difficulty |
| --- | --- | --- |
| **L1 — which appliance** | "Is the mustard in the kitchen fridge or the loggia one?" | Solved today, no new hardware, one extra tap |
| **L2 — which zone** | "Door shelf, middle shelf, or the crisper drawer?" | Achievable at *hint* quality (60–80%), real hardware, real work |
| **L3 — exact position** | "Second from the left, behind the milk" | Not worth it. See "Occlusion" below |
The value is overwhelmingly at L1, and it is the level this household actually has a
problem at — the whole premise of the question is *multiple fridges*, because the
freezer compartment of the main one is too small. Somebody standing in the kitchen
wondering whether to walk to the loggia is asking an L1 question. L2 saves ten seconds
of looking; L3 saves none, because by the time you have the door open you can see.
**Do not build downward from L3.** Every camera-in-the-fridge design starts there,
because that is the impressive demo, and every one of them founders on the same rock.
## L1: available now, and mostly a data-model decision
`pantry-vision` already resolves the confirm screen's fridge/freezer/cupboard answer
to a **Grocy location**, created by name (`PLACEMENT_LOCATION_NAMES` in `server.py`,
`PANTRY_LOCATION_*` in the env file), and passes `location_id` on the stock add. Grocy
has modelled multiple locations natively since long before this project existed.
So L1 is not a vision problem at all. It is three small changes, and two and a half of
them are now written:
1. **The placement answer becomes the real appliances** — "Fridge (kitchen)", "Freezer
(loggia)", "Cupboard" — named in `PANTRY_LOCATION_*` and resolved to Grocy
locations. *Done, with a caveat*: the confirm screen still offers the four
categories (fridge / freezer / cupboard / counter) and each maps to one location
name, so a household with two fridges has to pick which one that word means and
move the other by hand. Reading the appliance list straight off Grocy's locations
is the obvious next step and has not been taken.
2. **`/inventory` carries the location through**, so every list says where. *Done* —
`location_id`/`location` on every row, `locations` on the response.
3. **Moving something between fridges needs a "moved" action**, or the data rots
within a fortnight. *Done* — `POST /transfer` onto Grocy's own transfer endpoint,
with a picker on the edit screen.
Point 3 is the whole ballgame, and it is the thing every camera proposal below is
really trying to buy its way out of: **the expensive part is not learning where things
are, it is noticing when they move.**
## L2/L3: what the camera approaches actually run into
### Occlusion is the rock
A single camera sees the front row. Groceries are stored in rows. The mustard behind
the milk is invisible, and no model — local, cloud, or otherwise — recovers it from a
picture that does not contain it. Multi-camera stereo does not fix this either; it is
not a depth problem, it is a line-of-sight problem.
This is why L3 is off the table and why L2 is a *hint*, not an assertion. Any design
that needs the camera to enumerate a shelf's contents is buying an answer that is
silently incomplete, which for an inventory is the worst failure available: "we have
no mustard" from a system that simply could not see it produces a second jar of
mustard.
### The interior-camera problems, in the order they will bite
- **Power.** Fridges have no interior outlet. The three options are a battery (poor:
cold cuts usable lithium capacity substantially, and this is a duty-cycled camera,
not a doorbell), a flat ribbon cable through the door gasket (works — this is how
retrofit fridge cams do it — but it is a modification to the seal of an appliance
that may be rented), or drilling a grommet (irreversible).
- **The freezer is out of scope for interior cameras, full stop.** Off-the-shelf
camera modules — ESP32-CAM, Pi camera, USB webcams — are specified to roughly 0 °C
at best. −18 °C is outside every consumer part's rating: lubricants stiffen,
electrolytics lose capacitance, and condensation cycling on every door open
eventually gets inside the enclosure. Industrial cold-chain cameras exist and cost
more than the freezer. **Given that the second appliance exists precisely because
the built-in freezer is too small, this rules interior cameras out of the exact
compartment the question started from.**
- **Condensation.** Every door opening puts warm humid air on a cold lens. It fogs,
and it fogs for minutes — longer than the interaction. Mitigations (sealed housing
with desiccant, conformal coating, a lens heater) all cost power, which brings back
problem one.
- **Light.** The interior lamp is on only while the door is open, so the only usable
frames are the ones during an opening — at which point the camera might as well be
outside.
Every one of those bullets is a *class* of problem, not a part-selection detail. Taken
together they say: **do not put the camera inside.**
### The approach that survives all four: watch the doorway, not the shelf
Point a camera at the **plane of the door opening**, from outside, and trigger on a
door sensor. Then you are not parsing a packed shelf — you are watching one item, held
in one hand, unoccluded, at a known moment, against a known background, with the
kitchen's own lighting. That is the *same* recognition problem `/identify` already
solves, which means it reuses the prompt, the model, and the "propose, never
auto-commit" guardrail instead of needing a new perception stack.
What it buys: **transactions, not state.** "A jar of X went into the loggia fridge at
18:42." Transactions are what keep L1 from rotting (the problem above), and a
sequence of them gives L2 for free if the camera can also see *which shelf the hand
went to* — a much weaker inference than reading the shelf, and one that degrades
gracefully into "somewhere in this fridge".
What it costs in honesty: it misses. Two items at once, an arm across the lens, a foot
closing the door. So its output is a **hint with a timestamp** — "last seen going into
the loggia fridge, Tuesday 18:42" — never an assertion the UI treats as fact. That
framing is what makes a 70%-accurate system useful instead of infuriating, and it is
the same distinction `digest-engine` already draws between what a source *says* and
what is *so*.
This is what `pantry-vision/doorway.py` implements. An HA automation on the contact
sensor POSTs `/doorway-event`, the service answers 202 and takes a short burst off the
request thread (three frames, stopping at the first that recognises anything), and the
result lands in **its own SQLite file** — not in Grocy, because Grocy owns stock and
has nowhere to put "a camera thinks it saw something like this go past that door, and
might be wrong". Hints expire after 30 days, because "last seen a month ago" tells
nobody anything they didn't know. The edit screen shows the sighting next to the
location Grocy records, and acting on it is a tap on *Move*.
### Where the cameras go, concretely
- **One camera per appliance**, mounted above the door on the hinge side, ~1.8–2 m,
angled down so its field of view is the door aperture and the ~40 cm in front of it.
Not inside the door swing. `docs/components.md` already lists **2× spare webcam
(destination TBD)** — this is a plausible destination for one of them, which makes
the experiment nearly free.
- **A contact sensor on every fridge and freezer door.** Zigbee, on the coordinator
this project already has. This is the single highest value-per-euro item in this
whole note and it is worth doing *even if no camera is ever installed*: it is what
triggers the capture, it is what tells you which appliance a transaction belongs to,
and on its own it already supports "the loggia freezer has been open for four
minutes" — which prevents more spoilage per euro than any amount of localisation.
- **No new inter-VLAN paths.** Both devices sit where the existing camera/IoT devices
sit; the capture is pulled by the container host, same direction as everything else.
### Approaches ruled out, and why
- **RFID/NFC tags per item.** The only technology that genuinely answers L3. It also
requires tagging every single item by hand, which is a *worse* manual step than the
one tap L1 costs — and a fridge is a metal box full of water, which detunes UHF tags
badly. Rules itself out twice.
- **Load cells under each shelf.** Excellent at "how much is left", blind to identity,
and a per-shelf retrofit on appliances that may be rented.
- **BLE/UWB.** Nothing to attach a transmitter to. Not applicable to passive goods.
## The compute question nobody asks until it's too late
Reading a shelf is not the same task as reading one held-up jar: small objects,
partial labels, angles, occlusion. A 7–13B local VLM will be markedly worse at it than
at the current task, and the honest architecture is detect-then-crop — a small object
detector proposing boxes, the VLM reading each crop — which is a Frigate-class,
always-on GPU workload.
There is one RTX 3060 in this design, and contention on it is already an open
question (project plan open decision #4, `MAX_LOADED_MODELS=1`). Adding a
continuous detector to a GPU that also has to answer Assist within voice latency and
run four digests a day is a scheduling problem, not a spare-capacity problem. **The
doorway-camera approach avoids this too**: it is event-driven, a handful of inferences
per day, on the model that is already loaded.
## Recommendation
In order, stopping wherever the household stops caring:
1. **Make L1 real in software** — appliances as Grocy locations, location shown in
every list, and a transfer action so moves get recorded. No hardware. This is the
only step that is unambiguously worth doing, and it must come first regardless:
without it there is nowhere to *put* a camera's answer. **Written.**
2. **Contact sensors on every fridge/freezer door.** Cheap, Zigbee, immediately useful
on their own, and the prerequisite for everything below. See
`docs/components.md`'s "Appliance door sensors" — ~€10–15 each, mounted on the
*outside* face of the door, because a coin cell at −18 °C is a false economy.
**Not bought.**
3. **Live with 1+2 for a month.** The real finding will be whether anyone ever wanted
L2, or whether "which fridge" plus "the door's been open" was the whole need. This
step is not padding — it is the only way to avoid building L2 for a question nobody
turned out to be asking. **Not done, and writing step 4 early does not skip it.**
4. **Only then**, a doorway camera on one appliance, producing timestamped *hints*,
reusing `/identify` unchanged. One appliance, not all of them, until it earns the
second. **Written** (`pantry-vision/doorway.py`), **untested against a real camera,
and configurable per appliance so it can stay off.**
5. **Never** an interior camera in the freezer, and probably not in the fridge either.
A related chore was added alongside this: `chores`' **`groceries_out_of_place`** watch
point, which notices food standing out on a counter and nudges whoever the camera last
saw — litter's twin, with a 30-minute fuse instead of four hours, because the failure
there is spoilage rather than untidiness. It is a *different* system looking at a
*different* surface for a *different* reason, and deliberately so: it knows food is
out, not what the food is, and it never touches Grocy. See `chores/README.md`.
## What in here is unverified
Everything with a number in it. Specifically: consumer camera modules' actual
low-temperature behaviour (the 0 °C figure is a datasheet-class generalisation, not a
part this project has picked or tested); how badly a lens really fogs on a door
opening in this kitchen's humidity; whether a local VLM can identify an item held in a
moving hand at doorway distance at all — that last one is the assumption step 4 lives
or dies on, and it is testable today with the kitchen display's existing camera and
about twenty minutes, long before anything is bought.
Related open decisions in `docs/project-plan.md`: #18 (no vision model picked,
benchmarked, or measured for latency) and #4 (GPU contention). Both apply here
unchanged, and neither gets easier with a second camera pointed at anything.

288
docs/pebble-presence-watchface.md
View File

@ -1,288 +0,0 @@
# A Pebble watchapp that shows who is in which room
Feasibility note. The question: take the floorplan drawn in `identity`'s admin panel,
put it on a Pebble, and mark each room with who is standing in it.
> **Status: built** — `pebble-presence/`. The payload came out at ~80 bytes for a
> three-room test plan including names, well inside the budget below, and the format is
> round-trip tested between the JS writer and the C reader. Two things remain unverified
> and both are about the *new* hardware rather than the code: the Round 2's platform
> identifier, and its round-screen detection in the JS. Nothing hardcodes a resolution,
> so both are one-line changes once the SDK names them.
**Verdict: the watch is the easy part.** Drawing the plan is a few hundred bytes and
some `gpath` calls, and the data already exists at `GET /floorplan/presence`. It is a
**watchapp**, not a watchface — see "Zooming into a room" for why the buttons decide
that.
The hard part is upstream and unchanged by any of this: **room-level presence
itself.** Build it if you want it, but do not expect it to be more right than the
presence data feeding it, which today is "which HA area a trusted BLE entity reports" and has
never been measured for room-level accuracy in this house.
## The hardware: a Pebble Round 2
**That is the target — it is the watch that exists.** 260×260, 64-colour e-paper,
touch. For reference, the rest of the line:
| Model | Screen | Colour | Notes |
|---|---|---|---|
| **Pebble Round 2** | 260×260 | 64-colour e-paper, touch | **The target.** The most pixels in the line, and circular — see the next section, which is entirely about that |
| Pebble Time 2 | 200×228 | 64-colour e-paper, touch | Rectangular, so a plan fits without the geometry below. Fewer pixels, more usable ones |
| Pebble 2 Duo | 144×168 | Black and white | The per-person colour collapses to a grey; two people sharing an initial stop being distinguishable |
Classic Pebbles (144×168) run the same code, cramped.
## Fitting a rectangular plan on a round screen
A floorplan is a rectangle and the screen is a circle, so the plan gets inscribed in
the circle and the corners are simply not available. The arithmetic, for a plan of
aspect ratio `a = w/h` on a usable diameter `D`:
```
w = D·a / √(a²+1)
h = D / √(a²+1)
```
With `D = 240` (260 less a ~10 px margin, because a circular screen's outermost pixels
are where you least want a room boundary):
| Plan aspect | Usable box |
|---|---|
| 1:1 | 170 × 170 |
| 4:3 | 192 × 144 |
| 16:9 | 209 × 118 |
So a squarish plan gets **170×170** — comparable to what a Time 2 gives after chrome,
and more than the 144×168 classics ever had. The Round is not a downgrade for this;
it just has to be told the truth about its shape.
Three consequences to design to:
- **Compute the box in PebbleKit JS, not on the watch.** The projection step already
planned for (normalised polygons → pixels) simply takes the inscribed box instead of
the full screen. No watch-side change at all.
- **No corner furniture.** The "3 of 4 home placed" footer has nowhere to live on a
circle — a line of text at the bottom gets clipped by the bezel radius. Put it at the
vertical centre-bottom, short, or move it into the room-detail screen where there is
a full-width line to use.
- **Round screens have less usable area than their diameter suggests**, and a ten-room
plan at 170×170 gives each room roughly 40×35 px. That is still enough for a 16 px
occupant dot with an initial in it — which is the number that decided against
profile pictures above, and it does not change here.
## The data path
```
identity ──HTTP──> PebbleKit JS ──AppMessage/BT──> watchapp (C)
(container host) (runs on the Android ~2 KB budget gpath + text
/floorplan/presence phone, inside the rendering
Pebble app)
```
Three consequences, all of which shape the design:
- **The watch has no network of its own.** Everything goes through PebbleKit JS, which
runs on the phone, inside the Pebble app, and — this is the important part —
**only while the app is open.** It cannot poll in the background — which bounds the
battery cost, and also means every launch begins with a fetch.
- **The phone has to be able to reach `identity`.** On the home network that is direct.
Off it, that is the WireGuard split tunnel this project already uses for exactly this
class of problem — nothing here justifies putting `identity` on the WAN. If the
tunnel is down it shows its last state with an age on it, not a blank plan.
- **Android only**, which this household is anyway, so no iOS-side caveats apply.
## The payload budget, and why it is not a problem
Pebble's guaranteed AppMessage buffers are small — 124 bytes in / 636 out at the
documented minimum, ~2 KB each way in practice for a JS-backed app. That sounds
alarming until you count what a floorplan actually is:
- Room polygons are already stored **normalised 0.0–1.0** (`floorplan_rooms.points`).
Quantise each coordinate to one byte and a vertex costs 2 bytes.
- A ten-room plan at eight vertices a room: `10 × (1 + 8×2) = 170 bytes`.
- An occupant marker is 3 bytes: room index, colour index, initial.
So a whole floor with everybody on it lands around **200–250 bytes** — comfortably
inside one message, with room to spare for a level name and a timestamp. Send it as a
single byte-array tuple rather than one tuple per room; the dictionary overhead is
what would actually cost you.
**Do the projection on the phone, not the watch.** PebbleKit JS picks the level, scales
the normalised polygons to the watch's pixel box — the inscribed box, on the Round —
drops rooms too small to draw, and sends integers. The watch does `gpath_create` → `gpath_draw_filled` → outline → text.
No floating point, no layout logic, no second copy of the floorplan model on a device
with 64 KB to its name.
## Empty rooms dark, occupied rooms lit
The plan should read as a *state* at a glance, not as a drawing you have to search. So
occupancy is carried by the room fill itself, before you look at any marker:
| | Fill | Outline | Contents |
|---|---|---|---|
| **Empty** | near-black, barely above the background | dim | nothing, or the room's initial letter in the dim outline colour |
| **Occupied** | light — a pale warm grey/white | bright | the occupant dots, in their own colours |
Three details that decide whether this works:
- **Carry it in lightness, not hue.** The occupant dots are already using colour to
mean *who*; if the room fill also used colour to mean *occupied*, the two would
compete on the one channel that matters most on a 64-colour panel. Dark-vs-light is
the strongest signal e-paper has and it costs nothing.
- **A lit room must be lighter than any occupant dot is dark**, or the dot vanishes into
its own room. With `PERSON_COLORS` being mid-to-bright, a near-white fill and a dark
dot outline keeps every one of the eight readable — this is the same reason the admin
panel draws initials in near-black on the person's colour.
- **"Unknown" is not "empty".** A room the plan has drawn but whose area HA never
reports is neither occupied nor confirmed-empty, and rendering it as empty is a
quiet lie. Give it the empty fill with a **dashed or dotted outline** — a third state
that costs one drawing call and is the difference between "nobody is in the study"
and "nothing can see the study".
E-paper is reflective, so "lit" here means a lighter fill, not a backlight. The effect
is exactly the one you want in a dark hallway at 2am: the rooms with people in them are
the bright shapes.
## Profile pictures: no, and the arithmetic says so
On the Round 2, a ten-room plan inscribed at 170×170 gives each room about 40×35 px. A
face inside one, with the room outline still visible, gets about **18×18 px, in 64
colours, on e-paper**. That is not a picture of a person; it is four skin-toned blobs.
And each one costs ~330 bytes to ship, so three of them exceed the entire message
budget that currently carries the whole floor. The rectangular Time 2 is no better —
45×35 px rooms, a 20×20 face — so this is a conclusion about the class of device, not
about the shape of this one.
**Initial plus colour is the right answer, and it is why the colour exists.** A filled
16 px circle in the person's own colour with their initial in `GOTHIC_14_BOLD` reads
at arm's length, costs 3 bytes, and degrades honestly: on a black-and-white Pebble the
colour becomes a grey and the letter still works.
`identity` now serves both fields ready-made — `color` and `initial` on `/people`,
`/presence`, and each occupant in `/floorplan/presence` — so no consumer has to derive
an initial from a name or invent a palette. **The eight palette colours are chosen on
the 2-bits-per-channel lattice (`00/55/AA/FF`) that a colour Pebble renders natively**,
precisely so the colour on the watch is the colour in the admin panel and not a
dithered approximation of it.
## Zooming into a room: why this is an app and not a face
The wanted behaviour — press a button, cycle to the next room, see a plain list of who
is in it — is easy to draw and cheap in bytes. **The catch is that it forces a
watchapp instead of a watchface**, and that is a real product decision, not a detail:
- **Watchfaces do not receive button events.** The buttons belong to the system there
(Select opens the app menu, Up/Down are the system shortcuts), and Pebble's own docs
say **touch is deliberately restricted to watchapps too** — "easier to allow it later
than to take it away once apps depend on it."
- A **watchapp** gets buttons and touch, and can do exactly the requested cycling. What
it does not get is being your default screen: you launch it from the menu, look, and
leave.
So pick which the thing actually is:
| | Watchface | Watchapp |
|---|---|---|
| Shows without launching | **yes** — this is the whole glance-at-wrist value | no, it's a menu entry |
| Button cycling through rooms | no | **yes** |
| Shake to cycle | yes (`accel_tap_service_subscribe`, the long-standing shake-to-reveal trick — worth confirming on your firmware) | yes |
| Practical shape | the plan, occupant dots, nothing else | the plan **plus** the per-room drill-down |
**Decided: build the watchapp.** Button cycling is the point, so the drill-down wins
over being the default screen. What that costs, stated plainly so it isn't a surprise
later: you launch it from the app menu rather than seeing it by raising your wrist, and
its JS — and therefore its data — only lives while it is open, so every launch starts
with one fetch and a moment of "loading". Design for that: draw the last-known state
immediately with its age on it, then repaint when the fetch lands, rather than showing
a spinner on a screen that already has something true to say.
One package is either a face or an app — no binary is both — but the rendering, the
AppMessage handler and the JS are all shared, so **a watchface variant later is a
second `main()` and a build target, not a second project.** Worth keeping that seam
clean while writing it, in case the glance turns out to be what you actually reach for.
### What the drill-down shows
Rooms in the order they are drawn, wrapping at both ends, with Back leaving. (On the
Round 2, confirm which of buttons and touch you actually want to drive this — it has
both, and a circular screen makes a swipe more natural than a button press for
"next". The click config provider and a touch handler are the same twenty lines either
way.)
```
Kitchen <- room name, Up/Down cycles
─────────────
● Amir <- the person's colour, then their full name
● Anna
<- "Nobody here" when empty, never a blank screen
3 of 4 home placed <- the honest footer, see below
```
Full names cost bytes the overview does not need — ten people at ~12 bytes is ~120,
still nothing against the ~2 KB budget, so send them with the plan rather than making a
second request per room. The colour dot stays even though there is room for the name:
it is what ties this screen back to the marker on the plan.
**Give the empty and the unknown cases real text.** "Nobody here" is a finding.
"3 of 4 home placed" is the truth that the overview can only gesture at — and the
`unplaced` list from `/floorplan/presence` deserves its own entry at the end of the
cycle ("Somewhere in the house: Bibi"), because a person the system cannot locate is
exactly who you were looking for when you picked up the watch.
Where a photo *does* belong: nowhere here either. Once you have room to print
"Amir" you have already solved the problem the picture was for.
## Update cadence
The JS only lives while the face is displayed, so "polling" means "while you are
looking at it":
- Fetch once on load.
- Refresh on a `tick_timer` every 2–5 minutes while visible.
- Refresh on tap/shake, for the "who's home *right now*" glance that is the actual use
case.
Do not refresh every second, and do not attempt a background service to keep it warm:
Bluetooth wakeups are the battery cost on both devices, and a presence display that is
four minutes stale is not wrong in any way that matters.
## What is already in place, and what is missing
Already there:
- `GET /floorplan/presence` — the drawn plan joined to who is in each room, with
`unplaced` for people who are home but not locatable and `unmapped_areas` for areas
HA reports that nothing on the plan claims. Both matter on a small screen: "3 home,
1 not locatable" is honest, and quietly dropping two people is not.
- `color` and `initial` on every person, in every presence payload.
- Bearer-token auth on the whole API — the token would have to live in the app's
Clay settings, which is a real consideration: it is stored on the phone in the Pebble
app's config, and it is a token that reads the household's presence history.
Missing, in the order it would need doing:
1. **A compact serialisation** — built, in the JS as planned, so the server stays
general. See `pebble-presence/WIREFORMAT.md`. The fallback if hand-drawn plans turn
out to have 20-vertex rooms is unchanged: a `?format=compact` on
`/floorplan/presence` that simplifies server-side.
2. **The watchapp itself** — built. `gpath` rendering, a click config provider, the
room-detail window, an AppMessage handler, and a settings page for the URL and
token.
3. **A level picker** — not built. The settings page takes a level id and defaults to
the first; folding levels into the same Up/Down cycle after the last room is the
obvious next move, and costs one wrap-around.
## The thing that decides whether this is worth building
**Room-level presence.** `identity` resolves a person's room from whatever
`AREA_ATTRIBUTE` holds on their trusted BLE entity — which is as good as the BLE
proxy layout and HA's area assignment, and this project has never measured it. If in
practice everyone resolves to "home, room unknown", the app is a picture of a
floorplan with everybody sitting in the `unplaced` list at the bottom, and no amount
of watch-side work fixes that.
That is testable today, without buying anything: open the admin panel's floorplan tab
with **Live** ticked, walk between two rooms, and see whether the marker moves. If it
does, the app is a weekend. If it doesn't, the work is in the BLE proxies, and
the watch is a distraction from it.

199
docs/project-plan.md
View File

@ -113,13 +113,10 @@ from Phase 1, same as every other MQTT-connected host in this plan.)*
|---|---|---|
| Mini PC or SBC with a touchscreen, mounted near the fridge/pantry | €150–300 | Prefers a native Wayland `wl_touch` device, same as §1.14 — unlike that host, `hosts/kitchen-display/` does not (yet) ship the `type:pointer`/udev-override fallback described there; a misclassified touchscreen here is still an open problem |
| USB webcam, pointed at wherever items get held up for scanning | €15–40 | Either built into the panel or on a short cable/gooseneck so its angle can be set independently of the screen. Any UVC webcam works — this is the same "no depth camera, no accelerator needed" bar as the thin client's gesture-control camera (§1.11), just used for a photo instead of continuous tracking |
| Zigbee door contact sensor, one per cold appliance door | €10–15 | ~3 to start (fridge, freezer compartment, second freezer). Mounted on the **outside** face of the door — a coin cell at −18 °C is a false economy and every opening condenses room air onto whatever is inside. Worth buying on their own merits ("has the freezer been open four minutes") before any camera exists; see `docs/fridge-item-location.md` |
| Doorway camera, one per appliance (optional, later) | €30–40 | Outward-facing, above the door on the hinge side, ~1.8–2 m, ingested by Frigate like every other camera in this project. **Never inside the appliance.** Buy one, not two — and only after the door sensors have been lived with, per `docs/fridge-item-location.md`'s recommended order |
*(No new container-host hardware — `pantry-vision` is a container on the existing
Phase 1 host, calling the existing Phase 3/9 LLM host for vision inference and the
already-running `grocy` container for storage. It does gain one small bind mount,
`/data`, for the doorway-hint database — see Phase 17 item 4a.)*
already-running `grocy` container for storage.)*
### 1.16 Door/wardrobe panel hardware (Phase 18)
| Item | Est. Price (EUR) | Notes |
@ -218,7 +215,7 @@ real hardware" callouts for everything downstream of this.)*
| Grocery vision recognition | **Ollama** (a vision-capable model, e.g. `llava`/`qwen2.5vl` — TBD, not yet pulled or benchmarked) | Identifies a grocery item from one photo and estimates its shelf life; the proposal is always human-reviewed before anything is written (see `pantry-vision/README.md`) |
| Grocery inventory backend | **pantry-vision** (custom Python, stdlib `http.server`) | `POST /identify` (photo → proposal), `POST /confirm` (human-reviewed proposal → Grocy stock write), `GET /inventory`/`GET /recipes` (proxy Grocy, reshaped). Bearer-token gated, **and — unlike admin-canvas — LAN-published**, since the kitchen display's kiosk browser calls it directly rather than through Home Assistant |
| Grocery inventory storage | **Grocy** (already running, Phase 1/7) | The single source of truth for stock/best-before-dates/recipes; `pantry-vision` is a client of Grocy's own REST API, not a replacement for it |
| Kitchen-display static serving | **pantry-web** (nginx:alpine) | Serves `pantry-vision/frontend/`'s single-page app (unload / consume / expired / edit, plus Inventory and Recipes) read-only to the kitchen display — same role `digest-web`/`admin-web` play for their own hosts |
| Kitchen-display static serving | **pantry-web** (nginx:alpine) | Serves `pantry-vision/frontend/`'s Scan/Inventory/Recipes single-page app read-only to the kitchen display — same role `digest-web`/`admin-web` play for their own hosts |
| Identity registry backend | **identity** (custom Python, stdlib `http.server` + `sqlite3`) | Person <-> BLE-identifier registry: registration (voice or touchscreen), presence resolution, visit history, per-device rights, a weather proxy. **Published**, unlike admin-canvas — `hosts/kitchen-display/`'s and `hosts/door-panel/`'s kiosk browsers call it directly, bearer-token gated |
| Identity static serving | **identity-web** (nginx:alpine) | Serves `identity/frontend/`'s `register.html`/`dashboard.html`/`admin.html` read-only — same role as `pantry-web`/`digest-web`/`admin-web`. The two kiosk pages are what the wall panels load; `admin.html` is deliberately not linked from either (see Phase 6b) |
| Door-panel OS build | **live-build** (custom config, `hosts/door-panel/live-build/`) | Reuses the thin client's build tool/convention, structurally `hosts/kitchen-display/`'s twin — see Phase 18 |
@ -630,34 +627,7 @@ service) plus `hosts/kitchen-display/` (a third, simpler kiosk image).
whole phase is designed around, the same "propose, never auto-commit" rule this
project already applies to identity-merge confirmation (the *Identity store* row
in §2). A wrong camera guess costs one tap to fix on the kiosk screen, not a wrong
fact silently written into the household's inventory. The book-out direction obeys
it too, and it costs a tap there as well: recognition never consumes anything by
itself, an ambiguous brand is asked about instead of picked, and a throw-away
states its amount first.
2a. **Stock only moves in four ways, so the display has four screens** — *Unload
groceries* (the camera runs in a loop, one confirm per item, straight on to the
next), *Consume article* (hold up what you are eating; multi-unit rows ask how
many, multi-brand kinds ask which), *List expired foods* (cleared by scanning what
you are binning, booked out as **spoiled** so Grocy keeps waste apart from eating),
and *Edit inventory* (`−`/`+` and a freeform amount — the one screen that
deliberately does not use the camera, because it is where you go when the camera
got something wrong). Corrections use Grocy's inventory-correction endpoint, not
consume/add, so the stock journal doesn't fill up with corrections dressed as
meals.
2b. **Two invariants the whole feature rests on.** (i) *Stock is counted in individual
units, never packages* — a twelve-pack of eggs is twelve, because that is the
question people ask and it makes consuming three of them arithmetic instead of
fractions of a pack; the model's `units_per_package` is a multiplier applied once
on the confirm screen, where it is editable and where the result is stated before
anything is written. (ii) *The fold key is the brand-free product kind* — twelve
eggs of brand X plus ten of brand Y are twenty-two eggs, recorded as a Grocy
**product group** so the grouping lives in Grocy's own data model rather than in a
second classification scheme here, with the per-brand rows kept underneath because
"take ten off brand Y" has to stay possible on the edit screen. The folding is
deliberately dumb (casefold, strip the brand suffix); the clever version is a
synonym problem nobody wants adjudicated by a kitchen display at 19:00, and its
failure mode — two lines that should be one — is at least visible, which a wrongly
merged line is not.
fact silently written into the household's inventory.
3. **A real, LAN-published network listener — the one deliberate exception to this
project's usual "the LLM only ever reaches a device through HA→MQTT" shape.**
`admin-canvas` (Phase 13) has no published port because only Home Assistant calls
@ -666,51 +636,18 @@ service) plus `hosts/kitchen-display/` (a third, simpler kiosk image).
with no HA round-trip in the loop for that specific call. The bearer token — not
network placement — is the actual boundary here; see
`pantry-vision/README.md`'s "A real network listener, unlike admin-canvas"
section. Controlling *which screen is showing* is a separate, narrower path that
does stay HA/MQTT-mediated: `kitchen-display-agent`, identical in shape to every
other host's agent in this project. Its "Show scan" button still publishes the
fragment `scan`, which `frontend/app.js` treats as an alias for the unload screen —
an old name living on costs nothing next to an HA button that silently stops
working.
section. Controlling *which screen is showing* (Scan/Inventory/Recipes) is a
separate, narrower path that does stay HA/MQTT-mediated: `kitchen-display-agent`,
identical in shape to every other host's agent in this project.
4. **Grocy is the system of record, not a new inventory store** — `pantry-vision` is
a client of Grocy's own REST API (already running unconditionally since Phase 1/7)
for the reads and for every stock write (`/confirm`, `/consume`, `/adjust`,
`/transfer`). Nothing here duplicates or replaces Grocy's own data model. The one
thing `pantry-vision` keeps in a file of its own is the doorway hints below, and
that is precisely because they are *not* inventory.
4a. **Multiple cold appliances, answered twice over — and only one of the answers is
authoritative.** Grocy **locations** are the record: the confirm screen's placement
resolves to one, `/inventory` reports it, and `POST /transfer` moves an amount
between appliances. That transfer action is the whole point — without it a location
decays into "where it was when it was bought", which is worse than no answer.
Alongside it, **doorway hints** (`pantry-vision/doorway.py`): a Zigbee contact
sensor on each appliance door drives an HA automation that POSTs `/doorway-event`,
which answers 202 immediately and pulls a short camera burst off the request
thread. Whatever the vision model recognises is written to its own SQLite file with
a timestamp and a confidence, **never to stock** — a camera at a door cannot tell
in from out, misses two-items-at-once, and sees nothing behind an arm, so it
produces "last seen at Freezer (loggia), 20 min ago" for a person to evaluate.
`docs/fridge-item-location.md` is the full argument, including why nothing goes
*inside* an appliance (no interior power, condensation on every opening, −18 °C
outside every consumer camera module's rating) and why the door sensors are worth
buying even if no camera ever is. **An appliance configured with a sensor and no
camera is a supported, recommended starting point.**
for both reads (`/inventory`, `/recipes`) and the one write (`/confirm` → Grocy
stock). Nothing here duplicates or replaces Grocy's own data model.
5. **The vision-identification prompt asks for one photo, not a live video stream**,
and degrades to a low-confidence placeholder proposal (never an error page) if the
model call fails or its response isn't parseable JSON — same "degrade, don't
blank" rule as the digest/admin canvas renderers and `llm_client.py`'s own
`_fallback_document`. The unload screen's "camera searching for products" is still
built out of single photos: `frontend/app.js` samples a 32×24 greyscale thumbnail
every 700 ms and spends an `/identify` call only when the frame has **settled** and
**changed** since the last identification. Both gates are latency economics — see
open decision #18 — and a degraded answer stops the loop rather than
re-photographing the counter at a model that is down.
5a. **The model is asked to read a printed best-before date off the packaging**, and
that is the one place the "propose, never auto-commit" rule is deliberately looser
than a pure estimate would be. The screen always labels which of the two it is
showing ("read off the packaging — check it" vs "estimated from the category"), and
the server discards any date more than a year past or ten years out, because a
misread label is this feature's most likely failure and that is its shape.
`_fallback_document`.
6. **The kitchen display has no native camera-capture app at all** — Chromium's own
`getUserMedia()`, called from `pantry-vision/frontend/app.js`, talks to the webcam
directly inside the kiosk page and hands a captured frame straight to
@ -718,12 +655,11 @@ service) plus `hosts/kitchen-display/` (a third, simpler kiosk image).
auto-accepts the permission prompt that would otherwise sit unanswered on an
unattended screen.
7. **`hosts/kitchen-display/` is one workspace, one app — no touch dock, no
multi-app switching** the way `hosts/touch-panel` has: Home/Inventory/Recipes is
in-page tab navigation inside `pantry-vision/frontend/`'s single-page app, with the
four stock-movement screens opening from Home, since there's only one thing this
device does. `kitchen-display-agent`'s MQTT surface is correspondingly narrow:
three "Show X" buttons and nothing else — no media_player, no
capture/audio/remote-desktop entities.
multi-app switching** the way `hosts/touch-panel` has: Scan/Inventory/Recipes is
in-page tab navigation inside `pantry-vision/frontend/`'s single-page app, since
there's only one thing this device does. `kitchen-display-agent`'s MQTT surface
is correspondingly narrow: three "Show X" buttons and nothing else — no
media_player, no capture/audio/remote-desktop entities.
8. **The Grocy API integration is written from documentation, not verified against a
live instance** — the exact `GET /api/stock` response shape (whether product
names arrive nested by default), the minimum required fields for
@ -822,28 +758,13 @@ entirely container-host services plus additional Frigate camera sources.
"Camera face recognition" section) specifically so that Tapo cameras plug into
the exact same anti-spoofing-respecting presence pipeline as everything else,
rather than this phase inventing its own.
3. **Trash-bin fullness / dishes / litter / groceries-left-out checks are `chores/`'s
job**: every ~2 hours (systemd timer, `RandomizedDelaySec=1800` — the "+/-30 min in
case something else is running" jitter, systemd's own built-in feature, not custom
3. **Trash-bin fullness / dishes / litter checks are `chores/`'s job**: every ~2
hours (systemd timer, `RandomizedDelaySec=1800` — the "+/-30 min in case
something else is running" jitter, systemd's own built-in feature, not custom
code), `chores/check.py` grabs a Frigate snapshot per configured watch point
(optionally moving a PTZ camera to a preset first) and asks an Ollama vision
model a one-word question. A "needs attention" result opens a chore; a "clear"
result auto-closes one.
3a. **`groceries_out_of_place` is `litter`'s twin, with a shorter fuse.** Same
culprit-attribution (whoever the camera last recognised nearby), same
exemption-proofing, same unassignability — because it is the same situation,
somebody carried something in and put it down. The one deliberate difference is
`GROCERIES_NEGLECT_THRESHOLD_HOURS` (30 minutes against the usual four), and it is
the only place in `chores/` where one chore type is treated as more urgent than
another: litter left for an afternoon is untidiness, a tub of ice cream left for an
afternoon is a bin bag. Its prompt names what to **ignore** (fruit bowls, bread
bins, coffee, spices, cookware) as well as what to look for, because the general
question — "is anything out of place?" — is one a vision model answers YES to for a
kitchen that is simply a kitchen, every two hours, until the household mutes the
topic. It knows food is out; it never knows *what* the food is and never writes to
Grocy. That is `pantry-vision`'s job, at a display someone is standing at — two
systems looking at the same counter for two different reasons, which is the
intended shape rather than a duplication to collapse.
4. **Household chore distribution — presence/calendar-driven nudging, not LLM
assignment.** The governing principle, stated in `chores/check.py`'s own module
docstring: *"I don't care who does it, as long as it gets done."* This is a
@ -935,89 +856,9 @@ entirely container-host services plus additional Frigate camera sources.
- Does `chores`' optional LLM message-phrasing (`chore_reminder_style`) ever change *who* gets nudged or *when*, rather than only the wording of the notification? (It must not — and a failed/empty LLM call must fall back to the plain template, never block the nudge from going out.)
- Does `chores`' household calendar busy-check, if unreachable or misconfigured, ever become the reason nudges stop going out entirely? (It must not — it fails open, treating an error as "not busy.")
### Phase 21 — Steam TV box (local gaming + media, living room)
New host directory `hosts/steam-tv-box/`. No container-host changes.
1. **A different device from the Phase 11 thin client, not a variant of it** — and the
distinction is the whole point. The thin client runs *Steam Link*: it streams a game
rendered on some other PC and needs no graphics performance at all. This box is the
other end of that — a real GPU/CPU machine that renders games **locally**. It reuses
the thin client's live-build toolchain and `configs/`+`agent/` split, not its
live-build tree, exactly the relationship `hosts/touch-panel/` already has to it.
2. **It is the first host to override `household.debian_release`.** A new optional
per-kiosk `debian_release` key (emitted for every kiosk by `config-export.py`, used
only by this builder) pins this image to **trixie**. bookworm's Mesa is 22.3, which
is fine for every other host here — they are a browser and a Python agent — and not
fine for the one machine that renders. The graphics stack is also the only thing on
this box that cannot be fixed later by editing a config file and rebuilding a
service, which is why it is worth a config-schema change rather than a comment.
3. **Native Steam, not the Flatpak**: `steam-installer` from contrib, installed by a
hook rather than the package list because it is unusable until `dpkg
--add-architecture i386` has run and live-build installs package lists *before*
hooks. Plus the i386 half of the Mesa/Vulkan/SDL/PulseAudio stack (a great many
shipped Linux builds are still 32-bit and fail with an opaque GL-context error
without it), `steam-devices`' udev rules (without which pads are root-only and Big
Picture shows no controller at all), `gamemode`, `mangohud`, and `gamescope` when the
release has it. The Flatpak was rejected deliberately: its sandbox complicates GPU
driver matching, controller udev access and mounting a games disk, all of which are
this host's whole job. Spotify and Prism, which have none of those constraints, do
come from Flathub.
4. **`gpu_vendor` is configuration, not detection** (`amd`/`intel`/`nvidia`, validated).
The build host cannot see the target machine, and installing the NVIDIA driver on an
AMD box actively breaks it. `nvidia` additionally sets `WLR_NO_HARDWARE_CURSORS=1`,
whose absence shows up as an invisible pointer — easy to miss on a box driven with a
gamepad.
5. **The media apps do not exist until you leave Big Picture.** The box boots to Steam
and nothing else; Firefox (uBlock Origin + SponsorBlock), Spotify and mpv are started
the first time somebody actually leaves it. Not tidiness: this is the only machine
here where a background browser and a background Electron app cost something visible
— GPU memory, timer wakeups, and an open Spotify audio stream that turns up as
stutter in a frame-limited title. **Two independent triggers**, because there are two
ways to leave and neither can see the other: `steam-session` runs `steam-big-picture`
in the foreground and continues when the client exits ("Exit Steam"), and
`session-watcher` subscribes to sway workspace-focus events ("Exit Big Picture mode",
alt-tab, the remote's channel keys, HA's Screen select — Steam is still running, so
the first trigger never fires). Both call an idempotent `media-session start`, guarded
per-app by `pgrep` and as a whole by `flock`, so whichever fires first wins.
6. **Window-title matching was rejected.** The obvious implementation — poll for a window
titled "Steam Big Picture Mode" — loses to Steam client rewrites, and a media session
that silently stops appearing after a Steam update reads as a broken image rather than
a moved string. A sway workspace name is a contract this repo owns and can only change
by editing two files together.
7. **Re-entering Steam tears nothing down.** People play music over games on purpose, and
killing a running Spotify because a game started would lose whatever was playing.
`media-session stop` exists and is wired to one explicit HA button; nothing calls it
automatically, and neither direction of the Mode select does either.
8. **Prism Launcher runs *through* Steam.** Launched directly it gets a Flatpak window
and a Minecraft that sees a raw evdev pad. `steam-shortcut-prism` writes Prism into
the user's binary `shortcuts.vdf` as a non-Steam game and `prism-launch` starts it via
`steam://rungameid/…`, so it runs inside the Steam Runtime with **Steam Input active**
— configurable pads with per-game bindings, a working overlay, gyro/back-buttons, and
an entry in the Big Picture library. This cannot be baked into the ISO (`shortcuts.vdf`
lives under a `userdata/<steamID3>/` that does not exist until somebody logs in, and no
Steam credentials go into an image), so it runs per-session, idempotently, and
**before Steam starts** — Steam rewrites that file from memory on exit, so anything
written under a running client is discarded. It falls back to a direct launch when
Steam is absent or nobody has logged in, and refuses to rewrite a `shortcuts.vdf` that
already holds other entries rather than risk eating them.
9. **`enable_installer` is `true` here, unlike every other kiosk.** Every other kiosk in
this project is stateless by design and boots live. This one is not: a live system
keeps its writable layer in RAM, so a Steam library and a few modpacks — hundreds of
gigabytes — would vanish on reboot. `validate-config.py` warns when this type has the
installer off.
10. **`steamtv-agent` adds one genuinely new entity to the kiosk vocabulary**: a
**session-mode sensor** (`gaming`/`steam`/`media`/`idle`, read from two cheap local
facts — is Steam's process alive, and which workspace is focused), which is what lets
HA answer "is somebody playing?" without guessing from power draw: don't dim the room
mid-game, don't announce the doorbell over a raid, count the room as occupied though
nobody has moved in forty minutes. The rest — media transport, volume, audio-output
select, workspace select, launch buttons, CEC display switch — is the thin client's
surface, over the same MQTT-only control boundary.
---
## 4. Open decisions (Phases 6, 11–21)
## 4. Open decisions (Phases 6, 11–20)
These need a decision before their respective implementation steps can be built — everything above is written to accommodate any answer, but nothing should be built against an unresolved item.
@ -1060,7 +901,3 @@ These need a decision before their respective implementation steps can be built
37. ~~iOS cannot receive `identity`'s arrival notifications without WAN egress through a third party~~ — **household decision made** (Phase 6b): **no Apple devices are used here**, which removes the only forcing function for exposing ntfy at all. ntfy therefore **stays LAN-only** — no DMZ, no port forward, no NAT-reflection hairpin; `identity → ntfy` is a container-to-container call that never reaches OPNsense, and remote delivery rides a **WireGuard split tunnel** routing only the smart-home VLAN. The full comparison (including why the hairpin variant, which avoids an inter-VLAN rule, was rejected for making an internal path depend on the ISP) is recorded in `docs/network-integration.md` §2.2 so it doesn't get re-litigated. **Still genuinely open**: nothing has been delivered to a real phone yet, and the split tunnel's `AllowedIPs` is the fiddly part — routing all of `192.168.0.0/16` would collide with typical café/hotel LANs and break the phone's connectivity there. Revisit §2.2 only if an Apple device ever joins the household.
38. **Both of `chores`' shipped container URLs pointed at `127.0.0.1` and could never have worked** (found while wiring Phase 6b's arrival notifications) — inside the `chores` container that address is the chores container itself, not the host. `NTFY_URL=http://127.0.0.1:8090` meant every nudge failed to send, and `IDENTITY_URL=http://127.0.0.1:8097` meant `_presence()` never reached identity at all, so **`chores` would have done nothing whatsoever** — and silently, since both failure paths log and continue by design (a deliberate never-block-on-a-dependency choice that here hid a total outage). Corrected in the template to `http://ntfy` and `http://identity:8097` (compose-bridge DNS; ntfy on its internal port 80, not the 8090 published to the LAN). **An already-deployed `/opt/smart-home/chores/chores.env` still has the old values** — templates are copied once at setup and never re-synced, so existing installs need this edited by hand. The broader lesson worth acting on: several services fail soft on an unreachable dependency, which means a misconfigured address produces silence rather than an error — nothing in this repo currently distinguishes "nothing to do" from "never reached the thing that would have told me".
39. **Nothing enforces that voice/TTS consumers actually read `speak_name` rather than `nickname`** (new, Phase 6b) — the field exists, is documented, and `chores/` uses it, but a future HA intent script that reaches for the friendlier-looking `nickname` would break the "the assistant uses real names" rule silently. Worth checking whenever a new consumer of `/presence` or `/resolve` is written.
40. **Whether a vision model can identify a grocery item in a moving hand at doorway distance** (new, Phase 17) — the assumption the entire `/doorway-event` path rests on, and a materially harder task than the kitchen display's held-still-against-a-plain-background one. It is also the cheapest open decision on this list to close: point the kitchen's existing webcam at somebody walking past with a carton and run `/identify` on the frames. Until then the door sensors are the part worth deploying, and `PANTRY_DOOR_APPLIANCES` entries can be configured without a camera — see `docs/fridge-item-location.md`, whose recommended order (sensors, then a month of living with them, *then* one camera) the code deliberately does not shortcut.
41. **Whether `groceries_out_of_place` can tell a carton of milk from a fruit bowl** (new, Phase 20) — its prompt lists what to ignore precisely because the general question gets a YES for any normal kitchen, but that is a mitigation written blind. The failure mode is not a missed chore, it is a false one every two hours, which is how a household learns to ignore the notification channel entirely. Point it at a *clean* counter for a day before trusting a YES, and note this is the first chore type whose false positives cost more than its false negatives.
42. **Contact-sensor battery life on an appliance door is unverified** (new, Phase 17) — a fridge door opens far more often than the window these sensors are sold for, and the sensor sits in a cold, humid draught even when mounted outside the compartment (which `docs/components.md` says to do, for the separate reason that a coin cell at −18 °C is a false economy). Check one after a month before buying more.
43. **Nothing in `hosts/steam-tv-box/` has been built, flashed, or booted, and no hardware has been chosen** (new, Phase 21) — the standing fleet-wide caveat, but it bites harder here because every claim this host makes is a hardware claim. Four things are specifically reasoned rather than verified, in rough order of how loudly they fail: the **`shortcuts.vdf` binary format and non-Steam AppID derivation** (CRC32 of `"Exe"`+`AppName` with the high bit set — the long-standing community format, documented by Valve nowhere; if it is wrong, Prism simply never appears in the library and nothing logs why), **`gamescope`'s availability** in trixie (the hook installs it if apt has it and `steam-big-picture` falls back to plain Xwayland if not, so this costs polish rather than function), the **Flathub application IDs** (`com.spotify.Client`, `org.prismlauncher.PrismLauncher` — same unverified-app-ID caveat as #17), and the **NVIDIA driver package names** for the target release. The cheapest of these to close is the first: log into Steam once, restart the session, and look at the Big Picture library.

78
docs/rooms-and-endpoints.md
View File

@ -1,78 +0,0 @@
# Rooms, and what is in them
One vocabulary for rooms across the whole project, and one answer to "what hardware is
in this room". Written because three separate features now need it — the floorplan
presence view, the Pebble app, and anything room-scoped like the workshop assistant —
and each of them was about to invent its own.
## The room id is an HA `area_id`, everywhere
`living_room`, `kitchen`, `workshop`. Lowercase, digits, underscores; Home Assistant
slugifies area names into exactly that shape.
That one string is the join key for every part of this project that cares where
something is:
| Where it appears | As what |
|---|---|
| Home Assistant | the device's **area** |
| `CoreSystemConfig.json` | `room` on each kiosk and audio endpoint |
| Each host's agent config | `<PREFIX>_ROOM`, baked in at build time |
| MQTT discovery | `suggested_area` on the device object |
| `identity` | `floorplan_rooms.ha_area_id` — the drawn polygon's tie to reality |
| `identity`'s `/presence` | the `room` field, read off `AREA_ATTRIBUTE` of a trusted entity |
Nothing translates between these. A room named `Living Room` in one file and
`living_room` in another is two rooms as far as every join above is concerned, which is
why `tools/validate-config.py` rejects anything that is not already an area_id rather
than helpfully slugifying it — helpfully slugifying it is how you end up with two.
## How a device learns which room it is in
```
CoreSystemConfig.json tools/config-export.py the ISO builder
kiosks[].room ────────────> CORE_KIOSK_ROOM ───────────> <PREFIX>_ROOM in the
agent's config file
│
HA files the device in that area <───────────┘
(suggested_area, on first discovery)
```
Declared once, in the same file that already knows every other fact about that device.
Nobody drags devices into areas in the HA UI, and nobody types a room name twice.
**The one real limitation: `suggested_area` is a suggestion, and it is only honoured
when HA first discovers the device.** Move a panel to another room, rebuild its image,
and HA keeps it in the old area — the suggestion is not reapplied. Moving a device
means moving it in HA too, once. This is a property of HA's discovery, not something
this project can paper over, and it is why the field is named *suggested*.
An empty `room` is allowed and is a **warning, not an error**: a household that hasn't
settled its room names must still be able to build an image. What it loses is automatic
area assignment, which is a nuisance to fix by hand — not a broken device.
## What is not covered by this
- **Zigbee devices, cameras, and anything not built by `tools/`.** Their room lives in
HA only, set when they are paired or added. That is fine — HA is the registry; this
file is about the devices this repo builds images for, which otherwise had no way to
say where they were.
- **Where a *person* is.** That is `identity`'s `/presence`, and it is a different
problem with a different reliability story — see
`docs/pebble-presence-watchface.md`'s closing section.
- **Where an *item* is.** `pantry-vision` locations for food
(`docs/fridge-item-location.md`); nothing tracks tools or parts yet
(`docs/workshop-assistant.md`).
## Adding a room
1. Create the area in Home Assistant (or let a device's `suggested_area` create it).
2. Use its area_id as `room` on every kiosk/audio endpoint that lives there.
3. Draw it on the floorplan in `identity`'s admin panel and set its `ha_area_id` to
the same string. The room editor offers a pick-list from
`GET /floorplan/areas` — every area HA is currently reporting — so this step is
choosing from a list rather than retyping an id.
Step 3 is optional and independent: presence works without a drawn plan, and a drawn
plan is useful before presence is wired up. They only need each other for the views
that show people *on* the plan.

268
docs/stage-light-dimmer.md
View File

@ -1,268 +0,0 @@
# Stage light dimmer — driving an old filament fixture from HA over Zigbee
**Short answer: a SONOFF MINI Extreme ZBDIM (400 W halogen rating) wired inline
into a short Schuko extension lead, paired to the existing Zigbee2MQTT instance.
Roughly €25, appears in HA as a normal dimmable `light` with no HA config at
all.** The only reason this needs a document is that the lamp has to be verified
*before* anything is bought — the wattage decides whether this is a €25 job or a
€100 one, and getting it wrong is the one path here with a real fire risk.
This is guidance, not automation — **nothing under this repo touches the fixture,
the dimmer, or the Zigbee pairing.** Same convention as `network-integration.md`:
you do this by hand, at move-in, after Phase 1 is up.
## 0. Prerequisite
Phase 1 must be running first: the Docker stack from
`tools/setup-container-host.sh`, with Mosquitto, Home Assistant, and
Zigbee2MQTT talking to the Haozee CC2652P dongle (`docs/project-plan.md:16`).
Nothing below works until `docker logs zigbee2mqtt` shows a healthy coordinator.
Everything in this doc stays on the Zigbee backbone rather than adding a WiFi
dimmer, for the reasons in `docs/components.md:208`.
## 1. Verify the lamp before buying anything
The fixture as measured in 2026: **19.6 cm lens diameter, ~15 Ω cold across the
lamp's own terminals, no transformer, no cooling fan.** That works out to roughly
a 250 W mains filament lamp. Re-verify at move-in — the fixture may have been
relamped, and the whole plan hangs on this number.
### 1a. Read the lamp first
The wattage is usually stamped on the pinch, base, or glass, and the fixture
nameplate states max lamp wattage and voltage. A type code is enough — CP60 /
CP61 / CP62 = 1000 W, CP95 = 500 W, HPL 575 = 575 W. **If you can read it, skip
to §2** — a printed number beats every estimate below.
### 1b. Or measure it running — better than any estimate
Not required, but it ends the guessing completely and costs almost nothing. A
plug-in energy meter (~€10, rated 16 A / 3680 W, so fine up to a 2 kW lamp)
reports true wattage directly: no tungsten coefficient, no lead nulling, no
arithmetic. A clamp meter (~€15) on the live lead does the same job.
Worth owning regardless — it's the same instrument you'll want the first time an
automation's power figures look wrong anywhere else in this project.
### 1c. Otherwise, estimate from cold resistance
Disconnected from mains, probe **across the lamp's own two pins** — not at the
plug. The plug reading includes the fixture's switch, wiring, and socket
contacts, which added ~4–5 Ω of error last time and made the lamp look half its
actual size.
P ≈ V² / (R_cold × k) → at 230 V: P ≈ 3530 / R_cold
`k` is tungsten's hot/cold resistance ratio, ~15 for theatre halogen (range
13–17). Expect ±25% overall.
| Cold reading | Likely lamp | | Cold reading | Likely lamp |
|---|---|---|---|---|
| ~18 Ω | 200 W | | ~5.4 Ω | 650 W |
| ~15 Ω | 250 W | | ~4.7 Ω | 750 W |
| ~12 Ω | 300 W | | ~3.5 Ω | 1000 W |
| ~7 Ω | 500 W | | ~1.8 Ω | 2000 W |
| open / ∞ | blown filament — **or a discharge lamp, see §1c** |
Null your leads first (short the probes, subtract that reading). Irrelevant at
15 Ω, critical below 5 Ω where lead resistance is a large fraction of the total.
A reading that wanders means bad probe contact, or that you're measuring a
winding rather than a filament. A good filament reads stable.
### 1d. Optional cross-check: fixture size
A sanity check on the estimate, not a rule. Lens diameter tracks wattage across
*professional theatrical* ranges, but older and budget fixtures routinely pair a
large lens with a modest lamp — lens size is set by the beam optics you want, not
the power. **Where this table disagrees with a terminal measurement, the
measurement wins.**
| Fresnel lens Ø | Typical lamp | | PAR can Ø | Typical lamp |
|---|---|---|---|---|
| 75 mm (3") | 150–300 W | | PAR36 · 114 mm | 6–12 V + **transformer** |
| 125 mm (5") | 500–650 W | | PAR56 · 178 mm | 300–500 W |
| 150 mm (6") | 1000 W | | PAR64 · 204 mm | 500–1000 W |
| 200 mm (8") | 2000 W | | | |
This fixture's 196 mm lens sits on PAR64, which would normally imply 500–1000 W,
while the terminal reading says ~250 W. That conflict is exactly what the caveat
above covers — but it's also why §1a/§1b are worth doing rather than trusting
either number alone.
### 1e. Three things that stop this plan dead
**Discharge lamp.** If the lamp or fixture says HMI, MSR, CSI, CDM, or HQI, or
there's a ballast/igniter in the base — **it cannot be phase-dimmed at all.**
Attempting it destroys the lamp and possibly the ballast. On/off relay only.
**Transformer in the fixture.** Confirm by pulling the lamp and re-measuring at
the plug: open circuit means the path was the filament and you're fine; a
remaining reading of tens of ohms means there's a winding in there. That's an
inductive load, generic dimmer modules are not automatically safe on it, and the
lamp is low-voltage rather than mains. Stop and reassess.
**Cooling fan.** If the fixture has one, it is usually wired in parallel with the
lamp downstream of the dimmer. Dim the circuit and the fan stalls while the lamp
still throws full heat. It must be fed from unswitched mains, ahead of the
dimmer.
## 2. Bill of materials — ~€25
| Item | Price | Note |
|---|---|---|
| [SONOFF MINI Extreme ZBDIM](https://www.amazon.de/SONOFF-elektronische-Halogenlampen-Neutralleiter-Leistungsmessung/dp/B0GFN145VT) | €15–20 | 400 W halogen, Zigbee 3.0, Z2M-supported, power monitoring, acts as a router |
| Short Schuko extension lead | ~€5 | Gets cut in half |
| Surface junction box (Abzweigdose) | ~€3 | Houses the module |
| Inline fuse holder + 2 A fuse | ~€3 | The module has no internal fuse |
At 250 W into a 400 W rating you sit at ~62% load — the headroom you want for
cold-filament inrush.
**Do not buy a plug-in Zigbee dimmer.** The whole category caps around 200 W
because a plug body has nowhere to dump triac heat. The
[AduroSmart ERIA dimmable plug](https://www.amazon.de/AduroSmart-Steckdose-Dimmbar-kompatibel-81855/dp/B08PKQVQTQ)
is the only real Schuko option and it's rated 200 W — *under* this lamp's draw.
[Candeo C204](https://www.amazon.de/Candeo-Dimmermodul-erforderlich-kompatibel-Smartthings/dp/B091GTK61Y)
(~€35) is an equally valid module if the SONOFF is unavailable. Same 400 W
halogen rating, same build job, no power monitoring.
## 3. Build the inline dimmer
Cut the extension lead, wire the module in a junction box: Schuko plug in, Schuko
socket out. The module needs neutral, which an extension lead has.
- Proper enclosure, no exposed conductors.
- Strain relief on **both** cable entries.
- Fuse in the live leg, upstream of the module.
- Don't bury the box anywhere unventilated. Thermal is the dominant failure mode.
If cutting mains cable isn't something you want to do, this is a 15-minute job
for an electrician — and there's no off-the-shelf product that avoids it, given
the 200 W plug ceiling above.
## 4. Pair with Zigbee2MQTT
`tools/setup-container-host.sh:503` sets `permit_join: false`, which is the right
default and means pairing is a deliberate act:
1. Z2M web UI → **Permit join**, scoped to the coordinator.
2. Power the module. It should appear within ~30 s.
3. Let permit-join time out. Don't leave it open.
4. **Rename it immediately** — `stage_lamp` or similar. Otherwise
`light.0x00124b00...` ends up baked into every automation you write.
`tools/setup-container-host.sh:502` already sets `homeassistant: true`, so MQTT
discovery creates the HA `light` entity automatically. **No HA YAML, no manual
entity definition, nothing to commit to this repo** — the pairing lives in Z2M's
`database.db`, which the restic backups already cover
(`docs/project-plan.md:459`).
If Z2M reports the device as unsupported, pull a newer image rather than writing
a converter — the compose file already tracks `koenkk/zigbee2mqtt:latest`
(`tools/setup-container-host.sh:1184`).
## 5. Configure
One setting genuinely matters:
- **`power_on_behavior` → `off`.** Defaults are often `previous` or `on`, which
means a power blip or a Z2M container restart brings a 250 W stage lamp to full
brightness in an empty room.
Three worth doing:
- **Tune `min_brightness` by walking it down.** The default is set to stop LEDs
flickering at the bottom of the range, and a filament doesn't need that
protection — it dims smoothly to near-zero. But don't just set it to 1: at very
low phase angles the triac stops latching reliably, which shows up as flicker
or dropout rather than a dim glow. On the Z2M device page, lower it a step at a
time until the lamp goes unstable, then back off one step. That point is the
real bottom of the usable range, and it's fixture-specific — worth writing the
final value down.
- **Cap `max_brightness` at 90%.** Filament life is brutally sensitive to
voltage: roughly `life ∝ V⁻¹³` against `light ∝ V³·⁴`. Running at 90% buys
about **4× the lamp life for ~30% less light** — a strong trade on a fixture
whose lamp type may well be discontinued (§6), and this is a mood/accent light
rather than something being asked to actually illuminate a room. Back off to
95% (≈2× life, ~15% less light) if it turns out dimmer than you want. Rules of
thumb, not exact, but the direction is very strong.
- **Pass `transition: 2` in `light.turn_on` calls.** Ramping over a couple of
seconds cuts the cold-filament inrush, which is the biggest single wear event
on the lamp and the main stress on the triac.
Optional and situational:
- **`switch_type`** — only relevant if a physical wall switch is wired to the
module's input. Irrelevant for an inline extension-lead build; set it if the
fixture ever gets a permanent installation.
- **Edge mode**, if the module exposes it. Trailing edge (Phasenabschnitt) and
leading edge (Phasenanschnitt) both work fine on a pure resistive filament, so
leave the default alone. It only becomes a real decision if a transformer ever
enters the picture (§1e), where leading edge is the more tolerant of the two.
- **Surface the power reading in HA.** The ZBDIM's power monitoring arrives as
its own sensor entity. A dashboard card, or a template alert if draw exceeds
the expected figure, is cheap early warning on a failing lamp or a wrong
relamp — and it's the only thing here that would catch someone plugging a
bigger fixture into this dimmer.
- **Don't bother with a dimming curve.** Filament light output is very nonlinear
against phase angle, so the HA brightness slider won't feel perceptually even.
Correcting it is fiddly and buys little on a fixture like this; live with it
unless it actually bothers you.
## 6. First run
- Non-flammable surface, module in open air, not boxed into anything yet.
- RCD-protected circuit, ideally on a 6 A or 10 A MCB rather than 16 A.
- Stay with it for the first 30 minutes and sweep the full dim range rather than
parking at one level — **mid-dim is peak triac dissipation, not full-on**.
- **Check the power reading against the ~250 W estimate.** This is what the
SONOFF's power monitoring is for; it replaces the resistance estimate with a
direct measurement. Materially higher than expected → stop, reassess headroom.
- Check the fixture's clearance. Old stage cans run 200 °C+ housings and want
~0.5 m from anything flammable. If the triac ever fails short, the lamp sits at
100% indefinitely — fine for the fixture, less fine for a curtain.
Worth doing once while the lamp is out: clean the lamp socket contacts. The
~4–5 Ω gap between the plug and terminal readings in 2026 was fixture wiring and
contacts, dissipating several watts as heat inside an already-hot fixture.
Two more that aren't required but pay for themselves:
- **Buy a spare lamp at the same time.** Theatre lamp types get discontinued and
an old fixture is worth nothing without one. Record the exact type code once
you've read it in §1a.
- **Label the finished dimmer box** with both numbers — lamp draw and module
rating (250 W / 400 W). Future-you plugging a different fixture into a box that
looks like a generic extension lead is the most plausible way this ends up
overloaded.
## 7. If the lamp turns out bigger than ~400 W
Everything above is void; a 1 kW lamp pulls ~4.3 A steady and 50 A of cold
inrush, well past any in-wall module. Two routes, both built around a used
theatrical dimmer pack (€50–100 for 4×1200 W), which handles inrush, fusing,
heatsinking, and EMI properly:
| | Approach | Cost | Note |
|---|---|---|---|
| **B** | Zigbee 0–10 V dimmer (YSRSAI YSR-Mini-01 or Lonsonho VM-Zigbee-S02, ~€20–25) into the pack's **0–10 V analog input** | €75–125 | Stays on the Zigbee backbone. Most analog packs already have this input. |
| **C** | ESP32 + RS485 module (~€10) running [andyboeh/esphome-dmx512](https://github.com/andyboeh/esphome-dmx512) driving the pack over DMX512 | €60–110 | Cheapest, fits the existing `firmware/` ESPHome pattern, but breaks the Zigbee-only lighting rule in `docs/components.md:208`. |
Prefer **B** — €20 more to keep lighting on one backbone is noise, and either
way you get three spare channels for future fixtures.
## 8. Troubleshooting
| Symptom | Likely cause |
|---|---|
| Device never appears in Z2M | Permit-join timed out; module has no neutral; too far from coordinator for its first hop |
| Appears in Z2M, not in HA | `homeassistant: true` missing, or Mosquitto auth failing — check `docker logs zigbee2mqtt` |
| Lamp stuck at full, unresponsive | Triac failed short. Unplug at the wall. Usually means the load exceeded rating — re-measure before replacing |
| Buzzing or humming from the lamp | Normal for phase-cut dimming of big filaments (magnetostriction). Worse at mid-dim. A dimmer pack's choke fixes it; an inline module won't |
| Flicker at low brightness | Raise `min_brightness` |
| Module warm to the touch | Expected. Hot enough to be uncomfortable means it's under-rated for the load |
| Resistance readings that don't reconcile | A parallel path can never *raise* total resistance. If pulling the lamp makes the reading go **down**, the two measurements weren't the same configuration — re-probe both at the lamp terminals |
| Lamp reads open but fixture still conducts | The lamp is blown *and* there's a fan or transformer across the input. Both §1e cases at once — replace the lamp, then re-read §1e before wiring anything |

341
docs/workshop-assistant.md
View File

@ -1,341 +0,0 @@
# A workshop/office assistant: what's worth building
Feasibility note. The ask: identify laptops and computers by camera, identify
components (mainboards especially), pull up disassembly guides, research part specs
automatically, and hold a conversation about planning a project — all scoped per room.
**The one finding that reorganises the whole thing: stop trying to recognise the
object, and read the label on it.** Everything below follows from that.
## Why "identify it by shape" is the wrong problem
Ask a local 7–13B vision model what laptop it is looking at and you will get a
confident, wrong answer, because:
- **Laptops are visually identical by design.** A closed ThinkPad T480 and a T490 are
the same black rectangle. So are half of Dell's Latitude line. There is no visual
feature to learn, and a model that claims otherwise is pattern-matching on the
wallpaper.
- **Mainboards are worse, not better.** Board layout varies more than a laptop shell
does, but the discriminating detail — where the fourth M.2 slot is, which VRM
configuration — is exactly what a photo at workbench distance and a general-purpose
VLM cannot resolve.
- **Getting it wrong is expensive here**, unlike in the pantry. A wrong yoghurt costs a
tap. A disassembly guide for the wrong laptop revision costs a broken clip or a
ribbon cable, and a wrong pinout costs the board.
Meanwhile, every one of these objects **carries its own identity in printed text**:
| Object | Where its identity actually is |
|---|---|
| Laptop | Service tag / serial sticker on the base — and on Dell/Lenovo/HP that tag resolves to the *exact factory configuration* |
| Mainboard | Model silkscreened on the PCB (`PRIME B450M-A`), usually near the RAM slots or the PCIe bracket |
| GPU / PSU / drives | Model and part number on the label |
| ICs | Top-marking, when it hasn't been sanded |
So the pipeline is **OCR-first**: photo → detect text regions → read them → match
against a parts database. The VLM's job is the small, tractable one it is good at —
"what kind of thing is this, and where on it is the label" — and the identification
comes from characters, not from vibes. This also degrades honestly: an unreadable
label produces "I can't read it, hold it closer" rather than a plausible wrong model.
**What stays hard, permanently:** an unmarked part. A capacitor, a sanded IC, an
anonymous barrel connector. Nothing here will identify those, and the assistant should
say so rather than guess.
## The four capabilities, ranked by whether they're worth it
### 1. Conversational project planning — build this first
The cheapest and the most useful, and the only one with no perception problem in it.
It is a text LLM with a notebook: what am I building, what have I got, what's the next
step, what did I decide last Tuesday and why. The value is entirely in the *notebook*
being persistent and per-project, not in the model being clever.
This is also the piece that makes the others worth having: an identified mainboard is
only useful if there is somewhere to put it ("this is for the NAS build").
Shape: a small service in the pattern this repo already uses four times over
(`identity`, `chores`, `pantry-vision`) — SQLite, an HTTP API, bearer token, a static
frontend on a display. Projects, notes, parts, photos, decisions.
### 2. Part-spec research — build it, with one hard rule
**Every spec must be quoted from a fetched document, with its URL, and never generated
from the model's memory.** This is the same rule `digest-engine`'s political prompt
enforces on claims about the world and `pantry-vision` enforces on stock writes, and
here it is a safety property rather than a quality one: a hallucinated TDP wastes an
afternoon, a hallucinated pinout destroys hardware.
Practically: fetch the vendor page or datasheet PDF, cache it locally, extract, and
show the extract next to its source. If nothing was fetched, the answer is "I couldn't
find a datasheet", not a paragraph of plausible numbers.
**This is the first component in this project that deliberately reaches the open
internet for content**, which is worth stating plainly given the local-first doctrine
everything else follows. It is outbound-only, it does not require anything to be
exposed, and it should run against an explicit allowlist of sources (vendor domains,
iFixit, datasheet archives) with everything it fetches cached on disk — so the same
board is looked up once, not once per question.
### 3. Disassembly guides — mostly a link, and that's fine
iFixit is the corpus, and it has a public API. The honest scope is: resolve the
identified model to a guide, show the steps and images on the workshop display, cache
what you fetch. Do not rewrite the guide through an LLM — the steps are the thing, and
an LLM paraphrase of "disconnect the battery before removing the board" is strictly
worse than the sentence itself.
Vendor service manuals (Lenovo's HMM PDFs, Dell's service manuals) cover what iFixit
doesn't, and are the reason the cache is worth having: they are large PDFs you want
locally, once.
### 4. Camera identification — build it last, and OCR-first
By the time the three above exist, this is a convenience: it saves typing a model
number. Useful convenience — hands are usually dirty or full — but it is the piece
with the perception risk, and it should not be the first thing attempted.
The interaction that actually works: hold the label to the camera, get the text back
with a confidence, **confirm before anything acts on it** — the identical
propose-never-auto-commit rule `pantry-vision` runs on. Scanning a barcode or QR code
where one exists (most service tags have one) is strictly better than OCR and should
be tried first.
## Per-room scoping, which the plumbing now supports
"Make this a per-room system" splits into two questions, and the boring one is now
answered:
**Which devices are in which room** — done. Every kiosk and audio endpoint declares a
`room` (an HA area_id) in `CoreSystemConfig.json`; it is baked into the agent and
published as `suggested_area`, so HA files each device in the right area by itself. See
`docs/rooms-and-endpoints.md`. The workshop is a room like any other: give it a display
and it declares `room: workshop`.
**Which assistant is active in which room** — the design question, and the answer that
fits this project is: *the room selects the toolset, not the personality.* A voice
request in the workshop can reach the parts database and the project notebook; the same
words in the kitchen reach `pantry-vision` and the recipes. That is a routing table
from area_id to capability set, sitting in front of HA Assist, and it is small.
Two reasons to do it that way rather than one assistant with everything:
- **Ambiguity collapses.** "What have I got?" means stock in the kitchen and parts in
the workshop, and no amount of prompt engineering makes one agent reliably guess
which room the words came from — but `identity`'s `/speaker` already answers "who
spoke in this area", and the area is right there in the request.
- **The dangerous tools stay where they belong.** Nothing in the kitchen should be able
to open a disassembly guide, and nothing in the workshop needs to book out groceries.
## The workspace, and why it is an SMB share
The assistant needs somewhere to put things: fetched datasheets, extracted specs,
generated diagrams, photos of the board, project notes, a scratch directory per
project. Making that a **plain directory on the container host, exported over SMB**, is
right for a reason worth stating: it means every artefact is a file you can open from a
laptop with no API, no export step, and no dependence on this project still existing in
two years. The assistant's output should outlive the assistant.
```
/opt/smart-home/workshop/
projects/<slug>/
notes/ markdown the assistant and you both write
datasheets/ everything fetched, cached, named by part
diagrams/ generated SVGs (see the next section)
photos/ what the camera captured
scratch/ the assistant's working directory
```
**This project already runs Samba** — `gallery-smb`, from `setup-container-host.sh`.
Two things follow, and the second is a real trap:
- **Reuse the container, add a share.** Samba serves many shares on one port; a second
Samba container would fight the first for **445, which is already claimed**
(`ports.gallery_smb`). That is the same class of collision already recorded as open
decision #31 for Music Assistant, and it is avoidable here by construction: one
`smb` container, a `gallery` share and a `workshop` share, separate accounts.
- **The gallery share is `read only = yes`. This one cannot be**, and that is the
entire security difference between them. A writable share is fine; a writable share
that an LLM writes into deserves three limits, none of which are exotic:
- **Its own volume**, not a subdirectory of anything else. The blast radius of a bad
path is then "the workshop workspace", which is recoverable.
- **Its own account**, not the gallery's. Guest access off, same as the gallery.
- **No execution.** Nothing in the workspace is ever run by anything on the host —
it is a place for documents, and the moment it becomes a place for scripts it is a
different security question than this note answers.
Keep the assistant's *state* — projects, parts, decisions — in SQLite as with every
other service here, and the *artefacts* on the share. The database is the index; the
share is the filing cabinet. Mixing them (blobs in SQLite, or state in files) gets you
the worst of both: a database you cannot browse and files nothing can query.
## Technical display widgets, and the one decision that makes them cheap
The thin-client canvas SDK today has `stat`, `chart`, `image`, `video`, and plain
markdown-ish text. The wanted additions — schematics, board plans, code-flow and
data-structure diagrams, breadboard layouts, 3D — look like six new widgets. They are
mostly one:
> **Render to SVG on the container host; show it in the canvas.** One new `svg` window
> kind, plus renderers server-side.
That keeps the SDK's dependency-free, no-build-step property (the thing that makes it
maintainable), puts heavy tooling on the machine that already has heavy tooling, and
means a diagram is a *file in the workspace share* as well as something on a screen.
Client-side rendering libraries would put a megabyte of JavaScript on a kiosk to
produce a picture the server could have produced once and cached.
| Widget | How | Honest difficulty |
|---|---|---|
| **Circuit diagrams** | `netlistsvg` (or KiCad's own SVG export) from a netlist the assistant produces | Medium — see the notation note below, which is the whole job |
| **Board plans** | KiCad `.kicad_pcb`/Gerber → SVG, server-side, pre-rendered | Medium, and only for boards you have files for. There is no path from a *photo* of a board to a layout drawing |
| **Code-flow diagrams** | Graphviz `dot` → SVG | **Easy.** Graphviz is small, deterministic, and an LLM writes `dot` reliably |
| **Data-structure diagrams** | Graphviz (records/HTML-like labels) → SVG | Easy, same pipeline |
| **Assembly guides** | The guide's own step images in sequence — see the iFixit section | Easy in 2D |
| **Breadboard diagrams** | Fritzing-style, and Fritzing's value *is* its part library, which is not callable | **Hard, and least worth it** — a photo of the actual breadboard is usually better and always more honest |
| **3D models** | glTF + a vendored three.js on the kiosk | The only one that genuinely needs client-side code — see below |
### EU/IEC notation is a symbol-library decision, not a rendering one
Rectangular resistors, not zigzags. This matters more than it sounds: **most schematic
tooling defaults to ANSI/US symbols**, and a renderer that cannot be given a symbol set
will quietly produce American schematics forever.
So the selection criterion for the schematic renderer is *"can I supply my own symbol
library?"* before anything about output quality. `netlistsvg` takes custom skins, which
is the concrete lever — an IEC skin, written once, vendored in the repo like every
other asset here. KiCad can be configured the same way. Anything that hardcodes its
symbols is disqualified regardless of how good its output looks, because the first
zigzag resistor is the last time anyone trusts the diagram.
The same applies downstream: state the convention in the prompt that generates the
netlist, and check it on the way out. A diagram in the wrong notation is not a style
complaint — it is a diagram that reads wrong to the person holding the soldering iron.
### The holographic look is a variable override, not a new stylesheet
The canvas SDK is already a holo aesthetic — glow text, glowing panel edges, a pulse
animation, a dark blue base — and, importantly, it is already parameterised. Every
colour in `glow.css` comes from custom properties on `:root`
(`--admin-accent`, `--admin-glow-color`, `--admin-edge`, `--admin-panel`, `--admin-bg`).
So "make the workshop look holographic in purple/magenta" is **one small override file
loaded after the SDK**, not a fork of it:
```css
/* workshop-theme.css — loaded after glow.css. Overrides only. */
:root {
--admin-bg: #0a0510; /* near-black, violet-biased */
--admin-panel: rgba(28, 12, 44, 0.82);
--admin-edge: rgba(200, 120, 255, 0.32);
--admin-accent: #c084fc; /* the purple everything glows in */
--admin-text: #eadcff;
--admin-muted: #a98fc4;
--admin-glow-color: #ff3ec8; /* magenta glow against a purple accent */
}
```
Two magenta-on-purple choices worth making deliberately: the **accent** (borders,
labels, chrome) reads better as the lighter purple, and the **glow** as the magenta —
glow bleeds and saturates, so the more aggressive colour belongs to the effect rather
than to the thing being read. Keep `--admin-good`/`--admin-warn` as they are; green
and amber mean something, and recolouring them into the theme would cost that meaning.
**Do not tint the technical drawings.** This is the one rule that keeps the look from
becoming a liability: a schematic, a board plan or a pinout table needs contrast and
neutral colour far more than it needs to match the frame — trace colours mean things,
and a magenta glow over a resistor value is how you misread it. So:
- Holo treatment on the **chrome**: window frames, titles, edges, the pulse, the
background.
- The **drawing surface stays neutral** — a light panel for schematics (they are drawn
for paper and read best that way), or high-contrast line art on near-black. The
frame around it can glow all it likes.
- Anything conveying state by colour — a red LED in a diagram, an error, a
live measurement — keeps its own colour. The theme owns the furniture, not the data.
The same override trick works if a future room wants its own palette; the SDK never
needed to know about themes for this to be possible, which is why it costs one file.
### 3D: worth doing, but not first, and not with a live viewer at first
3D is the one item that cannot be an SVG. Two paths:
- **Pre-rendered orbit frames** — render N views server-side, ship them as an image
sequence, scrub with a drag. Cheap, works on every surface including the ones with
no GPU, and covers "let me see what this connector looks like from behind", which is
most of the actual need.
- **A real viewer** — vendored three.js plus glTF. It is roughly a megabyte of
JavaScript on a kiosk that currently ships none, which is a real break with the
SDK's "vendored, dependency-free" rule and should be a deliberate decision rather
than a drift into one.
Start with the frames. Move to a viewer if and only if manipulating the model turns
out to be what people do, rather than glancing at it.
**3D assembly guides** — exploded views with per-step visibility — are the most
speculative item in this entire note. They need a model *with assembly structure*,
which almost nothing you download has, and authoring one per project is more work than
the repair. Park it.
## Hardware
The workshop needs what the kitchen already has: a screen you can touch with one
knuckle, a camera, a microphone. `hosts/kitchen-display/` is exactly this shape
already — a Sway kiosk, a webcam, an agent, one workspace — so a workshop display is
that image with a different frontend, not a new class of device.
The camera wants to be better than the kitchen's, though, and differently: reading a
service tag or a PCB silkscreen is a **macro** problem. Autofocus at 10–20 cm and
enough resolution to resolve 2 mm text matter far more than field of view. A cheap
USB microscope/inspection camera is a better fit than a webcam, and is worth pricing
before assuming the C920 answer carries over.
## The compute problem, again
One RTX 3060, already carrying Assist, four digests a day, `pantry-vision` and
`chores`. Adding OCR is fine (it is small, and CPU OCR is viable). Adding a second
large model that has to be resident is not — and "the workshop assistant felt slow
because someone asked for a digest" is exactly the contention already flagged as open
decision #4. Design for one model swapped predictably, not two resident.
## Recommended order
1. **The project notebook + conversational planning, and the workspace share.** No
camera, no internet, no new hardware. Useful on day one, and it is where everything
else lands — including every artefact the later steps produce.
2. **Spec research with mandatory citation**, against an allowlist, cached on disk.
3. **The `svg` window kind + a Graphviz renderer.** One kind and one small tool buys
code-flow and data-structure diagrams immediately, and is the pipeline the schematic
and board-plan renderers then plug into.
4. **Guide lookup**, by typed model number.
5. **Schematics with an IEC symbol skin** — the notation work is the work.
6. **Camera identification**, barcode first and OCR second, proposing rather than
acting.
7. **3D**, as pre-rendered orbit frames. A live viewer only if manipulation turns out
to be what people actually do.
The purple/magenta holo theme is not in that order because it is not a step: it is one
override file and can land whenever.
Steps 1–3 need no new hardware at all — a browser on any existing thin client reaches
them. That matters: it means the whole idea can be proved before buying a workshop
display, and if the conversation-and-notebook half turns out to be the only part
anyone uses, that is a complete and useful outcome rather than a failure.
## What is unverified here
Everything past step 1. Specifically: whether iFixit's API terms permit caching what
this would cache; whether a local VLM can read a PCB silkscreen at all under workshop
lighting (testable today with any webcam and a spare board — do this before buying an
inspection camera); whether service-tag lookups work without a vendor account; and
whether the OCR-first pipeline holds up on the labels that are scratched, curved, or
under a warranty sticker, which in a workshop is most of them.
On the display side specifically: whether `netlistsvg`'s skin mechanism really can
express a full IEC symbol set (the notation requirement rests entirely on that, and it
is worth a two-hour spike before committing to the tool); whether an LLM produces
netlists that are *correct* rather than merely well-formed, which is a different and
much harder property; and how big a vendored three.js actually is against the kiosk's
budget before deciding 3D is affordable.

5
hosts/door-panel/agent/door_panel_agent/main.py
View File

@ -71,9 +71,6 @@ def main() -> int:
hostname = socket.gethostname()
node_id = "".join(c if c.isalnum() else "_" for c in hostname).strip("_") or "doorpanel"
friendly_name = config.get("DOOR_PANEL_NAME") or f"Door panel ({hostname})"
# The HA area this device sits in, published as suggested_area — see
# docs/rooms-and-endpoints.md. Blank is fine and means "no suggestion".
room = config.get("DOOR_PANEL_ROOM", "")
broker_host = config.get("MQTT_BROKER_HOST", "")
broker_port = int(config.get("MQTT_BROKER_PORT") or 1883)
@ -82,7 +79,7 @@ def main() -> int:
if config.get("MQTT_USERNAME"):
client.username_pw_set(config["MQTT_USERNAME"], config.get("MQTT_PASSWORD") or None)
discovery = Discovery(client, node_id, friendly_name, room)
discovery = Discovery(client, node_id, friendly_name)
def on_show(screen: str) -> None:
if screen == "register":

10
hosts/door-panel/agent/door_panel_agent/mqtt_discovery.py
View File

@ -25,11 +25,10 @@ DISCOVERY_PREFIX = "homeassistant"
class Discovery:
def __init__(self, client, node_id: str, friendly_name: str, room: str = ""):
def __init__(self, client, node_id: str, friendly_name: str):
self.client = client
self.node_id = node_id
self.friendly_name = friendly_name
self.room = (room or "").strip()
self.base = f"doorpanel/{node_id}"
self.availability_topic = f"{self.base}/availability"
self._handlers: dict[str, Callable[[str], None]] = {}
@ -41,13 +40,6 @@ class Discovery:
"model": "Sway door panel",
"sw_version": __version__,
}
# Which room this physically sits in, as an HA area_id. `suggested_area`
# is honoured by HA only when the device is FIRST discovered — moving a
# device later means moving it in HA too, this cannot un-file it. Omitted
# entirely when unset, because an empty suggested_area is not the same
# request as no suggestion. See docs/rooms-and-endpoints.md.
if self.room:
self.device["suggested_area"] = self.room
def _publish_config(self, component: str, object_id: str, payload: dict) -> None:
payload = {

11
hosts/kitchen-display/README.md
View File

@ -9,9 +9,8 @@ inventory (soonest-to-expire first) and Grocy's recipes, all touch-driven.
that's [`../../pantry-vision/`](../../pantry-vision/README.md), a container-host
service this device's kiosk browser calls directly. This host is deliberately thin:
a Sway session, one Chromium kiosk window, and `kitchen-display-agent`, which only
ever controls *which screen is showing* (the unload-groceries scanner, Inventory or
Recipes, plus Registration — see below) over MQTT — the same "the agent controls the
surface, a
ever controls *which screen is showing* (Scan / Inventory / Recipes, plus
Registration — see below) over MQTT — the same "the agent controls the surface, a
separate write API owns the content" split `hosts/thin-client`'s admin canvas already
established.
@ -89,11 +88,7 @@ would otherwise sit unanswered on a screen nobody is there to click "Allow" on.
`kitchen-display-agent` publishes four buttons on connect: **Show scan**, **Show
inventory**, **Show recipes**, **Show registration** — the first three kill and
relaunch the kiosk Chromium window at `PANTRY_WEB_URL/index.html?...#<tab>`. ("Show
scan" still publishes the fragment `scan`, which the frontend treats as an alias for
its unload-groceries screen; the four stock-movement screens are reached by tapping,
not over MQTT, since they are all things a person is standing at the display doing.)
The
relaunch the kiosk Chromium window at `PANTRY_WEB_URL/index.html?...#<tab>`; the
fourth relaunches it at `IDENTITY_WEB_URL/register.html?...` instead (a different
backend, `identity`, behind the same "Show X" shape — see
`kitchen_display_agent/main.py`'s `on_show()`). That's the entire MQTT surface; there

5
hosts/kitchen-display/agent/kitchen_display_agent/main.py
View File

@ -72,9 +72,6 @@ def main() -> int:
hostname = socket.gethostname()
node_id = "".join(c if c.isalnum() else "_" for c in hostname).strip("_") or "kitchendisplay"
friendly_name = config.get("KITCHEN_DISPLAY_NAME") or f"Kitchen display ({hostname})"
# The HA area this device sits in, published as suggested_area — see
# docs/rooms-and-endpoints.md. Blank is fine and means "no suggestion".
room = config.get("KITCHEN_DISPLAY_ROOM", "")
broker_host = config.get("MQTT_BROKER_HOST", "")
broker_port = int(config.get("MQTT_BROKER_PORT") or 1883)
@ -83,7 +80,7 @@ def main() -> int:
if config.get("MQTT_USERNAME"):
client.username_pw_set(config["MQTT_USERNAME"], config.get("MQTT_PASSWORD") or None)
discovery = Discovery(client, node_id, friendly_name, room)
discovery = Discovery(client, node_id, friendly_name)
def on_show(fragment: str) -> None:
# "register" is a different backend (identity's register.html) behind the

10
hosts/kitchen-display/agent/kitchen_display_agent/mqtt_discovery.py
View File

@ -28,11 +28,10 @@ DISCOVERY_PREFIX = "homeassistant"
class Discovery:
def __init__(self, client, node_id: str, friendly_name: str, room: str = ""):
def __init__(self, client, node_id: str, friendly_name: str):
self.client = client
self.node_id = node_id
self.friendly_name = friendly_name
self.room = (room or "").strip()
self.base = f"kitchendisplay/{node_id}"
self.availability_topic = f"{self.base}/availability"
self._handlers: dict[str, Callable[[str], None]] = {}
@ -44,13 +43,6 @@ class Discovery:
"model": "Sway kitchen display",
"sw_version": __version__,
}
# Which room this physically sits in, as an HA area_id. `suggested_area`
# is honoured by HA only when the device is FIRST discovered — moving a
# device later means moving it in HA too, this cannot un-file it. Omitted
# entirely when unset, because an empty suggested_area is not the same
# request as no suggestion. See docs/rooms-and-endpoints.md.
if self.room:
self.device["suggested_area"] = self.room
def _publish_config(self, component: str, object_id: str, payload: dict) -> None:
payload = {

290
hosts/steam-tv-box/README.md
View File

@ -1,290 +0,0 @@
# Steam TV box
A living-room machine that boots straight into **Steam Big Picture** and runs games
**locally** — real GPU, real CPU, games installed on this box. When you leave Big
Picture it becomes the same media station as every other monitor client in the house:
Spotify, a browser with uBlock Origin, and a general mp3/mp4 player.
**This is not the thin client.** `hosts/thin-client/` runs *Steam Link*, which streams
a game rendered on some other PC and needs no graphics performance at all. This image
is the other end of that: it is the machine that renders. Different distro release,
different driver stack, different session.
What ends up on the image:
| | |
|---|---|
| Compositor | Sway, no bars, no lock screen, workspaces `1:steam` / `2:games` / `3:web` / `4:media` / `5:music` |
| Autologin | greetd, `default_session` straight into `/usr/local/bin/kiosk-session` |
| Default app | **Steam Big Picture** (`-gamepadui`), under gamescope when available |
| Gaming | Native `steam-installer` + i386 multiarch + full Mesa/Vulkan stack, `steam-devices` udev rules, gamemode, MangoHud |
| Minecraft | **Prism Launcher** (Flathub), registered as a non-Steam game so it runs *through* Steam with Steam Input live |
| Browser | Firefox ESR, minimal chrome, uBlock Origin + SponsorBlock force-installed |
| Music | Spotify (Flathub, the real GUI client) |
| Player | mpv + mpv-mpris, opened idle on `4:media` |
| Audio | PipeWire/WirePlumber, persistent output selection from HA |
| Remote control (human) | wayvnc on `0.0.0.0:5900`, authentication required, fails closed |
| Remote control (HA/LLM) | `steamtv-agent`, a systemd service publishing HA MQTT-discovery entities |
| TV power | HDMI-CEC (`cec-ctl`) + Sway DPMS, driven by HA presence |
| Debian release | **trixie**, not the household default — see below |
## The one design decision worth reading
**The media apps do not exist until you leave Big Picture.**
The box boots to Steam and nothing else. No Firefox, no Spotify, no mpv. They are
started the first time somebody actually leaves Big Picture, and they stay up from then
on.
That is not tidiness. This is the only machine in the project where a background
browser and a background Electron app cost something you can see: they hold GPU memory,
they wake the CPU on timers, and Spotify keeps an audio stream open that turns up as
stutter in a title that is already frame-limited. A gaming session should not pay for
media features it is not using.
There are **two triggers**, because there are two different ways to leave and neither
one can see the other:
| Way out | What notices | File |
|---|---|---|
| "Exit Steam", or the client crashing | `steam-big-picture` runs in the foreground; when it returns, the wrapper carries on to the next line | `configs/session/steam-session` |
| "Exit Big Picture mode", alt-tab, the remote's channel keys, HA's Screen select | a sway IPC subscription to workspace-focus events | `configs/session/session-watcher` |
Both call `media-session start`, which is idempotent — every launch is guarded by its
own `pgrep` and the whole path holds an `flock` — so whichever fires first wins and the
second is free. Leaving and re-entering Big Picture repeatedly costs three `pgrep`s.
**Deliberately rejected:** matching on a window titled "Steam Big Picture Mode". Steam's
window titles, classes, and whether Big Picture is even a separate window have all
changed across client rewrites, and a media session that silently stops appearing after
a Steam update looks like a broken image rather than a moved string. A sway workspace
name is a contract this repo owns.
**Going back into Steam does not stop anything.** People play music over games on
purpose, and killing a running Spotify because a game started would lose whatever was
playing. `media-session stop` exists, is wired to an HA button ("Stop media apps"), and
nothing calls it automatically.
## Prism Launcher runs through Steam
Launched directly, Prism gets you a Flatpak window and a Minecraft that sees a raw
evdev gamepad — which is to say, no controller support worth the name. So it does not
launch directly. `/usr/local/bin/steam-shortcut-prism` registers Prism as a **non-Steam
game** in `shortcuts.vdf`, and `/usr/local/bin/prism-launch` asks Steam to run it via
`steam://rungameid/…`.
What that buys: the game runs inside the Steam Runtime with **Steam Input active**, so
the Steam Controller API is present for it. Pads arrive as configurable Steam
controllers with per-game bindings, the overlay works, gyro and back-buttons on
Deck-style pads work, and Prism shows up in the Big Picture library instead of being a
hole you fall out of the UI into.
Two consequences you need to know about:
- **It cannot be baked into the ISO.** `shortcuts.vdf` lives under
`~/.steam/steam/userdata/<steamID3>/config/`, which does not exist until somebody has
logged into Steam on the machine — and no Steam credentials go into an image. So
registration happens per-session, is idempotent, and quietly no-ops until the first
login. **After you first log into Steam, restart the session (or reboot) once.**
- **It runs before Steam starts, from `steam-session`.** Steam reads `shortcuts.vdf` at
startup and rewrites it from memory at shutdown, so anything written underneath a
running client is silently discarded.
If Steam is not installed, or nobody has logged in yet, `prism-launch` falls back to
running Prism directly. That is a working Minecraft with worse controller support, which
beats refusing to start.
> **VERIFY:** the binary VDF encoding and the non-Steam AppID derivation (CRC32 of
> `"Exe"`+`AppName`, high bit set) are the long-standing community format — Valve
> documents neither, and neither was checked against a real client from this
> environment. The test is simply: does *Prism Launcher* appear in the Big Picture
> library, and does the pad work in Minecraft. `steam-shortcut-prism` refuses to
> rewrite a `shortcuts.vdf` that already has other entries in it (it backs it up and
> tells you to add the entry by hand) rather than risk eating shortcuts you added
> yourself.
## Why trixie and not the household release
`household.debian_release` is `bookworm`, whose Mesa is 22.3. That is fine for every
other host here — they are a browser and a Python agent, and none of them care what
Mesa they have. It is not fine for the one machine that renders games.
So this kiosk sets a per-kiosk override:
```json
"debian_release": "trixie"
```
`tools/config-export.py` emits `CORE_KIOSK_DEBIAN_RELEASE` for every kiosk, falling back
to the household value, so the override is available to any builder but only this one
uses it. `validate-config.py` warns if a `steam-tv-box` is pinned to bookworm. The
graphics stack is the one thing on this box that cannot be fixed later by editing a
config file and rebuilding a service.
## Before you build
**No hardware has been chosen and nothing here has been booted on real metal.** That is
the same standing caveat as the rest of the fleet, and it bites harder on this host
than on the others: everything about a gaming box is a hardware claim.
Set these on the kiosk entry in `CoreSystemConfig.json` (see
`CoreSystemConfig.json.template` for a complete example):
| Key | What to put in it |
|---|---|
| `type` | `steam-tv-box` |
| `gpu_vendor` | `amd`, `intel` or `nvidia`. **Not detected** — the build host cannot see this machine, and installing the NVIDIA driver on an AMD box actively breaks it. `amd`/`intel` need nothing beyond Mesa; `nvidia` pulls the non-free driver and sets `WLR_NO_HARDWARE_CURSORS=1`. |
| `debian_release` | `trixie` |
| `enable_installer` | `true`, unlike every other kiosk here — see below |
| `enable_cec` | `true` unless the set ignores HDMI-CEC |
| `room` | the HA area, e.g. `living_room` |
Then:
```sh
sudo -E tools/build-steam-tv-box-iso.sh # the only steam-tv-box in the config
sudo -E tools/build-steam-tv-box-iso.sh <hostname> # a specific one, if several
```
The build is noticeably longer than the other images: i386 multiarch plus Steam's
dependency chain is a few hundred extra packages.
### `enable_installer` should be true here
Every other kiosk in this project boots live, because every other kiosk is stateless by
design. This one is not. A live system keeps its writable layer in RAM, so a Steam
library and a few modpacks would vanish on reboot — and they are hundreds of gigabytes.
Install to the disk, and give `/home` its own large partition, or mount a games disk at
`/home/<kiosk_username>/Games` (created by `0100-user-setup.hook.chroot`, and the
directory Prism's Flatpak is granted access to).
The validator warns if `enable_installer` is false on this type.
## After the first boot
1. **Confirm it is actually accelerated** before blaming any game. From the maintenance
shell (`Super+Shift+Ctrl+M`):
```sh
vulkaninfo | head -n 20 # should name your GPU, not llvmpipe
glxinfo -B # same
```
2. **Set the wayvnc password.** wayvnc will not be running until you do — it fails
closed, same as the thin client:
```sh
sudo sh -c 'openssl rand -base64 24 > /etc/wayvnc/wayvnc-password'
sudo chmod 600 /etc/wayvnc/wayvnc-password
sudo chown <kiosk_username>:<kiosk_username> /etc/wayvnc/wayvnc-password
swaymsg reload
```
It is for administering the box, not for playing over: wayvnc streams the
compositor, so a game at 120fps arrives as a slideshow. That is what a
screen-scraping protocol does, not a bug to work around.
3. **Log into Steam**, then restart the session once so the Prism shortcut registers.
4. **Set the audio output.** On a TV box the right sink is usually HDMI, and if the set
was switched off at boot WirePlumber may have defaulted to a headphone jack with
nothing in it — the classic "the game has no sound" report. Pick it in HA's *Audio
output* select; the choice persists across reboots (it is stored as a WirePlumber
`node.name`, which is stable, not as a numeric ID, which is not).
5. **Leave Big Picture** and confirm Firefox, mpv and Spotify appear. Then go back into
a game and confirm they are *not* killed.
6. **Pull the power on the container host** and re-check: the box must still boot into
Big Picture and play a game. `steamtv-agent` connects asynchronously and keeps
retrying.
## Home Assistant entities
Published by `steamtv-agent` via MQTT discovery, under a device named after the kiosk's
`friendly_name`:
| Entity | Type | Notes |
|---|---|---|
| Session | sensor | `gaming` / `steam` / `media` / `idle`, plus `steam_running`, `media_running`, `workspace` attributes |
| Mode | select | `gaming` / `media`. Both directions are **additive** — see below |
| Screen | select | the five workspaces |
| Launch Steam Big Picture / Prism Launcher / web browser / media player / Spotify | buttons | |
| Stop media apps | button | the only thing in the whole surface that shuts anything down |
| Audio output | select | persisted across reboots |
| Display | switch | HDMI-CEC standby + Sway DPMS. The **panel**, never the machine |
| Volume | number | |
| Playback state | sensor | mpv/Spotify via MPRIS |
| Play/pause, Next, Previous, Stop | buttons | |
**Why *Mode* is additive in both directions:** choosing `gaming` launches or focuses
Steam and leaves the media apps alone (music over a game is a thing people want).
Choosing `media` starts the media session and does not quit Steam, because quitting
Steam from a phone while somebody is mid-match is not something this should be able to
do by accident.
**Playback state reads `off` for most of a gaming session.** That is correct, not
broken: the media players genuinely do not exist yet. Steam exposes no MPRIS bus and a
game's audio is not something you "pause" — for a noisy game, the volume control is the
right surface. Watch *Session* for what the box is doing.
## Security posture
Same as every other kiosk here, and it matters more on this one, because this is the
machine with a Steam login and a games library on it.
- **MQTT discovery is the only inbound control surface.** No HTTP listener, no websocket
server, no exposed Sway IPC socket, no shell endpoint. Every handler is a fixed,
enumerated action; a payload never becomes an argv element, a shell string or a URL
host. See `agent/steamtv_agent/mqtt_discovery.py`.
- The local LLM has no network path here. The only chain is: LLM tool call → Home
Assistant service call → MQTT → the dispatcher.
- Nothing listens on a port except sshd (key-only, root login off, passwords off) and
wayvnc (password mandatory, refuses to start without one).
- **Steam's remote-play ports are not opened by anything in this image.** If you want
them, that is a deliberate firewall decision, not a default this ships. Nothing here
is port-forwarded; remote access is WireGuard, same as the rest of the house.
- The built ISO contains the Wi-Fi PSK and MQTT credentials from
`CoreSystemConfig.json`. Treat it as a credential. `.gitignore` covers `iso-out/`.
## Files
```
hosts/steam-tv-box/
├── configs/
│ ├── audio/audio-config.json template; the runtime copy lives in /var/lib
│ ├── firefox/ policies.json (uBlock Origin/SponsorBlock), chrome, launcher
│ ├── greetd/ autologin config + the session wrapper
│ ├── installer/preseed.cfg source; the builder generates live-build/config/preseed.cfg
│ ├── mpv/mpv.conf IPC socket + hwdec
│ ├── session/
│ │ ├── steam-session boots Steam, starts the media session when it exits
│ │ ├── steam-big-picture gamescope-or-Xwayland launcher, idempotent
│ │ ├── steam-shortcut-prism registers Prism as a non-Steam game (Steam Input)
│ │ ├── session-watcher sway IPC: the other "left Big Picture" trigger
│ │ ├── media-session idempotent start/stop of the media half
│ │ ├── media-player mpv wrapper
│ │ ├── prism-launch routes through Steam; falls back to direct
│ │ └── spotify-launch Flathub Spotify
│ ├── sway/ the session config + display-toggle
│ └── wayvnc/ config + the fails-closed launcher
├── agent/
│ ├── steamtv-agent.service
│ └── steamtv_agent/ main, sway_control, mqtt_discovery, mpris_bridge,
│ audio_control, runtime_state, display_power, session_mode
└── live-build/config/
├── package-lists/steam-tv-box.list.chroot
└── hooks/normal/ 0100-user-setup, 0250-wayvnc, 0200-greetd,
0300-steam, 0400-flatpak-apps, 0700-steamtv-agent,
0900-firefox
```
`live-build/config/includes.chroot/` and `live-build/config/preseed.cfg` are
**generated** — wiped and rewritten by the builder on every run. Edit `configs/`.
## Status
Written, syntax-checked, and validated through `tools/validate-config.py` and
`tools/config-export.py`. **Never built, never flashed, never booted.** No hardware has
been chosen. Specific things that are reasoned rather than verified:
- `gamescope`'s availability in the target Debian release (the hook installs it if
present, `steam-big-picture` falls back to plain Xwayland if not).
- The Flathub application IDs (`com.spotify.Client`,
`org.prismlauncher.PrismLauncher`) — believed correct, not checked against Flathub.
- The `shortcuts.vdf` binary format and non-Steam AppID derivation (see above).
- The NVIDIA driver package names for the target release.
- Whether the television honours CEC standby at all.

8
hosts/steam-tv-box/agent/requirements.txt
View File

@ -1,8 +0,0 @@
# The ISO installs this from apt (python3-paho-mqtt) rather than pip — see
# live-build/config/package-lists/steam-tv-box.list.chroot. This file is for running
# the agent outside the image (development, a venv on a test box). The code works
# against both the 1.x and 2.x callback APIs.
paho-mqtt>=1.6
# Config is read from /etc/steamtv-agent/config.env by a small parser in main.py, so
# there is no python-dotenv dependency.

39
hosts/steam-tv-box/agent/steamtv-agent.service
View File

@ -1,39 +0,0 @@
[Unit]
Description=Steam TV box agent (Home Assistant MQTT control surface for Sway)
Documentation=file:///opt/steamtv-agent
# A system unit rather than a `systemctl --user` unit: the agent has to be reachable
# over MQTT whether or not a graphical session is up, and it has to survive sway
# restarting (a user unit bound to graphical-session.target would go down with it).
# The session-scoped bits it needs — SWAYSOCK, XDG_RUNTIME_DIR,
# DBUS_SESSION_BUS_ADDRESS — are derived at call time from the kiosk user's runtime
# directory in sway_control.SwayControl.session_env(), which also means they are
# re-resolved after every sway restart instead of being frozen at unit start.
#
# Deliberately NOT After=network-online.target: this box must boot to Big Picture and
# play a game with the container host powered off, same rule as the thin client's
# Phase 11.10.
After=network.target
[Service]
Type=simple
User=@KIOSK_USERNAME@
Group=@KIOSK_USERNAME@
WorkingDirectory=/opt/steamtv-agent
Environment=PYTHONPATH=/opt/steamtv-agent
Environment=PYTHONUNBUFFERED=1
EnvironmentFile=-/etc/steamtv-agent/config.env
ExecStart=/usr/bin/python3 -m steamtv_agent.main
Restart=always
RestartSec=5
# Deliberately gentle on shutdown, and more so than the other agents: this process is
# the parent of nothing important, but `media-session stop` reaches Firefox and Spotify,
# both of which persist state on exit.
KillMode=mixed
TimeoutStopSec=20
# Sandboxing stops here on purpose: this unit's whole job is to spawn GUI child
# processes (Firefox, the Spotify and Prism flatpaks, Steam) that need the real /tmp,
# the user's home, and the session bus.
NoNewPrivileges=true
[Install]
WantedBy=multi-user.target

3
hosts/steam-tv-box/agent/steamtv_agent/__init__.py
View File

@ -1,3 +0,0 @@
"""steamtv-agent — Home Assistant MQTT control surface for the Steam TV box."""
__version__ = "0.1.0"

211
hosts/steam-tv-box/agent/steamtv_agent/audio_control.py
View File

@ -1,211 +0,0 @@
"""Persistent audio-output selection via WirePlumber's wpctl.
Two identifiers are in play and confusing them is the main failure mode here. `wpctl`
takes a numeric object ID, which WirePlumber reassigns on every boot and every device
hotplug — useless for persistence. `node.name` is stable across both, which is what
audio-config.json stores. Home Assistant is shown neither: the select lists the human
descriptions ("Built-in Audio Analog Stereo"), because those are what someone picking
an output in a mobile app can actually recognise.
Per the security note in mqtt_discovery.py, the payload from HA is only ever used to
look up an entry in the sink table parsed from `wpctl status`. It never reaches a
command line: what does is the integer ID that lookup returns.
"""
from __future__ import annotations
import logging
import re
import subprocess
from dataclasses import dataclass
from .runtime_state import ensure_runtime_copy, load_json, save_json
log = logging.getLogger(__name__)
CONFIG_FILENAME = "audio-config.json"
SYSTEM_DEFAULT = "system-default"
# `wpctl status` prints a tree; sink rows look like
# │ * 47. Built-in Audio Analog Stereo [vol: 0.65]
# with a leading "*" on the current default. The box-drawing prefix varies between
# WirePlumber versions, so the row is matched from the ID onwards rather than anchored.
_SINK_ROW = re.compile(r"(\*?)\s*(\d+)\.\s+(.*?)(?:\s+\[vol:.*)?$")
_SECTION = re.compile(r"^\s*[^\w]*\s*(\w[\w /]*):\s*$")
@dataclass(frozen=True)
class Sink:
node_id: int
description: str
node_name: str
is_default: bool
class AudioControl:
def __init__(self, env_provider):
# wpctl talks to the user's PipeWire session, so it needs the same XDG_RUNTIME_DIR
# derivation SwayControl uses for swaymsg and playerctl.
self._env_provider = env_provider
self.config_path = ensure_runtime_copy(CONFIG_FILENAME)
config = load_json(self.config_path)
self.preferred_sink = str(config.get("preferred_sink") or "")
self.fallback = str(config.get("fallback") or SYSTEM_DEFAULT)
self._sinks: list[Sink] = []
# --- wpctl --------------------------------------------------------------
def _wpctl(self, *args: str) -> str | None:
try:
result = subprocess.run(
["wpctl", *args],
env=self._env_provider(),
capture_output=True,
text=True,
timeout=10,
check=False,
)
except (OSError, subprocess.TimeoutExpired) as exc:
log.warning("wpctl %s failed: %s", " ".join(args), exc)
return None
if result.returncode != 0:
log.warning("wpctl %s: %s", " ".join(args), result.stderr.strip())
return None
return result.stdout
def _node_name(self, node_id: int) -> str:
output = self._wpctl("inspect", str(node_id)) or ""
for line in output.splitlines():
if "node.name" in line:
_, _, value = line.partition("=")
return value.strip().strip('"')
return ""
def list_sinks(self) -> list[Sink]:
output = self._wpctl("status")
if output is None:
self._sinks = []
return self._sinks
sinks: list[Sink] = []
in_sinks = False
for line in output.splitlines():
section = _SECTION.match(line)
if section:
# Sources, Filters and Streams also carry numbered rows, so the parser
# has to stop at the next heading rather than read to end of output.
in_sinks = section.group(1).strip() == "Sinks"
continue
if not in_sinks:
continue
match = _SINK_ROW.search(line)
if not match:
continue
node_id = int(match.group(2))
sinks.append(
Sink(
node_id=node_id,
description=match.group(3).strip() or f"Sink {node_id}",
node_name=self._node_name(node_id),
is_default=match.group(1) == "*",
)
)
self._sinks = sinks
return sinks
# --- entity surface -----------------------------------------------------
def options(self) -> list[str]:
"""Select options for HA: descriptions, plus the "let WirePlumber decide" entry."""
seen: dict[str, int] = {}
result = [SYSTEM_DEFAULT]
for sink in self._sinks:
label = sink.description
if label in seen:
# Two identical descriptions (e.g. a pair of matched HDMI outputs) would
# otherwise collapse into one unselectable option.
seen[label] += 1
label = f"{label} ({seen[label]})"
else:
seen[label] = 1
result.append(label)
return result
def current_option(self) -> str:
for sink in self._sinks:
if self.preferred_sink and sink.node_name == self.preferred_sink:
return sink.description
if self.preferred_sink:
# Configured but not present right now — say so rather than silently
# reporting whatever WirePlumber happens to be using.
return SYSTEM_DEFAULT
for sink in self._sinks:
if sink.is_default:
return sink.description
return SYSTEM_DEFAULT
def _find(self, option: str) -> Sink | None:
for sink in self._sinks:
if sink.description == option:
return sink
# Match the disambiguating "(2)" suffix options() may have added.
base = re.sub(r"\s+\(\d+\)$", "", option)
matches = [s for s in self._sinks if s.description == base]
return matches[0] if matches else None
def apply_preferred(self) -> None:
"""Called once at startup, after the sink list has been read."""
if not self.preferred_sink:
log.info("no preferred audio sink configured; leaving WirePlumber's default")
return
for sink in self._sinks:
if sink.node_name == self.preferred_sink:
self._set_default(sink)
return
# Never fatal: a docked machine booted undocked, or an HDMI display that is off,
# legitimately has no such sink. The preference stays on file for next boot.
log.warning(
"preferred audio sink %r is not currently available; falling back to %s",
self.preferred_sink,
self.fallback,
)
def _set_default(self, sink: Sink) -> None:
log.info("setting default audio sink to %s (id=%s)", sink.description, sink.node_id)
self._wpctl("set-default", str(sink.node_id))
def select(self, option: str) -> str:
"""Handle the HA select. Returns the option to publish back as state."""
option = option.strip()
self.list_sinks()
if option == SYSTEM_DEFAULT:
self.preferred_sink = ""
self._save()
log.info("audio output preference cleared; WirePlumber's default applies")
return self.current_option()
sink = self._find(option)
if sink is None:
log.warning("ignoring unknown audio output %r", option)
return self.current_option()
self._set_default(sink)
if sink.node_name:
self.preferred_sink = sink.node_name
self._save()
else:
# Without a node.name there is nothing stable to persist; the change still
# applies to this boot.
log.warning(
"sink %r has no node.name; the change applies now but will not survive a reboot",
sink.description,
)
return sink.description
def _save(self) -> None:
save_json(
self.config_path,
{"preferred_sink": self.preferred_sink, "fallback": self.fallback},
)

146
hosts/steam-tv-box/agent/steamtv_agent/display_power.py
View File

@ -1,146 +0,0 @@
"""Turning the attached TV on and off, so an empty room does not power a panel.
The television this box is plugged into draws 60-150 W while it shows a paused game
or an empty Big Picture that nobody is in the room to look at. This module is what
Home Assistant calls when presence says the room is occupied or empty — the decision
lives in HA (an area's occupancy, the same presence system everything else here
uses), and the doing lives here.
Note what this deliberately does not do: it never touches the machine's own power.
Suspending a box mid-game loses the game. Turning the panel off while the machine
keeps running is the whole point — see also /usr/local/bin/display-toggle, the local
no-network version of this, bound to the remote's power button.
TWO MECHANISMS, IN THIS ORDER
-----------------------------
1. **HDMI-CEC** (`cec-ctl`, from v4l-utils). This box is the HDMI *source*,
so it can put the display into standby and wake it again over the HDMI cable
itself. That is the one that actually saves the panel's power, and it needs no
network path to the TV, no pairing, no credentials, and no account — it keeps
working with the LAN down, which is this project's whole posture. Android TV
and Google TV sets implement CEC as "HDMI-CEC", "Bravia Sync", "Anynet+",
"Simplink" and a dozen other brand names for the same standard; it usually has
to be enabled in the TV's settings once.
2. **Sway DPMS** (`swaymsg output <name> power on|off`) as the fallback, and as a
belt-and-braces companion: it stops the compositor driving pixels and drops the
HDMI signal, which most panels treat as "go to sleep" on their own. It always
works because it needs nothing but the compositor already running here — but on
its own it may leave a TV showing a "no signal" banner rather than sleeping,
which is why CEC is tried first.
Both are attempted on every call unless CEC is switched off, because they fail in
different ways and neither reports reliably.
ONE CAVEAT SPECIFIC TO THIS IMAGE: when Steam is running under gamescope, gamescope
owns the output, and `swaymsg output … power off` blanks gamescope's surface rather
than the game inside it. CEC is what actually darkens the panel in that case, which
is another reason it is tried first here rather than kept as the fallback.
WHAT "OFF" HONESTLY MEANS
-------------------------
Standby, not disconnected. A TV in CEC standby still draws roughly half a watt to
keep listening on the HDMI line — that is what makes waking it possible at all.
This turns 60-150 W of lit panel into ~0.5 W of standby; it is not a smart plug
and does not pretend to be. If a set is one of the ones that ignores CEC standby
entirely, you will see it immediately (the panel stays lit) — that is what the
verification note in hosts/steam-tv-box/README.md is for.
SECURITY POSTURE, UNCHANGED
---------------------------
This is another enumerated MQTT command, exactly like the workspace switch and the
canvas buttons: HA -> MQTT -> a fixed action here. A payload never becomes an argv
element — `set_power()` takes a boolean, and the device names come from local
configuration, never from the message. See mqtt_discovery.py's module docstring.
"""
from __future__ import annotations
import logging
import os
import shutil
import subprocess
log = logging.getLogger(__name__)
CEC_TIMEOUT_SECONDS = 10
class DisplayPower:
def __init__(self, sway, cec_device: str | None = None, outputs: str = "*",
use_cec: bool = True):
self.sway = sway
# The CEC adapter, e.g. /dev/cec0. Most systems have exactly one and cec-ctl
# finds it on its own; this is for the machine that has two.
self.cec_device = cec_device or os.environ.get("CEC_DEVICE", "")
# Which Sway outputs to power down. "*" is every output, which is right for a
# box driving one TV; name an output (e.g. "HDMI-A-1") on a machine
# where only one of several screens is the TV.
self.outputs = outputs or "*"
self.use_cec = use_cec
self.state = True
# --- CEC ----------------------------------------------------------------
def _cec(self, *args: str) -> bool:
binary = shutil.which("cec-ctl")
if not binary:
log.info("cec-ctl is not installed; falling back to DPMS only")
return False
command = [binary]
if self.cec_device:
command += ["-d", self.cec_device]
command += list(args)
try:
result = subprocess.run(
command, capture_output=True, text=True, timeout=CEC_TIMEOUT_SECONDS
)
except (OSError, subprocess.SubprocessError) as exc:
log.warning("cec-ctl %s failed: %s", " ".join(args), exc)
return False
if result.returncode != 0:
log.warning("cec-ctl %s: %s", " ".join(args), (result.stderr or "").strip())
return False
return True
# --- the one public action ---------------------------------------------
def set_power(self, on: bool) -> bool:
"""Turn the display on or off. Returns the state it believes it left it in.
Deliberately not idempotent-by-early-return: HA asking for "on" when this
object already thinks it is on must still send the wake, because the TV may
have been turned off with its own remote and nothing here would know. The
state field is for reporting, never for skipping work.
"""
log.info("display: turning the panel %s", "on" if on else "off")
if self.use_cec:
# --to 0 addresses the TV specifically (logical address 0) rather than
# broadcasting, so a soundbar or receiver on the same bus is left alone.
if on:
self._cec("--to", "0", "--image-view-on")
# Ask to become the active source too: waking a TV that then shows a
# different input is the same as not waking it.
self._cec("--to", "0", "--active-source", "phys-addr=0.0.0.0")
else:
self._cec("--to", "0", "--standby")
# Always also drive the compositor: on a set that ignores CEC this is what
# stops it displaying, and on one that honours CEC it stops this machine
# rendering to a panel nobody is looking at.
self.sway.swaymsg("output", self.outputs, "power", "on" if on else "off")
self.state = on
return self.state
def handle_command(self, payload: str) -> bool:
"""MQTT payload -> action. Anything that isn't a known ON/OFF word is ignored
rather than guessed at, per the enumerated-command rule."""
value = (payload or "").strip().upper()
if value in ("ON", "TRUE", "1"):
return self.set_power(True)
if value in ("OFF", "FALSE", "0"):
return self.set_power(False)
log.warning("display: ignoring unknown power payload %r", payload)
return self.state

292
hosts/steam-tv-box/agent/steamtv_agent/main.py
View File

@ -1,292 +0,0 @@
"""steamtv-agent entrypoint."""
from __future__ import annotations
import itertools
import logging
import os
import signal
import socket
import sys
import threading
from dataclasses import dataclass, field
import paho.mqtt.client as mqtt
from .audio_control import AudioControl
from .display_power import DisplayPower
from .mpris_bridge import MprisBridge
from .mqtt_discovery import Discovery
from .session_mode import MODES, SessionMode
from .sway_control import WS_GAMES, WS_MEDIA, WS_MUSIC, WS_STEAM, WS_WEB, SwayControl
CONFIG_PATH = os.environ.get("STEAMTV_AGENT_CONFIG", "/etc/steamtv-agent/config.env")
CONFIG_KEYS = (
"MQTT_BROKER_HOST",
"MQTT_BROKER_PORT",
"MQTT_USERNAME",
"MQTT_PASSWORD",
"HA_URL",
"KIOSK_USERNAME",
"STEAMTV_NAME",
"STEAMTV_ROOM",
"GPU_VENDOR",
"ENABLE_CEC",
)
WORKSPACES = (WS_STEAM, WS_GAMES, WS_WEB, WS_MEDIA, WS_MUSIC)
# How often the session-mode sensor is recomputed. Slower than the MPRIS poll on
# purpose: "is a game running" changes on a timescale of minutes, and each check costs
# two pgreps and a swaymsg. Riding the MPRIS loop rather than adding a second thread.
SESSION_POLL_EVERY = 5
log = logging.getLogger("steamtv-agent")
@dataclass(frozen=True)
class App:
name: str
command: list[str] = field(default_factory=list)
process_pattern: str | None = None
workspace: str | None = None
focus_criteria: str | None = None
icon: str = "mdi:application"
def load_config(path: str = CONFIG_PATH) -> dict[str, str]:
values: dict[str, str] = {}
try:
with open(path, encoding="utf-8") as handle:
for line in handle:
line = line.strip()
if not line or line.startswith("#") or "=" not in line:
continue
key, _, value = line.partition("=")
values[key.strip()] = value.strip().strip('"').strip("'")
except OSError as exc:
log.warning("could not read %s (%s); falling back to the environment", path, exc)
for key in CONFIG_KEYS:
if key in os.environ:
values[key] = os.environ[key]
return values
def build_apps(_config: dict[str, str]) -> dict[str, App]:
"""The launch table. Every command is built from constants in this file — a payload
only ever selects a key here, it never contributes an argv element. See the security
note in mqtt_discovery.py."""
return {
"steam": App(
name="Steam Big Picture",
command=["/usr/local/bin/steam-big-picture"],
# No process_pattern: steam-big-picture does its own already-running check
# and knows how to ask a live client to re-open Big Picture, which is more
# than "focus the window" can do.
workspace=WS_STEAM,
icon="mdi:steam",
),
"prism": App(
name="Prism Launcher",
# Goes through Steam (steam://rungameid/…) so Steam Input and the Steam
# Controller API are live for Minecraft — see the script's header. Which is
# also why there is no workspace here: when Steam runs it, the window
# belongs to Steam's own workspace, and forcing 2:games would move it away
# from the client that owns it. 2:games is where it lands only on the
# direct-launch fallback path, and sway puts it there by focus anyway.
command=["/usr/local/bin/prism-launch"],
process_pattern="org.prismlauncher.PrismLauncher",
focus_criteria='app_id="org.prismlauncher.PrismLauncher"',
icon="mdi:minecraft",
),
"web_browser": App(
name="web browser",
command=["/usr/local/bin/web-browser"],
workspace=WS_WEB,
icon="mdi:web",
),
"media_player": App(
name="media player",
command=["/usr/local/bin/media-player"],
process_pattern="media-player-idle",
workspace=WS_MEDIA,
focus_criteria='app_id="mpv"',
icon="mdi:play-box",
),
"spotify": App(
name="Spotify",
command=["/usr/local/bin/spotify-launch"],
process_pattern="com.spotify.Client",
workspace=WS_MUSIC,
focus_criteria='app_id="com.spotify.Client"',
icon="mdi:spotify",
),
}
def make_client(client_id: str) -> mqtt.Client:
# paho-mqtt 2.x requires an explicit callback API version; older Debian releases
# ship 1.6.x and have no such argument. VERSION1 is requested when available so the
# callback signatures below are identical under both.
callback_api = getattr(mqtt, "CallbackAPIVersion", None)
if callback_api is not None:
return mqtt.Client(callback_api.VERSION1, client_id=client_id)
return mqtt.Client(client_id=client_id)
def main() -> int:
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s %(levelname)s %(name)s: %(message)s",
stream=sys.stdout,
)
config = load_config()
hostname = socket.gethostname()
node_id = "".join(c if c.isalnum() else "_" for c in hostname).strip("_") or "steamtv"
friendly_name = config.get("STEAMTV_NAME") or f"Steam TV box ({hostname})"
# The HA area this device sits in, published as suggested_area — see
# docs/rooms-and-endpoints.md. Blank is fine and means "no suggestion".
room = config.get("STEAMTV_ROOM", "")
broker_host = config.get("MQTT_BROKER_HOST", "")
broker_port = int(config.get("MQTT_BROKER_PORT") or 1883)
sway = SwayControl()
apps = build_apps(config)
audio = AudioControl(sway.session_env)
session = SessionMode(sway)
display = DisplayPower(sway, use_cec=config.get("ENABLE_CEC", "true") != "false")
client = make_client(f"steamtv-agent-{node_id}")
if config.get("MQTT_USERNAME"):
client.username_pw_set(config["MQTT_USERNAME"], config.get("MQTT_PASSWORD") or None)
discovery = Discovery(client, node_id, friendly_name, room)
mpris = MprisBridge(discovery.publish_media_state, sway.session_env)
def on_launch(key: str) -> None:
app = apps[key]
sway.launch_app(
app.command,
process_pattern=app.process_pattern,
workspace=app.workspace,
focus_criteria=app.focus_criteria,
)
if app.workspace:
discovery.publish_workspace(app.workspace)
def on_workspace(payload: str) -> None:
name = payload.strip()
# Enumerated, never passed through: see the security note in mqtt_discovery.py.
if name not in WORKSPACES:
log.warning("ignoring unknown workspace %r", name)
return
sway.switch_workspace(name)
discovery.publish_workspace(name)
def on_session_mode(payload: str) -> None:
mode = session.select(payload)
discovery.publish_session_state(mode, session.attributes())
def on_stop_media() -> None:
mode = session.stop_media()
discovery.publish_session_state(mode, session.attributes())
def on_audio_output(payload: str) -> None:
discovery.publish_audio_output(audio.select(payload))
def on_display(payload: str) -> None:
discovery.publish_display_power(display.handle_command(payload))
def on_connect(_client, _userdata, _flags, rc):
if rc != 0:
log.error("MQTT connection refused (rc=%s)", rc)
return
log.info("connected to MQTT broker %s:%s", broker_host, broker_port)
discovery.register_media_player(mpris.handle_command, mpris.set_volume)
discovery.register_app_launchers(apps, on_launch)
discovery.register_workspace_select(WORKSPACES, on_workspace, WS_STEAM)
discovery.register_session_mode(MODES, on_session_mode, on_stop_media)
# The sink list is read here rather than at construction because WirePlumber may
# not be up yet when this service starts (it is a system unit; the session is
# not). By the time the broker connects, the session normally is.
audio.list_sinks()
audio.apply_preferred()
discovery.register_audio_output(audio.options(), audio.current_option(), on_audio_output)
discovery.register_display_power(on_display, display.state)
discovery.subscribe_all()
discovery.publish_available(True)
discovery.publish_session_state(session.current(), session.attributes())
def on_disconnect(_client, _userdata, rc):
log.warning("disconnected from MQTT broker (rc=%s); paho will retry", rc)
def on_message(_client, _userdata, message):
discovery.dispatch(message.topic, message.payload.decode("utf-8", "replace"))
client.on_connect = on_connect
client.on_disconnect = on_disconnect
client.on_message = on_message
client.will_set(discovery.availability_topic, "offline", qos=1, retain=True)
stop_event = threading.Event()
def handle_signal(_signum, _frame):
stop_event.set()
signal.signal(signal.SIGTERM, handle_signal)
signal.signal(signal.SIGINT, handle_signal)
if not broker_host:
log.error("MQTT_BROKER_HOST is not set in %s — running without HA control", CONFIG_PATH)
else:
# connect_async + loop_start, never a blocking connect(): this box must boot to
# Big Picture and play a game with the container host powered off. Same
# "reactive path never depends on a remote service" rule as the thin client's
# Phase 11.10 — and it matters more here, since nobody wants their console to
# need the house's server to be up.
client.connect_async(broker_host, broker_port, keepalive=60)
client.loop_start()
log.info("steamtv-agent %s started (node_id=%s)", node_id, node_id)
ticks = itertools.count(1)
def poll_session() -> None:
if next(ticks) % SESSION_POLL_EVERY:
return
mode = session.current()
if mode == session.last_published:
return
session.last_published = mode
if broker_host:
discovery.publish_session_state(mode, session.attributes())
# The MPRIS bridge owns the main loop; the session sensor rides along on it rather
# than starting a second thread to do the same waiting.
original_poll = mpris.poll_once
def poll_both() -> None:
original_poll()
poll_session()
mpris.poll_once = poll_both # type: ignore[method-assign]
try:
mpris.run_forever(stop_event)
finally:
log.info("shutting down")
if broker_host:
discovery.publish_available(False)
client.loop_stop()
client.disconnect()
return 0
if __name__ == "__main__":
raise SystemExit(main())

138
hosts/steam-tv-box/agent/steamtv_agent/mpris_bridge.py
View File

@ -1,138 +0,0 @@
"""Bridges mpv's and Spotify's MPRIS state to the HA media_player entity via playerctl.
Identical in shape and reasoning to
hosts/thin-client/agent/thinclient_agent/mpris_bridge.py — see that file's docstring
for why this polls via subprocess rather than holding a dbus connection open.
PLAYER_PRIORITY lists mpv first, then the Spotify GUI client. That order is the answer
to "both are alive, which one does the remote's play button reach?", and mpv wins
because it is the one holding something somebody deliberately opened.
What is NOT in that list is any game. Steam exposes no MPRIS bus, and a game's audio is
not something you "pause" from a phone — the volume controls in audio_control.py are
the right surface for a game making noise, not these transport buttons.
This bridge reports "off" for most of a gaming session, and that is correct rather than
broken: on this image the media players genuinely do not exist until somebody leaves
Big Picture (see configs/session/media-session). The entity to watch for "what is this
box doing" is the session-mode sensor in session_mode.py.
"""
from __future__ import annotations
import logging
import subprocess
import threading
log = logging.getLogger(__name__)
PLAYER_PRIORITY = "mpv,spotify,%any"
_STATUS_TO_HA = {
"Playing": "playing",
"Paused": "paused",
"Stopped": "idle",
}
_METADATA_FORMAT = "{{title}}\x1f{{artist}}\x1f{{album}}\x1f{{mpris:length}}\x1f{{mpris:artUrl}}"
class MprisBridge:
def __init__(self, publish_state, env_provider, poll_interval: float = 2.0):
self._publish_state = publish_state
# playerctl needs DBUS_SESSION_BUS_ADDRESS, which this system service does not
# inherit; SwayControl.session_env() derives it from the kiosk user's runtime dir.
self._env_provider = env_provider
self._poll_interval = poll_interval
self._last_state: dict | None = None
def _playerctl(self, *args: str) -> str | None:
try:
result = subprocess.run(
["playerctl", "-p", PLAYER_PRIORITY, *args],
env=self._env_provider(),
capture_output=True,
text=True,
timeout=5,
check=False,
)
except (OSError, subprocess.TimeoutExpired) as exc:
log.debug("playerctl %s failed: %s", " ".join(args), exc)
return None
if result.returncode != 0:
return None
return result.stdout.strip()
def read_state(self) -> dict:
status = self._playerctl("status")
if status is None:
return {"state": "off"}
state = {"state": _STATUS_TO_HA.get(status, "idle")}
metadata = self._playerctl("metadata", "--format", _METADATA_FORMAT)
if metadata:
title, artist, album, length, art_url = (metadata.split("\x1f") + [""] * 5)[:5]
state["title"] = title
state["artist"] = artist
state["album"] = album
state["art_url"] = art_url
if length.isdigit():
state["duration"] = int(length) // 1_000_000
position = self._playerctl("position")
if position:
try:
state["position"] = int(float(position))
except ValueError:
pass
volume = self._playerctl("volume")
if volume:
try:
state["volume"] = round(float(volume), 3)
except ValueError:
pass
return state
def poll_once(self) -> None:
state = self.read_state()
if state != self._last_state:
self._last_state = state
self._publish_state(state)
def run_forever(self, stop_event: threading.Event) -> None:
while not stop_event.is_set():
try:
self.poll_once()
except Exception:
log.exception("MPRIS poll failed")
stop_event.wait(self._poll_interval)
# --- command side -------------------------------------------------------
def handle_command(self, command: str) -> None:
command = command.strip().upper()
action = {
"PLAY": ("play",),
"PAUSE": ("pause",),
"PLAY_PAUSE": ("play-pause",),
"TOGGLE": ("play-pause",),
"STOP": ("stop",),
"NEXT": ("next",),
"PREVIOUS": ("previous",),
"PREV": ("previous",),
}.get(command)
if action is None:
log.warning("ignoring unknown media command %r", command)
return
log.info("media command %s", command)
self._playerctl(*action)
self.poll_once()
def set_volume(self, level: float) -> None:
level = max(0.0, min(1.0, level))
self._playerctl("volume", f"{level:.3f}")
self.poll_once()

308
hosts/steam-tv-box/agent/steamtv_agent/mqtt_discovery.py
View File

@ -1,308 +0,0 @@
"""Home Assistant MQTT Discovery payloads and command dispatch.
SECURITY BOUNDARY — this module is the entire remote-control API of the Steam TV box.
Same principle as hosts/thin-client/agent/thinclient_agent/mqtt_discovery.py and the
touch panel's, applied to this device: the local LLM never gets a network path to this
machine. The only chain is: LLM tool call -> Home Assistant service call -> MQTT ->
this dispatcher. That property holds only as long as this stays the sole inbound
control surface: no HTTP listener, no websocket server, no exposed Sway IPC socket, no
shell endpoint. A new feature belongs as another entity below, not as another listener.
It is worth restating on this host in particular, because it is the one machine here
with a real GPU, a Steam login and a games library on it — i.e. the one with something
worth taking. Nothing in this image listens on a port except sshd (key-only) and wayvnc
(password-mandatory, fails closed). Steam's own remote-play ports are not opened by
anything here; if you want them, that is a deliberate decision to make in the firewall,
not a default this image ships.
Every command handler here is a fixed, enumerated action. A payload never becomes an
argv element, a shell string, or a URL host — see the launch table in main.py, which
builds every command from local constants and uses the payload only to pick between
known values.
"""
from __future__ import annotations
import json
import logging
from typing import Callable
from . import __version__
log = logging.getLogger(__name__)
DISCOVERY_PREFIX = "homeassistant"
class Discovery:
def __init__(self, client, node_id: str, friendly_name: str, room: str = ""):
self.client = client
self.node_id = node_id
self.friendly_name = friendly_name
self.room = (room or "").strip()
self.base = f"steamtv/{node_id}"
self.availability_topic = f"{self.base}/availability"
self.media_state_topic = f"{self.base}/media/state"
self.session_state_topic = f"{self.base}/session/state"
self._handlers: dict[str, Callable[[str], None]] = {}
self.device = {
"identifiers": [f"steamtv_{node_id}"],
"name": friendly_name,
"manufacturer": "SmartestHome",
"model": "Steam TV box",
"sw_version": __version__,
}
# Which room this physically sits in, as an HA area_id. `suggested_area` is
# honoured by HA only when the device is FIRST discovered — moving a device
# later means moving it in HA too, this cannot un-file it. Omitted entirely
# when unset, because an empty suggested_area is not the same request as no
# suggestion. See docs/rooms-and-endpoints.md.
if self.room:
self.device["suggested_area"] = self.room
# --- plumbing -----------------------------------------------------------
def _publish_config(self, component: str, object_id: str, payload: dict) -> None:
payload = {
"availability_topic": self.availability_topic,
"device": self.device,
"unique_id": f"{self.node_id}_{object_id}",
**payload,
}
topic = f"{DISCOVERY_PREFIX}/{component}/{self.node_id}/{object_id}/config"
self.client.publish(topic, json.dumps(payload), qos=1, retain=True)
def _command_topic(self, suffix: str, handler) -> str:
topic = f"{self.base}/{suffix}"
self._handlers[topic] = handler
return topic
def subscribe_all(self) -> None:
for topic in self._handlers:
self.client.subscribe(topic, qos=1)
def dispatch(self, topic: str, payload: str) -> None:
handler = self._handlers.get(topic)
if handler is None:
log.warning("no handler for %s", topic)
return
try:
handler(payload)
except Exception:
log.exception("handler for %s failed", topic)
def publish_available(self, available: bool = True) -> None:
self.client.publish(
self.availability_topic,
"online" if available else "offline",
qos=1,
retain=True,
)
def publish_media_state(self, state: dict) -> None:
self.client.publish(self.media_state_topic, json.dumps(state), qos=0, retain=True)
# --- media --------------------------------------------------------------
def register_media_player(self, on_command, on_volume) -> None:
command_topic = self._command_topic("media/command", on_command)
volume_topic = self._command_topic("media/volume/set", lambda p: on_volume(float(p)))
# Core Home Assistant's MQTT integration has NO media_player platform — see
# hosts/thin-client/README.md's identical caveat. The button/number entities
# below give the same transport control with stock HA.
self._publish_config(
"media_player",
"media",
{
"name": "Media",
"state_topic": self.media_state_topic,
"state_template": "{{ value_json.state }}",
"command_topic": command_topic,
"volume_command_topic": volume_topic,
"volume_state_topic": self.media_state_topic,
"volume_template": "{{ value_json.volume }}",
"title_template": "{{ value_json.title }}",
"artist_template": "{{ value_json.artist }}",
"album_template": "{{ value_json.album }}",
},
)
for object_id, name, payload, icon in (
("media_play_pause", "Play/pause", "PLAY_PAUSE", "mdi:play-pause"),
("media_next", "Next track", "NEXT", "mdi:skip-next"),
("media_previous", "Previous track", "PREVIOUS", "mdi:skip-previous"),
("media_stop", "Stop", "STOP", "mdi:stop"),
):
self._publish_config(
"button",
object_id,
{
"name": name,
"command_topic": command_topic,
"payload_press": payload,
"icon": icon,
},
)
self._publish_config(
"sensor",
"media_state",
{
"name": "Playback state",
"state_topic": self.media_state_topic,
"value_template": "{{ value_json.state }}",
"json_attributes_topic": self.media_state_topic,
"icon": "mdi:play-circle",
},
)
self._publish_config(
"number",
"media_volume",
{
"name": "Volume",
"command_topic": volume_topic,
"state_topic": self.media_state_topic,
"value_template": "{{ value_json.volume }}",
"min": 0,
"max": 1,
"step": 0.05,
"mode": "slider",
"icon": "mdi:volume-high",
},
)
# --- apps / workspaces --------------------------------------------------
def register_app_launchers(self, apps, on_launch) -> None:
for key, app in apps.items():
self._publish_config(
"button",
f"launch_{key}",
{
"name": f"Launch {app.name}",
"command_topic": self._command_topic(
f"app/{key}/launch",
lambda _payload, key=key: on_launch(key),
),
"icon": app.icon,
},
)
def register_workspace_select(self, workspaces, on_workspace, state_topic_value) -> None:
self._publish_config(
"select",
"workspace",
{
"name": "Screen",
"command_topic": self._command_topic("workspace/set", on_workspace),
"state_topic": f"{self.base}/workspace/state",
"options": list(workspaces),
"icon": "mdi:view-dashboard",
},
)
self.client.publish(
f"{self.base}/workspace/state", state_topic_value, qos=1, retain=True
)
def publish_workspace(self, name: str) -> None:
self.client.publish(f"{self.base}/workspace/state", name, qos=1, retain=True)
# --- session mode -------------------------------------------------------
def register_session_mode(self, modes, on_select, on_stop_media) -> None:
"""The "what is this box doing" surface — see session_mode.py.
A sensor and a select rather than one entity, because the two are not the same
question. The sensor reports four states (gaming / steam / media / idle); the
select offers only the two that are meaningful to *ask for*. "idle" is not
something you can request, and "steam" (Steam up but not focused) is a
transitional state nobody sets on purpose.
"""
self._publish_config(
"sensor",
"session_mode",
{
"name": "Session",
"state_topic": self.session_state_topic,
"value_template": "{{ value_json.mode }}",
"json_attributes_topic": self.session_state_topic,
"icon": "mdi:gamepad-variant",
},
)
self._publish_config(
"select",
"session_mode_select",
{
"name": "Mode",
"command_topic": self._command_topic("session/mode/set", on_select),
"state_topic": self.session_state_topic,
"value_template": "{{ value_json.mode }}",
"options": list(modes),
"icon": "mdi:gamepad-variant",
},
)
# The only thing in this whole surface that shuts something down, and therefore
# its own explicit button rather than a side effect of switching mode. See
# SessionMode.select()'s docstring for why the mode switch is additive.
self._publish_config(
"button",
"stop_media",
{
"name": "Stop media apps",
"command_topic": self._command_topic(
"session/media/stop", lambda _payload: on_stop_media()
),
"icon": "mdi:close-circle-outline",
},
)
def publish_session_state(self, mode: str, attributes: dict | None = None) -> None:
payload = {"mode": mode}
if attributes:
payload.update(attributes)
self.client.publish(
self.session_state_topic, json.dumps(payload), qos=1, retain=True
)
# --- audio output -------------------------------------------------------
def register_audio_output(self, options, current, on_select) -> None:
self._publish_config(
"select",
"audio_output",
{
"name": "Audio output",
"command_topic": self._command_topic("audio/output/set", on_select),
"state_topic": f"{self.base}/audio/output/state",
"options": list(options),
"icon": "mdi:speaker",
},
)
self.publish_audio_output(current)
def publish_audio_output(self, option: str) -> None:
self.client.publish(
f"{self.base}/audio/output/state", option, qos=1, retain=True
)
# --- display power ------------------------------------------------------
def register_display_power(self, on_command, initial: bool = True) -> None:
self._publish_config(
"switch",
"display",
{
"name": "Display",
"command_topic": self._command_topic("display/set", on_command),
"state_topic": f"{self.base}/display/state",
"payload_on": "ON",
"payload_off": "OFF",
"icon": "mdi:television",
},
)
self.publish_display_power(initial)
def publish_display_power(self, on: bool) -> None:
self.client.publish(
f"{self.base}/display/state", "ON" if on else "OFF", qos=1, retain=True
)

75
hosts/steam-tv-box/agent/steamtv_agent/runtime_state.py
View File

@ -1,75 +0,0 @@
"""Seeds writable runtime copies of the read-only config templates in the image.
One of this agent's config files (audio-config.json) is rewritten at runtime — the HA
"Audio output" select has to survive a reboot, and it cannot live where the build put
it, because the live image's /etc is inside a squashfs. (The thin client's copy of this
module also covers rdp-vnc.json; this box has no outbound remote-desktop client, so
audio is the only file here. The module is kept whole rather than trimmed so the two
stay comparable.)
So the same split the wayvnc password already uses applies here — a committed template
that the build bakes in read-only, plus a real file created on the booted machine that
is never committed. The difference is that wayvnc's real file is written by a human and
fails closed if they forget, whereas these two are seeded automatically from the
template, because "no audio-output preference yet" is a perfectly safe state and there
is nothing to fail closed about.
"""
from __future__ import annotations
import json
import logging
import os
import shutil
import tempfile
log = logging.getLogger(__name__)
TEMPLATE_DIR = os.environ.get("STEAMTV_TEMPLATE_DIR", "/etc/steamtv-agent")
STATE_DIR = os.environ.get("STEAMTV_STATE_DIR", "/var/lib/steamtv-agent")
def ensure_runtime_copy(filename: str) -> str:
"""Return the writable path for `filename`, seeding it from the template if new."""
runtime_path = os.path.join(STATE_DIR, filename)
if os.path.exists(runtime_path):
return runtime_path
template_path = os.path.join(TEMPLATE_DIR, filename)
try:
os.makedirs(STATE_DIR, exist_ok=True)
shutil.copyfile(template_path, runtime_path)
log.info("seeded %s from %s", runtime_path, template_path)
except OSError as exc:
log.warning("could not seed %s from %s: %s", runtime_path, template_path, exc)
return runtime_path
def load_json(path: str) -> dict:
try:
with open(path, encoding="utf-8") as handle:
data = json.load(handle)
except (OSError, ValueError) as exc:
log.warning("could not read %s (%s); using defaults", path, exc)
return {}
return data if isinstance(data, dict) else {}
def save_json(path: str, data: dict) -> bool:
"""Write atomically — a half-written config on a power cut would be worse than a
stale one, since these files are read unattended at boot."""
directory = os.path.dirname(path) or "."
try:
os.makedirs(directory, exist_ok=True)
with tempfile.NamedTemporaryFile(
"w", encoding="utf-8", dir=directory, delete=False
) as handle:
json.dump(data, handle, indent=2)
handle.write("\n")
temp_path = handle.name
os.replace(temp_path, path)
except OSError as exc:
log.warning("could not write %s: %s", path, exc)
return False
return True

146
hosts/steam-tv-box/agent/steamtv_agent/session_mode.py
View File

@ -1,146 +0,0 @@
"""What is this box doing right now — gaming, media, or idle — and switching between.
WHY THIS EXISTS
---------------
Home Assistant needs to be able to answer "is somebody playing on the TV box?" without
guessing from power draw. That one sensor is worth a lot in automations: don't dim the
living room during a game, don't announce the doorbell over a raid, count the room as
occupied even though nobody has moved for forty minutes.
WHAT IT READS
-------------
Two cheap facts, both local:
- is the Steam client running (pgrep);
- which sway workspace is focused (swaymsg -t get_workspaces).
Deliberately NOT the window title. Steam's window titles and the existence of a
separate Big Picture window have both changed across client rewrites, and a sensor that
silently goes wrong after a Steam update is worse than one that is slightly coarse. The
workspace name is a contract this repo owns — see the header in
configs/session/session-watcher, which makes the same call for the same reason.
MODES
-----
gaming — Steam is running and 1:steam is focused. The screen is showing a game or
Big Picture.
steam — Steam is running but the user has moved elsewhere (this is "exited Big
Picture but left Steam up", the case session-watcher reacts to).
media — Steam is not running; the media session is what is on screen.
idle — nothing is up. Only really seen in the seconds after boot, or after
somebody quit Steam and the media session has not been started yet.
SECURITY POSTURE, UNCHANGED
---------------------------
The two commands below take no arguments derived from any MQTT payload. The mode
*switch* is an enumerated action — see mqtt_discovery.py's module docstring.
"""
from __future__ import annotations
import json
import logging
from .sway_control import WS_STEAM
log = logging.getLogger(__name__)
MODE_GAMING = "gaming"
MODE_STEAM = "steam"
MODE_MEDIA = "media"
MODE_IDLE = "idle"
MODES = (MODE_GAMING, MODE_MEDIA)
STEAM_PROCESS = "steam"
MEDIA_SESSION = "/usr/local/bin/media-session"
STEAM_BIG_PICTURE = "/usr/local/bin/steam-big-picture"
# What media-session itself guards on. Kept in sync with that script by hand; there is
# no shared file to read, and inventing one to hold three strings would be worse.
MEDIA_PATTERNS = (
"com.spotify.Client",
"media-player-idle",
"firefox.*--profile.*/firefox/web",
)
class SessionMode:
def __init__(self, sway):
self.sway = sway
# What current() last computed, versus what was last put on the broker. Two
# fields because current() updates the first every time it is called, so it
# cannot also serve as "have we told HA about this yet".
self.last_mode = MODE_IDLE
self.last_published = ""
# --- reading ------------------------------------------------------------
def focused_workspace(self) -> str:
output = self.sway.swaymsg("-t", "get_workspaces")
if not output:
return ""
try:
workspaces = json.loads(output)
except ValueError:
log.warning("could not parse get_workspaces output")
return ""
for workspace in workspaces:
if isinstance(workspace, dict) and workspace.get("focused"):
return str(workspace.get("name") or "")
return ""
def steam_running(self) -> bool:
# is_process (pgrep -x), not is_running (pgrep -f) — see that method's comment
# for why Steam specifically needs the exact-name match.
return self.sway.is_process(STEAM_PROCESS)
def media_running(self) -> bool:
return any(self.sway.is_running(pattern) for pattern in MEDIA_PATTERNS)
def current(self) -> str:
steam = self.steam_running()
if steam and self.focused_workspace() == WS_STEAM:
mode = MODE_GAMING
elif steam:
mode = MODE_STEAM
elif self.media_running():
mode = MODE_MEDIA
else:
mode = MODE_IDLE
self.last_mode = mode
return mode
def attributes(self) -> dict:
return {
"steam_running": self.steam_running(),
"media_running": self.media_running(),
"workspace": self.focused_workspace(),
}
# --- switching ----------------------------------------------------------
def select(self, mode: str) -> str:
"""Handle the HA select. Enumerated: anything else is ignored, not guessed at.
Note the asymmetry, which is intentional. Choosing "gaming" launches or focuses
Steam and leaves the media apps alone — someone may well want music over a
game, and killing a running Spotify because a game started would lose whatever
was playing. Choosing "media" starts the media session and likewise does not
quit Steam, because quitting Steam from a phone while somebody is mid-match is
not a thing this should be able to do by accident. Both directions are additive;
the only thing that shuts anything down is the explicit "Stop media apps"
button.
"""
mode = (mode or "").strip().lower()
if mode == MODE_GAMING:
log.info("session mode -> gaming")
self.sway.launch_app([STEAM_BIG_PICTURE])
elif mode == MODE_MEDIA:
log.info("session mode -> media")
self.sway.launch_app([MEDIA_SESSION, "start", "--focus"])
else:
log.warning("ignoring unknown session mode %r", mode)
return self.current()
def stop_media(self) -> str:
log.info("stopping the media session")
self.sway.launch_app([MEDIA_SESSION, "stop"])
return self.current()

135
hosts/steam-tv-box/agent/steamtv_agent/sway_control.py
View File

@ -1,135 +0,0 @@
"""Thin wrapper around swaymsg and local process launching.
Identical in shape to hosts/thin-client/agent/thinclient_agent/sway_control.py and
hosts/touch-panel/agent/touchpanel_agent/sway_control.py — duplicated rather than
imported across hosts, same convention as the rest of this repo's agents.
"""
from __future__ import annotations
import glob
import logging
import os
import subprocess
log = logging.getLogger(__name__)
# Contract with configs/sway/config — these strings must match the `set $ws_*` lines,
# and with configs/session/media-session and configs/session/session-watcher, which
# hardcode the same names in shell.
WS_STEAM = "1:steam"
WS_GAMES = "2:games"
WS_WEB = "3:web"
WS_MEDIA = "4:media"
WS_MUSIC = "5:music"
def runtime_dir() -> str:
return os.environ.get("XDG_RUNTIME_DIR") or f"/run/user/{os.getuid()}"
class SwayControl:
def _socket_path(self) -> str | None:
path = os.environ.get("SWAYSOCK")
if path and os.path.exists(path):
return path
# sway names the socket sway-ipc.<uid>.<pid>.sock, so the path changes every
# time sway restarts. steamtv-agent is a system service that outlives the
# session, so the socket is re-resolved per call instead of cached at startup.
matches = sorted(glob.glob(os.path.join(runtime_dir(), "sway-ipc.*.sock")))
return matches[-1] if matches else None
def session_env(self) -> dict[str, str]:
env = dict(os.environ)
env["XDG_RUNTIME_DIR"] = runtime_dir()
env.setdefault("DBUS_SESSION_BUS_ADDRESS", f"unix:path={runtime_dir()}/bus")
env.setdefault("WAYLAND_DISPLAY", "wayland-1")
sock = self._socket_path()
if sock:
env["SWAYSOCK"] = sock
return env
def swaymsg(self, *args: str) -> str | None:
if self._socket_path() is None:
log.warning("no sway IPC socket found; dropping command %s", " ".join(args))
return None
try:
result = subprocess.run(
["swaymsg", *args],
env=self.session_env(),
capture_output=True,
text=True,
timeout=10,
check=False,
)
except (OSError, subprocess.TimeoutExpired) as exc:
log.warning("swaymsg %s failed: %s", " ".join(args), exc)
return None
if result.returncode != 0:
log.warning("swaymsg %s: %s", " ".join(args), result.stderr.strip())
return None
return result.stdout
def switch_workspace(self, name: str) -> None:
log.info("switching to workspace %s", name)
self.swaymsg("workspace", name)
def focus_window(self, criteria: str) -> None:
self.swaymsg(f"[{criteria}] focus")
def is_running(self, pattern: str) -> bool:
"""Match `pattern` against full command lines (pgrep -f)."""
return self._pgrep("-f", pattern)
def is_process(self, name: str) -> bool:
"""Match `name` against process names only (pgrep -x).
Separate from is_running() because Steam is the one thing here that needs it:
its command line is `/bin/sh /usr/games/steam -gamepadui …` and it spawns a
dozen helpers (steamwebhelper, reaper, steamerrorreporter) whose command lines
all contain the word "steam". A -f match would report Steam as running long
after the client has gone, which would leave the session-mode sensor stuck on
"gaming" for the rest of the evening.
"""
return self._pgrep("-x", name)
def _pgrep(self, flag: str, pattern: str) -> bool:
try:
result = subprocess.run(
["pgrep", "-u", str(os.getuid()), flag, pattern],
capture_output=True,
timeout=5,
check=False,
)
except (OSError, subprocess.TimeoutExpired):
return False
return result.returncode == 0
def launch_app(
self,
command: list[str],
process_pattern: str | None = None,
workspace: str | None = None,
focus_criteria: str | None = None,
) -> None:
if workspace:
self.switch_workspace(workspace)
if process_pattern and self.is_running(process_pattern):
log.info("%s already running; focusing instead of launching", command[0])
if focus_criteria:
self.focus_window(focus_criteria)
return
log.info("launching %s", " ".join(command))
try:
subprocess.Popen(
command,
env=self.session_env(),
stdin=subprocess.DEVNULL,
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL,
start_new_session=True,
)
except OSError as exc:
log.error("could not launch %s: %s", " ".join(command), exc)

10
hosts/steam-tv-box/configs/audio/audio-config.json
View File

@ -1,10 +0,0 @@
{
"_comment": "TEMPLATE. Baked into the image read-only at /etc/steamtv-agent/audio-config.json. On first boot steamtv_agent/audio_control.py copies it to /var/lib/steamtv-agent/audio-config.json and from then on reads and REWRITES only that copy — the HA 'Audio output' select has to survive a reboot, and a file inside a squashfs image cannot. Same split as the wayvnc password: a committed template plus a runtime-populated real file that is gitignored.",
"_comment_preferred_sink": "A WirePlumber node.name, not a description and not a numeric ID. IDs are reassigned on every boot and on every device hotplug; node.name is stable. Find yours on the booted machine with: wpctl status then wpctl inspect <id> | grep node.name",
"_comment_this_box": "On a machine plugged into a television this is usually the HDMI sink (alsa_output.pci-....hdmi-stereo) rather than the analog one, and getting it wrong is the classic 'the game has no sound' report. Worth setting explicitly here rather than trusting WirePlumber's pick, because a TV that is switched off at boot can leave the HDMI sink absent and WirePlumber will happily default to the motherboard's headphone jack — which nothing is plugged into.",
"preferred_sink": "",
"fallback": "system-default"
}

63
hosts/steam-tv-box/configs/firefox/policies.json
View File

@ -1,63 +0,0 @@
{
"_comment_path": "Installed to /etc/firefox/policies/policies.json, the documented Linux location for Firefox enterprise policy. 0900-firefox.hook.chroot also links it into firefox-esr's distribution/ directory, an older location some builds read instead; whichever one the installed build honours wins and the other is ignored. Note that every key below lives OUTSIDE the 'policies' object on purpose — Firefox flags unrecognised keys *inside* it as invalid policies in about:policies.",
"_comment_extensions": "VERIFY BEFORE THE FIRST REAL BUILD. The two AMO slugs ('ublock-origin', 'sponsorblock') were confirmed against the live addons.mozilla.org listings, and /firefox/downloads/latest/<slug>/latest.xpi is AMO's documented always-current download URL, so install_url should be right. The ExtensionSettings *keys* must be each extension's real add-on ID as Firefox sees it, and those were NOT verifiable from the AMO listing pages — they are widely-published values reproduced here, not checked. If an extension silently fails to install, that key is the first suspect: install the XPI by hand once, read the ID off about:debugging#/runtime/this-firefox, and correct it here.",
"_comment_kiosk_ui": "The Disable*/UserMessaging/FirefoxHome blocks all remove a prompt, tour, or nag that would otherwise sit on an unattended screen in a shared room with nobody there to dismiss it.",
"_comment_updates": "The browser is part of the image and is replaced by rebuilding and reflashing it, so in-browser updates would only produce version drift between rooms plus a restart banner nobody is there to click.",
"policies": {
"ExtensionSettings": {
"uBlock0@raymondhill.net": {
"installation_mode": "force_installed",
"install_url": "https://addons.mozilla.org/firefox/downloads/latest/ublock-origin/latest.xpi",
"default_area": "menupanel"
},
"sponsorBlocker@ajay.app": {
"installation_mode": "force_installed",
"install_url": "https://addons.mozilla.org/firefox/downloads/latest/sponsorblock/latest.xpi",
"default_area": "menupanel"
}
},
"DisableProfileImport": true,
"DisableProfileRefresh": true,
"DisableFirefoxAccounts": true,
"DisableFirefoxStudies": true,
"DisableTelemetry": true,
"DisablePocket": true,
"DisableFeedbackCommands": true,
"DisableSetDesktopBackground": true,
"DontCheckDefaultBrowser": true,
"NoDefaultBookmarks": true,
"OfferToSaveLogins": false,
"PasswordManagerEnabled": false,
"PromptForDownloadLocation": false,
"AppAutoUpdate": false,
"DisableAppUpdate": true,
"UserMessaging": {
"WhatsNew": false,
"ExtensionRecommendations": false,
"FeatureRecommendations": false,
"UrlbarInterventions": false,
"SkipOnboarding": true,
"MoreFromMozilla": false
},
"FirefoxHome": {
"Search": true,
"TopSites": true,
"SponsoredTopSites": false,
"Highlights": false,
"Pocket": false,
"SponsoredPocket": false,
"Snippets": false
},
"OverrideFirstRunPage": "",
"OverridePostUpdatePage": ""
}
}

55
hosts/steam-tv-box/configs/firefox/user.js
View File

@ -1,55 +0,0 @@
// Prefs for the Steam TV box's Firefox profile. Installed by
// build-steam-tv-box-iso.sh to /etc/steamtv-firefox/user.js, from where
// /usr/local/bin/web-browser copies it into
// /home/<KIOSK_USERNAME>/.mozilla/firefox/web on every launch.
//
// One profile here, not two: this image has no digest canvas — the browser is the
// general-browsing window and nothing else. Otherwise these are the thin client's
// prefs verbatim, and for the same reasons.
//
// user.js rather than prefs.js: user.js is re-applied to prefs.js on every startup, so
// nothing a stray click changes at runtime survives a restart. That matters on a kiosk
// nobody logs into to fix things.
//
// Anything expressible as enterprise policy lives in policies.json instead — policy is
// enforced and shows up in about:policies, whereas a pref is merely a default. The
// prefs here are the ones with no policy equivalent.
// The only reason userChrome.css is read at all. Without this the stylesheet in
// chrome/ is silently ignored and the window comes up with full default chrome.
user_pref("toolkit.legacyUserProfileCustomizations.stylesheets", true);
// First-run / post-update interstitials. An unattended screen has nobody to close them.
user_pref("browser.startup.homepage_override.mstone", "ignore");
user_pref("browser.aboutwelcome.enabled", false);
user_pref("browser.messaging-system.whatsNewPanel.enabled", false);
user_pref("trailhead.firstrun.didSeeAboutWelcome", true);
user_pref("datareporting.policy.firstRunURL", "");
user_pref("datareporting.policy.dataSubmissionPolicyBypassNotification", true);
// Session restore prompts after a power cut would leave the room's display stuck on a
// "restore your session?" page instead of the digest.
user_pref("browser.sessionstore.resume_from_crash", false);
user_pref("browser.tabs.warnOnClose", false);
user_pref("browser.tabs.warnOnCloseOtherTabs", false);
user_pref("browser.warnOnQuit", false);
// No permission doorhangers on a display nobody is standing at. Autoplay is allowed
// because the digest canvas and embedded video are the point of the machine.
user_pref("permissions.default.desktop-notification", 2);
user_pref("permissions.default.geo", 2);
user_pref("media.autoplay.default", 0);
user_pref("media.autoplay.blocking_policy", 0);
// Wayland-native rather than Xwayland, matching MOZ_ENABLE_WAYLAND in kiosk-session.
// VA-API is left at its default: no thin-client hardware has been chosen yet
// (project-plan §4 #7), so forcing hardware decode here could just as easily produce a
// black video surface as a working one.
user_pref("gfx.webrender.all", true);
// The address bar in the minimal-chrome window is for typing a URL, not for a dropdown
// of suggestions covering the page.
user_pref("browser.urlbar.suggest.topsites", false);
user_pref("browser.urlbar.suggest.quicksuggest.sponsored", false);
user_pref("browser.newtabpage.activity-stream.showSponsored", false);
user_pref("browser.newtabpage.activity-stream.showSponsoredTopSites", false);

63
hosts/steam-tv-box/configs/firefox/userChrome.css
View File

@ -1,63 +0,0 @@
/* Minimal browser chrome for the Steam TV box's browsing window.
* Installed to <profile>/chrome/userChrome.css by /usr/local/bin/web-browser, and only
* read because user.js sets toolkit.legacyUserProfileCustomizations.stylesheets=true.
*
* What survives: back, forward, reload, and the address bar. Everything else goes.
* That set is deliberate — those four are what someone driving this from the sofa with
* the remote's little keyboard actually needs, and every extra control is one more
* thing to mis-click on a television at two metres.
*
* Taken verbatim from hosts/thin-client/configs/firefox/userChrome.css; the two are
* meant to stay identical, so the browser behaves the same in every room.
*/
/* Tab strip. The kiosk browses one page at a time; a tab bar on a shared display just
* accumulates whatever the last person left open. */
#TabsToolbar,
#tabbrowser-tabs,
#alltabs-button {
visibility: collapse !important;
}
/* Menu bar and bookmarks bar. */
#toolbar-menubar,
#PersonalToolbar,
#PlacesToolbar {
visibility: collapse !important;
}
/* Everything on the nav bar except back / forward / reload / the address bar. */
#home-button,
#library-button,
#sidebar-button,
#fxa-toolbar-menu-button,
#unified-extensions-button,
#PanelUI-button,
#downloads-button,
#save-to-pocket-button,
#pageActionButton,
#star-button-box,
#reader-mode-button,
#customizableui-special-spring1,
#customizableui-special-spring2 {
display: none !important;
}
/* The urlbar keeps its identity box (padlock) — dropping it would hide the only
* on-screen signal that a page is or is not https. */
#urlbar-container {
min-width: 0 !important;
}
#nav-bar {
border: none !important;
box-shadow: none !important;
}
/* Titlebar spacers left over once the tab strip is collapsed. */
.titlebar-buttonbox-container,
.titlebar-spacer {
display: none !important;
}
/* Findbar and notification popups stay: they are transient and user-initiated. */

47
hosts/steam-tv-box/configs/firefox/web-browser
View File

@ -1,47 +0,0 @@
#!/bin/sh
# The general web browser — minimal chrome, not kiosk mode. Installed to
# /usr/local/bin/web-browser.
#
# The requirement this satisfies is "a web browser with uBlock Origin", which is
# enforced by /etc/firefox/policies/policies.json (force_installed, so it cannot be
# removed by a stray click on the TV) rather than by anything here.
#
# Unlike the thin client there is only one profile on this image: no digest canvas
# means no second Firefox process to keep apart. The profile still lives under a named
# directory rather than the default, because media-session's pgrep guard matches on the
# --profile path to decide whether the browser is already up.
#
# SponsorBlock is in the policy set too. On a box whose browser exists mostly to play
# video on a television, skipping sponsor segments without a remote in hand is worth
# more here than anywhere else in the house.
set -eu
PROFILE_DIR="${HOME:-/home/$(id -un)}/.mozilla/firefox/web"
if command -v firefox-esr >/dev/null 2>&1; then
FIREFOX=firefox-esr
else
FIREFOX=firefox
fi
mkdir -p "$PROFILE_DIR/chrome"
# Re-copied on every launch rather than once at build time: this is a locked-down
# profile with no interactive customisation expected, so keeping it in lockstep with
# /etc/steamtv-firefox/ (edited by rebuilding the image) is simpler than a one-shot seed
# that could drift after a userChrome.css update.
cp /etc/steamtv-firefox/userChrome.css "$PROFILE_DIR/chrome/userChrome.css" 2>/dev/null || true
cp /etc/steamtv-firefox/user.js "$PROFILE_DIR/user.js" 2>/dev/null || true
# Focus an already-open window rather than stacking a second one: this window holds
# state (history, a half-typed URL, a logged-in video site) that a kill-and-relaunch
# would throw away — and media-session calls this every time somebody leaves Big
# Picture, so "already open" is the common case, not the exception.
if pgrep -u "$(id -u)" -f "$FIREFOX .*--profile $PROFILE_DIR" >/dev/null 2>&1; then
if [ -n "${1:-}" ]; then
exec "$FIREFOX" --profile "$PROFILE_DIR" --new-tab "$1"
fi
exit 0
fi
exec "$FIREFOX" --profile "$PROFILE_DIR" --new-instance --new-window "${1:-about:blank}"

21
hosts/steam-tv-box/configs/greetd/config.toml
View File

@ -1,21 +0,0 @@
# greetd — autologin straight into the kiosk Sway session, no greeter UI.
# Installed to /etc/greetd/config.toml by build-steam-tv-box-iso.sh, which substitutes
# @KIOSK_USERNAME@ with its own KIOSK_USERNAME variable on the way in.
#
# Schema note (getting these key names wrong fails silently — you get a black VT with
# no login, and greetd logs nothing obvious): the table names are exactly [terminal]
# and [default_session], and the session keys are exactly `command` and `user`.
#
# Why [default_session] and not [initial_session]: [initial_session] fires once, on the
# first greetd start after boot, and greetd falls back to [default_session] the moment
# that session ends. For a TV box that would mean one crashed sway drops the living
# room to an agreety login prompt on a screen with no keyboard in front of it. Pointing
# [default_session] itself at the kiosk user makes the autologin permanent and
# self-healing. [initial_session] is deliberately absent.
[terminal]
vt = 1
[default_session]
command = "/usr/local/bin/kiosk-session"
user = "@KIOSK_USERNAME@"

43
hosts/steam-tv-box/configs/greetd/kiosk-session
View File

@ -1,43 +0,0 @@
#!/bin/sh
# greetd's default_session command. Installed to /usr/local/bin/kiosk-session.
#
# Exists so that MQTT_BROKER_HOST / HA_URL / GPU_VENDOR are in Sway's environment:
# sway's config file has no way to read an env file itself, but every `exec` line it
# runs inherits this process's environment, so sourcing here is what lets the session
# scripts read configuration without the sway config being templated.
set -eu
# An `if` rather than `[ -r … ] && . …`: under `set -e` the && form exits the whole
# script when the file is absent, which would leave greetd with a session that dies
# instantly and a black screen.
if [ -r /etc/steamtv-agent/config.env ]; then
. /etc/steamtv-agent/config.env
fi
export MQTT_BROKER_HOST HA_URL GPU_VENDOR
export XDG_CURRENT_DESKTOP=sway
export XDG_SESSION_TYPE=wayland
export XDG_SESSION_DESKTOP=sway
export MOZ_ENABLE_WAYLAND=1
: "${XDG_RUNTIME_DIR:=/run/user/$(id -u)}"
export XDG_RUNTIME_DIR
# Steam's client, Proton, Prism's Minecraft windows and most native titles are X11
# applications. Telling them the display is 96dpi-normal and letting Xwayland scale is
# the sane default for a TV at 1-2m viewing distance; a 4K set that looks
# postage-stamp-sized wants this raised, not the compositor's scale changed (which
# would blur Big Picture).
export GDK_BACKEND=wayland,x11
export QT_QPA_PLATFORM="wayland;xcb"
export SDL_VIDEODRIVER=wayland,x11
# NVIDIA's driver still mishandles hardware cursor planes under wlroots on a number of
# releases; the symptom is an invisible or corrupt pointer, which on a box you drive
# with a gamepad is easy to miss until someone picks up a mouse. Harmless when set on
# hardware that doesn't need it, so it is scoped to the vendor rather than always on.
if [ "${GPU_VENDOR:-}" = "nvidia" ]; then
export WLR_NO_HARDWARE_CURSORS=1
fi
exec sway

24
hosts/steam-tv-box/configs/installer/preseed.cfg
View File

@ -1,24 +0,0 @@
# Only relevant when `enable_installer` is true for this kiosk in CoreSystemConfig.json.
#
# Unlike the thin client, this one is worth turning on: a gaming box wants games on a
# real disk rather than in a live session's RAM overlay, so the normal path here is an
# actual install. See hosts/steam-tv-box/README.md.
#
# THIS FILE IS THE SOURCE, NOT THE ONE LIVE-BUILD READS.
# build-steam-tv-box-iso.sh substitutes @KEYBOARD_LAYOUT@ and writes the result to
# live-build/config/preseed.cfg on every run, so that the installer's default layout
# follows household.keyboard_layout in CoreSystemConfig.json instead of being a second
# place the layout can be wrong. The generated copy is regenerated every build — edit
# this one. (The thin client hardcodes its layout here because its installer is off by
# default; this image's is not.)
#
# live-build auto-includes config/preseed.cfg into the debian-installer's preseed when
# --debian-installer is not "none". VERIFY: this is live-build's documented mechanism
# for this, but has not been exercised — if enable_installer is turned on, confirm the
# installer's keyboard step actually shows the right layout pre-selected rather than
# falling back to its own default.
#
# No "keyboard-configuration/xkb-keymap seen true" line on purpose: this preseeds the
# *default* answer, it does not skip the question, so whoever runs the installer can
# still pick a different layout for this specific machine.
d-i keyboard-configuration/xkb-keymap select @KEYBOARD_LAYOUT@

31
hosts/steam-tv-box/configs/mpv/mpv.conf
View File

@ -1,31 +0,0 @@
# mpv defaults for the kiosk user. Installed to
# /home/<KIOSK_USERNAME>/.config/mpv/mpv.conf by build-steam-tv-box-iso.sh.
#
# input-ipc-server is what lets `media-player <file>` load into the mpv window that is
# already sitting on 4:media instead of opening a second one — two mpvs would mean two
# MPRIS players and a coin-flip as to which one the remote's play button reaches.
#
# VERIFY: `~/` expansion in an mpv.conf value is documented behaviour but has not been
# checked on this image. If the socket never appears, replace the path below with the
# absolute one for the kiosk user.
input-ipc-server=~/.mpv-socket
# Hardware decode. This box has a real GPU, which is the one place in this project
# where that is true, so 4K video should not be burning cores it could be spending on a
# game running alongside it. `auto-safe` picks a hardware decoder only from the
# combinations upstream considers reliable and silently falls back to software
# otherwise — the failure mode is a warm CPU, never a black window.
hwdec=auto-safe
# Sensible on a television: keep the aspect ratio, never letterbox into a window that
# is already fullscreen, and don't drop to a tiny window for a small file.
keep-open=yes
force-window=yes
# Audio goes to whatever WirePlumber's default sink is, which is what the agent's
# "Audio output" select actually changes. Naming a device here would silently override
# that select and make it look broken.
ao=pipewire
# Nothing here loads mpv-mpris: Debian's mpv-mpris package drops its .so into mpv's
# autoload directory, and naming it again would load the plugin twice.

38
hosts/steam-tv-box/configs/session/media-player
View File

@ -1,38 +0,0 @@
#!/bin/sh
# The general mp3/mp4 player. Installed to /usr/local/bin/media-player.
#
# mpv, same as the thin client's — it plays everything without a codec pack, it exposes
# MPRIS through mpv-mpris (which is what puts the transport controls in Home Assistant
# and on the remote's media keys), and it takes a file, a directory or a URL equally.
#
# --idle=yes --force-window is what makes it part of a *session* rather than a one-shot
# command: with no file given it opens an empty player window and waits, so it is
# already on 4:media with a working MPRIS bus before anyone has picked something to
# play. Dropping a file on it, opening one from a share, or `media-player <path>` from
# the maintenance shell all then load into the window that is already there.
#
# The "media-player-idle" title is not decoration — media-session and the sway config
# both match on it (mpv's app_id is just "mpv", which a second, file-playing mpv would
# share), so changing it means changing those two too.
set -eu
# A second invocation with a file loads it into the running instance rather than
# opening a competing window: two mpvs means two MPRIS players and a coin-flip as to
# which one the remote's play button reaches. mpv.conf sets input-ipc-server for this.
SOCKET="${HOME:-/home/$(id -un)}/.mpv-socket"
if [ -n "${1:-}" ] && [ -S "$SOCKET" ]; then
# loadfile via the JSON IPC. The path is passed as a JSON string argument, never
# interpolated into a shell command.
if printf '{"command":["loadfile","%s","replace"]}\n' "$1" | socat - "$SOCKET" 2>/dev/null; then
exit 0
fi
# If that failed the socket is stale (mpv died without cleaning up) — fall through to
# a plain launch, which is the right outcome rather than an error.
fi
if [ -n "${1:-}" ]; then
exec mpv --title=media-player-idle "$@"
fi
exec mpv --idle=yes --force-window=yes --title=media-player-idle

133
hosts/steam-tv-box/configs/session/media-session
View File

@ -1,133 +0,0 @@
#!/bin/sh
# Brings up (or takes down) the media half of this box. Installed to
# /usr/local/bin/media-session.
#
# WHAT THIS IS FOR
# ----------------
# The requirement was: boot into Big Picture, and on leaving it have the same media
# functions as the other clients that drive a monitor — Spotify, a general web browser
# with uBlock Origin, a general mp3/mp4 player, and working audio — with those apps
# only launched when Big Picture is actually left. This script is the "launch them"
# half; steam-session and session-watcher are the two things that call it.
#
# WHY LAZY AT ALL
# ---------------
# Not tidiness. A machine that is playing a game is the one machine here where a
# background Firefox and a background Electron/Spotify client cost something visible:
# they hold GPU memory, they wake the CPU on timers, and Spotify in particular keeps an
# audio stream open that shows up as stutter in a title that is already frame-limited.
# Starting them the first time somebody leaves the game means a gaming session pays
# nothing for media features it isn't using.
#
# IDEMPOTENCE
# -----------
# Every launch below is guarded by its own pgrep, and the whole start path holds an
# flock, so:
# - the two triggers firing at once cannot produce two Spotifys;
# - leaving and re-entering Big Picture repeatedly costs one pgrep per app;
# - an app the user closed by hand comes back the next time they leave the game,
# which is what "the media session is up" should mean.
# There is deliberately no "already started" flag file: a flag would go stale the
# moment somebody quit one of the apps, and the per-app check is the honest question.
#
# WHAT LEAVING BIG PICTURE DOES *NOT* DO
# --------------------------------------
# Going back into Steam does not stop any of this. Killing a running Spotify because
# someone launched a game would lose whatever was playing — and people do play music
# over a game on purpose. `media-session stop` exists for when you actually want the
# machine quiet (it is wired to an HA button); nothing calls it automatically.
set -eu
WS_WEB="3:web"
WS_MEDIA="4:media"
WS_MUSIC="5:music"
LOCK_DIR="${XDG_RUNTIME_DIR:-/tmp}"
LOCK_FILE="${LOCK_DIR}/steamtv-media-session.lock"
log() { echo "media-session: $*"; }
# Launch `command` on `workspace` unless something matching `pattern` is already this
# user's. The workspace switch happens before the launch for the same reason
# steamtv_agent.sway_control.launch_app does it that way: sway places a new window on
# whatever workspace is focused when it maps, and there is no reliable per-app `assign`
# rule for two Firefox profiles or for a flatpak's app_id.
launch_unless_running() {
pattern="$1"
workspace="$2"
shift 2
if pgrep -u "$(id -u)" -f "$pattern" >/dev/null 2>&1; then
log "$1 already running"
return 0
fi
log "launching $1 on $workspace"
swaymsg workspace "$workspace" >/dev/null 2>&1 || true
setsid "$@" >/dev/null 2>&1 &
# Give sway a moment to map the window on the workspace we just switched to, before
# the next launch switches away again. Crude, and correct: the alternative is
# subscribing to window events for each app, which is a lot of machinery for a path
# that runs once per session.
sleep 1
}
start() {
focus_after="${1:-}"
launch_unless_running "com.spotify.Client" "$WS_MUSIC" /usr/local/bin/spotify-launch
launch_unless_running "media-player-idle" "$WS_MEDIA" /usr/local/bin/media-player
# web-browser has its own focus-an-existing-window logic (it holds state a relaunch
# would throw away), so the pattern here matches its profile directory rather than
# the bare binary — a second Firefox on a different profile is a different window.
launch_unless_running "firefox.*--profile.*/firefox/web" "$WS_WEB" /usr/local/bin/web-browser
if [ "$focus_after" = "--focus" ]; then
swaymsg workspace "$WS_WEB" >/dev/null 2>&1 || true
fi
log "media session up"
}
stop() {
# pkill by the same patterns the launches are guarded by, so start/stop cannot
# disagree about what counts as "running". SIGTERM only — Spotify and Firefox both
# persist state on exit, and a SIGKILL here is how you get a "Firefox didn't shut
# down properly" dialog on the TV next time.
for pattern in "com.spotify.Client" "media-player-idle" "firefox.*--profile.*/firefox/web"; do
pkill -u "$(id -u)" -f "$pattern" >/dev/null 2>&1 || true
done
log "media session stopped"
}
case "${1:-start}" in
start)
shift 2>/dev/null || true
# flock serialises the two triggers. -n: if another start is already in flight,
# this one has nothing to add — the in-flight one will launch the same apps.
if command -v flock >/dev/null 2>&1; then
exec 9>"$LOCK_FILE"
if ! flock -n 9; then
log "another start is already running; nothing to do"
exit 0
fi
fi
start "${1:-}"
;;
stop)
stop
;;
status)
for pattern in "com.spotify.Client" "media-player-idle" "firefox.*--profile.*/firefox/web"; do
if pgrep -u "$(id -u)" -f "$pattern" >/dev/null 2>&1; then
echo "running: $pattern"
else
echo "stopped: $pattern"
fi
done
;;
*)
echo "usage: media-session [start [--focus] | stop | status]" >&2
exit 2
;;
esac

99
hosts/steam-tv-box/configs/session/prism-launch
View File

@ -1,99 +0,0 @@
#!/bin/sh
# Prism Launcher (Minecraft). Installed to /usr/local/bin/prism-launch.
#
# Prism rather than the official launcher: it runs offline once an account has been
# authenticated, it manages several instances/modpacks side by side, and it is packaged
# on Flathub, so it comes from the same channel as Spotify here and Steam Link on the
# thin client. 0400-flatpak-apps.hook.chroot installs it.
#
# THIS RUNS THROUGH STEAM, NOT BESIDE IT
# --------------------------------------
# Called with no arguments, this does not start Prism directly — it asks Steam to start
# it, via the non-Steam-game shortcut that /usr/local/bin/steam-shortcut-prism
# registers. That indirection is the entire point: a Prism launched by Steam runs inside
# the Steam Runtime with Steam Input active, so the Steam Controller API is present for
# it. Gamepads arrive as configurable Steam controllers with per-game bindings, the
# overlay works, gyro and back-buttons on Deck-style pads work, and Minecraft appears in
# Big Picture's library like any other title instead of being a hole you fall out of the
# UI into. Launched directly, Minecraft sees a raw evdev pad and none of that exists.
#
# `--steam` is what Steam passes back in when it runs the shortcut (it is the shortcut's
# LaunchOptions), and it is how this script knows not to bounce the request back to
# Steam a second time. Without that flag the two would call each other forever.
#
# FALLBACKS, IN ORDER: if Steam is not installed, or nobody has logged into it yet (so
# there is no shortcut and no game id), this runs Prism directly. That is a working
# Minecraft with worse controller support, which is the right outcome — better than
# refusing to start on a box where somebody has not signed into Steam.
#
# Java: the image ships openjdk-21-jre and openjdk-8-jre, and the hook grants the
# sandbox read access to /usr/lib/jvm, so Prism's auto-detection finds both without
# needing to download a runtime on first launch. 21 covers current Minecraft, 8 covers
# anything before 1.17 — which is most of what a modpack older than a couple of years
# actually wants.
#
# Login is interactive and on-device the first time (a Microsoft account is required —
# Mojang's device-code flow, so it is a short code typed on a phone rather than a
# password typed with a gamepad). The Flatpak's own persistent data directory
# (~/.var/app/org.prismlauncher.PrismLauncher) keeps that across restarts, and nothing
# here reseeds it — unlike the kiosk Firefox profiles, staying logged in is the point.
#
# Instances live under ~/Games if that is where you put them; see the README on
# mounting a real disk there, since this is the other thing on this box (with the Steam
# library) that will not fit in a live session's RAM overlay.
set -eu
GAME_ID_FILE="${HOME:-/home/$(id -un)}/.local/state/steamtv/prism-gameid"
DIRECT=""
# Strip the marker so it is never passed on to Prism itself, and keep any real
# arguments (an instance name, a .mrpack to import) for the direct launch below.
if [ "${1:-}" = "--steam" ]; then
DIRECT="yes"
shift
fi
run_direct() {
if ! flatpak info org.prismlauncher.PrismLauncher >/dev/null 2>&1; then
echo "prism-launch: Prism Launcher is not installed. Install it with:" >&2
echo " flatpak install -y flathub org.prismlauncher.PrismLauncher" >&2
exit 1
fi
exec flatpak run org.prismlauncher.PrismLauncher "$@"
}
# Steam is already the parent — run the real thing.
if [ -n "$DIRECT" ]; then
run_direct "$@"
fi
# Arguments mean somebody is opening a specific instance or file from a shell; hand
# that straight to Prism rather than through a Steam URL that cannot carry it.
if [ -n "${1:-}" ]; then
run_direct "$@"
fi
if ! command -v steam >/dev/null 2>&1; then
echo "prism-launch: Steam is not installed; launching Prism directly (no Steam Input)."
run_direct
fi
if [ ! -r "$GAME_ID_FILE" ]; then
# steam-shortcut-prism writes this on every session, so its absence means that script
# has never run — not that Steam is unusable.
echo "prism-launch: no Steam shortcut registered yet; launching Prism directly."
echo " Run /usr/local/bin/steam-shortcut-prism (with Steam closed) to fix that."
run_direct
fi
GAME_ID="$(head -n 1 "$GAME_ID_FILE" | tr -dc '0-9')"
if [ -z "$GAME_ID" ]; then
echo "prism-launch: $GAME_ID_FILE is malformed; launching Prism directly." >&2
run_direct
fi
# `steam -applaunch` does not work for non-Steam shortcuts — the rungameid URL is the
# only handle Steam exposes for them. A running client picks this up immediately; a
# stopped one starts first and then launches it.
exec steam "steam://rungameid/${GAME_ID}"

60
hosts/steam-tv-box/configs/session/session-watcher
View File

@ -1,60 +0,0 @@
#!/bin/sh
# Watches sway for the user leaving Big Picture. Installed to /usr/local/bin/session-watcher.
#
# THIS IS THE "MEDIA APPS ONLY START WHEN YOU LEAVE BIG PICTURE" MECHANISM, HALF TWO.
# Read steam-session's header first — it explains why there are two halves.
#
# The case this one covers: Steam is *still running*, but the user is no longer looking
# at it. "Exit Big Picture mode" (which drops to Steam's desktop client), the remote's
# channel keys, $mod+3, or Home Assistant's Screen select — all of those change the
# focused sway workspace and none of them make the Steam process exit, so steam-session
# never resumes and something else has to notice.
#
# WHY WORKSPACE EVENTS AND NOT WINDOW TITLES
# ------------------------------------------
# The obvious implementation is to poll for a window titled "Steam Big Picture Mode"
# and react when it goes away. That was rejected: the title, the window class and the
# very existence of a separate Big Picture window have all changed across Steam client
# rewrites, and a media session that silently stops appearing after a Steam update is a
# bad failure — it looks like the image is broken, not like a string moved. A sway
# workspace name is a contract this repo owns and can only change by editing the sway
# config and steamtv_agent/sway_control.py together.
#
# Cost: one blocked read on a socket, and a jq per workspace switch. It publishes
# nothing and listens on nothing — the session-mode sensor in Home Assistant is
# steamtv-agent asking sway, not this script pushing anywhere. (Adding an inbound
# listener here would breach the MQTT-is-the-only-control-surface rule; see
# steamtv_agent/mqtt_discovery.py.)
set -eu
WS_STEAM="1:steam"
if ! command -v jq >/dev/null 2>&1; then
echo "session-watcher: jq is missing; the media session will only start when Steam" >&2
echo " itself exits (steam-session's path). Install jq and reload sway." >&2
exit 1
fi
# `swaymsg -m` streams one JSON object per event line and blocks forever. If sway goes
# away the pipe closes and this exits — which is correct, since sway restarting means
# `exec_always` starts a fresh watcher.
swaymsg -t subscribe -m '["workspace"]' 2>/dev/null | while read -r event; do
# "focus" is the only change that means the user went somewhere; "init", "empty" and
# "rename" fire during ordinary window churn and must not trigger a launch.
change=$(printf '%s' "$event" | jq -r '.change // empty' 2>/dev/null) || continue
[ "$change" = "focus" ] || continue
current=$(printf '%s' "$event" | jq -r '.current.name // empty' 2>/dev/null) || continue
[ -n "$current" ] || continue
# An `if` rather than `[ … ] && continue`: the && form returns non-zero when the test
# is false, which under `set -e` would kill the watcher the first time somebody moved
# to a non-Steam workspace — i.e. exactly once, silently, on the first use.
if [ "$current" = "$WS_STEAM" ]; then
continue
fi
# No --focus: the user has already chosen where they want to be, and yanking them to
# the browser workspace because they pressed the channel key would be worse than
# doing nothing. This only makes sure the apps exist.
/usr/local/bin/media-session start >/dev/null 2>&1 || true
done

26
hosts/steam-tv-box/configs/session/spotify-launch
View File

@ -1,26 +0,0 @@
#!/bin/sh
# Full Spotify GUI client. Installed to /usr/local/bin/spotify-launch.
#
# The real, official Spotify Linux client — its own library browser, search and
# playlists — not spotifyd/librespot. Those are the thin client's and audio-endpoint's
# headless Spotify Connect *receivers*, which have no UI at all; this machine is sat in
# front of with a remote in hand, so it wants the browsable app, same as the touch
# panel. Spotify ships no apt package, so it comes from Flathub
# (0400-flatpak-apps.hook.chroot installs it).
#
# Login is interactive, on-device, on first launch (a Spotify account is needed;
# Premium for playback). The Flatpak's persistent data directory
# (~/.var/app/com.spotify.Client) keeps that session across restarts.
#
# Started lazily — see media-session. Spotify is one of the two apps that measurably
# costs a running game something if it sits in the background, which is most of why the
# media half of this image does not start at boot.
set -eu
if ! flatpak info com.spotify.Client >/dev/null 2>&1; then
echo "spotify-launch: Spotify is not installed. Install it with:" >&2
echo " flatpak install -y flathub com.spotify.Client" >&2
exit 1
fi
exec flatpak run com.spotify.Client "$@"

57
hosts/steam-tv-box/configs/session/steam-big-picture
View File

@ -1,57 +0,0 @@
#!/bin/sh
# Launches Steam in Big Picture (gamepad UI). Installed to /usr/local/bin/steam-big-picture.
#
# This is the image's default application: the sway config execs steam-session, which
# execs this, and this is what the TV shows within a few seconds of power-on. Nothing
# else is running at that point — see media-session for why.
#
# GAMESCOPE
# ---------
# When gamescope is present (0300-steam.hook.chroot installs it if the release has it)
# Steam runs nested inside it. That is what Valve ships on the Deck and it buys three
# things that matter on a television: a fixed output resolution and refresh rate that a
# game cannot change out from under the compositor, integer/FSR scaling so a 1080p game
# on a 4K set is sharp rather than smeared, and a framerate limiter. Without gamescope
# Steam runs directly on Xwayland — fully functional, just without those.
#
# IDEMPOTENT: relaunching while Steam is already up focuses the existing client instead
# of starting a second one (Steam would refuse anyway, but noisily, and the second
# process would sit in the session doing nothing). This is what makes the HA "Launch
# Steam" button and the remote's own key safe to press repeatedly.
set -eu
WS_STEAM="1:steam"
if ! command -v steam >/dev/null 2>&1; then
echo "steam-big-picture: Steam is not installed on this image — see" >&2
echo " live-build/config/hooks/normal/0300-steam.hook.chroot, which logs why." >&2
exit 1
fi
# Already running: switch to it and focus rather than starting another client.
if pgrep -u "$(id -u)" -x steam >/dev/null 2>&1; then
swaymsg workspace "$WS_STEAM" >/dev/null 2>&1 || true
swaymsg '[class="^[Ss]team$"] focus' >/dev/null 2>&1 || true
# Steam's own URL handler is the documented way to ask a *running* client to go back
# into Big Picture; there is no command-line flag that does it to an existing process.
exec steam steam://open/bigpicture
fi
# -gamepadui is the current Big Picture. -tenfoot is the old one and is gone; if this
# ever stops opening the gamepad UI, that flag name is the first thing to check against
# the installed client (`steam -help`).
STEAM_ARGS="-gamepadui -nochatui -nofriendsui"
if command -v gamescope >/dev/null 2>&1 && [ "${STEAMTV_USE_GAMESCOPE:-auto}" != "false" ]; then
# -f fullscreen, -e Steam integration (lets Steam drive resolution per game),
# --adaptive-sync hands VRR through to a set that supports it.
# Output geometry is deliberately NOT pinned here: gamescope defaults to the
# connected display's native mode, and hardcoding 1920x1080 would be wrong on the 4K
# set this is most likely plugged into. Pin it in STEAMTV_GAMESCOPE_ARGS if a
# specific title needs it.
# shellcheck disable=SC2086
exec gamescope -f -e --adaptive-sync ${STEAMTV_GAMESCOPE_ARGS:-} -- steam $STEAM_ARGS
fi
# shellcheck disable=SC2086
exec steam $STEAM_ARGS

47
hosts/steam-tv-box/configs/session/steam-session
View File

@ -1,47 +0,0 @@
#!/bin/sh
# The session's first and normally only process. Installed to /usr/local/bin/steam-session.
#
# THIS IS THE "MEDIA APPS ONLY START WHEN YOU LEAVE BIG PICTURE" MECHANISM, HALF ONE.
#
# The box boots to Steam and nothing else: no browser, no Spotify, no player. Those are
# started the first time somebody actually leaves Big Picture. Two independent triggers
# do that, because there are two different ways to leave and neither one can see the
# other:
#
# 1. THIS SCRIPT — "Exit Steam" / the client crashing. steam-big-picture runs in the
# foreground, so when the Steam client goes away this script resumes on the line
# after it and brings the media session up, focused.
# 2. /usr/local/bin/session-watcher — "Exit Big Picture mode", alt-tabbing, the
# remote's channel keys, or the HA workspace select. Steam is still running, so
# trigger 1 never fires; what changes is the focused sway workspace, which the
# watcher subscribes to.
#
# Both call `media-session start`, which is idempotent, so whichever fires first wins
# and the second is a no-op. That is the whole design — no window-title matching, no
# polling for Big Picture's internal state, nothing that breaks when Valve reshuffles
# the UI. See media-session for the guards that make double-firing free.
#
# WHY THIS IS NOT A LOOP: if Steam exits, it stays exited. Relaunching it automatically
# would make "Exit Steam" impossible to act on from the sofa, and the box would never
# be usable as anything but a games console. Getting back in is a button (HA, the
# remote's Home key, or $mod+s) — see the sway config.
set -eu
# Register Prism Launcher as a non-Steam game BEFORE Steam starts. Ordering is not
# incidental: Steam reads shortcuts.vdf at startup and rewrites it from memory when it
# exits, so anything written while it is running is silently discarded. This is also
# why the registration lives here rather than in an `exec_always` in the sway config,
# where it would race the client. Idempotent, and a no-op until somebody has logged
# into Steam on this machine — see the script's own header for why it cannot be baked
# into the image.
/usr/local/bin/steam-shortcut-prism || \
echo "steam-session: could not register the Prism shortcut; Prism will still launch," \
"just without Steam Input"
/usr/local/bin/steam-big-picture || \
echo "steam-session: Steam exited non-zero; falling through to the media session anyway"
# --focus because this path means the screen is now showing nothing at all: Steam has
# quit and the user is looking at an empty compositor. The watcher's path deliberately
# does not focus, since there the user already chose where to be.
exec /usr/local/bin/media-session start --focus

268
hosts/steam-tv-box/configs/session/steam-shortcut-prism
View File

@ -1,268 +0,0 @@
#!/usr/bin/env python3
"""Registers Prism Launcher as a Steam "non-Steam game". Installed to
/usr/local/bin/steam-shortcut-prism, run by steam-session before Steam starts.
WHY
---
Launching Prism directly gets you a plain Flatpak window and a Minecraft that sees a
raw evdev gamepad, which is to say: no controller support worth the name. Launching it
*through* Steam puts it inside the Steam Runtime with Steam Input active, which is what
makes the Steam Controller API present for it. Concretely that means the overlay works,
the pad shows up as a configurable controller with Steam's own per-game bindings, the
box's Steam Controller / Deck-style pads get their gyro and back buttons, and Big
Picture treats Minecraft like any other title in the library instead of a hole you fall
out of the UI into.
There is no command-line flag for this. Steam's only mechanism for a non-Steam game is
an entry in the user's binary `shortcuts.vdf`, which is what this writes.
WHY IT CANNOT BE BAKED INTO THE IMAGE
-------------------------------------
shortcuts.vdf lives under ~/.steam/steam/userdata/<steamID3>/config/, and that
directory does not exist until somebody has logged into Steam on this machine. The
account is not known at build time and must not be — no Steam credentials go into an
ISO. So this runs per-session instead, is idempotent, and no-ops quietly when Steam has
never been logged into (the common state on a freshly flashed box, right up until
somebody signs in on the TV).
WHY IT RUNS BEFORE STEAM STARTS
-------------------------------
Steam reads shortcuts.vdf at startup and rewrites it from memory at shutdown. Writing
it underneath a running client means the change is silently reverted the next time
Steam exits. steam-session calls this first for that reason; running it by hand while
Steam is up is refused below rather than being allowed to quietly do nothing.
FORMAT NOTE / VERIFY BEFORE TRUSTING IT
---------------------------------------
The binary VDF encoding below (0x00 map, 0x01 string, 0x02 int32, 0x08 end) and the
non-Steam AppID derivation (CRC32 of Exe+AppName, high bit set) are the long-standing,
widely-reimplemented community format — Valve documents neither. They are believed
correct but were NOT verified against a real Steam client from this environment. The
check on first boot is simply: does "Prism Launcher" appear in Big Picture's library,
and does the pad work inside Minecraft. If the file turns out to be malformed Steam
discards it silently, which is why this keeps a .bak (see below) rather than writing in
place.
"""
from __future__ import annotations
import binascii
import glob
import os
import shutil
import subprocess
import sys
import time
APP_NAME = "Prism Launcher"
# The wrapper, not `flatpak run …` directly: prism-launch is where the "is it even
# installed" check and the Java/instance notes live, and pointing Steam at a stable
# path means this entry does not change when the Flatpak app ID does.
EXE = "/usr/local/bin/prism-launch"
START_DIR = "/usr/local/bin"
# --steam tells prism-launch it is already inside Steam's runtime, so it does not
# recurse back through `steam steam://rungameid/...` and launch itself forever.
LAUNCH_OPTIONS = "--steam"
USERDATA_GLOB = os.path.expanduser("~/.steam/steam/userdata/*/config")
# Flatpak'd and Snap'd Steam put userdata elsewhere; this image installs Steam from
# apt, so the path above is the real one. The alternates are checked anyway because
# somebody debugging on a laptop will have one of them.
ALT_GLOBS = (
os.path.expanduser("~/.local/share/Steam/userdata/*/config"),
os.path.expanduser("~/.var/app/com.valvesoftware.Steam/data/Steam/userdata/*/config"),
)
def log(message: str) -> None:
print(f"steam-shortcut-prism: {message}")
# --- binary VDF -----------------------------------------------------------------
def _string(key: str, value: str) -> bytes:
return b"\x01" + key.encode("utf-8") + b"\x00" + value.encode("utf-8") + b"\x00"
def _int32(key: str, value: int) -> bytes:
return b"\x02" + key.encode("utf-8") + b"\x00" + value.to_bytes(4, "little", signed=False)
def shortcut_app_id(exe: str, app_name: str) -> int:
"""The 32-bit ID Steam gives a non-Steam shortcut.
CRC32 of the Exe field concatenated with AppName, with the top bit set. Steam
quotes the Exe field in the file it writes, and the CRC is taken over the quoted
form — getting that wrong produces an ID that no `steam://rungameid/` URL matches,
which looks exactly like "the shortcut didn't work" with nothing in any log.
"""
key = f'"{exe}"{app_name}'.encode("utf-8")
return binascii.crc32(key) | 0x80000000
def run_game_id(app_id: int) -> int:
"""The 64-bit ID `steam://rungameid/` wants for a shortcut."""
return (app_id << 32) | 0x02000000
def encode_shortcuts(entries: list[dict]) -> bytes:
out = bytearray(b"\x00shortcuts\x00")
for index, entry in enumerate(entries):
out += b"\x00" + str(index).encode("ascii") + b"\x00"
out += _int32("appid", entry["appid"])
out += _string("AppName", entry["AppName"])
out += _string("Exe", entry["Exe"])
out += _string("StartDir", entry["StartDir"])
out += _string("icon", entry.get("icon", ""))
out += _string("ShortcutPath", entry.get("ShortcutPath", ""))
out += _string("LaunchOptions", entry.get("LaunchOptions", ""))
out += _int32("IsHidden", 0)
# AllowDesktopConfig + AllowOverlay are the two that matter for the whole point
# of this file: the overlay is what carries Steam Input's binding UI, and
# desktop-config is what lets a pad still work when Big Picture is not focused.
out += _int32("AllowDesktopConfig", 1)
out += _int32("AllowOverlay", 1)
out += _int32("OpenVR", 0)
out += _int32("Devkit", 0)
out += _string("DevkitGameID", "")
out += _int32("DevkitOverrideAppID", 0)
out += _int32("LastPlayTime", entry.get("LastPlayTime", 0))
out += b"\x00tags\x00\x08"
out += b"\x08"
out += b"\x08\x08"
return bytes(out)
def decode_app_names(data: bytes) -> list[str]:
"""Just enough parsing to answer "is our entry already in here?".
A full VDF reader is not needed and would be more to get wrong: this only has to
decide between rewriting the file and leaving it alone.
"""
names = []
marker = b"\x01AppName\x00"
position = data.find(marker)
while position != -1:
start = position + len(marker)
end = data.find(b"\x00", start)
if end == -1:
break
names.append(data[start:end].decode("utf-8", "replace"))
position = data.find(marker, end)
return names
# --- the work -------------------------------------------------------------------
def config_dirs() -> list[str]:
found = sorted(glob.glob(USERDATA_GLOB))
for pattern in ALT_GLOBS:
found += sorted(glob.glob(pattern))
# userdata/0/ is Steam's placeholder for "no account", not a real profile.
return [d for d in found if os.path.basename(os.path.dirname(d)) != "0"]
def steam_is_running() -> bool:
try:
return subprocess.run(
["pgrep", "-u", str(os.getuid()), "-x", "steam"],
capture_output=True,
check=False,
timeout=5,
).returncode == 0
except (OSError, subprocess.SubprocessError):
return False
def write_shortcut(config_dir: str, app_id: int) -> bool:
path = os.path.join(config_dir, "shortcuts.vdf")
existing = b""
if os.path.exists(path):
try:
with open(path, "rb") as handle:
existing = handle.read()
except OSError as exc:
log(f"could not read {path}: {exc}")
return False
if APP_NAME in decode_app_names(existing):
log(f"{APP_NAME} is already in {path}")
return True
if existing:
# Anything already in shortcuts.vdf was put there by hand, and this parser is
# not good enough to rewrite the file without losing it. So: back it up, tell
# the human exactly what to do, and refuse rather than destroy their entries.
backup = f"{path}.bak-{int(time.time())}"
try:
shutil.copyfile(path, backup)
except OSError as exc:
log(f"could not back up {path}: {exc}")
return False
log(f"{path} already has other shortcuts in it; backed it up to {backup}.")
log("Refusing to rewrite it — this script only knows how to write a file it")
log("owns entirely, and rewriting would drop the entries already there.")
log(f"Add {APP_NAME} by hand in Steam (Games -> Add a Non-Steam Game -> {EXE}),")
log(f"then set its launch options to: {LAUNCH_OPTIONS}")
return False
entry = {
"appid": app_id,
"AppName": APP_NAME,
"Exe": f'"{EXE}"',
"StartDir": f'"{START_DIR}"',
"LaunchOptions": LAUNCH_OPTIONS,
}
try:
os.makedirs(config_dir, exist_ok=True)
temporary = f"{path}.tmp"
with open(temporary, "wb") as handle:
handle.write(encode_shortcuts([entry]))
os.replace(temporary, path)
except OSError as exc:
log(f"could not write {path}: {exc}")
return False
log(f"registered {APP_NAME} in {path} (appid {app_id})")
return True
def main() -> int:
app_id = shortcut_app_id(EXE, APP_NAME)
game_id = run_game_id(app_id)
# Written unconditionally, even when there is no Steam profile yet: prism-launch
# reads this to build its steam://rungameid/ URL, and it is derived from two
# constants in this file, so it is correct whether or not the shortcut exists yet.
state_dir = os.path.expanduser("~/.local/state/steamtv")
try:
os.makedirs(state_dir, exist_ok=True)
with open(os.path.join(state_dir, "prism-gameid"), "w", encoding="utf-8") as handle:
handle.write(f"{game_id}\n")
except OSError as exc:
log(f"could not record the game id: {exc}")
if steam_is_running():
log("Steam is running — it would overwrite shortcuts.vdf on exit and discard")
log("anything written now. Quit Steam and re-run, or just let the next session")
log("do it (steam-session runs this before Steam starts).")
return 1
dirs = config_dirs()
if not dirs:
log("no Steam userdata directory yet — nobody has logged into Steam on this")
log("machine. Nothing to do; this will register itself on the session after")
log("the first Steam login.")
return 0
ok = True
for config_dir in dirs:
# Every logged-in account on the box gets the entry: which one is signed in at
# any moment is not knowable here, and a stale entry for an account that never
# plays Minecraft costs nothing.
ok = write_shortcut(config_dir, app_id) and ok
return 0 if ok else 1
if __name__ == "__main__":
sys.exit(main())

203
hosts/steam-tv-box/configs/sway/config
View File

@ -1,203 +0,0 @@
# Sway session for the Steam TV box.
# Installed to /home/<KIOSK_USERNAME>/.config/sway/config by build-steam-tv-box-iso.sh.
#
# Every `exec` below inherits the environment set by /usr/local/bin/kiosk-session,
# which sources /etc/steamtv-agent/config.env — that is how GPU_VENDOR and the MQTT
# settings get here without this file being templated.
#
# THE SHAPE OF THIS SESSION, IN ONE PARAGRAPH: the box powers on, greetd autologins,
# sway starts, and the only thing it launches is Steam in Big Picture. The browser,
# Spotify and the player do not exist yet. The first time somebody leaves Big Picture —
# by quitting Steam, by exiting Big Picture mode, by pressing a workspace key, or via
# Home Assistant — the media session is brought up behind them. See steam-session and
# session-watcher, which are the two halves of that.
set $mod Mod4
set $ws_steam 1:steam
set $ws_games 2:games
set $ws_web 3:web
set $ws_media 4:media
set $ws_music 5:music
# Workspace names are a contract with steamtv_agent/sway_control.py,
# steamtv_agent/session_mode.py, /usr/local/bin/media-session and
# /usr/local/bin/session-watcher — changing one side means changing all of them.
# ---------------------------------------------------------------------------
# Output / input
# ---------------------------------------------------------------------------
output * bg #0b0b10 solid_color
input type:keyboard {
xkb_layout @KEYBOARD_LAYOUT@
}
# A gamepad is not an input device to the compositor — it goes straight to Steam and to
# games via evdev, so there is nothing to configure here for it. What does need
# configuring is a mouse/trackball on the coffee table, if there is one.
input type:pointer {
accel_profile adaptive
}
# ---------------------------------------------------------------------------
# Look — no bars, no borders, no gaps. This is a television.
# ---------------------------------------------------------------------------
default_border none
default_floating_border none
hide_edge_borders both
gaps inner 0
gaps outer 0
# Games and Big Picture must never be tiled next to something else; every workspace
# here holds exactly one thing that fills the screen.
for_window [class="^[Ss]team$"] fullscreen enable
for_window [class="^gamescope$"] fullscreen enable
for_window [app_id="^gamescope$"] fullscreen enable
for_window [app_id="mpv"] fullscreen enable
# ---------------------------------------------------------------------------
# Remote control (administration, not play)
# ---------------------------------------------------------------------------
# exec_always so a `swaymsg reload` re-establishes it. start-wayvnc refuses to run
# until /etc/wayvnc/wayvnc-password has been set on this machine.
#
# Worth being explicit: this is for fixing the box, not for playing over. wayvnc streams
# the compositor, so a game rendering at 120fps arrives as a slideshow — that is not a
# defect to work around, it is what a screen-scraping protocol does.
exec_always /usr/local/bin/start-wayvnc
# The workspace-focus half of the leave-Big-Picture detection. Its own exit-on-
# sway-death behaviour makes exec_always safe across a reload: the old watcher's pipe
# closes with the old sway, and reload starts one watcher, not a second.
exec_always /usr/local/bin/session-watcher
# steamtv-agent is NOT started here. systemd owns it (steamtv-agent.service, enabled by
# 0700-steamtv-agent.hook.chroot) so that it is up and connected to Mosquitto whether or
# not a graphical session ever came up, and so it survives a sway restart. Starting it
# from sway too would give two competing MQTT clients.
# ---------------------------------------------------------------------------
# The default application
# ---------------------------------------------------------------------------
# This is the whole boot behaviour: land on 1:steam and start Big Picture. steam-session
# blocks on the Steam client and starts the media session when it exits.
#
# No `assign [class="steam"]` rule: Steam maps several windows (splash, client, an
# optional gamescope surface) and an assign rule would scatter them across workspaces
# mid-launch. Switching workspace first and letting them map where focus already is
# gets the same result without the race.
exec swaymsg workspace $ws_steam
exec /usr/local/bin/steam-session
# ---------------------------------------------------------------------------
# Idle
# ---------------------------------------------------------------------------
# Never lock: this is a shared living-room machine, and a lock screen would make the TV
# unusable to anyone not holding a keyboard.
#
# 20 minutes, and only the display — noticeably longer than the thin client's 15,
# because "nobody has touched an input device" is a much weaker signal here. Watching a
# two-hour film, or a cutscene, or a turn in a slow strategy game are all legitimately
# input-free, and the inhibit rules below are what keep the screen alive through them.
exec swayidle -w \
timeout 1200 'swaymsg "output * power off"' \
resume 'swaymsg "output * power on"'
# Anything playing keeps the screen on. Steam gets `focus` rather than `fullscreen`
# because Big Picture is fullscreen essentially always, and a game that alt-tabs itself
# briefly should not start the idle countdown.
for_window [class="^[Ss]team$"] inhibit_idle focus
for_window [class="^gamescope$"] inhibit_idle focus
for_window [app_id="^gamescope$"] inhibit_idle focus
for_window [app_id="mpv"] inhibit_idle visible
for_window [app_id="firefox-esr"] inhibit_idle fullscreen
for_window [app_id="com.spotify.Client"] inhibit_idle focus
# ---------------------------------------------------------------------------
# Local keys. A gamepad drives Steam and games; these are for the keyboard on the back
# of the remote, and for standing in front of the machine.
# ---------------------------------------------------------------------------
# Maintenance shell. A deliberately obscure chord (not $mod+Return, which is the
# ordinary local-terminal key below) so it is not something a visitor bumps into,
# floating so it overlays whatever is running instead of tiling against it.
bindsym $mod+Shift+Ctrl+m exec foot --title maintenance-shell
for_window [title="maintenance-shell"] floating enable, resize set width 900 height 550, move position center
bindsym $mod+Return exec foot
bindsym $mod+q kill
bindsym $mod+f fullscreen toggle
bindsym $mod+Shift+c reload
# Getting back into Steam after "Exit Steam" — the counterpart to steam-session's
# deliberate refusal to relaunch it in a loop. Idempotent, so holding the key does
# nothing worse than focusing the client.
bindsym $mod+s exec /usr/local/bin/steam-big-picture
# And bringing the media half up by hand, for the case where somebody wants Spotify
# without leaving the game.
bindsym $mod+m exec /usr/local/bin/media-session start
# Prism/Minecraft. This goes through Steam (steam://rungameid/…) so Steam Input is
# active for it — see prism-launch. Which is also why its window lands on 1:steam and
# not 2:games when Steam is running: it *is* a Steam game as far as the client is
# concerned, and Big Picture shows it in the library.
bindsym $mod+p exec /usr/local/bin/prism-launch
bindsym $mod+1 workspace $ws_steam
bindsym $mod+2 workspace $ws_games
bindsym $mod+3 workspace $ws_web
bindsym $mod+4 workspace $ws_media
bindsym $mod+5 workspace $ws_music
bindsym $mod+Left focus left
bindsym $mod+Right focus right
bindsym $mod+Up focus up
bindsym $mod+Down focus down
# ---------------------------------------------------------------------------
# Remote control — the standardised media-key set.
#
# Same device class as the thin client's: a wireless USB remote that presents as two
# HID keyboards (TV controls on the front, a small keyboard on the back). Nothing to
# configure per-device; the front buttons arrive as the XF86* keysyms below.
#
# DELIBERATELY NOT BOUND: plain arrows and Return. Steam's Big Picture, mpv and every
# web page need them, and a remote's D-pad and OK button send exactly those. Stealing
# them at the compositor would make Big Picture unnavigable with the remote — which is
# most of what the remote is for here. Window focus stays on $mod+arrows above.
#
# `wev` from the maintenance shell prints the keysym for any button that is not bound.
# ---------------------------------------------------------------------------
# Transport. `playerctl -p mpv,spotify` matches the two players this image runs; the
# order is the priority when both are alive. Steam's own media has no MPRIS bus and is
# not a target — a game's audio is not something you "pause".
bindsym XF86AudioPlay exec playerctl -p mpv,spotify play-pause
bindsym XF86AudioPause exec playerctl -p mpv,spotify pause
bindsym XF86AudioStop exec playerctl -p mpv,spotify stop
bindsym XF86AudioNext exec playerctl -p mpv,spotify next
bindsym XF86AudioPrev exec playerctl -p mpv,spotify previous
bindsym XF86AudioForward exec playerctl -p mpv,spotify position 30+
bindsym XF86AudioRewind exec playerctl -p mpv,spotify position 10-
# Volume. Sinks, not players — the volume rocker should move the room's volume whether
# the noise is coming from Spotify or from a game, and a game is the common case here.
bindsym XF86AudioRaiseVolume exec wpctl set-volume @DEFAULT_AUDIO_SINK@ 5%+
bindsym XF86AudioLowerVolume exec wpctl set-volume @DEFAULT_AUDIO_SINK@ 5%-
bindsym XF86AudioMute exec wpctl set-mute @DEFAULT_AUDIO_SINK@ toggle
# Navigation.
bindsym XF86ChannelUp workspace next
bindsym XF86ChannelDown workspace prev
# Home goes back to the games, because on this box that is home.
bindsym XF86HomePage exec /usr/local/bin/steam-big-picture
bindsym XF86Back workspace back_and_forth
bindsym XF86Forward workspace back_and_forth
# Power. **The display, not the machine** — same reasoning as the thin client, and more
# so here: `poweroff` from the sofa would drop whatever game is running, unsaved.
bindsym XF86PowerOff exec display-toggle
bindsym XF86Sleep exec display-toggle
bindsym XF86ScreenSaver exec display-toggle
# Deliberately no exit binding: `swaymsg exit` would drop to a black VT and greetd would
# autologin straight back in. Use SSH or the maintenance shell to administer.
workspace $ws_steam

29
hosts/steam-tv-box/configs/sway/display-toggle
View File

@ -1,29 +0,0 @@
#!/bin/sh
# Turn every output off, or back on. Installed to /usr/local/bin/display-toggle and
# bound to the remote's power/sleep buttons in the Sway config.
#
# WHY THE POWER BUTTON DOES NOT POWER ANYTHING OFF: on a TV, that button turns the
# picture off. Here, `poweroff` would drop whatever game is running — unsaved — and
# take the room's screen away until somebody walks over to the machine. So the remote's
# power button does the thing the person pressing it actually meant.
#
# THE WAKE SIDE IS THE HARD HALF. With outputs powered off, Sway is still running and
# still receiving keys, so pressing power again lands here and turns them back on. That
# is why this is a toggle rather than two bindings: there is no other way back. It also
# means a stuck remote button cannot leave the screen dark — the next press fixes it.
#
# This is the local, no-network path. Home Assistant's "Display" switch goes through
# steamtv_agent/display_power.py instead, which also drives HDMI-CEC so the television
# itself goes to standby rather than just showing "no signal".
set -eu
# `swaymsg -t get_outputs` reports each output's power state in a field still named
# "dpms" (the *command* was renamed to `power`; the JSON key was not). If ANY output is
# still on, the intent of a press is "turn it off"; only when everything is already
# dark does a press mean "wake up". That ordering matters on a multi-output machine,
# where asking per-output would leave the remote toggling one screen at a time.
if swaymsg -t get_outputs | grep -q '"dpms": true'; then
swaymsg 'output * power off'
else
swaymsg 'output * power on'
fi

26
hosts/steam-tv-box/configs/wayvnc/config
View File

@ -1,26 +0,0 @@
# wayvnc — installed to /etc/wayvnc/config by build-steam-tv-box-iso.sh.
# Format is plain key=value, one per line (not TOML, not INI sections).
#
# There is deliberately NO `password=` line in this file. wayvnc only accepts the
# password inline, so committing one here would put a live credential for a full
# remote-control channel into git. Instead /usr/local/bin/start-wayvnc reads
# /etc/wayvnc/wayvnc-password (mode 0600, never committed) and writes a merged config
# into $XDG_RUNTIME_DIR at session start. If that file still holds the build-time
# sentinel, start-wayvnc refuses to launch — no unauthenticated VNC server, ever.
# Bound to all interfaces on purpose: wayvnc is this project's remote-control channel
# (the confirmed replacement for RDP), so it has to be reachable from the LAN, not
# just loopback. That is exactly why the auth below is not optional.
address=0.0.0.0
port=5900
enable_auth=true
username=@KIOSK_USERNAME@
# wayvnc >= 0.7 uses this for RSA-AES auth.
rsa_private_key_file=/etc/wayvnc/rsa_key.pem
# wayvnc <= 0.6 authenticates over TLS instead and needs these two; harmless on newer
# builds. Both are generated by 0300-wayvnc.hook.chroot, self-signed.
private_key_file=/etc/wayvnc/tls_key.pem
certificate_file=/etc/wayvnc/tls_cert.pem

36
hosts/steam-tv-box/configs/wayvnc/start-wayvnc
View File

@ -1,36 +0,0 @@
#!/bin/sh
# Launches wayvnc with a password that is never stored in the repo or in /etc/wayvnc/config.
# Installed to /usr/local/bin/start-wayvnc, started from the sway config.
set -eu
BASE_CONFIG=/etc/wayvnc/config
PASSWORD_FILE=/etc/wayvnc/wayvnc-password
SENTINEL='CHANGEME-SET-ON-FIRST-BOOT'
if [ ! -r "$PASSWORD_FILE" ]; then
echo "start-wayvnc: $PASSWORD_FILE is missing or unreadable — refusing to start." >&2
exit 1
fi
PASSWORD="$(head -n 1 "$PASSWORD_FILE" | tr -d '\r\n')"
# Fail closed. An operator who forgets this step gets no remote access, rather than a
# remote-control channel anyone on the LAN can open.
if [ -z "$PASSWORD" ] || [ "$PASSWORD" = "$SENTINEL" ]; then
echo "start-wayvnc: no wayvnc password set. Run, as root, on this machine:" >&2
echo " openssl rand -base64 24 > $PASSWORD_FILE && chmod 600 $PASSWORD_FILE" >&2
echo " chown $(id -un):$(id -gn) $PASSWORD_FILE" >&2
echo "Then restart the session. Refusing to start an unauthenticated VNC server." >&2
exit 1
fi
RUNTIME_DIR="${XDG_RUNTIME_DIR:-/run/user/$(id -u)}/wayvnc"
mkdir -p "$RUNTIME_DIR"
chmod 700 "$RUNTIME_DIR"
RUNTIME_CONFIG="$RUNTIME_DIR/config"
umask 077
cp "$BASE_CONFIG" "$RUNTIME_CONFIG"
printf 'password=%s\n' "$PASSWORD" >> "$RUNTIME_CONFIG"
exec wayvnc --config="$RUNTIME_CONFIG"

61
hosts/steam-tv-box/live-build/config/hooks/normal/0100-user-setup.hook.chroot
View File

@ -1,61 +0,0 @@
#!/bin/sh
# Creates the kiosk account the whole image is built around.
#
# live-build runs chroot_local-includes BEFORE chroot_local-hooks, so
# /etc/steamtv-agent/config.env (written by build-steam-tv-box-iso.sh) already exists
# here. Sourcing it is why hooks don't need placeholder/sed templating.
set -eu
. /etc/steamtv-agent/config.env
if ! id "$KIOSK_USERNAME" >/dev/null 2>&1; then
useradd --create-home --shell /bin/bash --comment "Steam TV box kiosk session" "$KIOSK_USERNAME"
fi
# Same list as the thin client, plus `games` — some titles and emulators expect it, and
# `gamemode` needs the user in a group its daemon accepts before it will honour a
# governor request.
for grp in audio video input render dialout netdev plugdev seat _seatd games gamemode bluetooth; do
if getent group "$grp" >/dev/null 2>&1; then
adduser "$KIOSK_USERNAME" "$grp" >/dev/null
fi
done
# No password is baked in: the account is locked so it can never be used to log in
# remotely, while the physical console still autologins via greetd.
passwd --lock "$KIOSK_USERNAME" >/dev/null
adduser "$KIOSK_USERNAME" sudo >/dev/null
# Passwordless sudo is a deliberate call, not laziness: this image autologins to an
# unattended interactive Sway session at the physical console, so anyone standing in
# front of the machine already has the equivalent of a root shell. Requiring a password
# here would buy nothing while making the locked account unadministrable. The
# boundaries that actually matter are the wayvnc password and key-only SSH below.
cat > "/etc/sudoers.d/010-${KIOSK_USERNAME}" <<EOF
${KIOSK_USERNAME} ALL=(ALL) NOPASSWD: ALL
EOF
chmod 0440 "/etc/sudoers.d/010-${KIOSK_USERNAME}"
mkdir -p /etc/ssh/sshd_config.d
cat > /etc/ssh/sshd_config.d/10-steam-tv-box.conf <<'EOF'
PermitRootLogin no
PasswordAuthentication no
KbdInteractiveAuthentication no
PubkeyAuthentication yes
EOF
if [ -d "/home/${KIOSK_USERNAME}/.ssh" ]; then
chmod 700 "/home/${KIOSK_USERNAME}/.ssh"
[ -f "/home/${KIOSK_USERNAME}/.ssh/authorized_keys" ] && \
chmod 600 "/home/${KIOSK_USERNAME}/.ssh/authorized_keys"
fi
# Games are large and this box has a real disk, so the library lives outside the live
# image's writable overlay by convention — see hosts/steam-tv-box/README.md for
# mounting a games disk here. Created either way so Steam's own first-run path exists.
mkdir -p "/home/${KIOSK_USERNAME}/Games"
chown -R "${KIOSK_USERNAME}:${KIOSK_USERNAME}" "/home/${KIOSK_USERNAME}"
systemctl enable ssh >/dev/null 2>&1 || true

25
hosts/steam-tv-box/live-build/config/hooks/normal/0200-greetd.hook.chroot
View File

@ -1,25 +0,0 @@
#!/bin/sh
# Makes greetd the boot target so the machine comes up straight in the kiosk Sway
# session, which in turn launches Steam Big Picture (config in /etc/greetd/config.toml,
# shipped via includes.chroot).
set -eu
. /etc/steamtv-agent/config.env
chmod 0755 /usr/local/bin/kiosk-session
# greetd's own package user; it still needs to exist even though no greeter UI runs.
if ! id greeter >/dev/null 2>&1; then
useradd --system --create-home --home-dir /var/lib/greetd --shell /usr/sbin/nologin greeter
fi
systemctl enable greetd
systemctl set-default graphical.target
# live-config would otherwise autologin its own account on tty1 and fight greetd for
# the VT. build-steam-tv-box-iso.sh passes `noautologin` on the kernel command line;
# masking the getty on vt1 makes that robust even if someone edits the boot args.
systemctl mask getty@tty1.service
mkdir -p "/home/${KIOSK_USERNAME}/.config/sway"
chown -R "${KIOSK_USERNAME}:${KIOSK_USERNAME}" "/home/${KIOSK_USERNAME}/.config"

35
hosts/steam-tv-box/live-build/config/hooks/normal/0250-wayvnc.hook.chroot
View File

@ -1,35 +0,0 @@
#!/bin/sh
# Prepares wayvnc's auth material. Deliberately does NOT set a password.
set -eu
. /etc/steamtv-agent/config.env
mkdir -p /etc/wayvnc
chmod 0755 /usr/local/bin/start-wayvnc
# wayvnc's RSA-AES auth needs a key pair; it is machine-local and carries no secret
# that belongs in git, so generating it at build time is fine.
if [ ! -f /etc/wayvnc/rsa_key.pem ]; then
openssl genrsa -out /etc/wayvnc/rsa_key.pem 2048 2>/dev/null
fi
# Self-signed TLS material, needed only by wayvnc <= 0.6 whose auth path is TLS-based
# rather than RSA-AES. Harmless on newer versions.
if [ ! -f /etc/wayvnc/tls_key.pem ]; then
openssl req -x509 -newkey rsa:2048 -nodes -days 3650 \
-keyout /etc/wayvnc/tls_key.pem -out /etc/wayvnc/tls_cert.pem \
-subj "/CN=steam-tv-box" 2>/dev/null
fi
# Sentinel, not a password. start-wayvnc refuses to launch while this value is still
# here, so the failure mode of "operator forgot to set a password" is "no VNC server"
# rather than "an unauthenticated VNC server on the LAN". A real value must never be
# committed — see hosts/steam-tv-box/README.md.
if [ ! -f /etc/wayvnc/wayvnc-password ]; then
printf 'CHANGEME-SET-ON-FIRST-BOOT\n' > /etc/wayvnc/wayvnc-password
fi
chmod 0600 /etc/wayvnc/wayvnc-password /etc/wayvnc/rsa_key.pem /etc/wayvnc/tls_key.pem
chown "${KIOSK_USERNAME}:${KIOSK_USERNAME}" \
/etc/wayvnc/wayvnc-password /etc/wayvnc/rsa_key.pem /etc/wayvnc/tls_key.pem
chmod 0644 /etc/wayvnc/tls_cert.pem /etc/wayvnc/config

130
hosts/steam-tv-box/live-build/config/hooks/normal/0300-steam.hook.chroot
View File

@ -1,130 +0,0 @@
#!/bin/sh
# Native Steam, the 32-bit graphics stack it needs, and the optional extras around it.
#
# WHY THIS IS A HOOK AND NOT A LINE IN steam-tv-box.list.chroot
# ------------------------------------------------------------
# live-build installs config/package-lists/* BEFORE it runs config/hooks/normal/*, and
# `steam-installer` is unusable until i386 is a known architecture: it is an amd64
# package whose entire job is to pull in an i386 dependency chain. Enabling multiarch
# has to happen first, and the only place in a live-build tree that can run a command
# before apt sees a package name is here.
#
# `steam-installer` (not `steam`) is Debian's own wrapper: it fetches Valve's bootstrap
# on first launch and keeps itself updated from Valve thereafter, which is what you
# want for a client that talks to a moving service. It lives in contrib, so
# build-steam-tv-box-iso.sh passes `--archive-areas "main contrib non-free
# non-free-firmware"` to `lb config`; without that this hook fails on the first
# apt-get install and the image comes up with no Steam at all.
set -eu
. /etc/steamtv-agent/config.env
export DEBIAN_FRONTEND=noninteractive
echo "0300-steam: enabling i386 multiarch"
dpkg --add-architecture i386
apt-get update
# --- The 32-bit graphics/audio stack -----------------------------------------------
# Proton's own runtime is 64-bit, but a great many shipped Linux builds and every
# pre-2015 title in a Steam library are 32-bit, and they fail with an opaque "failed to
# create GL context" rather than anything that names the missing library. Installing
# these up front is much cheaper than diagnosing that on a TV with a gamepad.
I386_PACKAGES="
libgl1-mesa-dri:i386
libglx-mesa0:i386
mesa-vulkan-drivers:i386
libvulkan1:i386
libasound2-plugins:i386
libpulse0:i386
libsdl2-2.0-0:i386
libopenal1:i386
"
# shellcheck disable=SC2086
if apt-get install -y --no-install-recommends $I386_PACKAGES; then
echo "0300-steam: installed the i386 graphics/audio stack."
else
echo "0300-steam: WARNING — the i386 stack did not install cleanly. 64-bit titles will"
echo " still run; 32-bit ones will fail with GL/Vulkan context errors. Retry on the"
echo " booted image with: sudo apt-get install $(echo $I386_PACKAGES | tr '\n' ' ')"
fi
# --- Steam itself ------------------------------------------------------------------
# steam-devices ships the udev rules for Steam Controllers, Steam Decks used as pads,
# and the DualShock/DualSense/Xbox families. Without it the pads are visible only to
# root and Big Picture shows no controller at all — which on a machine with no keyboard
# in front of it is indistinguishable from a broken image.
if apt-get install -y steam-installer steam-devices; then
echo "0300-steam: installed steam-installer + steam-devices."
else
echo "0300-steam: ERROR — could not install steam-installer. The most likely cause is"
echo " that contrib is missing from the archive areas (see this file's header), or"
echo " the chroot had no network. This image will boot to the media session with no"
echo " Steam on it. Fix and rebuild rather than shipping it."
fi
# --- gamescope (optional) ----------------------------------------------------------
# The micro-compositor Valve uses on the Steam Deck. On a TV it is what gives Big
# Picture a fixed resolution and refresh rate independent of the desktop, integer
# scaling, and a framerate limiter — noticeably better than handing a game the raw
# output. Not required: /usr/local/bin/steam-big-picture detects it at launch and runs
# Steam directly under Xwayland when it is absent, so this failing is cosmetic.
if apt-get install -y --no-install-recommends gamescope; then
echo "0300-steam: installed gamescope; Big Picture will run nested inside it."
else
echo "0300-steam: note — gamescope is not available in this release. steam-big-picture"
echo " will fall back to plain Xwayland, which works but has no framerate limiter"
echo " and no integer scaling."
fi
# --- Vendor GPU driver -------------------------------------------------------------
# GPU_VENDOR comes from `gpu_vendor` in CoreSystemConfig.json. AMD and Intel need
# nothing beyond the Mesa packages already in the package list plus the firmware blobs;
# NVIDIA needs its own non-free driver, and installing that on an AMD box actively
# breaks it, which is why this is a config value and not a guess.
case "${GPU_VENDOR:-amd}" in
nvidia)
echo "0300-steam: GPU_VENDOR=nvidia — installing the non-free NVIDIA driver"
# libnvidia-gl:i386 is the 32-bit GL/Vulkan half, the exact counterpart of the
# Mesa i386 packages above and needed for the same 32-bit titles.
if apt-get install -y nvidia-driver libnvidia-gl-535:i386 2>/dev/null \
|| apt-get install -y nvidia-driver nvidia-driver-libs:i386; then
echo "0300-steam: NVIDIA driver installed."
echo "0300-steam: VERIFY ON FIRST BOOT — sway on NVIDIA still wants"
echo " WLR_NO_HARDWARE_CURSORS=1 on some driver versions; kiosk-session sets it"
echo " when GPU_VENDOR=nvidia. Check \`vulkaninfo | head\` reports the NVIDIA ICD."
else
echo "0300-steam: WARNING — the NVIDIA driver did not install. Check that non-free"
echo " is in the archive areas and that the driver package name matches this"
echo " release, then install it on the booted machine."
fi
;;
intel)
echo "0300-steam: GPU_VENDOR=intel — Mesa (already installed) is the whole driver."
if ! apt-get install -y --no-install-recommends intel-media-va-driver-non-free; then
echo "0300-steam: note — intel-media-va-driver-non-free unavailable; video decode"
echo " falls back to the free driver, which is fine for games and slower for 4K video."
fi
;;
*)
echo "0300-steam: GPU_VENDOR=${GPU_VENDOR:-amd} — Mesa (already installed) is the whole driver."
;;
esac
# --- Let games ask for more file descriptors and memory maps -----------------------
# Proton/DXVK open a lot of both, and the distro defaults are what produce the
# "shader cache failed" and "vm.max_map_count" class of crashes late in a session.
cat > /etc/security/limits.d/90-steam-tv-box.conf <<'EOF'
# Proton and DXVK hold thousands of file descriptors open per running game.
* soft nofile 1048576
* hard nofile 1048576
EOF
cat > /etc/sysctl.d/90-steam-tv-box.conf <<'EOF'
# Several Proton titles (and anything using esync/fsync) exceed the default map count
# and die with an allocation failure that names nothing useful.
vm.max_map_count = 2147483642
EOF
apt-get clean

59
hosts/steam-tv-box/live-build/config/hooks/normal/0400-flatpak-apps.hook.chroot
View File

@ -1,59 +0,0 @@
#!/bin/sh
# Flathub remote + the two apps this image needs that Debian does not package:
# the official Spotify client and Prism Launcher.
#
# Same distribution channel and the same "don't pin a version that will just rot"
# reasoning as hosts/touch-panel's 0300-flatpak-spotify.hook.chroot and
# hosts/thin-client's 0400-flatpak-steamlink.hook.chroot.
#
# Note what is NOT here: Steam. Steam is installed natively from apt by
# 0300-steam.hook.chroot, deliberately — the Flatpak runs sandboxed with its own
# runtime, which complicates GPU driver matching, controller udev access and mounting
# a games disk, all of which matter on a machine whose whole job is running games
# locally. Prism and Spotify have none of those constraints, so the sandbox is a
# straightforward win for them.
set -eu
flatpak remote-add --if-not-exists flathub https://dl.flathub.org/repo/flathub.flatpakrepo
# VERIFY BEFORE THE FIRST REAL BUILD: confirm both application IDs against the live
# Flathub listing (`flatpak search Spotify`, `flatpak search "Prism Launcher"`).
# Believed correct — both are the publishers' own official listings — but not checked
# against Flathub from this environment.
SPOTIFY_APP_ID="com.spotify.Client"
PRISM_APP_ID="org.prismlauncher.PrismLauncher"
install_app() {
app_id="$1"
label="$2"
if flatpak install -y --noninteractive flathub "$app_id"; then
echo "0400-flatpak-apps: installed ${app_id}."
else
echo "0400-flatpak-apps: WARNING — could not install ${app_id} during the build (no"
echo " network in the chroot, or the app ID is wrong). Run this on the booted image"
echo " instead: flatpak install -y flathub ${app_id}"
echo " Until then the ${label} launcher opens nothing."
fi
}
install_app "$SPOTIFY_APP_ID" "Spotify"
install_app "$PRISM_APP_ID" "Prism Launcher"
# Prism needs a Java runtime it can actually see. The Flatpak's sandbox has its own
# filesystem view, so the openjdk packages in the package list are not visible inside
# it by default — this grants read access to the host's JVMs so Prism's auto-detection
# finds them and Minecraft launches with no network on first run.
if flatpak override --filesystem=/usr/lib/jvm:ro "$PRISM_APP_ID" 2>/dev/null; then
echo "0400-flatpak-apps: granted Prism Launcher read access to /usr/lib/jvm."
else
echo "0400-flatpak-apps: note — could not set the /usr/lib/jvm override (Prism is"
echo " probably not installed yet). Prism will download its own JRE on first launch,"
echo " which needs network at that moment. To fix it offline later:"
echo " flatpak override --user --filesystem=/usr/lib/jvm:ro ${PRISM_APP_ID}"
fi
# The games directory, so both Prism instances and any Steam library added later sit in
# one predictable place — see hosts/steam-tv-box/README.md on mounting a real disk there.
if flatpak override --filesystem=~/Games "$PRISM_APP_ID" 2>/dev/null; then
echo "0400-flatpak-apps: granted Prism Launcher access to ~/Games."
fi

20
hosts/steam-tv-box/live-build/config/hooks/normal/0700-steamtv-agent.hook.chroot
View File

@ -1,20 +0,0 @@
#!/bin/sh
# Installs the steamtv-agent systemd unit.
#
# The package itself arrives under /opt/steamtv-agent via includes.chroot and runs on
# the system interpreter against apt's python3-paho-mqtt, so there is nothing to
# pip-install and no PEP 668 problem to work around here.
set -eu
install -m 0644 /opt/steamtv-agent/steamtv-agent.service \
/etc/systemd/system/steamtv-agent.service
chmod 0644 /etc/steamtv-agent/config.env
chown -R root:root /opt/steamtv-agent
# Writable runtime state (the persisted audio-output choice) — steamtv_agent's
# runtime_state.py seeds this from the read-only templates in /etc on first start, and
# cannot create the directory itself once /etc is a squashfs.
install -d -m 0755 /var/lib/steamtv-agent
systemctl enable steamtv-agent

22
hosts/steam-tv-box/live-build/config/hooks/normal/0900-firefox.hook.chroot
View File

@ -1,22 +0,0 @@
#!/bin/sh
# Firefox enterprise policy + kiosk-chrome plumbing. Identical in mechanism to
# hosts/thin-client's 0900-firefox.hook.chroot — see that file for the full reasoning.
#
# policies.json is installed to /etc/firefox/policies/policies.json by includes.chroot
# (the documented Linux location), and linked here into firefox-esr's own distribution/
# directory too — an older location some builds read instead of the system one.
# Whichever the installed firefox-esr honours wins; the other is inert.
set -eu
FIREFOX_LIB_DIR="/usr/lib/firefox-esr"
if [ -d "$FIREFOX_LIB_DIR" ]; then
mkdir -p "$FIREFOX_LIB_DIR/distribution"
ln -sf /etc/firefox/policies/policies.json "$FIREFOX_LIB_DIR/distribution/policies.json"
echo "0900-firefox: linked policies.json into $FIREFOX_LIB_DIR/distribution/"
else
echo "0900-firefox: $FIREFOX_LIB_DIR not found (firefox-esr not installed yet, or a"
echo " different path in this Debian release) — /etc/firefox/policies/policies.json"
echo " still applies if the package's install order runs after this hook; verify"
echo " about:policies shows uBlock Origin/SponsorBlock as force-installed on first boot."
fi

157
hosts/steam-tv-box/live-build/config/package-lists/steam-tv-box.list.chroot
View File

@ -1,157 +0,0 @@
# Steam-TV-Box package list (live-build .list.chroot format: one package per line).
#
# This machine is a LOCAL gaming box first and a media station second: real GPU, real
# CPU, games run here rather than being streamed in. That is the difference from
# hosts/thin-client, which runs Steam *Link* (a client for a PC somewhere else) and
# needs no graphics performance at all.
#
# Because of that, this image is built against Debian trixie rather than the household
# default — see the debian_release note in tools/build-steam-tv-box-iso.sh. Mesa 22.3
# (bookworm) predates most of the driver work that current titles rely on, and a
# gaming box on an old Mesa is the one thing here that cannot be fixed later by
# rebuilding a config file.
#
# NOT in this list and installed by hooks instead:
# steam-installer, steam-devices -> 0300-steam.hook.chroot (needs `dpkg
# --add-architecture i386` + `apt-get update` first, and live-build installs
# package lists BEFORE it runs hooks, so it cannot be done from here)
# the i386 halves of the graphics stack -> same hook, same reason
# Spotify, Prism Launcher -> 0400-flatpak-apps.hook.chroot (Flathub)
# --- Wayland compositor / session ---
sway
swayidle
swaybg
# Steam, Prism Launcher's Minecraft windows and most native games are X11 clients, so
# Xwayland is load-bearing here rather than a nicety.
xwayland
greetd
foot
# --- Remote view/control (same wayvnc-with-a-mandatory-password posture as the thin
# client; it is for administering the box, not for playing over) ---
wayvnc
# --- Graphics: the amd64 half. The i386 half is installed by 0300-steam.hook.chroot
# after multiarch is enabled, and 32-bit games/Proton runtimes genuinely still
# need it in 2026. ---
mesa-vulkan-drivers
mesa-va-drivers
mesa-vdpau-drivers
libgl1-mesa-dri
libglx-mesa0
libvulkan1
# vulkaninfo/vkcube — the fastest way to answer "is this box actually accelerated?"
# from the maintenance shell before blaming a game.
vulkan-tools
# glxinfo, for the same question on the GL side.
mesa-utils
libva-drm2
vainfo
# --- Gaming runtime bits ---
# Lets a game ask the system for the performance governor while it runs and hand it
# back afterwards, instead of pinning this box to "performance" around the clock.
gamemode
# On-screen FPS/frametime overlay, opt-in per launch (MANGOHUD=1). Diagnostics, not
# decoration: it is how you tell a CPU-bound stutter from a GPU-bound one.
mangohud
# Controller support. Steam's own controller stack covers Steam games; these cover
# everything outside it (Prism/Minecraft, emulators, the browser).
joystick
evtest
bluez
bluetooth
# Steam's runtime and many native titles link against these directly.
libsdl2-2.0-0
libopenal1
# Proton/Wine prerequisites that apt will otherwise pull only as recommends.
cabextract
p7zip-full
# Xbox/PlayStation pad firmware, plus the general non-free firmware blobs a real GPU
# needs. Requires the contrib + non-free + non-free-firmware archive areas, which
# tools/build-steam-tv-box-iso.sh passes to `lb config`.
firmware-linux
firmware-amd-graphics
firmware-realtek
firmware-iwlwifi
firmware-misc-nonfree
# --- Browser / media (the "same media functions as the other monitor clients" half) ---
firefox-esr
mpv
mpv-mpris
libmpv2
playerctl
# wev prints the keysym for each key press — how to find out what a given TV remote's
# buttons actually send. Same reasoning as the thin client's.
wev
# --- Audio ---
pipewire
pipewire-pulse
pipewire-alsa
wireplumber
alsa-utils
# 32-bit games play through pipewire-pulse's ALSA/Pulse shim; libpulse0:i386 comes with
# the i386 batch in 0300-steam.hook.chroot.
# --- Flatpak apps: Spotify and Prism Launcher (0400-flatpak-apps.hook.chroot) ---
flatpak
# Portals — Flatpak apps use these for file pickers and screen sharing under Wayland.
# Without them Prism's "choose a Java binary" dialog and Spotify's file dialogs come up
# empty rather than failing loudly, which is a confusing way to lose an evening.
xdg-desktop-portal
xdg-desktop-portal-wlr
xdg-desktop-portal-gtk
# --- Java for Prism Launcher's own use ---
# Prism can download its own JREs, but only if it has network at the time. Shipping
# both the LTS and the current runtime means Minecraft launches offline too: 8 for
# anything pre-1.17, 21 for current versions.
openjdk-21-jre
openjdk-8-jre
# --- Agent runtime ---
python3
python3-venv
# trixie ships paho-mqtt 2.x; steamtv_agent is written against both the 1.x and 2.x
# callback APIs (see make_client() in main.py), so apt's package is used as-is.
python3-paho-mqtt
procps
# --- Networking ---
network-manager
# --- Out-of-band admin ---
openssh-server
sudo
# --- Session plumbing ---
# session-watcher parses `swaymsg -t subscribe` events with jq; the sway config's
# workspace logic depends on it.
jq
# HDMI-CEC (cec-ctl) for turning the TV itself on and off — steamtv_agent/display_power.py.
v4l-utils
# media-player talks to a running mpv over its JSON IPC socket (mpv.conf's
# input-ipc-server) so a second `media-player <file>` loads into the existing window
# instead of opening a competing MPRIS player.
socat
# --- Misc ---
ca-certificates
curl
openssl
less
vim-tiny
git
# NOTE: gamescope is NOT listed here on purpose. It is the right way to run Big
# Picture on a TV (integer-scaled, its own framerate limiter, no compositor tearing),
# but its availability in the target release was not verified from this environment —
# 0300-steam.hook.chroot installs it if apt has it and steam-big-picture falls back to
# plain Xwayland if it does not, so a missing gamescope costs polish rather than a
# broken image.
# NOTE: NVIDIA users — see 0300-steam.hook.chroot. The driver is not in this list
# because it must not be installed on an AMD/Intel box, and which one this is comes
# from `gpu_vendor` in CoreSystemConfig.json.

46
hosts/thin-client/README.md
View File

@ -294,44 +294,6 @@ vendor/product IDs) and passes it to mpv as `--external-file=alsa://hw:X,0` if
found. Video-only playback (not a crash) if nothing matches. **Unverified against
real hardware** — see the checklist below.
## A wireless USB remote works out of the box
The kind with TV buttons on the front and a small keyboard on the back. To Linux that
dongle is two ordinary HID keyboards — a normal one and a "consumer control" one — so
there is nothing to configure per device: the front buttons arrive as `XF86*` keysyms
and the back keyboard arrives as keys. `configs/sway/config`'s remote-control section
binds the full standardised set:
| Buttons | What they do |
|---|---|
| ▶ ⏸ ⏹ ⏭ ⏮ | `playerctl -p mpv,spotifyd` — play/pause, pause, stop, next, previous |
| ⏪ ⏩ | seek 10s back / 30s forward. Skip, not scan: remotes get *pressed*, not held, and the podcast convention is already in people's fingers |
| Volume, mute, mic mute | the **sink**, not the player — the rocker should move the room's volume whatever is making the noise |
| Channel ± | next/previous workspace. A media station's "channels" are its workspaces, which is the closest honest analogy |
| Home / Back | the media workspace / `back_and_forth` |
| Power, Sleep | **the display, not the machine** — see below |
**Two deliberate choices worth knowing before you remap anything.**
*Plain arrows and Return are not bound, on purpose.* A remote's D-pad and OK send
exactly those, unmodified, and Chromium, mpv and every kiosk page need them — binding
them at the compositor would break scrolling a web page with the remote, which is most
of what the remote is for. Window focus stays on `$mod`+arrows.
*The power button turns the screen off, not the computer.* On a TV that button turns
the picture off; on a thin client that autologins into a kiosk, `poweroff` takes the
room's screen away until somebody walks over to press a physical button. So it runs
`display-toggle`, which is a **toggle** rather than two bindings because with the
outputs dark there is no other way back — Sway is still running and still receiving
keys, so the next press wakes it. A stuck button therefore cannot leave the screen
dark either.
**Remotes vary more than their marketing does.** Run `wev` (or
`sudo libinput debug-events`) from the maintenance shell (`$mod+Shift+Ctrl+m`), press
every button, and add any that comes back with an unbound keysym. A button that reports
*no keysym at all* is one the kernel has no mapping for — that is a udev hwdb entry,
not a Sway binding, and is worth knowing before blaming the config.
## Home Assistant entities
`thinclient-agent` publishes MQTT-discovery configs on connect. Under the MQTT
@ -490,14 +452,6 @@ default on most sets, and the reason a TV that sleeps fine refuses to wake.
## Manual verification still outstanding
0. **The remote.** No remote has been plugged into anything — the keysym list above is
the standardised set, not one read off a specific device. Expect one or two buttons
on any given remote to report something unbound (or nothing at all); `wev` from the
maintenance shell is the two-minute check, and `display-toggle`'s grep for
`'"dpms": true'` in `swaymsg -t get_outputs` is worth confirming against the Sway
version this image actually ships, since that field's spelling is the one thing
that would make the power button silently do nothing.
None of this has been run on hardware. In rough order:
1. The ISO builds at all (`lb build` is network-heavy and can fail on mirror hiccups).

5
hosts/thin-client/agent/thinclient_agent/main.py
View File

@ -152,9 +152,6 @@ def main() -> int:
hostname = socket.gethostname()
node_id = "".join(c if c.isalnum() else "_" for c in hostname).strip("_") or "thinclient"
friendly_name = config.get("THINCLIENT_NAME") or f"Thin client ({hostname})"
# The HA area this device sits in, published as suggested_area — see
# docs/rooms-and-endpoints.md. Blank is fine and means "no suggestion".
room = config.get("THINCLIENT_ROOM", "")
broker_host = config.get("MQTT_BROKER_HOST", "")
broker_port = int(config.get("MQTT_BROKER_PORT") or 1883)
@ -186,7 +183,7 @@ def main() -> int:
if config.get("MQTT_USERNAME"):
client.username_pw_set(config["MQTT_USERNAME"], config.get("MQTT_PASSWORD") or None)
discovery = Discovery(client, node_id, friendly_name, room)
discovery = Discovery(client, node_id, friendly_name)
mpris = MprisBridge(discovery.publish_media_state, sway.session_env)
def on_detail_level(payload: str) -> None:

10
hosts/thin-client/agent/thinclient_agent/mqtt_discovery.py
View File

@ -29,11 +29,10 @@ DISCOVERY_PREFIX = "homeassistant"
class Discovery:
def __init__(self, client, node_id: str, friendly_name: str, room: str = ""):
def __init__(self, client, node_id: str, friendly_name: str):
self.client = client
self.node_id = node_id
self.friendly_name = friendly_name
self.room = (room or "").strip()
self.base = f"thinclient/{node_id}"
self.availability_topic = f"{self.base}/availability"
self.media_state_topic = f"{self.base}/media/state"
@ -52,13 +51,6 @@ class Discovery:
"model": "Sway thin client",
"sw_version": __version__,
}
# Which room this physically sits in, as an HA area_id. `suggested_area`
# is honoured by HA only when the device is FIRST discovered — moving a
# device later means moving it in HA too, this cannot un-file it. Omitted
# entirely when unset, because an empty suggested_area is not the same
# request as no suggestion. See docs/rooms-and-endpoints.md.
if self.room:
self.device["suggested_area"] = self.room
# --- plumbing -----------------------------------------------------------
def _publish_config(self, component: str, object_id: str, payload: dict) -> None:

52
hosts/thin-client/configs/sway/config
View File

@ -133,64 +133,12 @@ bindsym $mod+Right focus right
bindsym $mod+Up focus up
bindsym $mod+Down focus down
# ---------------------------------------------------------------------------
# Remote control — the full standardised media-key set.
#
# THE DEVICE THIS IS FOR: a wireless USB remote with TV controls on the front and a
# small keyboard on the back. To Linux that is just two HID keyboards on one dongle
# (a normal keyboard plus a "consumer control" device), so there is nothing to
# configure per-device — the front buttons arrive as the XF86* keysyms below and the
# back keyboard arrives as ordinary keys. Any remote in that class works; the bindings
# are what make it useful.
#
# WHAT IS DELIBERATELY *NOT* BOUND: plain Up/Down/Left/Right and Return. A remote's
# D-pad and OK button send exactly those, unmodified — and Chromium, mpv and every
# kiosk page need them. Stealing them at the compositor would break scrolling a web
# page with the remote, which is most of what the remote is for. Window focus stays on
# $mod+arrows above; the bare arrows belong to whatever is on screen.
#
# CHECKING WHAT YOUR REMOTE ACTUALLY SENDS: remotes vary more than their marketing
# does. `wev` (or `sudo libinput debug-events`) prints the keysym for each button —
# run it once from the maintenance shell and add any button that comes back unbound.
# A button that reports no keysym at all is one the kernel has no mapping for, which
# is a udev hwdb entry, not a Sway binding.
# ---------------------------------------------------------------------------
# Transport. `playerctl -p mpv,spotifyd` matches the two players this image runs;
# the order is the priority when both are alive.
bindsym XF86AudioPlay exec playerctl -p mpv,spotifyd play-pause
bindsym XF86AudioPause exec playerctl -p mpv,spotifyd pause
bindsym XF86AudioStop exec playerctl -p mpv,spotifyd stop
bindsym XF86AudioNext exec playerctl -p mpv,spotifyd next
bindsym XF86AudioPrev exec playerctl -p mpv,spotifyd previous
# Skip rather than scan: a remote's ⏪/⏩ are pressed repeatedly, not held, and
# playerctl has no scan mode. 30s forward / 10s back is the podcast convention and is
# the one people already have in their fingers.
bindsym XF86AudioForward exec playerctl -p mpv,spotifyd position 30+
bindsym XF86AudioRewind exec playerctl -p mpv,spotifyd position 10-
# Volume. Sinks, not players — the remote's volume rocker should move the room's
# volume regardless of what is making the noise.
bindsym XF86AudioRaiseVolume exec wpctl set-volume @DEFAULT_AUDIO_SINK@ 5%+
bindsym XF86AudioLowerVolume exec wpctl set-volume @DEFAULT_AUDIO_SINK@ 5%-
bindsym XF86AudioMute exec wpctl set-mute @DEFAULT_AUDIO_SINK@ toggle
bindsym XF86AudioMicMute exec wpctl set-mute @DEFAULT_AUDIO_SOURCE@ toggle
# Navigation. A media station's "channels" are its workspaces, which is the closest
# honest analogy and means the channel rocker does something useful instead of nothing.
bindsym XF86ChannelUp workspace next
bindsym XF86ChannelDown workspace prev
bindsym XF86HomePage workspace $ws_media
bindsym XF86Back workspace back_and_forth
bindsym XF86Forward workspace back_and_forth
# Power. **The display, not the machine.** A remote's power button on a TV turns the
# picture off, and that is what people expect it to do here — while `poweroff` on a
# thin client that autologins is a button that takes the room's screen away until
# somebody walks over to it. See the display-toggle script for the wake side.
bindsym XF86PowerOff exec display-toggle
bindsym XF86Sleep exec display-toggle
bindsym XF86ScreenSaver exec display-toggle
# Deliberately no exit binding: `swaymsg exit` would drop to a black VT, and greetd
# would just autologin straight back in. Use SSH or the local terminal to administer.

26
hosts/thin-client/configs/sway/display-toggle
View File

@ -1,26 +0,0 @@
#!/bin/sh
# Turn every output off, or back on. Installed to /usr/local/bin/display-toggle and
# bound to the remote's power/sleep buttons in the Sway config.
#
# WHY THE POWER BUTTON DOES NOT POWER ANYTHING OFF: on a TV, that button turns the
# picture off. On a thin client that autologins into a kiosk, `poweroff` takes the
# room's screen away until somebody walks over and presses a physical button — which
# is a worse outcome than any it prevents. So the remote's power button does the thing
# the person pressing it actually meant.
#
# THE WAKE SIDE IS THE HARD HALF. With outputs powered off, Sway is still running and
# still receiving keys, so pressing power again lands here and turns them back on.
# That is why this is a toggle rather than two bindings: there is no other way back.
# It also means a *stuck* remote button cannot leave the screen dark — the next press
# fixes it.
set -eu
# `swaymsg -t get_outputs` reports each output's dpms state. If ANY output is still on,
# the intent of a press is "turn it off"; only when everything is already dark does a
# press mean "wake up". That ordering matters on a multi-output machine, where asking
# per-output would leave the remote toggling one screen at a time.
if swaymsg -t get_outputs | grep -q '"dpms": true'; then
swaymsg 'output * dpms off'
else
swaymsg 'output * dpms on'
fi

3
hosts/thin-client/live-build/config/package-lists/thin-client.list.chroot
View File

@ -22,9 +22,6 @@ mpv
mpv-mpris
libmpv2
playerctl
# wev prints the keysym for each key press — the two-minute way to find out what a
# given wireless remote's buttons actually send. See README's remote-control section.
wev
# --- Audio ---
pipewire

75
hosts/touch-panel/README.md
View File

@ -2,9 +2,8 @@
Phase 16 of `docs/project-plan.md`. Builds a Debian 12 live ISO for a touch-driven
wall/counter panel: autologin into Sway, three fixed apps (Spotify, Home Assistant,
a general web browser) — four where the panel asks for **Discord** — switched with a
finger via an always-on dock, and also controllable by Home Assistant/the local LLM
over MQTT.
a general web browser) switched with a finger via an always-on dock, and also
controllable by Home Assistant/the local LLM over MQTT.
**A different device from `../thin-client/`**, not a variant of it. The thin client is
a couch-distance media station whose primary control surface is HA/MQTT and wayvnc,
@ -18,11 +17,10 @@ What ends up on the image:
| | |
|---|---|
| Compositor | Sway, workspaces `1:spotify` / `2:home` / `3:web` (+ `4:discord` where enabled), always-on touch dock |
| Compositor | Sway, workspaces `1:spotify` / `2:home` / `3:web`, always-on touch dock |
| Autologin | greetd, `default_session` straight into `/usr/local/bin/kiosk-session` |
| Touch input | Native Wayland `wl_touch` preferred; degrades to a usable single-touch pointer if the hardware only exposes an emulated-mouse HID interface — see below |
| Music | Full Spotify GUI client (Flathub `com.spotify.Client`) — not a headless Connect receiver |
| Chat / voice | Discord (Flathub `com.discordapp.Discord`), **only where `enable_discord` is set** — see below |
| Home dashboard | Chromium in kiosk mode, `--app=$HA_URL`, auto-restart if it crashes |
| Web browser | Firefox, minimal chrome (back/forward/reload/address bar), general browsing |
| On-screen keyboard | wvkbd, toggled from the dock — see the caveat below, no auto-show |
@ -141,9 +139,8 @@ panel and adjust `PLAYER_PRIORITY` in `mpris_bridge.py`.
## Touch dock
An always-visible bar reserved at the bottom of the screen (`configs/eww/`, layer-shell
`:exclusive true` so nothing ever tiles under or draws over it) with four buttons —
five on a panel with Discord: **Spotify**, **Home**, **Web**, (**Discord**), and
**Keyboard**. The workspace ones call `swaymsg
`:exclusive true` so nothing ever tiles under or draws over it) with four buttons:
**Spotify**, **Home**, **Web**, and **Keyboard**. The first three call `swaymsg
workspace` directly — fixed constants in `eww.yuck`, nothing from MQTT/HA is ever
interpolated into them, same rule as the thin client's now-playing widget. This is the
touch-first equivalent of the thin client's HA **Screen** select entity — both exist,
@ -160,54 +157,6 @@ there is no on-screen app switcher at all. The panel still boots and shows Home;
LLM-driven app switching via `touchpanel-agent`'s **Screen** select and **Show
\<app\>** buttons still works either way, since that's a separate control path.
## Discord — a per-panel fourth app
Off by default; turned on per panel with `enable_discord` on that kiosk in
`CoreSystemConfig.json`. `tools/validate-config.py` rejects it on any other image
type — no other build installs the Flatpak or has a workspace for it, so the flag
would otherwise be silently ignored on a panel that boots looking fine.
**What it is for.** The Loggia panel is the balcony one, and the point is that a voice
call survives stepping outside for a cigarette: you move the call onto this machine on
the way out and back onto the desktop on the way in. That is a *switch device* action,
which is why Discord starts at session boot and stays logged in rather than launching
on demand — an app that needs fifteen seconds and a login is not one you use on the
way past. It is also why the HA **Show Discord** button and the dock button both
resolve to focus-the-running-app, exactly like Home and Spotify.
Three things follow from it being an Electron/XWayland app rather than a native
Wayland one:
- Sway matches it on `class="discord"`, **not** `app_id` — an `app_id` rule would
silently never fire.
- Touch arrives as emulated pointer events, so it lands in the same tier-2 behaviour
described under *Touch input* below: tap and drag work, two-finger scrolling in the
message list may not. Fine for join / mute / leave, which is the job.
- It deliberately does **not** get `inhibit_idle`. A call would otherwise hold the
panel's display on for an hour in an empty room; the screen blanks and the call
keeps running, because sway's idle timeout powers off the output, it does not
suspend the machine or the audio stream.
Which audio device the call actually uses is Discord's own setting, on-device, once —
this image does not manage it, and `touchpanel-agent` has no entity for it.
## This panel as a remote microphone
The Loggia panel doubles as a microphone for a desktop in another room
(`../../mic-follow/`): that machine opens an SSH session and runs `pw-record` here, so
somebody on the balcony stays in a voice call running on their PC. Nothing listens for
it — it is an ordinary SSH command over the admin path this image already has — and
audio is captured only while that session is open.
Because it is a microphone in a shared flat that somebody elsewhere can open, the dock
carries a red **MIC LIVE** badge, driven by `mic-in-use` (polled every 2s). It lights
whenever *anything* is capturing this panel's microphone, not only when mic-follow is:
an indicator whose job is to be believed should not depend on knowing who asked.
Monitor streams — something recording what the panel is *playing* — do not light it.
The image ships `pipewire-bin` (for `pw-record`) and `pulseaudio-utils` (for `pactl`)
for this. Neither is needed by anything else on the panel.
## On-screen keyboard — no auto-show, by design
`configs/keyboard/toggle-keyboard` toggles `wvkbd` on and off. There is deliberately
@ -228,14 +177,11 @@ placeholder shape as `eww` and the thin client's `spotifyd`/`librespot` install.
`touchpanel-agent` publishes MQTT-discovery configs on connect. Under the MQTT
integration you should get one device per touch panel with:
- **Show Spotify**, **Show Home**, **Show web browser**, and on a Discord panel
**Show Discord** (buttons) — switch workspace
- **Show Spotify**, **Show Home**, **Show web browser** (buttons) — switch workspace
and, for Spotify/Home, focus the already-running app rather than relaunching it
(both are meant to stay open and stateful, unlike the thin client's stateless
digest/admin kiosk pages)
- **Screen** (select) — `1:spotify` / `2:home` / `3:web`, plus `4:discord` on a panel
that has it. The options are derived from the app table rather than fixed, so a
dropdown never offers a workspace this image has no app for
- **Screen** (select) — `1:spotify` / `2:home` / `3:web`
- **Playback state** (sensor, with track metadata as attributes), **Volume**
(number), and play/pause / next / previous / stop (buttons), bridged from
Spotify's own MPRIS interface
@ -271,13 +217,6 @@ None of this has been run on hardware. In rough order:
4. `touchpanel-agent` connects to Mosquitto and the device appears in HA.
5. **Flathub app ID `com.spotify.Client` is correct** — flagged for verification in
`0300-flatpak-spotify.hook.chroot`, same as the thin client's Steam Link ID.
5b. **Flathub app ID `com.discordapp.Discord` is correct**, on a Discord panel —
flagged in `0350-flatpak-discord.hook.chroot` for the same reason.
5c. **Discord's real X11 `class`** — assumed `discord`, which is what the sway
fullscreen rule and the agent's focus criteria both match on. `swaymsg -t
get_tree` on the booted panel is the way to check; if it is capitalised or
suffixed, the app opens in a window that never goes fullscreen and **Show
Discord** switches workspace without focusing anything.
6. Spotify login persists in `~/.var/app/com.spotify.Client` across a reboot.
7. Spotify's real MPRIS bus name — assumed `spotify`, not confirmed (see the Spotify
section above); `playerctl -l` on the booted panel is the way to check.

41
hosts/touch-panel/agent/touchpanel_agent/main.py
View File

@ -14,7 +14,7 @@ import paho.mqtt.client as mqtt
from .mpris_bridge import MprisBridge
from .mqtt_discovery import Discovery
from .sway_control import WS_DISCORD, WS_HOME, WS_SPOTIFY, WS_WEB, SwayControl
from .sway_control import WS_HOME, WS_SPOTIFY, WS_WEB, SwayControl
CONFIG_PATH = os.environ.get("TOUCHPANEL_AGENT_CONFIG", "/etc/touchpanel-agent/config.env")
@ -26,14 +26,9 @@ CONFIG_KEYS = (
"HA_URL",
"KIOSK_USERNAME",
"TOUCHPANEL_NAME",
"ENABLE_DISCORD",
)
# The workspaces the panel offers over MQTT. Derived from the app table rather than
# fixed, because Discord is per-panel (kiosks[].enable_discord): a select whose
# options include a workspace this image has no app for is a dropdown entry that
# switches to a black screen.
BASE_WORKSPACES = (WS_SPOTIFY, WS_HOME, WS_WEB)
WORKSPACES = (WS_SPOTIFY, WS_HOME, WS_WEB)
log = logging.getLogger("touchpanel-agent")
@ -68,8 +63,8 @@ def load_config(path: str = CONFIG_PATH) -> dict[str, str]:
return values
def build_apps(config: dict[str, str]) -> dict[str, App]:
apps = {
def build_apps(_config: dict[str, str]) -> dict[str, App]:
return {
"spotify": App(
name="Spotify",
command=["/usr/local/bin/spotify-launch"],
@ -100,23 +95,6 @@ def build_apps(config: dict[str, str]) -> dict[str, App]:
),
}
# Discord, on the panels that asked for it. Launched at session start by the sway
# config, so this entry — like "home" — is normally a focus-and-switch rather than
# a cold start: launch_app() sees the process already running and just brings the
# workspace up, which is what "step outside without dropping the call" needs.
# Matched on X11 class, not app_id: it is an XWayland window (see discord-launch).
if str(config.get("ENABLE_DISCORD", "")).strip().lower() == "true":
apps["discord"] = App(
name="Discord",
command=["/usr/local/bin/discord-launch"],
process_pattern="com.discordapp.Discord",
workspace=WS_DISCORD,
focus_criteria='class="discord"',
icon="mdi:discord",
)
return apps
def make_client(client_id: str) -> mqtt.Client:
# paho-mqtt 2.x requires an explicit callback API version; bookworm's
@ -139,23 +117,18 @@ def main() -> int:
hostname = socket.gethostname()
node_id = "".join(c if c.isalnum() else "_" for c in hostname).strip("_") or "touchpanel"
friendly_name = config.get("TOUCHPANEL_NAME") or f"Touch panel ({hostname})"
# The HA area this device sits in, published as suggested_area — see
# docs/rooms-and-endpoints.md. Blank is fine and means "no suggestion".
room = config.get("TOUCHPANEL_ROOM", "")
broker_host = config.get("MQTT_BROKER_HOST", "")
broker_port = int(config.get("MQTT_BROKER_PORT") or 1883)
sway = SwayControl()
apps = build_apps(config)
workspaces = tuple(dict.fromkeys(
list(BASE_WORKSPACES) + [app.workspace for app in apps.values() if app.workspace]))
client = make_client(f"touchpanel-agent-{node_id}")
if config.get("MQTT_USERNAME"):
client.username_pw_set(config["MQTT_USERNAME"], config.get("MQTT_PASSWORD") or None)
discovery = Discovery(client, node_id, friendly_name, room)
discovery = Discovery(client, node_id, friendly_name)
mpris = MprisBridge(discovery.publish_media_state, sway.session_env)
def on_launch(key: str) -> None:
@ -172,7 +145,7 @@ def main() -> int:
def on_workspace(payload: str) -> None:
name = payload.strip()
# Enumerated, never passed through: see the security note in mqtt_discovery.py.
if name not in workspaces:
if name not in WORKSPACES:
log.warning("ignoring unknown workspace %r", name)
return
sway.switch_workspace(name)
@ -185,7 +158,7 @@ def main() -> int:
log.info("connected to MQTT broker %s:%s", broker_host, broker_port)
discovery.register_media_player(mpris.handle_command, mpris.set_volume)
discovery.register_app_launchers(apps, on_launch)
discovery.register_workspace_select(workspaces, on_workspace, WS_HOME)
discovery.register_workspace_select(WORKSPACES, on_workspace, WS_HOME)
discovery.subscribe_all()
discovery.publish_available(True)

10
hosts/touch-panel/agent/touchpanel_agent/mqtt_discovery.py
View File

@ -30,11 +30,10 @@ DISCOVERY_PREFIX = "homeassistant"
class Discovery:
def __init__(self, client, node_id: str, friendly_name: str, room: str = ""):
def __init__(self, client, node_id: str, friendly_name: str):
self.client = client
self.node_id = node_id
self.friendly_name = friendly_name
self.room = (room or "").strip()
self.base = f"touchpanel/{node_id}"
self.availability_topic = f"{self.base}/availability"
self.media_state_topic = f"{self.base}/media/state"
@ -47,13 +46,6 @@ class Discovery:
"model": "Sway touch panel",
"sw_version": __version__,
}
# Which room this physically sits in, as an HA area_id. `suggested_area`
# is honoured by HA only when the device is FIRST discovered — moving a
# device later means moving it in HA too, this cannot un-file it. Omitted
# entirely when unset, because an empty suggested_area is not the same
# request as no suggestion. See docs/rooms-and-endpoints.md.
if self.room:
self.device["suggested_area"] = self.room
# --- plumbing -----------------------------------------------------------
def _publish_config(self, component: str, object_id: str, payload: dict) -> None:

4
hosts/touch-panel/agent/touchpanel_agent/sway_control.py
View File

@ -18,10 +18,6 @@ log = logging.getLogger(__name__)
WS_SPOTIFY = "1:spotify"
WS_HOME = "2:home"
WS_WEB = "3:web"
# Only present on panels built with enable_discord — see build_apps() in main.py. The
# workspace exists in the sway config either way (an unused workspace name costs
# nothing); what is gated is the app, the dock button and the HA select's options.
WS_DISCORD = "4:discord"
def runtime_dir() -> str:

25
hosts/touch-panel/configs/discord/discord-launch
View File

@ -1,25 +0,0 @@
#!/bin/sh
# Discord, on the touch panel. Installed to /usr/local/bin/discord-launch.
#
# WHY IT IS HERE: this panel is in the Loggia, and the point is walking out for a
# cigarette without dropping out of a voice call — you join from the panel and leave
# the call on the desktop, or the other way round. That is a "switch device" action,
# not a "read messages" one, so the app stays logged in and running from session
# start rather than being launched on demand: an app that takes fifteen seconds to
# start and then wants a login is not something you use on the way past.
#
# Flathub, for the same reason as Spotify: Discord's own .deb exists but updates
# itself by nagging you to download a new one, which on a kiosk nobody logs into
# means it eventually refuses to connect. The Flatpak updates like everything else.
#
# Login is interactive, on-device, on first launch. The Flatpak's persistent data dir
# (~/.var/app/com.discordapp.Discord) keeps that session across reboots — nothing here
# reseeds it, unlike the kiosk Firefox profiles.
set -eu
# Discord runs under XWayland here (its Electron build does not default to Wayland),
# so touch arrives as emulated pointer events rather than wl_touch. Tap and drag
# work; two-finger scrolling in the message list may not. See README.md's touch
# section — this is the same tier-2 behaviour the pointer-fallback block in
# configs/sway/config exists for, and it is good enough for "join voice, mute, leave".
exec flatpak run com.discordapp.Discord "$@"

22
hosts/touch-panel/configs/eww/eww.scss
View File

@ -48,25 +48,3 @@ $accent: #6ea8fe;
background-color: rgba(255, 255, 255, 0.12);
margin: 8px 4px;
}
/* The remote-microphone indicator. Red, because there is exactly one thing on this
dock that means "somebody in another room can hear you" and it should not have to
compete with the app buttons for attention. */
.mic-badge {
background-color: #7a1020;
border-radius: 8px;
padding: 0 12px;
margin-left: 8px;
}
.mic-dot {
color: #ff4d5e;
font-size: 16px;
}
.mic-text {
color: #ffd7dc;
font-size: 13px;
font-weight: bold;
}

16
hosts/touch-panel/configs/eww/eww.yuck
View File

@ -12,13 +12,6 @@
(defpoll active_workspace :interval "1s" :initial "2:home"
"swaymsg -t get_workspaces | jq -r '.[] | select(.focused) | .name' 2>/dev/null || echo 2:home")
;; Is this panel's microphone being recorded right now — by anything, including a
;; desktop in another room using it as a remote microphone (mic-follow/). A room
;; microphone that can be opened remotely has to say so IN THE ROOM; Home Assistant
;; knowing is not the same as the person standing here knowing.
(defpoll mic_live :interval "2s" :initial "0"
"mic-in-use 2>/dev/null || echo 0")
(defwidget dock-btn [ws icon label]
(button :class {active_workspace == ws ? "dock-btn dock-btn-active" : "dock-btn"}
:onclick "swaymsg workspace ${ws}"
@ -32,16 +25,11 @@
(dock-btn :ws "1:spotify" :icon "🎵" :label "Spotify")
(dock-btn :ws "2:home" :icon "🏠" :label "Home")
(dock-btn :ws "3:web" :icon "🌐" :label "Web")
@DISCORD_DOCK_BTN@ (box :class "dock-sep")
(box :class "dock-sep")
(button :class "dock-btn" :onclick "toggle-keyboard"
(box :orientation "vertical" :space-evenly false :spacing 2
(label :class "dock-icon" :text "⌨")
(label :class "dock-label" :text "Keyboard")))
;; Not a button: nothing to press, it is a statement of fact. It takes no space
;; when the microphone is idle.
(box :class "mic-badge" :visible {mic_live == "1"} :space-evenly false :spacing 4
(label :class "mic-dot" :text "●")
(label :class "mic-text" :text "MIC LIVE"))))
(label :class "dock-label" :text "Keyboard")))))
;; :exclusive true reserves this strip so it is never overlapped by, and never steals
;; area from behind, the app on screen — unlike the thin client's overlay widgets,

47
hosts/touch-panel/configs/mic/mic-in-use
View File

@ -1,47 +0,0 @@
#!/bin/sh
# Is somebody recording this panel's microphone right now? Prints "1" or "0".
#
# Installed to /usr/local/bin/mic-in-use and polled by the touch dock
# (configs/eww/eww.yuck), which turns a "1" into a red LIVE badge.
#
# WHY THIS EXISTS: this panel can be used as a remote microphone by a desktop in
# another room (mic-follow/). That is a deliberate, useful feature, and it is also a
# microphone in a shared flat that somebody elsewhere can open. Such a thing has to be
# visible IN THE ROOM, not only in Home Assistant — anyone standing in the Loggia
# should be able to see that the panel is listening without knowing mic-follow exists.
#
# It reports what is actually capturing, from PipeWire, rather than trusting anything
# about who asked: a stray process recording for its own reasons lights the badge too,
# which is correct for an indicator whose whole job is to be believed.
#
# Monitor streams — something recording what the panel is PLAYING, like a visualiser —
# are not the microphone and must not light it up.
set -eu
command -v pactl >/dev/null 2>&1 || { echo 0; exit 0; }
if command -v jq >/dev/null 2>&1; then
sources="$(pactl -f json list sources 2>/dev/null || echo '[]')"
outputs="$(pactl -f json list source-outputs 2>/dev/null || echo '[]')"
jq -n --argjson s "$sources" --argjson o "$outputs" '
# The indexes of every monitor source, by either of the two ways one identifies
# itself.
[ $s[]
| select(((.properties["device.class"] // "") | ascii_downcase) == "monitor"
or (.name | endswith(".monitor")))
| .index ] as $monitors
| [ $o[] | select([.source] | inside($monitors) | not) ] | length > 0
| if . then 1 else 0 end
' 2>/dev/null || echo 0
exit 0
fi
# No jq: fall back to counting capture streams without being able to tell a monitor
# apart. That OVER-reports, which is the right direction to be wrong in for a warning
# light — a badge that is on too often gets questioned, one that is off too often gets
# trusted wrongly.
if pactl list source-outputs 2>/dev/null | grep -q "^Source Output #"; then
echo 1
else
echo 0
fi

24
hosts/touch-panel/configs/sway/config
View File

@ -14,7 +14,6 @@ set $mod Mod4
set $ws_spotify 1:spotify
set $ws_home 2:home
set $ws_web 3:web
set $ws_discord 4:discord
# Workspace names are a contract with touchpanel_agent/sway_control.py — changing one
# side means changing the other.
@ -108,23 +107,13 @@ exec_always eww open touch-dock
# separately-matched `assign` rule can't race it.
for_window [app_id="chromium.*"] fullscreen enable
for_window [app_id="spotify"] fullscreen enable
# Discord runs under XWayland, so it lands as an X11 window with a `class`, not a
# native `app_id` — matching on app_id alone would silently never fire.
for_window [class="discord"] fullscreen enable
# HA kiosk, Spotify and (where enabled) Discord all auto-launch at session start —
# this is a fixed panel of three or four apps, not an on-demand surface like the thin
# client's digest/admin workspaces, so there is no "nothing to show yet" state to
# guard against beyond an unset HA_URL.
# HA kiosk window and Spotify both auto-launch at session start — this is a fixed
# 3-app panel, not an on-demand surface like the thin client's digest/admin
# workspaces, so there is no "nothing to show yet" state to guard against beyond an
# unset HA_URL.
exec sh -c '[ -n "$HA_URL" ] && { swaymsg workspace $ws_home; /usr/local/bin/ha-kiosk; }'
exec sh -c 'swaymsg workspace $ws_spotify; /usr/local/bin/spotify-launch'
# Discord, on the panels that asked for it (kiosks[].enable_discord). Started at
# session boot rather than on demand for the reason in discord-launch's own header:
# the point is stepping out mid-call, and an app that needs fifteen seconds and a
# login is not something you use on the way past. $ENABLE_DISCORD is not a sway
# variable, so it passes through to sh, which reads it from the environment
# kiosk-session sourced out of /etc/touchpanel-agent/config.env.
exec sh -c '[ "$ENABLE_DISCORD" = "true" ] && { swaymsg workspace $ws_discord; /usr/local/bin/discord-launch; }'
exec swaymsg workspace $ws_home
# ---------------------------------------------------------------------------
@ -139,10 +128,6 @@ exec swayidle -w \
for_window [app_id="chromium.*"] inhibit_idle fullscreen
for_window [app_id="spotify"] inhibit_idle fullscreen
# Deliberately NOT inhibiting idle for Discord: a voice call would otherwise hold the
# panel's screen on for an hour in an empty room. The call keeps running with the
# output powered off — sway's idle timeout blanks the display, it does not suspend
# the machine or the audio stream.
# ---------------------------------------------------------------------------
# Local override keys — a fallback for standing in front of the machine (or a
@ -158,7 +143,6 @@ bindsym $mod+Shift+c reload
bindsym $mod+1 workspace $ws_spotify
bindsym $mod+2 workspace $ws_home
bindsym $mod+3 workspace $ws_web
bindsym $mod+4 workspace $ws_discord
bindsym XF86AudioPlay exec playerctl -p spotify play-pause
bindsym XF86AudioNext exec playerctl -p spotify next

37
hosts/touch-panel/live-build/config/hooks/normal/0350-flatpak-discord.hook.chroot
View File

@ -1,37 +0,0 @@
#!/bin/sh
# Discord from Flathub, for panels that ask for it (kiosks[].enable_discord).
#
# Gated rather than unconditional: a kitchen panel does not want a chat client, and
# the Loggia one is the whole reason this exists (see configs/discord/discord-launch).
# ENABLE_DISCORD comes from /etc/touchpanel-agent/config.env, which live-build has
# already copied in before hooks run — the same mechanism 0300-flatpak-spotify uses
# for nothing at all today, and the reason that file is written first.
set -eu
ENABLE_DISCORD=false
if [ -r /etc/touchpanel-agent/config.env ]; then
# shellcheck disable=SC1091
. /etc/touchpanel-agent/config.env
fi
if [ "${ENABLE_DISCORD:-false}" != "true" ]; then
echo "0350-flatpak-discord: ENABLE_DISCORD is not true — skipping."
exit 0
fi
flatpak remote-add --if-not-exists flathub https://dl.flathub.org/repo/flathub.flatpakrepo
# VERIFY BEFORE THE FIRST REAL BUILD: confirm this application ID against the live
# Flathub listing (`flatpak search Discord`). Believed correct (Discord's own
# published app) but not checked from this environment — same honesty as the Spotify
# hook next door.
DISCORD_APP_ID="com.discordapp.Discord"
if flatpak install -y --noninteractive flathub "$DISCORD_APP_ID"; then
echo "0350-flatpak-discord: installed ${DISCORD_APP_ID}."
else
echo "0350-flatpak-discord: WARNING — could not install ${DISCORD_APP_ID} during the"
echo " build (no network in the chroot, or the app ID is wrong). Run this on the"
echo " booted image instead: flatpak install -y flathub ${DISCORD_APP_ID}"
echo " The Discord workspace will be empty until this is done."
fi

7
hosts/touch-panel/live-build/config/package-lists/touch-panel.list.chroot
View File

@ -18,13 +18,6 @@ pipewire
pipewire-pulse
wireplumber
alsa-utils
# pw-record / pw-cat / pw-cli. Needed because this panel can act as a REMOTE
# MICROPHONE for a desktop in another room (mic-follow/): the desktop opens an SSH
# session and runs pw-record here. Nothing listens for that — it is an ordinary SSH
# command — and audio is only ever captured while that session is open.
pipewire-bin
# pactl, for checking what is capturing locally (mic-in-use, below).
pulseaudio-utils
# --- Spotify GUI client and Steam-Link-style apps come from Flathub, not apt
# (see 0300-flatpak-spotify.hook.chroot) ---

122
identity/README.md
View File

@ -72,95 +72,6 @@ actually decides who's registering. **If you want camera-based identity later,
Frigate's own face-recognition + enrollment (0.16+) is the piece to wire in, not a
new pipeline here.**
## How a person looks on a screen too small for their name
Every person carries two derived fields, served on `/people`, `/presence` and each
occupant in `/floorplan/presence`:
- **`initial`** — the first letter of the **real name**, never the nickname. Same rule
as `speak_name`, same reason: what a machine shows for somebody should follow who
they are, not what the household happens to call them this year.
- **`color`** — one of eight, assigned automatically at registration and editable in
the admin panel.
**The colour exists because initials collide.** A household with an Anna and an Amir
gets two identical "A"s on a wall panel or a watch face, and the colour is what makes
that readable. So the assignment rule is: first avoid any colour already worn by
somebody with the same initial — that is worth spending the whole palette on — then
take the least-used colour overall. Past eight people sharing one letter it repeats,
and you pick by hand.
**Why those eight values.** Every channel is `00`/`55`/`AA`/`FF`, which is exactly the
2-bits-per-channel space a colour Pebble renders natively. Anything else is snapped or
dithered by the watch, and a colour that shifts between the admin panel and the watch
defeats the point of having one. They also vary in lightness rather than only in hue,
so they stay distinguishable when a black-and-white screen reduces them to greys, and
they survive the common colour-vision deficiencies better than a rainbow would. A hex
outside the palette is accepted if you set one deliberately; it just won't be exact on
a watch. See `docs/pebble-presence-watchface.md`.
Existing people are backfilled with a colour at startup, oldest first, so the
assignment is stable across restarts and never reshuffles a colour somebody has
already learned.
## The photo, and where it comes from
Two ways a person gets a profile picture:
- **Registration** — the door panel's capture becomes the picture, "most recent
registration wins". Good for the picture registration took; useless for a device-less
household member registered by hand, who could never have one at all.
- **`POST /people/<id>/photo`** — upload one from the admin panel. Written into the
same photo directory with the same filename shape, so anything that serves or backs
up a registration capture handles this identically. Deliberately **not** recorded as
a registration event: nobody registered.
Removing a picture (`clear_photo`) blanks the reference but leaves the file on disk —
it may also be a `registration_events` audit artifact, and "stop showing this photo" is
a different request from "destroy the record that it was taken". The same reasoning
applies to replacing one.
A person with no picture is not a gap in a UI: their initial on their colour is the
fallback everywhere, and on the small screens it is the *preferred* rendering anyway.
## Exact positions inside a room
`/floorplan/presence` answers "which room" from BLE. It can also answer "**where in
it**", from a different sensor, and the two are kept apart on purpose.
| Signal | Source | Gives |
|---|---|---|
| who | BLE / Bermuda | a person, resolved to a room |
| where | mmWave radar (LD2450-class) | a coordinate, **with no name** |
**The fusion rule is deliberately timid:**
> exactly one occupant in the room **and** exactly one target in the room → that target
> is that person (`position.source: "fused"`). Anything else → targets stay anonymous
> in `room.targets`, occupants stay unpositioned, and `position_ambiguous` is true.
Two people in a room are two blobs that cannot be told apart. Guessing would put a name
on the wrong person, and a display that does that occasionally is worse than one that
never tries — you can't tell its wrong answers from its right ones.
Which room a target is in is computed **from the polygon**, never from which sensor saw
it: a radar in an open-plan kitchen sees into the living room, and attributing by sensor
puts people through walls.
Two things have to be drawn by a human first, because nothing can infer them:
- **`metres_wide` on the level** — the real width of its 0–1 extent. Without it,
positions are simply not computed, and the API says so rather than guessing a scale.
- **A sensor placement** (`POST /floorplan/sensors`): where the radar sits on the plan,
and **which way it faces**. A wrong `rotation_deg` mirrors every target it reports —
the most likely way to get positions that look plausible and are wrong.
Readings are assumed to be **millimetres** (ESPHome's LD2450 default);
`POSITION_UNIT_DIVISOR` overrides that, because "which unit is this number in" differs
between integrations and getting it wrong scales everything by a thousand instead of
failing visibly. A target at exactly (0,0) is how these radars say *nothing here*, so
those are dropped — otherwise every sensor grows a phantom person sitting on top of it.
## People without a device
Two paths, distinct on purpose because they solve different problems:
@ -648,12 +559,9 @@ not network placement.
| `POST /register` | `{"name", "device_id", "photo_id"?, "entity_id"?, "no_device"?}` -> registers, or returns a reason it couldn't (see above) |
| `POST /register/guest` | `{"device_id", "photo_id"?}` -> registers "Guest N", no name needed |
| `GET /people` | admin/audit list of every person: identifiers, device grants, chore assignments, `nickname`/`speak_name`, `last_visit_at`, `visit_count`, `currently_home_since` |
| `POST /people/<id>` | edit any editable field — `{"name"?, "nickname"?, "note"?, "color"?, "chore_exempt"?, "chore_reminder_style"?, "notify_on_arrival"?, "announce_arrivals"?, "notify_topic"?, "digest_sections"?, "clear_photo"?}`. Omitted keys are left alone; an empty `color` re-derives one rather than blanking it |
| `POST /people/<id>` | edit any editable field — `{"name"?, "nickname"?, "note"?, "chore_exempt"?, "chore_reminder_style"?, "notify_on_arrival"?, "announce_arrivals"?, "notify_topic"?, "digest_sections"?, "clear_photo"?}`. Omitted keys are left alone |
| `POST /people/<id>/test-notification` | push a test message to this person's ntfy topic, to prove it works |
| `GET /people/<id>/photo` | the person's profile picture (raw JPEG) |
| `POST /people/<id>/photo` | raw image bytes -> set the profile picture directly, without a walk to the door panel |
| `GET /person-colors` | the eight-colour palette the admin panel offers (see below) |
| `GET/POST /floorplan/sensors`, `DELETE /floorplan/sensors/<id>` | where each position radar sits on the plan, and which way it faces |
| `GET /people/<id>/photo` | the person's profile picture (raw JPEG) — their most recent registration photo |
| `POST /people/<id>/identifiers` | `{"entity_id"}` — attach an identifier by hand (a fixed BLE tag not in range yet). Still enforces `TRUSTED_ENTITY_PREFIXES` |
| `DELETE /people/<id>/identifiers/<id>` | revoke a mistaken or compromised identifier |
| `DELETE /people/<id>` | remove a person entirely (their identifiers, grants and visits go with them) |
@ -681,7 +589,7 @@ not network placement.
| `GET /floorplan/areas` | area values HA is currently reporting — the editor's pick list |
| `POST /presence/manual` | `{"person_id", "home"}` — hand-operated Home/Away for anyone with no identifiers |
| `POST /people/<id>/chore-settings` | `{"chore_exempt"?, "chore_reminder_style"?}` — see below; either field omitted/`null` leaves it unchanged |
| `GET /presence` | `{"people": [{"id", "name", "nickname", "speak_name", "home", "room", "has_device", "has_photo", "color", "initial", "chore_exempt", "chore_reminder_style", "chore_assignments"}], "generated_at"}` — `home` is `true`/`false`/`null` (unknown), `room` is best-effort floor-plan groundwork (see below) |
| `GET /presence` | `{"people": [{"id", "name", "nickname", "speak_name", "home", "room", "has_device", "has_photo", "chore_exempt", "chore_reminder_style", "chore_assignments"}], "generated_at"}` — `home` is `true`/`false`/`null` (unknown), `room` is best-effort floor-plan groundwork (see below) |
| `GET /weather` | proxies `smarthome/weather/current`, same JSON shape (`temperature`/`condition`/`location`) `hosts/thin-client`'s weather overlay already uses |
**Every person gets a profile picture, automatically** — whichever registration photo
@ -766,27 +674,3 @@ no way to send an `Authorization` header.
`digest-engine` needs all three places checked. `digest-engine`'s end of it (which
sections a run actually generates, and what a stopped `identity` does to a run) is
on that component's own verification list.
## Watch-reachable toggles
`GET /toggles` and `POST /toggles/<id>` (`{"action": "on"|"off"|"toggle"}`) read and flip
Home Assistant switches on behalf of the Pebble app — today, `mic-follow`'s Follow-me
microphone switches.
**It is an allowlist, not a proxy.** `TOGGLE_ALLOWLIST_JSON` names the exact entities;
anything not in it is a 404, only `switch.*` entities are accepted at all (an allowlist
that takes any domain is one that eventually contains a lock), and the only verbs are
on/off/toggle. The list is generated by `mic-follow/generate.py` — see
`mic-follow/README.md` §4.
It lives here for one reason: this service already holds an HA token, and the phone
already holds this service's. The alternative was an HA admin token in a watchapp's
settings to press one button, which is a much worse trade. Each entry may name a
`detail_entity` whose state is served alongside — for mic-follow that is the sensor whose
state is the microphone that is live right now, so a watch can display the answer instead
of deriving it.
Each response re-reads the state after the call rather than assuming it: "I sent the
command" is not the same fact as "the switch is on", and the caller is a watch that will
draw whatever this says.

22
identity/frontend/admin.html
View File

@ -203,28 +203,6 @@
<input id="edit-note" type="text" autocomplete="off" placeholder="e.g. cousin, visits at Christmas">
</label>
<fieldset class="field-group">
<legend>How they look on small screens</legend>
<p class="hint">
A wall panel or a watch face has no room for a name — it shows this colour and
the first letter of the real name. Two people whose names start the same letter
get different colours automatically; change one here if you'd rather pick.
</p>
<div class="field">Colour
<div id="edit-colors" class="swatch-row"></div>
</div>
<div class="field">Profile picture
<div class="photo-actions">
<input id="edit-photo-file" type="file" accept="image/*" hidden>
<button type="button" class="btn ghost" id="edit-photo-upload">Upload a picture</button>
<button type="button" class="btn ghost" id="edit-photo-clear">Remove</button>
</div>
<span class="hint">
Otherwise this is whatever the door panel captured at their last registration.
</span>
</div>
</fieldset>
<fieldset class="field-group">
<legend>Chores</legend>
<label class="check">

92
identity/frontend/admin.js
View File

@ -90,82 +90,6 @@ function duration(fromIso, toIso) {
// Same blob-fetch approach as register.js/dashboard.js — every identity endpoint
// requires an Authorization header, and a plain <img src="..."> has no way to send
// one. See identity/README.md.
// --- Colour + profile picture --------------------------------------------------------
// The palette comes from the server (GET /person-colors) rather than being repeated
// here: those eight values are chosen to render exactly on a colour Pebble's 2-bit-per-
// channel screen, and a second copy in this file would drift the first time somebody
// "improved" one of them. Falls back to whatever the person already has if the fetch
// fails, so the editor still opens.
let personColors = [];
function loadPersonColors() {
return api("/person-colors")
.then((data) => {
personColors = data.colors || [];
})
.catch(() => {
personColors = [];
});
}
function renderColorSwatches() {
const row = document.getElementById("edit-colors");
const current = (editing.color || "").toUpperCase();
const palette = personColors.length ? personColors : [current].filter(Boolean);
// A colour set by hand outside the palette still gets a swatch, so it is visible and
// reselectable rather than silently absent from its own editor.
const colors = palette.includes(current) || !current ? palette : palette.concat([current]);
row.innerHTML = colors
.map(
(color) =>
`<button type="button" class="swatch${color.toUpperCase() === current ? " selected" : ""}"
data-color="${escapeHtml(color)}" style="background:${escapeHtml(color)}"
title="${escapeHtml(color)}"><span>${escapeHtml(editing.initial || "")}</span></button>`
)
.join("");
row.querySelectorAll(".swatch").forEach((btn) =>
btn.addEventListener("click", () => {
editing.color = btn.dataset.color;
renderColorSwatches();
const avatar = document.getElementById("edit-avatar");
if (!editing.has_photo) avatar.style.background = editing.color;
})
);
}
document.getElementById("edit-photo-upload").addEventListener("click", () => {
document.getElementById("edit-photo-file").click();
});
document.getElementById("edit-photo-file").addEventListener("change", (event) => {
const file = event.target.files && event.target.files[0];
if (!file || !editing) return;
setStatus("Uploading…");
// Raw bytes, not multipart — same shape as the level-image upload and
// /register/photo. The picture is written immediately rather than waiting for Save,
// because it is a file on the server, not a field in this form.
api(`/people/${editing.id}/photo`, { method: "POST", body: file, headers: { "Content-Type": file.type || "image/jpeg" } })
.then((result) => {
if (!result.ok) throw new Error(result.message || "Could not upload.");
return refreshEditing("Picture updated.");
})
.catch((err) => setStatus(err.message, true))
.finally(() => {
event.target.value = "";
});
});
document.getElementById("edit-photo-clear").addEventListener("click", () => {
if (!editing) return;
setStatus("Removing…");
postJson(`/people/${editing.id}`, { clear_photo: true })
.then((result) => {
if (!result.ok) throw new Error(result.message || "Could not remove.");
return refreshEditing("Picture removed — showing their initial instead.");
})
.catch((err) => setStatus(err.message, true));
});
function loadAvatar(container, personId) {
fetch(`${API}/people/${personId}/photo`, { headers: { Authorization: `Bearer ${TOKEN}` } })
.then((res) => (res.ok ? res.blob() : Promise.reject()))
@ -230,9 +154,7 @@ function loadPeople() {
.map(
(p) =>
`<button type="button" class="card as-button" data-person="${p.id}">
<span class="avatar" data-avatar="${p.id}" style="background:${escapeHtml(
p.color || ""
)}">${escapeHtml(p.initial || "")}</span>
<span class="avatar" data-avatar="${p.id}">👤</span>
<span class="card-body">
<span class="card-name">${escapeHtml(p.name)}${p.currently_home_since ? ' <span class="dot-home" title="home now"></span>' : ""}</span>
<span class="card-meta">${escapeHtml(personSubtitle(p))}</span>
@ -286,12 +208,9 @@ function openEditor(personId) {
: '<span class="error">No ntfy topic reachable — set NTFY_URL and NTFY_DEFAULT_TOPIC on the server, or a topic here.</span>';
const avatar = document.getElementById("edit-avatar");
avatar.replaceChildren(document.createTextNode(editing.initial || "👤"));
avatar.style.background = editing.color || "";
avatar.replaceChildren(document.createTextNode("👤"));
if (editing.has_photo) loadAvatar(avatar, editing.id);
renderColorSwatches();
document.getElementById("edit-digest-sections").innerHTML = DIGEST_SECTIONS.map(
([key, label]) =>
`<label class="chip"><input type="checkbox" data-digest="${key}"${
@ -404,9 +323,6 @@ document.getElementById("edit-save").addEventListener("click", () => {
notify_on_arrival: document.getElementById("edit-notify-on-arrival").checked,
announce_arrivals: document.getElementById("edit-announce-arrivals").checked,
notify_topic: document.getElementById("edit-notify-topic").value.trim(),
// Whatever swatch is currently selected — the picker mutates `editing.color` and
// saves with the rest of the dialog, since it is one more column on `people`.
color: editing.color || "",
// Sent with the rest of the person's own fields rather than as a third call: unlike
// chore assignments (their own table), this is one column on `people`, so it saves
// or is refused together with everything else in the dialog.
@ -1076,6 +992,4 @@ document.getElementById("fp-image").addEventListener("change", (e) => {
document.getElementById("fp-live").addEventListener("change", loadFloorplanPresence);
// The palette first, so the first editor opened already has its swatches — it is one
// small request and it never has to be repeated.
loadPersonColors().then(loadPeople);
loadPeople();

41
identity/frontend/style.css
View File

@ -130,47 +130,6 @@ main.no-tabs {
object-fit: cover;
}
/* The initial is drawn on the person's own colour, so a household with an Anna and an
* Amir can be read at a glance on a surface with no room for names — the same pairing
* a floorplan marker and a watch face use. Dark text because every colour in
* PERSON_COLORS is mid-to-bright; a white glyph would vanish on the amber. */
.avatar {
color: #101014;
font-weight: 700;
}
.swatch-row {
display: flex;
flex-wrap: wrap;
gap: 8px;
margin-top: 6px;
}
.swatch {
width: 44px;
height: 44px;
border-radius: 50%;
border: 2px solid transparent;
cursor: pointer;
display: flex;
align-items: center;
justify-content: center;
color: #101014;
font-weight: 700;
font-size: 16px;
}
.swatch.selected {
border-color: #e8e8ec;
box-shadow: 0 0 0 2px rgba(0, 0, 0, 0.6) inset;
}
.photo-actions {
display: flex;
gap: 8px;
margin: 6px 0;
}
.card-meta {
font-size: 14px;
color: #9a9aa6;

16
identity/identity.env.example
View File

@ -119,19 +119,3 @@ IDENTITY_PHOTO_DIR=/data/photos
IDENTITY_FLOORPLAN_DIR=/data/floorplans
IDENTITY_MAX_IMAGE_MB=15
LOG_LEVEL=INFO
# --- Watch-reachable toggles -----------------------------------------------------------
# An ALLOWLIST of Home Assistant switches this service may read and flip on behalf of
# the Pebble app (GET /toggles, POST /toggles/<id>). Empty means the endpoints return
# nothing and accept nothing, which is the default.
#
# It exists so one button on a watch does not require an HA admin token on a phone:
# this service already has one. It must stay an allowlist — only `switch.*` entities
# are accepted, the verbs are on/off/toggle, and an id that is not listed is a 404.
#
# Generated by mic-follow/generate.py — copy the line out of
# mic-follow/generated/identity-toggles.env. One JSON array of
# {id, name, switch_entity, detail_entity}; detail_entity is what the watch DISPLAYS
# (for mic-follow, the sensor whose state is the microphone that is live right now).
TOGGLE_ALLOWLIST_JSON=

624
identity/server.py
View File

@ -61,9 +61,6 @@ Endpoints:
- POST /register the main call, described above
- GET /people admin/audit list of registered people + identifiers
- POST /people/<id> edit any editable field on a person (the admin panel)
- GET/POST /people/<id>/photo the profile picture — GET serves it, POST replaces
it from the admin panel without a walk to the door panel
- GET /person-colors the per-person colour palette (see PERSON_COLORS)
- DELETE /people/<id>/identifiers/<id> revoke a mistaken/compromised identifier
- GET /presence who's currently home, resolved from registered identifiers
- GET /resolve spoken name OR nickname -> the canonical person
@ -84,7 +81,6 @@ from __future__ import annotations
import json
import logging
import math
import os
import re
import sqlite3
@ -144,49 +140,6 @@ PRESENT_STATES = {"home"}
# project; there's no floor plan or room list to design against yet.
AREA_ATTRIBUTE = os.environ.get("AREA_ATTRIBUTE", "area_id")
# --- Watch-reachable toggles (mic-follow's Follow-me switches) --------------------------
# An ALLOWLIST, not a capability. The Pebble app needs to flip one Home Assistant
# switch, and it already talks to this service with this service's token — so this is
# the one place in the household that can do it without putting an HA admin token on a
# phone. What it must never become is a general-purpose HA proxy: only entities named
# in this list can be read or written, the verbs are on/off/toggle and nothing else,
# and an id that is not in the list is a 404 rather than a passthrough.
#
# Generated by tools/config-export.py from the mic_follow block; each entry is
# {id, name, switch_entity, detail_entity}. detail_entity is optional and is what the
# watch (and anything else) SHOWS: for mic-follow it is the status sensor whose state
# is the name of the microphone that is live right now.
def _load_toggle_allowlist() -> list[dict]:
raw = os.environ.get("TOGGLE_ALLOWLIST_JSON", "").strip()
if not raw:
return []
try:
parsed = json.loads(raw)
except json.JSONDecodeError:
LOG.warning("TOGGLE_ALLOWLIST_JSON is not valid JSON — no toggles will be served")
return []
allowed: list[dict] = []
for entry in parsed if isinstance(parsed, list) else []:
if not isinstance(entry, dict):
continue
toggle_id = str(entry.get("id") or "").strip()
switch_entity = str(entry.get("switch_entity") or "").strip()
# Only switches. An allowlist that accepts any domain is an allowlist that
# eventually contains a lock.
if not toggle_id or not switch_entity.startswith("switch."):
LOG.warning("ignoring toggle entry %r — needs an id and a switch.* entity", entry)
continue
allowed.append({
"id": toggle_id,
"name": str(entry.get("name") or toggle_id),
"switch_entity": switch_entity,
"detail_entity": str(entry.get("detail_entity") or "").strip(),
})
return allowed
TOGGLE_ALLOWLIST = _load_toggle_allowlist()
MAX_IMAGE_BYTES = int(os.environ.get("IDENTITY_MAX_IMAGE_MB", "15")) * 1024 * 1024
MAX_JSON_BYTES = 32 * 1024
@ -422,40 +375,8 @@ def init_db() -> None:
-- PDF, a photo of a sketch). Rooms can be drawn on bare canvas without
-- one; it just makes drawing them accurate rather than approximate.
image_path TEXT,
-- How wide this level is in REAL METRES, across its whole 0..1 extent.
-- Required for exact positions and for nothing else: a sensor reports
-- "target at 2.1m", and turning that into a point on a normalised plan
-- needs a scale. NULL means positions cannot be computed for this level,
-- which is reported honestly rather than guessed at.
metres_wide REAL,
created_at TEXT NOT NULL
);
-- WHERE A POSITION SENSOR PHYSICALLY SITS, on the plan.
--
-- A room-level presence signal answers "which room". Exact positions need a
-- sensor that reports coordinates — mmWave (LD2450-class) is the realistic
-- one — and its readings are RELATIVE TO ITSELF. So the plan has to know
-- where each sensor is and which way it faces, or the coordinates land
-- somewhere arbitrary. Drawn by a human in the admin panel, exactly like the
-- rooms, because nothing here can infer it.
CREATE TABLE IF NOT EXISTS floorplan_sensors (
id INTEGER PRIMARY KEY,
level_id INTEGER NOT NULL REFERENCES floorplan_levels(id) ON DELETE CASCADE,
name TEXT NOT NULL,
-- The HA entity prefix its targets appear under. An ESPHome LD2450
-- publishes sensor.<prefix>_target_1_x / _y and so on; this stores
-- <prefix> and the reader walks the targets.
ha_entity_prefix TEXT NOT NULL,
-- Its own spot on the plan, normalised 0..1 like the room polygons.
x REAL NOT NULL,
y REAL NOT NULL,
-- Which way it faces, degrees clockwise from "up" on the plan. Getting
-- this wrong mirrors or rotates every target it reports, which is the
-- single most likely reason positions look plausible but wrong.
rotation_deg REAL NOT NULL DEFAULT 0,
created_at TEXT NOT NULL
);
CREATE INDEX IF NOT EXISTS floorplan_sensors_level ON floorplan_sensors (level_id);
CREATE TABLE IF NOT EXISTS floorplan_rooms (
id INTEGER PRIMARY KEY,
level_id INTEGER NOT NULL REFERENCES floorplan_levels(id) ON DELETE CASCADE,
@ -515,99 +436,6 @@ def init_db() -> None:
_ensure_column(conn, "people", "announce_arrivals", "INTEGER NOT NULL DEFAULT 1")
_ensure_column(conn, "people", "notify_topic", "TEXT")
_ensure_column(conn, "people", "digest_sections", "TEXT")
_ensure_column(conn, "people", "color", "TEXT")
_ensure_column(conn, "floorplan_levels", "metres_wide", "REAL")
_backfill_colors(conn)
# --- Per-person colour -------------------------------------------------------------
# WHAT THIS IS FOR: telling two people apart at a glance on a surface too small for
# their name. The floorplan view marks an occupied room with each occupant's initial,
# and a household with an Anna and an Amir gets two identical "A"s — the colour is
# what makes that readable. Everything downstream (the admin panel, the dashboard's
# floorplan, and any watch/panel face built on GET /floorplan/presence) uses the same
# assignment rather than each picking its own, so a person is the same colour
# everywhere they appear.
#
# WHY THESE EIGHT: every channel is 0x00/0x55/0xAA/0xFF, which is exactly the 2-bit-
# per-channel colour space a colour Pebble renders natively (64 colours). Anything
# else gets dithered or snapped by the watch, and a colour that shifts between the
# admin panel and the watch defeats the entire point of having one. They are also
# spread across lightness, not just hue, so they stay distinguishable when a
# black-and-white watch reduces them to a grey — and for the same reason they survive
# the most common colour-vision deficiencies better than a rainbow would.
PERSON_COLORS = [
"#FF0000", # red
"#0055FF", # blue
"#FFAA00", # amber
"#00AA00", # green
"#AA00FF", # purple
"#00AAAA", # teal
"#FF55AA", # pink
"#AA5500", # brown
]
COLOR_RE = re.compile(r"^#[0-9a-fA-F]{6}$")
def _person_initial(name: str) -> str:
"""The single character a small display shows for this person.
First letter of the real name, never the nickname — the same rule speak_name
follows, and for the same reason: what a machine shows for somebody should be
derived from who they are, not from what the household happens to call them this
year. Non-alphanumeric leading characters are skipped so a name that starts with a
quote or an accent-mark artifact still yields a letter.
"""
for char in (name or "").strip():
if char.isalnum():
return char.upper()
return "?"
def _pick_color(conn: sqlite3.Connection, name: str, person_id: int | None = None) -> str:
"""The least-contended colour for this person.
Two rules, in order. First, avoid any colour already worn by somebody whose name
starts with the same letter — that collision is the entire reason this field
exists, and it is worth spending the whole palette on. Second, among what is left,
take the least-used colour overall, so a small household ends up with eight
distinct colours rather than three people sharing red.
Falls back to the least-used colour when the palette is exhausted (more than eight
people sharing an initial), because a repeat is better than an empty field — the
admin panel can always override it by hand.
"""
rows = conn.execute("SELECT id, name, color FROM people WHERE color IS NOT NULL").fetchall()
taken: dict[str, int] = {color: 0 for color in PERSON_COLORS}
same_initial: set[str] = set()
initial = _person_initial(name)
for row in rows:
if person_id is not None and row["id"] == person_id:
continue
color = str(row["color"]).upper()
taken[color] = taken.get(color, 0) + 1
if _person_initial(row["name"]) == initial:
same_initial.add(color)
preferred = [c for c in PERSON_COLORS if c not in same_initial] or PERSON_COLORS
return min(preferred, key=lambda c: (taken.get(c, 0), PERSON_COLORS.index(c)))
def _backfill_colors(conn: sqlite3.Connection) -> None:
"""Give everyone who predates this column a colour, oldest first.
Oldest first so the assignment is stable across restarts and matches the order the
household actually acquired people — re-running this must never reshuffle colours
somebody has already learned.
"""
rows = conn.execute(
"SELECT id, name FROM people WHERE color IS NULL OR color = '' ORDER BY id"
).fetchall()
for row in rows:
conn.execute("UPDATE people SET color = ? WHERE id = ?", (_pick_color(conn, row["name"], row["id"]), row["id"]))
if rows:
LOG.info("identity: assigned a colour to %d person/people that had none", len(rows))
def _ha_get(path: str):
@ -619,85 +447,6 @@ def _ha_get(path: str):
return json.loads(resp.read())
def _ha_post(path: str, payload: dict) -> None:
if not HA_TOKEN:
raise RuntimeError("HA_TOKEN is not configured")
req = urllib.request.Request(
f"{HA_URL}{path}",
data=json.dumps(payload).encode("utf-8"),
method="POST",
)
req.add_header("Authorization", f"Bearer {HA_TOKEN}")
req.add_header("Content-Type", "application/json")
urllib.request.urlopen(req, timeout=10).close()
def _toggle_by_id(toggle_id: str) -> dict | None:
for entry in TOGGLE_ALLOWLIST:
if entry["id"] == toggle_id:
return entry
return None
def _entity_state(entity_id: str) -> str:
if not entity_id:
return ""
try:
return str((_ha_get(f"/api/states/{entity_id}") or {}).get("state") or "")
except (urllib.error.URLError, urllib.error.HTTPError, RuntimeError, ValueError) as exc:
LOG.warning("could not read %s: %s", entity_id, exc)
return ""
def list_toggles() -> dict:
"""Every allowlisted toggle, with its state and the one line worth displaying.
`detail` is deliberately part of the payload rather than something each client
derives: a watch with 64 KB of RAM and a 72-pixel key on a desk should both be able
to show what is going on by printing a string somebody else already worked out.
"""
toggles = []
for entry in TOGGLE_ALLOWLIST:
state = _entity_state(entry["switch_entity"])
toggles.append({
"id": entry["id"],
"name": entry["name"],
"state": state or "unknown",
"on": state == "on",
"detail": _entity_state(entry["detail_entity"]) if entry["detail_entity"] else "",
})
return {"toggles": toggles}
def set_toggle(toggle_id: str, action: str) -> dict:
"""on | off | toggle, against one allowlisted switch. Returns the resulting state.
The state is re-read after the call rather than assumed: the caller is a watch that
will draw whatever this says, and "I sent the command" is not the same fact as
"the switch is on".
"""
entry = _toggle_by_id(toggle_id)
if entry is None:
return {"ok": False, "reason": "not_found"}
service = {"on": "turn_on", "off": "turn_off", "toggle": "toggle"}.get(action)
if service is None:
return {"ok": False, "reason": "bad_action"}
try:
_ha_post(f"/api/services/switch/{service}", {"entity_id": entry["switch_entity"]})
except (urllib.error.URLError, urllib.error.HTTPError, RuntimeError) as exc:
LOG.warning("toggle %s failed: %s", toggle_id, exc)
return {"ok": False, "reason": "home_assistant_unreachable"}
state = _entity_state(entry["switch_entity"])
return {
"ok": True,
"id": entry["id"],
"name": entry["name"],
"state": state or "unknown",
"on": state == "on",
"detail": _entity_state(entry["detail_entity"]) if entry["detail_entity"] else "",
}
def _trusted_present_candidates() -> list[dict]:
"""Entity_ids matching TRUSTED_ENTITY_PREFIXES whose current HA state indicates
presence right now. This is the ONLY source of registration candidates — see the
@ -770,10 +519,7 @@ def _find_or_create_person(conn: sqlite3.Connection, name: str) -> tuple[int, bo
raise AmbiguousName([m["name"] for m in matches])
if matches:
return matches[0]["id"], False
cur = conn.execute(
"INSERT INTO people (name, created_at, color) VALUES (?, ?, ?)",
(name, _now(), _pick_color(conn, name)),
)
cur = conn.execute("INSERT INTO people (name, created_at) VALUES (?, ?)", (name, _now()))
assert cur.lastrowid is not None
return cur.lastrowid, True
@ -1003,10 +749,6 @@ def _person_payload(conn: sqlite3.Connection, person: sqlite3.Row) -> dict:
# filesystem detail — just whether GET /people/<id>/photo has
# anything to serve.
"has_photo": person["photo_path"] is not None,
# How a display that has no room for a name shows this person: their colour and
# their initial, decided here so every surface agrees. See PERSON_COLORS.
"color": person["color"] or PERSON_COLORS[0],
"initial": _person_initial(person["name"]),
"chore_exempt": bool(person["chore_exempt"]),
"chore_reminder_style": person["chore_reminder_style"],
"chore_assignments": [a["chore_type"] for a in assignments],
@ -1314,26 +1056,6 @@ def update_person(person_id: int, fields: dict) -> dict:
("digest_sections", ",".join(s for s in DIGEST_SECTIONS if s in cleaned))
)
if "color" in fields:
color = str(fields["color"] or "").strip()
if not color:
# Emptying the field re-derives one rather than leaving a person with
# no colour: every surface that draws people needs *a* colour, and a
# blank would only push that decision into four different renderers.
updates.append(("color", _pick_color(conn, person["name"], person_id)))
elif not COLOR_RE.match(color):
return {
"ok": False,
"reason": "bad_color",
"message": "A colour has to look like #RRGGBB.",
}
else:
# Any valid hex is accepted, but only PERSON_COLORS render exactly on a
# colour Pebble (2 bits per channel) — anything else is snapped by the
# watch and will not match what the admin panel shows. The panel offers
# the palette; this permits going outside it knowingly.
updates.append(("color", color.upper()))
for text_field in ("chore_reminder_style", "note"):
if text_field in fields:
value = fields[text_field]
@ -1470,38 +1192,6 @@ def get_person_photo(person_id: int) -> bytes | None:
return path.read_bytes()
def set_person_photo(person_id: int, image: bytes) -> dict:
"""Set someone's profile picture directly, from the admin panel.
Until this existed the only way to get a profile picture was to walk to the door
panel and re-register, because `_set_profile_photo()` only ever runs on the
registration path ("most recent registration photo wins"). That is a fine rule for
the picture the *registration* captured and a poor one for "this is what this
person looks like" — a device-less household member registered by hand had no way
to have a face at all.
Written into the same PHOTO_DIR as a registration capture and with the same
filename shape, so anything that serves or backs up one serves and backs up the
other. It is deliberately NOT recorded as a registration_event: nobody registered.
"""
with _db_lock, _db() as conn:
person = conn.execute("SELECT id FROM people WHERE id = ?", (person_id,)).fetchone()
if person is None:
return {"ok": False, "reason": "no_such_person", "message": "No person with that id."}
PHOTO_DIR.mkdir(parents=True, exist_ok=True)
photo_path = PHOTO_DIR / f"{int(time.time())}{os.getpid() % 10000:04d}.jpg"
photo_path.write_bytes(image)
# The previous file is left on disk on purpose, the same reasoning as
# clear_photo's: it may also be a registration_events audit artifact, and
# replacing a profile picture is not a request to destroy the record that an
# earlier one was taken.
conn.execute("UPDATE people SET photo_path = ? WHERE id = ?", (str(photo_path), person_id))
updated = conn.execute("SELECT * FROM people WHERE id = ?", (person_id,)).fetchone()
LOG.info("identity: set profile photo for person %d (%d bytes)", person_id, len(image))
return {"ok": True, "person": _person_payload(conn, updated)}
def delete_identifier(person_id: int, identifier_id: int) -> bool:
with _db_lock, _db() as conn:
cur = conn.execute(
@ -1644,11 +1334,6 @@ def presence() -> dict:
"room": room,
"has_device": bool(person["identifiers"]),
"has_photo": person["has_photo"],
# Carried through from /people so a consumer that only ever calls
# /presence (the floorplan, a watch face) can draw somebody without a
# second request per person — see PERSON_COLORS.
"color": person["color"],
"initial": person["initial"],
"face_seen_recently": face_seen,
# Chore-system settings, straight passthrough — see chores/README.md
# for how these are used (never anything presence-related itself).
@ -2153,75 +1838,6 @@ def _validate_points(raw) -> tuple[list[list[float]] | None, str]:
return cleaned, ""
# How many targets to read per sensor. LD2450-class radars track three; asking for more
# is free and simply finds nothing.
POSITION_TARGETS_PER_SENSOR = int(os.environ.get("POSITION_TARGETS_PER_SENSOR", "3"))
# Readings are published in millimetres by ESPHome's LD2450 component. Overridable
# because "which unit is this number in" is exactly the kind of thing that differs
# between one integration and the next, and getting it wrong scales every position by
# a thousand rather than failing visibly.
POSITION_UNIT_DIVISOR = float(os.environ.get("POSITION_UNIT_DIVISOR", "1000"))
def _sensor_targets(states_by_id: dict, prefix: str) -> list[dict]:
"""The live targets one sensor reports, in metres relative to itself.
A target at exactly (0, 0) is how these radars say "nothing here" — they publish
zero rather than going unavailable — so those are dropped. Treating them as a real
detection would put a phantom person on top of every sensor on the plan.
"""
targets = []
for index in range(1, POSITION_TARGETS_PER_SENSOR + 1):
raw_x = states_by_id.get(f"sensor.{prefix}_target_{index}_x")
raw_y = states_by_id.get(f"sensor.{prefix}_target_{index}_y")
try:
x = float(raw_x) / POSITION_UNIT_DIVISOR
y = float(raw_y) / POSITION_UNIT_DIVISOR
except (TypeError, ValueError):
continue
if abs(x) < 0.01 and abs(y) < 0.01:
continue
targets.append({"x": x, "y": y})
return targets
def _plan_position(sensor, target, metres_wide: float) -> dict | None:
"""A sensor-relative reading, placed on the normalised plan.
Rotate by the sensor's own bearing, scale metres into plan units, offset by where
the sensor sits. Returns None when the result lands outside the plan, which is the
honest outcome for a bad rotation or a wrong scale — better a missing marker than a
confident one in the garden.
"""
if not metres_wide or metres_wide <= 0:
return None
angle = math.radians(sensor["rotation_deg"] or 0)
# Plan y grows downward, so a target "in front of" the sensor moves it up-plan.
rx = target["x"] * math.cos(angle) - target["y"] * math.sin(angle)
ry = target["x"] * math.sin(angle) + target["y"] * math.cos(angle)
x = sensor["x"] + rx / metres_wide
y = sensor["y"] - ry / metres_wide
if not (-0.05 <= x <= 1.05 and -0.05 <= y <= 1.05):
return None
return {"x": round(x, 4), "y": round(y, 4)}
def _point_in_polygon(x: float, y: float, points: list) -> bool:
"""Standard ray casting. Which room a target is in has to be computed rather than
assumed from which sensor saw it: a radar in an open-plan kitchen sees into the
living room, and attributing by sensor would put people through walls."""
inside = False
n = len(points)
for i in range(n):
x1, y1 = points[i]
x2, y2 = points[(i + 1) % n]
if (y1 > y) != (y2 > y):
xin = (x2 - x1) * (y - y1) / ((y2 - y1) or 1e-9) + x1
if x < xin:
inside = not inside
return inside
def list_floorplan() -> dict:
with _db_lock, _db() as conn:
levels = conn.execute(
@ -2239,7 +1855,6 @@ def list_floorplan() -> dict:
"name": level["name"],
"sort_order": level["sort_order"],
"has_image": level["image_path"] is not None,
"metres_wide": level["metres_wide"],
"rooms": [
{
"id": r["id"],
@ -2255,24 +1870,22 @@ def list_floorplan() -> dict:
return {"levels": result, "generated_at": _now()}
def save_level(level_id: int | None, name: str, sort_order: int,
metres_wide: float | None = None) -> dict:
def save_level(level_id: int | None, name: str, sort_order: int) -> dict:
name = (name or "").strip()
if not name:
return {"ok": False, "reason": "bad_name", "message": "A level needs a name."}
with _db_lock, _db() as conn:
if level_id:
cur = conn.execute(
"UPDATE floorplan_levels SET name = ?, sort_order = ?, metres_wide = ? WHERE id = ?",
(name, sort_order, metres_wide, level_id),
"UPDATE floorplan_levels SET name = ?, sort_order = ? WHERE id = ?",
(name, sort_order, level_id),
)
if cur.rowcount == 0:
return {"ok": False, "reason": "not_found", "message": "No such level."}
else:
cur = conn.execute(
"INSERT INTO floorplan_levels (name, sort_order, metres_wide, created_at) "
"VALUES (?, ?, ?, ?)",
(name, sort_order, metres_wide, _now()),
"INSERT INTO floorplan_levels (name, sort_order, created_at) VALUES (?, ?, ?)",
(name, sort_order, _now()),
)
level_id = cur.lastrowid
return {"ok": True, "level_id": level_id, "name": name}
@ -2375,24 +1988,6 @@ def floorplan_presence() -> dict:
resolved at all, which is the normal case without room-level BLE — come back under
`unplaced` rather than being dropped, so the UI can show "3 people home, 1 not
locatable" instead of quietly losing two of them.
EXACT POSITIONS ARE A SECOND, WEAKER SIGNAL, AND ARE KEPT SEPARATE
------------------------------------------------------------------
If the level has position sensors and a metre scale, each room also gets `targets`:
coordinates where a radar can see *somebody*. Those are ANONYMOUS — an mmWave sensor
reports a moving blob, not a name — and identity comes from BLE, which is room-level.
So attribution follows one rule, and the rule is deliberately timid:
exactly one occupant in the room AND exactly one target in the room
-> that target is that person, marked `source: "fused"`
anything else
-> the targets stay anonymous and the occupants stay unpositioned
Two people in a room produce two blobs that cannot be told apart, and guessing
which is which would put a name on a stranger — the failure that makes a presence
display worse than none. `position_ambiguous` says out loud when that has happened,
so a UI can show two unnamed dots rather than implying it knows.
"""
plan = list_floorplan()
people = presence().get("people", [])
@ -2402,45 +1997,18 @@ def floorplan_presence() -> dict:
for person in people:
if person.get("home") is not True:
continue
# Everything a floorplan occupant marker needs, and nothing else: a name for
# the surfaces with room for one, an initial and a colour for the ones without
# (a wall panel at a glance, a watch face — see docs/pebble-presence-watchface.md).
entry = {
"id": person["id"],
"name": person["name"],
"initial": person["initial"],
"color": person["color"],
"has_photo": person["has_photo"],
}
entry = {"id": person["id"], "name": person["name"], "has_photo": person["has_photo"]}
area = person.get("room")
if area:
by_area.setdefault(str(area), []).append(entry)
else:
unplaced.append(entry)
# Live targets, per level, projected onto the plan. Best-effort: no sensors, no
# scale, or an unreachable HA all end in the same place — rooms with no `targets`,
# which the renderer draws exactly as it always did.
targets_by_level = _live_targets(plan)
placed_areas = set()
for level in plan["levels"]:
level_targets = targets_by_level.get(level["id"], [])
for room in level["rooms"]:
occupants = by_area.get(room["ha_area_id"] or "", [])
room["occupants"] = occupants
# Which room a target is in is computed from the polygon, never from which
# sensor saw it — a radar in an open-plan kitchen sees into the living room.
inside = [t for t in level_targets if _point_in_polygon(t["x"], t["y"], room["points"])]
room["targets"] = inside
room["position_ambiguous"] = len(inside) > 1 or (len(inside) >= 1 and len(occupants) > 1)
if len(inside) == 1 and len(occupants) == 1:
occupants[0]["position"] = {**inside[0], "source": "fused"}
# Attributed, so it is no longer an unnamed dot for the renderer to draw.
room["targets"] = []
if room["ha_area_id"] and occupants:
placed_areas.add(room["ha_area_id"])
@ -2452,100 +2020,9 @@ def floorplan_presence() -> dict:
plan["unplaced"] = unplaced
plan["unmapped_areas"] = sorted(unmapped)
plan["positions_available"] = any(targets_by_level.values())
return plan
def _live_targets(plan: dict) -> dict[int, list[dict]]:
"""{level_id: [{x, y}]} — every position sensor's current targets, on the plan.
Read from the same HA state dump presence() uses. Fails soft in every direction:
an unreachable HA, a level with no metre scale, a sensor whose entities do not
exist yet — all produce no targets, and the floorplan renders room-level as before.
"""
result: dict[int, list[dict]] = {}
with _db_lock, _db() as conn:
sensors = conn.execute("SELECT * FROM floorplan_sensors").fetchall()
if not sensors:
return result
try:
states = _ha_get("/api/states")
except Exception:
LOG.warning("identity: could not read HA states for position sensors", exc_info=True)
return result
states_by_id = {s.get("entity_id"): s.get("state") for s in states if isinstance(s, dict)}
scale_by_level = {level["id"]: level.get("metres_wide") for level in plan["levels"]}
for sensor in sensors:
metres_wide = scale_by_level.get(sensor["level_id"])
if not metres_wide:
# A sensor on a level with no scale cannot be placed. Logged once per run
# rather than silently skipped, because the fix is one number in the editor.
LOG.info("identity: level %s has no metres_wide, so sensor %r cannot be placed",
sensor["level_id"], sensor["name"])
continue
for target in _sensor_targets(states_by_id, sensor["ha_entity_prefix"]):
point = _plan_position(sensor, target, float(metres_wide))
if point:
result.setdefault(sensor["level_id"], []).append(point)
return result
def save_sensor(payload: dict) -> dict:
"""Place (or move) a position sensor on the plan. Same human-drawn discipline as
the rooms: nothing here can infer where a radar is bolted to a wall."""
try:
level_id = int(payload.get("level_id"))
except (TypeError, ValueError):
return {"ok": False, "reason": "bad_field", "message": "'level_id' is required."}
name = str(payload.get("name") or "").strip()
prefix = str(payload.get("ha_entity_prefix") or "").strip()
if not name or not prefix:
return {"ok": False, "reason": "bad_field",
"message": "'name' and 'ha_entity_prefix' are both required. The prefix is "
"the part before _target_1_x in the sensor's entity ids."}
try:
x = float(payload.get("x"))
y = float(payload.get("y"))
rotation = float(payload.get("rotation_deg", 0))
except (TypeError, ValueError):
return {"ok": False, "reason": "bad_field", "message": "'x', 'y' must be numbers."}
if not (0 <= x <= 1 and 0 <= y <= 1):
return {"ok": False, "reason": "bad_field",
"message": "x and y are normalised 0..1, the same coordinate space the "
"room polygons use."}
with _db_lock, _db() as conn:
sensor_id = payload.get("id")
if sensor_id:
conn.execute(
"UPDATE floorplan_sensors SET name = ?, ha_entity_prefix = ?, x = ?, y = ?, "
"rotation_deg = ? WHERE id = ?",
(name, prefix, x, y, rotation, int(sensor_id)),
)
else:
cur = conn.execute(
"INSERT INTO floorplan_sensors (level_id, name, ha_entity_prefix, x, y, "
"rotation_deg, created_at) VALUES (?, ?, ?, ?, ?, ?, ?)",
(level_id, name, prefix, x, y, rotation, _now()),
)
sensor_id = cur.lastrowid
return {"ok": True, "id": sensor_id}
def list_sensors() -> dict:
with _db_lock, _db() as conn:
rows = conn.execute("SELECT * FROM floorplan_sensors ORDER BY level_id, name").fetchall()
return {"sensors": [dict(r) for r in rows]}
def delete_sensor(sensor_id: int) -> bool:
with _db_lock, _db() as conn:
return conn.execute("DELETE FROM floorplan_sensors WHERE id = ?", (sensor_id,)).rowcount > 0
def area_suggestions() -> dict:
"""Every area value HA is currently reporting on a trusted entity, so the room
editor can offer a pick-list instead of asking someone to retype an area_id from
@ -2722,17 +2199,6 @@ class Handler(BaseHTTPRequestHandler):
self._respond(HTTPStatus.OK, floorplan_presence())
elif path == "/floorplan/areas":
self._respond(HTTPStatus.OK, area_suggestions())
elif path == "/floorplan/sensors":
self._respond(HTTPStatus.OK, list_sensors())
elif path == "/toggles":
# The Pebble app's second screen. See TOGGLE_ALLOWLIST for why this is a
# fixed list rather than a proxy.
self._respond(HTTPStatus.OK, list_toggles())
elif path == "/person-colors":
# Served rather than duplicated in the admin panel's JS, so the palette
# has one definition — see PERSON_COLORS for what makes these eight
# specific values the palette.
self._respond(HTTPStatus.OK, {"colors": PERSON_COLORS})
elif level_image_match:
self._handle_level_image(int(level_image_match.group(1)))
elif assignments_match:
@ -2795,23 +2261,6 @@ class Handler(BaseHTTPRequestHandler):
self.end_headers()
self.wfile.write(data)
def _handle_set_person_photo(self, person_id: int) -> None:
"""POST /people/<id>/photo — raw image bytes, same shape as the level-image
upload and as /register/photo. No multipart: every client of this API is
either this project's own JS or a curl, and multipart parsing in the stdlib is
more failure surface than a Blob body is inconvenience.
"""
try:
data = self._read_body(MAX_IMAGE_BYTES)
except ValueError as exc:
self._respond(HTTPStatus.REQUEST_ENTITY_TOO_LARGE, {"error": str(exc)})
return
if not data:
self._respond(HTTPStatus.BAD_REQUEST, {"error": "empty image body"})
return
result = set_person_photo(person_id, data)
self._respond(HTTPStatus.OK if result.get("ok") else HTTPStatus.NOT_FOUND, result)
def _handle_level_image(self, level_id: int) -> None:
data = get_level_image(level_id)
if data is None:
@ -2840,15 +2289,11 @@ class Handler(BaseHTTPRequestHandler):
identifiers_match = re.match(r"^/people/(\d+)/identifiers$", path)
test_notify_match = re.match(r"^/people/(\d+)/test-notification$", path)
person_match = re.match(r"^/people/(\d+)$", path)
person_photo_match = re.match(r"^/people/(\d+)/photo$", path)
post_level_image_match = re.match(r"^/floorplan/levels/(\d+)/image$", path)
# /people/prune is checked before the bare /people/<id> edit route so it is
# never parsed as a person id (it can't be — it's not digits — but the ordering
# makes the intent explicit rather than incidental).
toggle_match = re.match(r"^/toggles/([A-Za-z0-9_-]+)$", path)
if toggle_match:
self._handle_set_toggle(toggle_match.group(1))
elif path == "/register/photo":
if path == "/register/photo":
self._handle_register_photo()
elif path == "/register":
self._handle_register()
@ -2879,12 +2324,8 @@ class Handler(BaseHTTPRequestHandler):
self._handle_save_level()
elif path == "/floorplan/rooms":
self._handle_save_room()
elif path == "/floorplan/sensors":
self._handle_save_sensor()
elif post_level_image_match:
self._handle_upload_level_image(int(post_level_image_match.group(1)))
elif person_photo_match:
self._handle_set_person_photo(int(person_photo_match.group(1)))
elif person_match:
self._handle_update_person(int(person_match.group(1)))
else:
@ -2964,28 +2405,13 @@ class Handler(BaseHTTPRequestHandler):
except (TypeError, ValueError):
self._respond(HTTPStatus.BAD_REQUEST, {"error": "'id' and 'sort_order' must be integers"})
return
# The real-world width of the level, needed for exact positions and nothing
# else. Absent leaves it NULL, which reports honestly as "positions cannot be
# computed here" rather than guessing a scale.
try:
metres_wide = float(payload["metres_wide"]) if payload.get("metres_wide") else None
except (TypeError, ValueError):
self._respond(HTTPStatus.BAD_REQUEST, {"error": "'metres_wide' must be a number"})
return
result = save_level(level_id, str(payload.get("name", "")), sort_order, metres_wide)
result = save_level(level_id, str(payload.get("name", "")), sort_order)
self._respond(
HTTPStatus.OK if result.get("ok")
else (HTTPStatus.NOT_FOUND if result.get("reason") == "not_found" else HTTPStatus.CONFLICT),
result,
)
def _handle_save_sensor(self) -> None:
payload = self._json_body()
if payload is None:
return
result = save_sensor(payload)
self._respond(HTTPStatus.OK if result.get("ok") else HTTPStatus.BAD_REQUEST, result)
def _handle_save_room(self) -> None:
payload = self._json_body()
if payload is None:
@ -3040,13 +2466,6 @@ class Handler(BaseHTTPRequestHandler):
return
path = urlsplit(self.path).path
sensor_match = re.match(r"^/floorplan/sensors/(\d+)$", path)
if sensor_match:
ok = delete_sensor(int(sensor_match.group(1)))
self._respond(HTTPStatus.OK if ok else HTTPStatus.NOT_FOUND,
{"ok": True} if ok else {"error": "no such sensor"})
return
id_match = re.match(r"^/people/(\d+)/identifiers/(\d+)$", path)
if id_match:
ok = delete_identifier(int(id_match.group(1)), int(id_match.group(2)))
@ -3168,31 +2587,6 @@ class Handler(BaseHTTPRequestHandler):
ok = set_manual_presence(person_id, home)
self._respond(HTTPStatus.OK if ok else HTTPStatus.NOT_FOUND, {"ok": ok})
def _handle_set_toggle(self, toggle_id: str) -> None:
"""POST /toggles/<id> {"action": "on"|"off"|"toggle"}
`toggle` is the default because the caller is usually a button being pressed by
somebody who can see the current state — on the watch, on a dock key — and
"flip it" is what a button means. Explicit on/off exists so an automation can be
idempotent.
"""
payload = self._json_body()
if payload is None:
return
action = str(payload.get("action") or "toggle").strip().lower()
result = set_toggle(toggle_id, action)
if result.get("ok"):
self._respond(HTTPStatus.OK, result)
elif result.get("reason") == "not_found":
# Deliberately the same answer as an id that does not exist: whether a
# given entity is merely absent from the allowlist is not something an
# unauthenticated-in-spirit surface should be able to probe for.
self._respond(HTTPStatus.NOT_FOUND, {"error": "no such toggle"})
elif result.get("reason") == "bad_action":
self._respond(HTTPStatus.BAD_REQUEST, {"error": "action must be on, off or toggle"})
else:
self._respond(HTTPStatus.BAD_GATEWAY, {"error": "Home Assistant did not answer"})
def _handle_chore_settings(self, person_id: int) -> None:
payload = self._json_body()
if payload is None:

192
mic-follow/README.md
View File

@ -1,192 +0,0 @@
# mic-follow — the microphone follows the person
A voice call that survives walking out of the room. The person leaves their desk, the
locator says which room they are in now, and their machine's live microphone switches
to one that can hear them there — then back to the good desk mic when they sit down.
Built for the specific case of a smoke break on the Loggia during a game, and written
**per client** so a second person with a second desktop is one more entry in
`CoreSystemConfig.json` and nothing else.
## 1. The one rule
**Off means the desk mic.**
The Follow-me switch being off is not "ignore me", it is an active guarantee that the
machine is on its own microphone. Being live on the wrong mic is the failure you notice
in front of five other people, so the safe state is reachable by one tap on the dock,
one button on a watch, or one MQTT message — and it does not depend on presence being
right, on the remote machine being up, or on the agent having seen a recent update.
The agent also returns to the desk mic when it shuts down.
## 2. Who decides what
```
locator (BLE via identity; RuView/Frigate corroborate — see §3)
│ presence_entity's state is an area_id
▼
Home Assistant ── generated automation ──> select.mic_follow_<node>_input
│ (only while switch.mic_follow_<node>_armed is on)
▼ MQTT
desktop_agent.py ──> pactl: set the default source, and MOVE the running
capture streams of the configured applications
```
Home Assistant decides **where the person is and therefore which mic should be live**.
The agent knows **how to change the input on this machine** and nothing about presence,
people or rooms beyond the names in its own config. Same rule as every other agent in
this repo: the inbound control surface is MQTT discovery entities and nothing else.
**Moving the streams matters as much as setting the default.** Changing the default
source only affects applications that asked for "default", and anyone who owns a studio
mic has picked it explicitly in Discord. `move_streams` names the applications whose
already-running capture stream gets moved too.
## 3. What the locator can and cannot tell you
`presence_entity` must be an entity whose **state is an HA `area_id`**
(`docs/rooms-and-endpoints.md`). Getting one is the part of this that is not built here:
| Source | Gives | Usable to drive this? |
|---|---|---|
| **BLE / Bermuda**, surfaced by `identity`'s `/presence` | a *person*, resolved to a room | **Yes** — the only identity-bearing room-level source in this household |
| **RuView** (CSI radar) | a room is occupied, by somebody | No on its own — it cannot say who. Good for corroboration |
| **Frigate** | a recognised face at the door | No — it is a peephole camera, not per-room |
So this rides on BLE, with everything else as confirmation. That matters because
**room-level presence has never been measured in this flat** — the repo says so
elsewhere and it is still true. Before trusting it with your microphone, watch the
entity in Developer Tools while you walk to the Loggia and back. If it lags by a minute
or reports a friendly room name instead of an `area_id`, the automation will never fire
and the mic will never move.
`dwell_seconds` (default 20) is the guard against a twitchy locator: a walk past a door
should not move your microphone. `return_dwell_seconds` (default 5) is deliberately
shorter — sitting down should give you the good mic back before you say anything into
it.
## 4. Three ways to flip it, one place to read it
Every surface shows the same entity: `sensor.mic_follow_<node>_status`, whose **state is
the name of the microphone that is live right now** — `Desk`, `Loggia`. That sensor
exists so no surface has to reimplement the same three-way template, and so the answer
to "what am I being heard through" is one string anything can print.
- **Stream Dock key** — displays that sensor and calls `switch.toggle`. The
Home Assistant plugin subscribes to HA's websocket, so the key updates when the state
changes rather than on a timer: **nothing polls**, and the key is right within a moment
of the mic actually moving, including when it was the watch or the automation that
moved it. Bindings are generated into `generated/dock-bindings.md`; leave the key's
title empty and let the state be the whole label, because `Loggia` in large type is
readable across a room and `Follow-me mic` over a small `Loggia` is not.
- **Pebble watchapp** — long-press Select on the plan screen. The screen shows the live
microphone in large type, `FOLLOWING` or `DESK ONLY` as a coloured pill, and Select
flips it. Up/Down cycles clients if there is more than one.
- **Home Assistant** — the switch and the select, like anything else.
The watch goes through `identity`'s `/toggles`, not straight at Home Assistant: identity
already holds an HA token and the phone already holds identity's, so one button on a
wrist does not put an HA admin token in a watchapp's settings. It is an **allowlist**,
not a proxy — only `switch.*` entities that are named in `TOGGLE_ALLOWLIST_JSON`, only
on/off/toggle, and an id that is not listed is a 404. Paste the line from
`generated/identity-toggles.env` into `identity.env` or the watch sees nothing.
## 5. Setting it up
1. Fill in `mic_follow` in `CoreSystemConfig.json`. Get the `desk_source` and each
`source` string by running `mic-follow/desktop_agent.py --list-sources` on the
machine in question — any part of a name or description matches.
2. `tools/validate-config.py`.
3. On each client machine: `mic-follow/setup-client.sh <node_id>`.
4. Put `generated/ha-package/mic_follow.yaml` in HA's `packages/` (the same directory
`stream-dock` uses) and reload YAML.
5. Paste `generated/identity-toggles.env` into `identity.env`, restart identity.
6. Bind the dock key from `generated/dock-bindings.md`.
## 6. A remote microphone — the Loggia case
The Loggia has no microphone you can plug into the desktop, but it does have a machine
with one: the touch panel that is already in there. `remote-mic.sh` makes that panel's
microphone appear on the desktop as an ordinary audio source, which the switch then
selects like any other.
```
panel: pw-record (its mic) ──ssh──> desktop: pw-cat --playback
into a null sink
│
module-remap-source ──────────┘
presented as a real source named micfollow_<room>
```
Configure it with a `remote` block on the source instead of a `source` pattern, and
the generator writes both hooks and the transport's parameters:
```jsonc
{ "room": "loggia",
"description": "Loggia panel mic",
"remote": { "host": "touch-panel-loggia", "user": "kiosk", "latency_ms": 40 } }
```
**Why SSH and not an audio protocol.** RTP and PipeWire's pulse-tunnel are both lower
latency, and both need a new listening service on the panel, an ACL, and config on two
machines that has to agree. The panel already runs sshd as its documented admin path
and already trusts this desktop's key, so the transport arrives with authentication and
encryption already solved and nothing new listening on the network. For a smoke-break
voice call, 40 ms of buffering is not the constraint — a hot microphone in an empty room
is. If you later want the latency, the null-sink half stays and only the transport
changes.
**Audio exists only while the stream runs.** There is no daemon: `start` opens one SSH
session, `stop` closes it, and the panel's microphone is not being read at any other
time. That is a property of the transport rather than a promise in a config file, which
is the reason to prefer it. The agent runs the stop hook on every transition away, when
a start fails, and on shutdown.
**And the panel says so, in the room.** `hosts/touch-panel/` now ships `mic-in-use` and
a red **MIC LIVE** badge on the touch dock, lit whenever anything is capturing that
panel's microphone — this feature or not. A room microphone somebody elsewhere can open
has to be visible to the person standing in front of it, and Home Assistant knowing is
not the same as them knowing. It reads PipeWire's actual capture streams rather than
trusting who asked, ignores monitor streams (recording what the panel is *playing* is
not the microphone), and without `jq` it falls back to over-reporting — the right
direction to be wrong in for a warning light.
**Why a null sink plus `module-remap-source`** rather than just using the sink's
monitor: a monitor is not a real source, applications treat it as "record what the
desktop is playing", and this component's own audio layer refuses to select one on
purpose (`test_selection.py`). remap-source turns it into an ordinary microphone with a
name and a description, which is what Discord's device list needs.
**The one command that decides whether any of this works:**
```
ssh <user>@<host> pw-record --help
```
That is the whole transport. If it works, the microphone works.
## 7. What is tested, and what is not
Tested here, and it runs anywhere:
- `test_selection.py` — 17 cases over the audio layer's parsing and selection: monitor
sources can never be selected (picking one transmits what the desktop is *playing*,
the worst outcome available), exact names beat substrings, an ambiguous pattern
resolves the same way after a reboot instead of coin-flipping, and only the
configured applications' streams get moved.
- `pebble-presence/test/run-tests.sh` — the toggles line is packed by the real JS and
parsed by the real C, so those two implementations of one format cannot drift.
- The generated HA package parses as YAML, for one client and for several.
- The remote-source generator: every `remote` block produces BOTH hooks, a local source
is left alone, and the transport parameters land in the env file. The stop hook is a
safety property, so it is pinned by a test rather than by having been written once.
- `mic-in-use`'s monitor-exclusion logic, against fixtures: a microphone capture lights
the badge, a monitor capture does not, nothing recording does not.
**Not tested, because it needs the actual machines:** the SSH audio pipe end to end
(`pw-record` on the panel, `pw-cat` here, and whether the latency is pleasant over that
wifi link), every command that changes state
(`pactl set-default-source`, `move-source-output`), the pactl JSON shapes the fixtures
imitate, the MQTT discovery payloads against a real Home Assistant, and — the one most
likely to bite — whether the presence entity reports rooms quickly and accurately
enough to be worth wiring to a microphone at all (§3).

224
mic-follow/audio_sources.py
View File

@ -1,224 +0,0 @@
#!/usr/bin/env python3
"""Finding, selecting and moving audio inputs on a PipeWire desktop.
Everything here goes through `pactl` (pipewire-pulse's compatibility layer) rather
than a mix of `wpctl`, `pw-dump` and `pw-metadata`. One tool covers all four things
this needs — list sources, list capture streams, set the default, move a live stream —
and it is the only one of them with a documented JSON output mode, which is the
difference between parsing a stable structure and scraping a table that changes
between releases.
WHY MOVING STREAMS MATTERS, and is not the same as setting the default: changing the
default source only affects applications that asked for "default". Discord, once you
have picked a specific microphone in its settings, holds that device — and the whole
point of this component is that somebody who has set up a studio mic has certainly
picked it explicitly. So the switch does both: it sets the default (for anything that
follows it) and moves the already-running capture streams of the configured
applications (for anything that does not).
Nothing in this file has run against a real PipeWire — see mic-follow/README.md. The
parsing and selection logic is tested against captured pactl output in
test_selection.py; what is unverified is the exact shape of that output on the user's
own version, and every command that changes state.
"""
from __future__ import annotations
import json
import logging
import subprocess
from dataclasses import dataclass
log = logging.getLogger(__name__)
@dataclass(frozen=True)
class Source:
index: int
name: str
description: str
def matches(self, pattern: str) -> bool:
pattern = pattern.strip().lower()
return (pattern == self.name.lower()
or pattern in self.name.lower()
or pattern in self.description.lower())
@dataclass(frozen=True)
class CaptureStream:
index: int
source: int
application: str
def _pactl_json(*args: str) -> list | dict | None:
try:
result = subprocess.run(["pactl", "-f", "json", *args],
capture_output=True, text=True, timeout=10, check=False)
except (OSError, subprocess.SubprocessError) as exc:
log.warning("pactl %s: %s", " ".join(args), exc)
return None
if result.returncode != 0:
log.warning("pactl %s failed: %s", " ".join(args), (result.stderr or "").strip()[:200])
return None
try:
return json.loads(result.stdout or "null")
except json.JSONDecodeError as exc:
log.warning("pactl %s returned unparseable JSON: %s", " ".join(args), exc)
return None
def parse_sources(payload) -> list[Source]:
"""pactl's `list sources` JSON -> Source objects, monitors dropped.
Monitor sources (the loopback of an output) are excluded deliberately: they match
name patterns surprisingly often, and selecting one means transmitting whatever the
desktop is playing instead of what the person is saying — the single worst outcome
this component could produce.
"""
sources: list[Source] = []
for entry in payload or []:
if not isinstance(entry, dict):
continue
name = str(entry.get("name") or "")
if not name or name.endswith(".monitor"):
continue
properties = entry.get("properties") or {}
if str(properties.get("device.class", "")).lower() == "monitor":
continue
sources.append(Source(
index=int(entry.get("index", -1)),
name=name,
description=str(entry.get("description") or properties.get("device.description") or name),
))
return sources
def parse_capture_streams(payload) -> list[CaptureStream]:
"""pactl's `list source-outputs` JSON -> the live recording streams."""
streams: list[CaptureStream] = []
for entry in payload or []:
if not isinstance(entry, dict):
continue
properties = entry.get("properties") or {}
application = str(
properties.get("application.name")
or properties.get("application.process.binary")
or ""
)
source = entry.get("source")
streams.append(CaptureStream(
index=int(entry.get("index", -1)),
source=int(source) if isinstance(source, int) else -1,
application=application,
))
return streams
def select_source(sources: list[Source], pattern: str) -> Source | None:
"""The configured pattern -> one source, preferring the least surprising match.
Exact `node.name` first, then a substring of the name, then a substring of the
human description. Ties inside a tier are resolved by lowest index (the order
pactl reports, which is stable within a boot) and logged, because a pattern that
matches two microphones is a configuration mistake the user should hear about
rather than a coin flip that lands differently after a reboot.
"""
pattern = (pattern or "").strip().lower()
if not pattern:
return None
exact = [s for s in sources if s.name.lower() == pattern]
by_name = [s for s in sources if pattern in s.name.lower()]
by_description = [s for s in sources if pattern in s.description.lower()]
for tier, label in ((exact, "exact name"), (by_name, "name"), (by_description, "description")):
if not tier:
continue
chosen = sorted(tier, key=lambda s: s.index)[0]
if len(tier) > 1:
log.warning("%r matches %d sources by %s (%s) — using %r",
pattern, len(tier), label,
", ".join(s.name for s in tier), chosen.name)
return chosen
return None
def streams_to_move(streams: list[CaptureStream], applications: list[str],
target: Source) -> list[CaptureStream]:
"""Which live capture streams belong to the configured apps and are on the wrong
source already. Streams already on the target are left alone: moving a stream that
is where it should be is a needless glitch in somebody's live audio."""
wanted = [a.strip().lower() for a in applications if a.strip()]
if not wanted:
return []
return [
stream for stream in streams
if stream.source != target.index
and any(pattern in stream.application.lower() for pattern in wanted)
]
# --- The four live commands. Everything above is pure and tested; these are not. -----
def list_sources() -> list[Source]:
return parse_sources(_pactl_json("list", "sources"))
def list_capture_streams() -> list[CaptureStream]:
return parse_capture_streams(_pactl_json("list", "source-outputs"))
def _run(*args: str) -> bool:
try:
result = subprocess.run(["pactl", *args], capture_output=True, text=True,
timeout=10, check=False)
except (OSError, subprocess.SubprocessError) as exc:
log.warning("pactl %s: %s", " ".join(args), exc)
return False
if result.returncode != 0:
log.warning("pactl %s failed: %s", " ".join(args), (result.stderr or "").strip()[:200])
return False
return True
def set_default_source(source: Source) -> bool:
return _run("set-default-source", source.name)
def move_stream(stream: CaptureStream, target: Source) -> bool:
return _run("move-source-output", str(stream.index), target.name)
def current_default_source_name() -> str:
payload = _pactl_json("info")
if isinstance(payload, dict):
return str(payload.get("default_source_name") or "")
return ""
def main() -> int:
"""`audio_sources.py` on its own prints what this desktop has, which is how you
fill in the source patterns in CoreSystemConfig.json."""
logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
sources = list_sources()
if not sources:
print("No sources found — is pactl installed and a PipeWire session running?")
return 1
default = current_default_source_name()
print("Audio inputs on this machine (any part of a name or description works as a")
print("`source` pattern in CoreSystemConfig.json):\n")
for source in sources:
marker = "*" if source.name == default else " "
print(f" {marker} {source.description}")
print(f" {source.name}")
print("\n* = current default")
streams = list_capture_streams()
if streams:
print("\nApplications recording right now:")
for stream in streams:
print(f" {stream.application or '(unnamed)'} (stream {stream.index}, source {stream.source})")
return 0
if __name__ == "__main__":
raise SystemExit(main())

354
mic-follow/desktop_agent.py
View File

@ -1,354 +0,0 @@
#!/usr/bin/env python3
"""mic-follow's desktop half: one MQTT device per client machine, exposing "which
microphone is live" to Home Assistant and nothing else.
THE DIVISION OF LABOUR, and why it is this way round:
Home Assistant decides WHERE the person is and therefore which mic should be live.
It is the only thing that sees the locator sources at all.
this agent knows HOW to change the input on this machine, and nothing about
presence, people, or rooms beyond the names in its own config.
Same rule as every other agent in this repo: the inbound control surface is MQTT
discovery entities and nothing else — no HTTP listener, no direct path from the LLM,
no presence logic on the desktop. It also means a second client machine is a second
copy of this file with a different config, which is what "prep it for multiple
users/clients" comes down to.
THE ONE RULE THAT MAKES IT PREDICTABLE: **off means the desk mic.** The Follow-me
switch being off is not "ignore me", it is an active guarantee that this machine is on
its own microphone. Being live on the wrong mic is the failure somebody notices in
front of their friends, so the safe state is reachable by one tap on the dock, one
button on a watch, or one MQTT message — and it does not depend on presence being
right, on the remote machine being up, or on this agent having seen a recent update.
Nothing here has run against a real PipeWire or a real Home Assistant. The audio layer
it calls is `audio_sources.py`, whose parsing is fixture-tested; the commands that
change state are not. See mic-follow/README.md.
"""
from __future__ import annotations
import argparse
import json
import logging
import os
import shlex
import signal
import socket
import subprocess
import sys
import threading
import time
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent))
import audio_sources # noqa: E402
import paho.mqtt.client as mqtt # noqa: E402
log = logging.getLogger("mic-follow")
DESK = "desk"
DISCOVERY_PREFIX = "homeassistant"
class Client:
"""One desktop machine, its microphones, and its Home Assistant entities."""
def __init__(self, config: dict) -> None:
self.config = config
self.node_id = config["node_id"]
self.friendly_name = config.get("friendly_name") or self.node_id
self.room = config.get("room", "")
self.desk_source = config["desk_source"]
self.move_streams = list(config.get("move_streams") or [])
self.reconcile_seconds = float(config.get("reconcile_seconds", 10))
# How long a source with a start hook gets to show up before it counts as
# absent. An SSH tunnel over the LAN is up well inside this; the timeout is
# what stops a dead panel from hanging the switch.
self.source_timeout = float(config.get("source_timeout_seconds", 6))
# option name -> {source, start_command, stop_command}
self.sources: dict[str, dict] = {
DESK: {"source": self.desk_source, "start_command": "", "stop_command": ""}
}
for entry in config.get("sources") or []:
self.sources[entry["room"]] = {
"source": entry["source"],
"start_command": entry.get("start_command", ""),
"stop_command": entry.get("stop_command", ""),
}
self.base = f"smarthome/mic_follow/{self.node_id}"
self.armed = False
self.selected = DESK
self.active = DESK
self.actual_description = ""
# --- entity plumbing ------------------------------------------------------------
@property
def availability_topic(self) -> str:
return f"{self.base}/availability"
def device_block(self) -> dict:
block = {
"identifiers": [f"mic_follow_{self.node_id}"],
"name": self.friendly_name,
"manufacturer": "SmartestHome",
"model": "mic-follow desktop agent",
}
if self.room:
block["suggested_area"] = self.room
return block
def discovery_payloads(self) -> list[tuple[str, dict]]:
device = self.device_block()
common = {
"device": device,
"availability_topic": self.availability_topic,
"payload_available": "online",
"payload_not_available": "offline",
}
return [
(f"{DISCOVERY_PREFIX}/switch/{self.node_id}/follow_me/config", {
**common,
"name": "Follow-me mic",
"unique_id": f"mic_follow_{self.node_id}_armed",
# object_id fixes the entity_id instead of letting HA derive one from
# the device and entity names. Everything downstream — the generated
# automations, the template sensor, the dock binding, identity's toggle
# allowlist, the watch — refers to these by name, and "probably
# switch.amirs_desktop_follow_me_mic" is not a thing to build four
# consumers on.
"object_id": f"mic_follow_{self.node_id}_armed",
"command_topic": f"{self.base}/armed/set",
"state_topic": f"{self.base}/armed/state",
"payload_on": "ON",
"payload_off": "OFF",
"icon": "mdi:microphone-message",
}),
(f"{DISCOVERY_PREFIX}/select/{self.node_id}/mic_input/config", {
**common,
"name": "Mic input",
"unique_id": f"mic_follow_{self.node_id}_input",
"object_id": f"mic_follow_{self.node_id}_input",
"command_topic": f"{self.base}/input/set",
"state_topic": f"{self.base}/input/state",
"options": list(self.sources.keys()),
"icon": "mdi:microphone",
}),
(f"{DISCOVERY_PREFIX}/sensor/{self.node_id}/mic_actual/config", {
**common,
"name": "Live microphone",
"unique_id": f"mic_follow_{self.node_id}_actual",
"object_id": f"mic_follow_{self.node_id}_actual",
"state_topic": f"{self.base}/actual/state",
"icon": "mdi:microphone-settings",
}),
]
# --- the actual switching --------------------------------------------------------
def _run_hook(self, command: str, label: str) -> None:
"""A source's start/stop hook. This is what brings a NETWORK microphone up and
down — and bringing it down matters more than bringing it up: a room mic that
keeps streaming after the switch left it is a hot mic in somebody's flat."""
if not command.strip():
return
try:
result = subprocess.run(shlex.split(command), capture_output=True,
text=True, timeout=20, check=False)
if result.returncode != 0:
log.warning("%s hook failed (%s): %s", label, result.returncode,
(result.stderr or "").strip()[:200])
except (OSError, subprocess.SubprocessError) as exc:
log.warning("%s hook could not run: %s", label, exc)
def apply(self, option: str) -> str:
"""Make `option` the live input. Returns the option actually applied.
ORDER MATTERS HERE, and it was wrong the first time: a remote microphone does
not EXIST as a PipeWire source until its start hook has run and the stream is
up. Looking for the source first meant a network mic could never be selected —
it fell back to the desk every time, correctly by the letter of the code and
uselessly in practice. So: hooks first, then wait for the source to appear,
then fall back only if it genuinely never does.
"""
wanted = self.sources.get(option)
if wanted is None:
log.warning("unknown input %r — falling back to %s", option, DESK)
option, wanted = DESK, self.sources[DESK]
if option != self.active:
previous = self.sources.get(self.active)
if previous:
self._run_hook(previous.get("stop_command", ""), f"{self.active} stop")
self._run_hook(wanted.get("start_command", ""), f"{option} start")
target = self._await_source(wanted["source"],
timeout=self.source_timeout if wanted.get("start_command") else 0)
if target is None and option != DESK:
log.warning("no audio source matching %r appeared — falling back to %s",
wanted["source"], DESK)
# Tear down whatever the failed start hook left running before leaving it
# behind: a half-started remote microphone is the hot-mic case.
self._run_hook(wanted.get("stop_command", ""), f"{option} stop (failed)")
option, wanted = DESK, self.sources[DESK]
target = self._await_source(wanted["source"], timeout=0)
if target is None:
log.error("no audio source matching %r either — leaving the input alone",
wanted["source"])
return self.active
audio_sources.set_default_source(target)
# And move what is already recording: an application that picked a specific
# microphone in its own settings — which anyone with a studio mic has — does
# not follow the default.
for stream in audio_sources.streams_to_move(
audio_sources.list_capture_streams(), self.move_streams, target):
log.info("moving %s's live stream to %s", stream.application, target.description)
audio_sources.move_stream(stream, target)
self.active = option
self.actual_description = target.description
log.info("input is now %s (%s)", option, target.description)
return option
def _await_source(self, pattern: str, timeout: float) -> audio_sources.Source | None:
"""Look for a source, giving a just-started one time to register.
A local device is found on the first pass and this costs nothing. A tunnelled
one takes a moment: the hook returns as soon as the transport is up, and the
node appears in PipeWire slightly after that.
"""
deadline = time.monotonic() + max(0.0, timeout)
while True:
found = audio_sources.select_source(audio_sources.list_sources(), pattern)
if found is not None or time.monotonic() >= deadline:
return found
time.sleep(0.25)
def make_mqtt_client(client_id: str) -> mqtt.Client:
# Same shim as every other agent here: paho 2.x wants an explicit callback API
# version, bookworm's 1.6.x has no such argument.
callback_api = getattr(mqtt, "CallbackAPIVersion", None)
if callback_api is not None:
return mqtt.Client(callback_api.VERSION1, client_id=client_id)
return mqtt.Client(client_id=client_id)
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(description="mic-follow desktop agent")
parser.add_argument("--config", default=os.environ.get(
"MIC_FOLLOW_CONFIG", str(Path.home() / ".config/mic-follow/client.json")))
parser.add_argument("--list-sources", action="store_true",
help="print this machine's audio inputs and exit")
args = parser.parse_args(argv)
logging.basicConfig(level=os.environ.get("MIC_FOLLOW_LOG_LEVEL", "INFO"),
format="%(asctime)s %(levelname)s %(name)s: %(message)s",
stream=sys.stdout)
if args.list_sources:
return audio_sources.main()
config_path = Path(args.config).expanduser()
if not config_path.exists():
log.error("no config at %s — generate one with mic-follow/generate.py", config_path)
return 2
client = Client(json.loads(config_path.read_text()))
broker = client.config.get("mqtt", {}) or {}
host = broker.get("host", "")
if not host:
log.error("mqtt.host is not set in %s", config_path)
return 2
mqtt_client = make_mqtt_client(f"mic-follow-{client.node_id}-{socket.gethostname()}")
if broker.get("username"):
mqtt_client.username_pw_set(broker["username"], broker.get("password") or None)
mqtt_client.will_set(client.availability_topic, "offline", qos=1, retain=True)
def publish_state() -> None:
mqtt_client.publish(f"{client.base}/armed/state", "ON" if client.armed else "OFF",
qos=1, retain=True)
mqtt_client.publish(f"{client.base}/input/state", client.active, qos=1, retain=True)
mqtt_client.publish(f"{client.base}/actual/state",
client.actual_description or "unknown", qos=1, retain=True)
def desired_option() -> str:
# The whole policy, in one line: armed follows the selection, unarmed is the
# desk mic. Everything else in this file is mechanism.
return client.selected if client.armed else DESK
def reconcile(force: bool = False) -> None:
wanted = desired_option()
if force or wanted != client.active:
client.apply(wanted)
publish_state()
return
# Nothing asked for a change — but something else on the desktop may have moved
# the default (plugging in a headset does exactly that), so the sensor has to be
# re-read rather than assumed.
current = audio_sources.current_default_source_name()
target = audio_sources.select_source(
audio_sources.list_sources(), client.sources[client.active]["source"])
if target is not None and current and current != target.name:
log.info("something else changed the default input — putting it back")
client.apply(wanted)
publish_state()
def on_connect(_client, _userdata, _flags, rc):
if rc != 0:
log.error("MQTT connection refused (rc=%s)", rc)
return
log.info("connected to MQTT %s:%s", host, broker.get("port", 1883))
for topic, payload in client.discovery_payloads():
mqtt_client.publish(topic, json.dumps(payload), qos=1, retain=True)
mqtt_client.subscribe([(f"{client.base}/armed/set", 1), (f"{client.base}/input/set", 1)])
mqtt_client.publish(client.availability_topic, "online", qos=1, retain=True)
reconcile(force=True)
def on_message(_client, _userdata, message):
payload = message.payload.decode("utf-8", "replace").strip()
if message.topic.endswith("/armed/set"):
client.armed = payload.upper() == "ON"
log.info("follow-me %s", "armed" if client.armed else "disarmed")
elif message.topic.endswith("/input/set"):
if payload not in client.sources:
log.warning("ignoring unknown input %r", payload)
return
client.selected = payload
log.info("input selection is now %r", payload)
reconcile()
mqtt_client.on_connect = on_connect
mqtt_client.on_message = on_message
mqtt_client.on_disconnect = lambda *_: log.warning("disconnected from MQTT; paho will retry")
stop = threading.Event()
signal.signal(signal.SIGTERM, lambda *_: stop.set())
signal.signal(signal.SIGINT, lambda *_: stop.set())
mqtt_client.connect_async(host, int(broker.get("port", 1883)), keepalive=60)
mqtt_client.loop_start()
log.info("mic-follow agent for %s started", client.node_id)
try:
while not stop.wait(client.reconcile_seconds):
reconcile()
finally:
log.info("shutting down — returning to the desk microphone")
# Leaving a machine on a remote microphone because a service stopped is exactly
# the surprise this component exists to avoid, and it also shuts down any
# network mic stream through the stop hook.
client.armed = False
client.apply(DESK)
mqtt_client.publish(client.availability_topic, "offline", qos=1, retain=True)
mqtt_client.loop_stop()
mqtt_client.disconnect()
return 0
if __name__ == "__main__":
raise SystemExit(main())

383
mic-follow/generate.py
View File

@ -1,383 +0,0 @@
#!/usr/bin/env python3
"""Turn CoreSystemConfig.json's `mic_follow` block into everything the feature needs.
Writes into mic-follow/generated/ (gitignored — the per-client configs carry the MQTT
password):
<node_id>/client.json the desktop agent's config
<node_id>/mic-follow-<node_id>.service the systemd --user unit for it
ha-package/mic_follow.yaml one status sensor + one automation per client
dock-bindings.md the Stream Dock toggle key, per client
identity-toggles.json the allowlist identity serves to the watch
One client is one desktop machine and the one person it follows. Adding a second
person with a second PC is a second entry in `clients`, and everything below comes out
twice with no other change — which is what "prep it for multiple users/clients" means
in practice.
Usage:
mic-follow/generate.py [CoreSystemConfig.json] [--out DIR]
"""
from __future__ import annotations
import json
import sys
from pathlib import Path
REPO = Path(__file__).resolve().parent.parent
DESK = "desk"
def room_label(room: str) -> str:
"""`living_room` -> `Living room`. Short and readable, because this ends up as the
text on a 72-pixel key and on a watch screen, not in a log line."""
if room == DESK:
return "Desk"
return room.replace("_", " ").strip().capitalize() or room
def entity_ids(node_id: str) -> dict[str, str]:
"""The four entity_ids every consumer refers to. They are deterministic because the
agent sets `object_id` in its discovery payloads — see desktop_agent.py."""
return {
"armed": f"switch.mic_follow_{node_id}_armed",
"input": f"select.mic_follow_{node_id}_input",
"actual": f"sensor.mic_follow_{node_id}_actual",
"status": f"sensor.mic_follow_{node_id}_status",
}
def remote_source_name(source: dict) -> str:
"""A remote microphone's PipeWire source name. Generated rather than configured:
it is created by remote-mic.sh on this desktop, so nobody has to go and look it up
on a machine where it does not exist yet."""
return str(source.get("source") or "").strip() or f"micfollow_{source['room']}"
def build_remote_env(source: dict, repo_root: Path) -> str:
"""One remote microphone's parameters, read by remote-mic.sh."""
remote = source.get("remote") or {}
name = remote_source_name(source)
description = source.get("description") or f"{room_label(source['room'])} microphone"
return "\n".join([
f"# Generated by mic-follow/generate.py for the '{source['room']}' microphone.",
"# Read by mic-follow/remote-mic.sh. Regenerate rather than editing.",
f"REMOTE_HOST={remote.get('host', '')}",
f"REMOTE_USER={remote.get('user', 'kiosk')}",
f"REMOTE_MIC={remote.get('mic_source', '')}",
f"SOURCE_NAME={name}",
f"SOURCE_DESCRIPTION={description}",
f"RATE={remote.get('rate', 48000)}",
f"CHANNELS={remote.get('channels', 1)}",
f"LATENCY_MS={remote.get('latency_ms', 40)}",
f"SSH_KEY={remote.get('ssh_key', '')}",
"",
])
def source_entry(source: dict, repo_root: Path) -> dict:
"""One entry of the agent's `sources` list.
A remote microphone's hooks are generated, not typed: the whole point of the
`remote` block is that "start the stream" and — much more importantly — "stop the
stream" are not things a person should be relying on themselves to have written
correctly in a config file.
"""
if source.get("remote"):
room = source["room"]
script = repo_root / "mic-follow" / "remote-mic.sh"
return {
"room": room,
"source": remote_source_name(source),
"start_command": f"{script} start {room}",
"stop_command": f"{script} stop {room}",
}
return {
"room": source["room"],
"source": source["source"],
"start_command": source.get("start_command", ""),
"stop_command": source.get("stop_command", ""),
}
def build_client_config(client: dict, cfg: dict) -> dict:
prefix = cfg["network"]["subnet_prefix"]
container_ip = f"{prefix}.{cfg['container_host']['ip_last_octet']}"
secrets = cfg.get("secrets", {}) or {}
return {
"node_id": client["node_id"],
"friendly_name": client.get("friendly_name") or client["node_id"],
"room": client.get("room", ""),
"mqtt": {
"host": container_ip,
"port": cfg["ports"]["mqtt"],
"username": secrets.get("mqtt_username", ""),
"password": secrets.get("mqtt_password", ""),
},
"desk_source": client["desk_source"],
"move_streams": list(client.get("move_streams") or []),
"reconcile_seconds": client.get("reconcile_seconds", 10),
"source_timeout_seconds": client.get("source_timeout_seconds", 6),
"sources": [source_entry(source, REPO) for source in client.get("sources") or []],
}
def build_unit(client: dict) -> str:
node_id = client["node_id"]
return f"""[Unit]
Description=mic-follow agent for {client.get('friendly_name') or node_id}
Documentation=file://{REPO}/mic-follow/README.md
After=network-online.target pipewire.service
Wants=pipewire.service
[Service]
Type=simple
ExecStart={REPO}/mic-follow/desktop_agent.py --config %h/.config/mic-follow/client.json
Restart=always
RestartSec=10
[Install]
WantedBy=default.target
"""
def build_ha_package(clients: list[dict]) -> str:
lines: list[str] = [
"# Generated by mic-follow/generate.py — do not hand-edit. Change the",
"# `mic_follow` block in CoreSystemConfig.json and regenerate.",
"#",
"# Two things per client:",
"#",
"# a status sensor whose STATE is the human-readable name of the microphone",
"# that is live right now. It exists so that every surface —",
"# the Stream Dock key, the Pebble app, a dashboard — can",
"# show the answer by displaying ONE entity's state, instead",
"# of each one reimplementing the same three-way template.",
"# an automation that moves the microphone when the person moves, and only",
"# while that client's Follow-me switch is on.",
"#",
"# The entity_ids below are fixed by the agent's `object_id`, not guessed.",
"",
"template:",
" - sensor:",
]
for client in clients:
node_id = client["node_id"]
ids = entity_ids(node_id)
labels = {DESK: "Desk"}
for source in client.get("sources") or []:
labels[source["room"]] = room_label(source["room"])
label_map = json.dumps(labels)
lines += [
f" - name: \"{client.get('friendly_name') or node_id} mic\"",
f" unique_id: mic_follow_{node_id}_status",
f" # object_id keeps this at {ids['status']} whatever the name becomes.",
f" object_id: mic_follow_{node_id}_status",
" state: >-",
f" {{% set labels = {label_map} %}}",
f" {{% set live = states('{ids['input']}') %}}",
" {{ labels.get(live, live | replace('_', ' ') | capitalize) }}",
" icon: >-",
f" {{{{ 'mdi:microphone-message' if is_state('{ids['armed']}', 'on')",
" else 'mdi:microphone' }}",
" attributes:",
f" armed: \"{{{{ is_state('{ids['armed']}', 'on') }}}}\"",
f" device: \"{{{{ states('{ids['actual']}') }}}}\"",
f" person: \"{client.get('person', '')}\"",
"",
]
lines += ["automation:"]
for client in clients:
node_id = client["node_id"]
ids = entity_ids(node_id)
presence = client["presence_entity"]
desk_room = client.get("room", "")
dwell = client.get("dwell_seconds", 20)
return_dwell = client.get("return_dwell_seconds", 5)
policy = client.get("on_unknown_room", "hold")
rooms = [source["room"] for source in client.get("sources") or []]
room_map = {room: room for room in rooms}
if desk_room:
room_map[desk_room] = DESK
lines += [
f" - id: mic_follow_{node_id}",
f" alias: \"Mic follow: {client.get('friendly_name') or node_id}\"",
" description: >-",
f" Moves {client.get('person') or 'this client'}'s live microphone to match",
" where they are, but only while the Follow-me switch is on. The switch",
" being off is an active guarantee of the desk mic, so this automation",
" never runs then.",
" mode: single",
" triggers:",
]
for room in rooms:
lines += [
" - trigger: state",
f" entity_id: {presence}",
f" to: \"{room}\"",
f" for: {{ seconds: {dwell} }}",
]
if desk_room:
lines += [
" # Coming back is faster than leaving: sitting down should give you",
" # the good microphone back before you say anything into it.",
" - trigger: state",
f" entity_id: {presence}",
f" to: \"{desk_room}\"",
f" for: {{ seconds: {return_dwell} }}",
]
if policy == "desk":
known = json.dumps(sorted(room_map.keys()))
lines += [
" # on_unknown_room: desk — a room with no microphone configured is a",
" # room where nothing can hear you, and this says so rather than",
" # leaving another room's mic live.",
" - trigger: state",
f" entity_id: {presence}",
f" not_to: {known}",
f" for: {{ seconds: {dwell} }}",
]
lines += [
" # And whenever it is switched on, catch up with where the person",
" # already is rather than waiting for them to move again.",
" - trigger: state",
f" entity_id: {ids['armed']}",
" to: \"on\"",
" conditions:",
" - condition: state",
f" entity_id: {ids['armed']}",
" state: \"on\"",
" actions:",
" - variables:",
f" room_map: {json.dumps(room_map)}",
f" current: \"{{{{ states('{presence}') }}}}\"",
f" wanted: \"{{{{ room_map.get(current, '{'desk' if policy == 'desk' else 'HOLD'}') }}}}\"",
" # HOLD is how on_unknown_room: hold is expressed — no service call at",
" # all, so the last microphone stays live while somebody walks through",
" # a room nothing can hear them in.",
" - condition: template",
" value_template: \"{{ wanted != 'HOLD' }}\"",
" - action: select.select_option",
" target:",
f" entity_id: {ids['input']}",
" data:",
" option: \"{{ wanted }}\"",
"",
]
return "\n".join(lines)
def build_dock_bindings(clients: list[dict]) -> str:
out = ["# Stream Dock — the follow-me toggle", "",
"Generated by `mic-follow/generate.py`. One key per client. Put it on whichever",
"OpenDeck layer you like — it does not have to share the lighting layer.", ""]
for client in clients:
node_id = client["node_id"]
ids = entity_ids(node_id)
out += [
f"## {client.get('friendly_name') or node_id}",
"",
"| Field | Value |",
"|---|---|",
f"| Entity | `{ids['status']}` |",
f"| Service | `switch.toggle` |",
"| Service data JSON | see below |",
"",
"```json",
json.dumps({"entity_id": ids["armed"]}, indent=2),
"```",
"",
"**Point the key's displayed entity at the status sensor, not at the switch.**",
f"The switch's state is `on`/`off`, which tells you nothing useful; "
f"`{ids['status']}`'s state is the *name of the microphone that is live right now*",
"— `Desk`, `Loggia` — which is what you actually want to read at a glance. Its",
"icon changes with the switch (`mdi:microphone-message` armed, `mdi:microphone`",
"off), so one key shows both facts: which mic, and whether it will follow you.",
"",
"The plugin subscribes to Home Assistant's websocket, so the key updates when",
"the state changes rather than on a timer — nothing here polls, and the key is",
"correct within a moment of the microphone actually moving, including when it",
"was the watch or an automation that moved it.",
"",
"Suggested title: leave it EMPTY and let the state be the whole label. A key",
"reading `Loggia` in large type is readable across a room; the same key reading",
"`Follow-me mic` over a small `Loggia` is not.",
"",
]
return "\n".join(out)
def main(argv: list[str]) -> int:
args = [a for a in argv[1:] if not a.startswith("--")]
out_dir = Path(__file__).resolve().parent / "generated"
if "--out" in argv:
out_dir = Path(argv[argv.index("--out") + 1])
config_path = Path(args[0]) if args else REPO / "CoreSystemConfig.json"
if not config_path.exists():
print(f"error: {config_path} not found", file=sys.stderr)
return 2
cfg = json.loads(config_path.read_text())
section = cfg.get("mic_follow") or {}
if not section.get("enabled"):
print("error: mic_follow.enabled is false — nothing to generate", file=sys.stderr)
return 2
clients = [c for c in (section.get("clients") or []) if isinstance(c, dict) and c.get("node_id")]
if not clients:
print("error: mic_follow.clients is empty", file=sys.stderr)
return 2
out_dir.mkdir(parents=True, exist_ok=True)
for client in clients:
node_id = client["node_id"]
client_dir = out_dir / node_id
client_dir.mkdir(parents=True, exist_ok=True)
config_file = client_dir / "client.json"
config_file.write_text(json.dumps(build_client_config(client, cfg), indent=2) + "\n")
config_file.chmod(0o600)
(client_dir / f"mic-follow-{node_id}.service").write_text(build_unit(client))
for source in client.get("sources") or []:
if source.get("remote"):
(client_dir / f"remote-{source['room']}.env").write_text(
build_remote_env(source, REPO))
package_dir = out_dir / "ha-package"
package_dir.mkdir(parents=True, exist_ok=True)
(package_dir / "mic_follow.yaml").write_text(build_ha_package(clients))
(out_dir / "dock-bindings.md").write_text(build_dock_bindings(clients))
allowlist = [
{
"id": client["node_id"],
"name": client.get("friendly_name") or client["node_id"],
"switch_entity": entity_ids(client["node_id"])["armed"],
"detail_entity": entity_ids(client["node_id"])["status"],
}
for client in clients
]
(out_dir / "identity-toggles.json").write_text(json.dumps(allowlist, indent=2) + "\n")
# The same thing as one line, ready to paste into identity.env on the container
# host — which is where it has to end up for the watch to see any toggles at all.
(out_dir / "identity-toggles.env").write_text(
"# Paste into /opt/smart-home/identity/identity.env on the container host,\n"
"# then: docker compose restart identity\n"
f"TOGGLE_ALLOWLIST_JSON={json.dumps(allowlist, separators=(',', ':'))}\n")
print(f"wrote {out_dir}/ for {len(clients)} client(s):")
for client in clients:
remotes = [s["room"] for s in client.get("sources") or [] if s.get("remote")]
extra = f", remote-*.env for {', '.join(remotes)}" if remotes else ""
print(f" {client['node_id']}/client.json, {client['node_id']}/mic-follow-{client['node_id']}.service{extra}")
print(" ha-package/mic_follow.yaml")
print(" dock-bindings.md")
print(" identity-toggles.json + identity-toggles.env (paste into identity.env)")
return 0
if __name__ == "__main__":
sys.exit(main(sys.argv))

166
mic-follow/remote-mic.sh
View File

@ -1,166 +0,0 @@
#!/usr/bin/env bash
#
# A microphone in another room, as a local audio source on this desktop.
#
# remote-mic.sh start <name>
# remote-mic.sh stop <name>
# remote-mic.sh status <name>
#
# Called by the mic-follow agent's start_command/stop_command hooks; `<name>` is the
# room, and its parameters come from ~/.config/mic-follow/remote-<name>.env, generated
# from CoreSystemConfig.json.
#
# HOW IT WORKS, and why this shape:
#
# panel: pw-record (its microphone) ──ssh──> desktop: pw-cat --playback
# into a null sink
# │
# module-remap-source ──────────┘
# presents it as a REAL source named <source_name>
#
# **SSH, not an audio protocol.** RTP and PipeWire's pulse-tunnel are both lower
# latency and both need a new listening service on the panel, an ACL, and config on two
# machines that has to agree. The panel already runs sshd as its documented admin path
# and already trusts this desktop's key, so the transport comes with authentication and
# encryption already solved, and nothing new listens on the network. For a smoke-break
# voice call, ~40 ms of extra buffering is not the constraint; a hot microphone in an
# empty room is.
#
# **Audio only exists while this runs.** No daemon, no always-on stream: `start` opens
# one SSH session, `stop` closes it, and the panel's microphone is not being read at
# any other time. That is a property of the transport rather than a promise in a
# config file, which is the reason to prefer it.
#
# **Why a null sink plus module-remap-source**, rather than just using the sink's
# monitor: a monitor is not a real source, applications treat it as "record what the
# desktop is playing", and mic-follow's own audio layer refuses to select one on
# purpose. remap-source turns it into an ordinary microphone with a name and a
# description, which is what Discord's device list needs to show.
#
# UNVERIFIED: this has never run against the real panel. The module names and the
# pw-record/pw-cat invocations are written from documentation. `status` exists to make
# checking it a one-liner.
set -uo pipefail
ACTION="${1:-}"
NAME="${2:-}"
[[ -n "$ACTION" && -n "$NAME" ]] || { echo "usage: remote-mic.sh <start|stop|status> <name>" >&2; exit 2; }
CONF="${MIC_FOLLOW_REMOTE_DIR:-$HOME/.config/mic-follow}/remote-${NAME}.env"
RUN_DIR="${XDG_RUNTIME_DIR:-/tmp}/mic-follow"
PID_FILE="${RUN_DIR}/${NAME}.pid"
MODULES_FILE="${RUN_DIR}/${NAME}.modules"
[[ -r "$CONF" ]] || { echo "remote-mic: no config at $CONF" >&2; exit 2; }
# shellcheck disable=SC1090
. "$CONF"
REMOTE_HOST="${REMOTE_HOST:-}"
REMOTE_USER="${REMOTE_USER:-kiosk}"
REMOTE_MIC="${REMOTE_MIC:-}" # empty = whatever the panel's default input is
SOURCE_NAME="${SOURCE_NAME:-micfollow_${NAME}}"
SOURCE_DESCRIPTION="${SOURCE_DESCRIPTION:-${NAME} microphone}"
RATE="${RATE:-48000}"
CHANNELS="${CHANNELS:-1}"
LATENCY_MS="${LATENCY_MS:-40}"
SSH_KEY="${SSH_KEY:-}"
mkdir -p "$RUN_DIR"
is_running() {
[[ -f "$PID_FILE" ]] || return 1
local pid; pid="$(cat "$PID_FILE" 2>/dev/null || true)"
[[ -n "$pid" ]] && kill -0 "$pid" 2>/dev/null
}
case "$ACTION" in
status)
if is_running; then
echo "running (pid $(cat "$PID_FILE"))"
pactl list short sources 2>/dev/null | grep -F "$SOURCE_NAME" || echo " WARNING: the source is not registered"
exit 0
fi
echo "stopped"
exit 1
;;
start)
[[ -n "$REMOTE_HOST" ]] || { echo "remote-mic: REMOTE_HOST is not set in $CONF" >&2; exit 2; }
if is_running; then
echo "remote-mic: ${NAME} is already running"
exit 0
fi
command -v pactl >/dev/null || { echo "remote-mic: pactl is not installed" >&2; exit 2; }
command -v pw-cat >/dev/null || { echo "remote-mic: pw-cat is not installed (pipewire-bin)" >&2; exit 2; }
# The sink and the source it is remapped into. Recorded so `stop` unloads exactly
# what `start` loaded, rather than pattern-matching modules somebody else may own.
: > "$MODULES_FILE"
SINK_MODULE="$(pactl load-module module-null-sink \
sink_name="${SOURCE_NAME}_sink" \
sink_properties="device.description='${SOURCE_DESCRIPTION} (transport)'" 2>/dev/null)" || {
echo "remote-mic: could not create the null sink" >&2; exit 1; }
echo "$SINK_MODULE" >> "$MODULES_FILE"
SOURCE_MODULE="$(pactl load-module module-remap-source \
master="${SOURCE_NAME}_sink.monitor" \
source_name="${SOURCE_NAME}" \
source_properties="device.description='${SOURCE_DESCRIPTION}'" 2>/dev/null)" || {
echo "remote-mic: could not create the remapped source" >&2
pactl unload-module "$SINK_MODULE" 2>/dev/null
exit 1; }
echo "$SOURCE_MODULE" >> "$MODULES_FILE"
SSH_ARGS=(-o BatchMode=yes -o ConnectTimeout=5 -o ServerAliveInterval=5
-o ServerAliveCountMax=2 -o StrictHostKeyChecking=accept-new)
[[ -n "$SSH_KEY" ]] && SSH_ARGS+=(-i "$SSH_KEY")
# The panel end. --target is left off entirely when REMOTE_MIC is empty, so the
# panel's own default input is used and nothing here has to know its device names.
REMOTE_CMD="pw-record --rate ${RATE} --channels ${CHANNELS} --format s16 --latency ${LATENCY_MS}ms"
[[ -n "$REMOTE_MIC" ]] && REMOTE_CMD="${REMOTE_CMD} --target '${REMOTE_MIC}'"
REMOTE_CMD="${REMOTE_CMD} -"
# setsid so the pipe survives the hook's own shell exiting, and so `stop` can kill
# the whole thing by process group rather than chasing two processes.
setsid bash -c "ssh ${SSH_ARGS[*]} '${REMOTE_USER}@${REMOTE_HOST}' \"${REMOTE_CMD}\" \
| pw-cat --playback --rate ${RATE} --channels ${CHANNELS} --format s16 \
--latency ${LATENCY_MS}ms --target '${SOURCE_NAME}_sink' -" \
</dev/null >/dev/null 2>&1 &
echo $! > "$PID_FILE"
sleep 0.5
if ! is_running; then
echo "remote-mic: the stream died immediately — check: ssh ${REMOTE_USER}@${REMOTE_HOST} pw-record --help" >&2
"$0" stop "$NAME" >/dev/null 2>&1
exit 1
fi
echo "remote-mic: ${NAME} up as source '${SOURCE_NAME}'"
;;
stop)
# Kill the transport FIRST, then tear down the local plumbing: the order that
# guarantees the panel stops being recorded even if unloading a module fails.
if [[ -f "$PID_FILE" ]]; then
pid="$(cat "$PID_FILE" 2>/dev/null || true)"
if [[ -n "$pid" ]]; then
kill -TERM -- "-${pid}" 2>/dev/null || kill -TERM "$pid" 2>/dev/null || true
sleep 0.3
kill -KILL -- "-${pid}" 2>/dev/null || true
fi
rm -f "$PID_FILE"
fi
if [[ -f "$MODULES_FILE" ]]; then
# Reverse order: the remapped source depends on the sink.
tac "$MODULES_FILE" | while read -r module; do
[[ -n "$module" ]] && pactl unload-module "$module" 2>/dev/null
done
rm -f "$MODULES_FILE"
fi
echo "remote-mic: ${NAME} stopped"
;;
*)
echo "usage: remote-mic.sh <start|stop|status> <name>" >&2
exit 2
;;
esac

88
mic-follow/setup-client.sh
View File

@ -1,88 +0,0 @@
#!/usr/bin/env bash
#
# Install the mic-follow agent for ONE client, on the machine that client is.
#
# mic-follow/setup-client.sh <node_id>
#
# Run it on that person's desktop, from a checkout of this repo. It generates from
# CoreSystemConfig.json, installs the client config (mode 0600 — it holds the MQTT
# password) and a systemd --user unit, and starts it.
#
# A second person's desktop runs the same command with their own node_id. Nothing here
# is specific to one machine except the argument.
set -euo pipefail
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
DIR="${REPO_ROOT}/mic-follow"
# shellcheck source=/dev/null
source "${REPO_ROOT}/tools/lib/coreconfig.sh"
NODE_ID="${1:-}"
[[ -n "$NODE_ID" ]] || core_die "usage: setup-client.sh <node_id> (from mic_follow.clients in CoreSystemConfig.json)"
core_load
[[ "$CORE_MIC_FOLLOW_ENABLED" == "true" ]] || core_die \
"mic_follow.enabled is false in $(basename "$CORE_CONFIG_PATH")"
core_log "Generating"
"${DIR}/generate.py" "$CORE_CONFIG_PATH"
GEN="${DIR}/generated/${NODE_ID}"
[[ -d "$GEN" ]] || core_die "no client '${NODE_ID}' in the config — have: $(ls -1 "${DIR}/generated" | grep -v '^ha-package$' | tr '\n' ' ')"
core_log "Checking this machine can see its microphones"
if ! command -v pactl >/dev/null; then
core_warn "pactl is not installed — the agent cannot switch anything without it"
core_warn " Debian/Ubuntu: apt install pulseaudio-utils Arch: pacman -S libpulse"
else
"${DIR}/desktop_agent.py" --list-sources || core_warn "could not list audio sources"
fi
core_log "Installing"
mkdir -p "$HOME/.config/mic-follow" "$HOME/.config/systemd/user"
install -m 600 "${GEN}/client.json" "$HOME/.config/mic-follow/client.json"
# Any remote microphones this client can switch to. Each one names a machine and an
# SSH user; remote-mic.sh reads them by room name.
shopt -s nullglob
for env_file in "${GEN}"/remote-*.env; do
install -m 600 "$env_file" "$HOME/.config/mic-follow/$(basename "$env_file")"
echo " remote microphone: $(basename "$env_file" .env | sed 's/^remote-//')"
done
shopt -u nullglob
install -m 644 "${GEN}/mic-follow-${NODE_ID}.service" \
"$HOME/.config/systemd/user/mic-follow.service"
if ! systemctl --user daemon-reload 2>/dev/null; then
core_warn "systemctl --user is not available here — the unit is installed but not loaded"
elif systemctl --user enable --now mic-follow.service; then
echo " started — follow it with: journalctl --user -fu mic-follow"
else
core_warn "could not start it — try: systemctl --user status mic-follow"
fi
cat <<EOF
Left to do, and none of it is on this machine:
[ ] Home Assistant: install the generated package
${DIR}/generated/ha-package/mic_follow.yaml
next to stream-dock's, in <HA config>/packages/, then reload YAML.
[ ] identity: paste the line from
${DIR}/generated/identity-toggles.env
into /opt/smart-home/identity/identity.env and restart identity.
Without it the Pebble app sees no toggles at all.
[ ] Stream Dock: bind the toggle key as described in
${DIR}/generated/dock-bindings.md
[ ] For each remote microphone, check the SSH path this desktop will use:
ssh <user>@<host> pw-record --help
That one command is the whole transport. If it works, the microphone works;
if it does not, no amount of Home Assistant configuration will help.
[ ] Check the presence entity actually reports area_ids:
Developer Tools -> States -> the presence_entity for this client.
If its state is a friendly room name rather than an area_id, the automation
will never match and the microphone will never move — see README.md section 3.
EOF

200
mic-follow/test_selection.py
View File

@ -1,200 +0,0 @@
#!/usr/bin/env python3
"""Fixture tests for the audio layer's parsing and selection.
python3 mic-follow/test_selection.py
The commands that CHANGE state cannot be tested without a real PipeWire session; what
can be tested is everything that decides which device those commands are pointed at,
and that is where the damaging mistakes live — picking a monitor source (transmitting
what the desktop is playing instead of what the person is saying), or picking a
different microphone after a reboot because two matched and the order changed.
The fixtures are hand-written in pactl's documented JSON shape. VERIFY THEM against
real output (`pactl -f json list sources`) the first time this runs on the desktop —
they are the assumption this whole component rests on.
"""
import sys
import unittest
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent))
import audio_sources as audio
import generate
SOURCES = [
{
"index": 44,
"name": "alsa_output.pci-0000_0c_00.4.analog-stereo.monitor",
"description": "Monitor of Family 17h HD Audio",
"properties": {"device.class": "monitor"},
},
{
"index": 46,
"name": "alsa_input.usb-Shure_Inc_MV6-00.mono-fallback",
"description": "MV6 Mono",
"properties": {"device.class": "sound"},
},
{
"index": 51,
"name": "bluez_input.AC:12:2F:9B:00:01.headset-head-unit",
"description": "Arctis Nova 7 (headset mic)",
"properties": {"device.class": "sound"},
},
{
"index": 58,
"name": "alsa_input.usb-DJI_MIC_MINI-00.mono-fallback",
"description": "DJI MIC MINI Mono",
"properties": {"device.class": "sound"},
},
]
STREAMS = [
{"index": 12, "source": 46, "properties": {"application.name": "Discord"}},
{"index": 13, "source": 46, "properties": {"application.name": "obs"}},
{"index": 14, "source": 58, "properties": {"application.process.binary": "Discord"}},
]
class ParsingTests(unittest.TestCase):
def test_monitors_are_dropped(self):
names = [s.name for s in audio.parse_sources(SOURCES)]
self.assertNotIn("alsa_output.pci-0000_0c_00.4.analog-stereo.monitor", names)
self.assertEqual(len(names), 3)
def test_monitor_dropped_by_property_even_without_the_suffix(self):
odd = [{"index": 1, "name": "weird_monitor_name", "description": "x",
"properties": {"device.class": "Monitor"}}]
self.assertEqual(audio.parse_sources(odd), [])
def test_description_falls_back_to_the_name(self):
bare = [{"index": 2, "name": "some.source", "properties": {}}]
self.assertEqual(audio.parse_sources(bare)[0].description, "some.source")
def test_capture_streams_read_either_application_property(self):
streams = audio.parse_capture_streams(STREAMS)
self.assertEqual([s.application for s in streams], ["Discord", "obs", "Discord"])
def test_garbage_does_not_raise(self):
self.assertEqual(audio.parse_sources(None), [])
self.assertEqual(audio.parse_sources(["not a dict"]), [])
self.assertEqual(audio.parse_capture_streams(None), [])
class SelectionTests(unittest.TestCase):
def setUp(self):
self.sources = audio.parse_sources(SOURCES)
def test_substring_of_the_description(self):
self.assertEqual(audio.select_source(self.sources, "MV6").index, 46)
def test_substring_of_the_node_name(self):
self.assertEqual(audio.select_source(self.sources, "DJI_MIC").index, 58)
def test_case_insensitive(self):
self.assertEqual(audio.select_source(self.sources, "dji mic mini").index, 58)
def test_exact_name_wins_over_a_substring_of_another(self):
sources = self.sources + [audio.Source(70, "mv6", "Something else entirely")]
self.assertEqual(audio.select_source(sources, "mv6").index, 70)
def test_ambiguous_match_is_deterministic(self):
# Two headsets, one pattern: the answer must be the same after a reboot, not
# whichever pactl happened to list first.
sources = self.sources + [audio.Source(9, "bluez_input.OTHER.headset", "Other headset mic")]
self.assertEqual(audio.select_source(sources, "headset").index, 9)
def test_no_match_is_none_not_a_guess(self):
self.assertIsNone(audio.select_source(self.sources, "rode wireless"))
def test_empty_pattern_is_none(self):
self.assertIsNone(audio.select_source(self.sources, " "))
def test_a_monitor_can_never_be_selected(self):
# The pattern below matches the monitor's description and nothing else; the
# right answer is None, never "transmit the desktop's audio".
self.assertIsNone(audio.select_source(self.sources, "Monitor of Family"))
class StreamMoveTests(unittest.TestCase):
def setUp(self):
self.streams = audio.parse_capture_streams(STREAMS)
self.dji = audio.Source(58, "alsa_input.usb-DJI_MIC_MINI-00.mono-fallback", "DJI")
self.mv6 = audio.Source(46, "alsa_input.usb-Shure_Inc_MV6-00.mono-fallback", "MV6")
def test_only_configured_applications_move(self):
moved = audio.streams_to_move(self.streams, ["Discord"], self.dji)
self.assertEqual([s.index for s in moved], [12])
def test_streams_already_on_the_target_are_left_alone(self):
moved = audio.streams_to_move(self.streams, ["Discord"], self.mv6)
self.assertEqual([s.index for s in moved], [14])
def test_no_applications_configured_moves_nothing(self):
self.assertEqual(audio.streams_to_move(self.streams, [], self.dji), [])
def test_matching_is_case_insensitive_and_partial(self):
moved = audio.streams_to_move(self.streams, ["discord"], self.dji)
self.assertEqual([s.index for s in moved], [12])
class RemoteSourceTests(unittest.TestCase):
"""The generated hooks for a microphone on another machine.
These are worth pinning because the STOP hook is a safety property: a room
microphone still streaming after the switch left it is a hot mic in a shared flat,
and "the generator emitted a stop command" is exactly the kind of thing that
silently stops being true.
"""
def setUp(self):
self.remote = {
"room": "loggia",
"description": "Loggia panel mic",
"remote": {"host": "touch-panel-loggia", "user": "kiosk", "latency_ms": 40},
}
self.local = {"room": "kitchen", "source": "DJI MIC MINI"}
def test_remote_source_gets_a_generated_name(self):
self.assertEqual(generate.remote_source_name(self.remote), "micfollow_loggia")
def test_an_explicit_name_wins(self):
named = dict(self.remote, source="my_own_name")
self.assertEqual(generate.remote_source_name(named), "my_own_name")
def test_remote_source_gets_both_hooks(self):
entry = generate.source_entry(self.remote, Path("/repo"))
self.assertEqual(entry["source"], "micfollow_loggia")
self.assertIn("remote-mic.sh start loggia", entry["start_command"])
self.assertIn("remote-mic.sh stop loggia", entry["stop_command"])
def test_every_remote_source_has_a_stop_hook(self):
for host in ("panel", "other-machine"):
entry = generate.source_entry(
{"room": "x", "remote": {"host": host}}, Path("/repo"))
self.assertTrue(entry["stop_command"], "a remote source without a stop hook")
def test_local_source_is_left_alone(self):
entry = generate.source_entry(self.local, Path("/repo"))
self.assertEqual(entry["source"], "DJI MIC MINI")
self.assertEqual(entry["start_command"], "")
self.assertEqual(entry["stop_command"], "")
def test_env_carries_the_transport_parameters(self):
env = generate.build_remote_env(self.remote, Path("/repo"))
self.assertIn("REMOTE_HOST=touch-panel-loggia", env)
self.assertIn("REMOTE_USER=kiosk", env)
self.assertIn("SOURCE_NAME=micfollow_loggia", env)
self.assertIn("LATENCY_MS=40", env)
# Empty means "the panel's own default input", which is what lets this work
# without knowing that machine's device names.
self.assertIn("REMOTE_MIC=", env)
def test_room_labels_are_short_and_readable(self):
self.assertEqual(generate.room_label("loggia"), "Loggia")
self.assertEqual(generate.room_label("living_room"), "Living room")
self.assertEqual(generate.room_label("desk"), "Desk")
if __name__ == "__main__":
unittest.main(verbosity=2)

6
pantry-vision/Dockerfile
View File

@ -8,8 +8,8 @@ ENV PYTHONUNBUFFERED=1 \
WORKDIR /app
# stdlib only (urllib for the Ollama/Grocy calls, sqlite3 for doorway.py's hint
# store) — no requirements.txt, same call as admin-canvas/server.py.
COPY server.py doorway.py ./
# stdlib only (urllib for the Ollama/Grocy calls) — no requirements.txt, same call as
# admin-canvas/server.py.
COPY server.py ./
CMD ["python", "server.py"]

206
pantry-vision/README.md
View File

@ -9,126 +9,21 @@ display then shows the resulting inventory ordered by what expires soonest, and
Grocy's recipes, on request.
- **`pantry-vision`** (this directory) — a small always-on Python HTTP service.
`POST /identify` (a photo → a proposal via an Ollama vision model), the writes
`POST /confirm` / `POST /consume` / `POST /adjust` / `POST /transfer`, the
door-sensor hook `POST /doorway-event`, and the reads `GET /inventory`,
`GET /expired`, `GET /doorway-events`, `GET /recipes`, `GET /shopping-list` (which
proxy Grocy, reshaped for a frontend). Every endpoint is bearer-token gated.
`/shopping-list` is also consumed by `hosts/door-panel/`'s dashboard (Phase 18,
"groceries running low") — it's a thin reshape of Grocy's own
`/api/stock/volatile` `missing_products`, not new inventory logic.
- **`doorway.py`** — the appliance-door half: a contact sensor fires, a camera burst
is pulled from Frigate, and what it recognised is recorded as a **hint**. See "Which
fridge is it in" below, and `docs/fridge-item-location.md` for the argument.
`POST /identify` (a photo → a proposal via an Ollama vision model), `POST /confirm`
(a human-reviewed proposal → written into Grocy stock), `GET /inventory`,
`GET /recipes`, and `GET /shopping-list` (all three proxy Grocy, reshaped for a
frontend). All five endpoints are bearer-token gated. `/shopping-list` is also
consumed by `hosts/door-panel/`'s dashboard (Phase 18, "groceries running low") —
it's a thin reshape of Grocy's own `/api/stock/volatile` `missing_products`, not
new inventory logic.
- **`frontend/`** — the static single-page app the kitchen display's kiosk browser
loads: the four stock-movement screens below, plus Inventory and Recipes. Vanilla
JS, no build step, no framework — same "vendored, dependency-free" choice as the
digest/admin canvas SDKs. Served read-only by a `pantry-web` nginx container
(`setup-container-host.sh`), the same role `digest-web`/`admin-web` already play for
their own hosts.
loads: Scan / Inventory / Recipes, vanilla JS, no build step, no framework — same
"vendored, dependency-free" choice as the digest/admin canvas SDKs. Served
read-only by a `pantry-web` nginx container (`setup-container-host.sh`), the same
role `digest-web`/`admin-web` already play for their own hosts.
- **`hosts/kitchen-display/`** — the touch kiosk image that runs the frontend. See
that directory's own README for the device side of this.
## The four ways stock moves
Stock only ever changes in four ways, so the display has four buttons, and three of
them are camera-first.
| Screen | What it does | Grocy call underneath |
| --- | --- | --- |
| **Unload groceries** | The camera runs in a loop. Hold up an item, get a stats card — description, best-before, where to put it, and how many individual things are in the pack — confirm, and it moves straight on to the next item without you touching the screen again. | `stock/products/{id}/add` |
| **Consume article** | Hold up what you are about to eat. If the same kind is in stock under more than one brand, it asks which; if the row holds more than one unit, it asks how many. | `stock/products/{id}/consume` |
| **List expired foods** | Everything already past its date. Clear a line by scanning the item you are about to bin (or with the row's own button when the label is unreadable). Booked out as **spoiled**, not eaten — Grocy keeps those apart, and that is the only way you will ever find out what the household keeps buying and throwing away. | `stock/products/{id}/consume` with `spoiled` |
| **Edit inventory** | The keyboard-and-buttons fallback: every article, `−`/`+`, and a freeform amount. This is the screen that does *not* use the camera, on purpose — it is where you go when the camera got something wrong. | `stock/products/{id}/inventory` |
Corrections go through Grocy's *inventory-correction* endpoint rather than
consume/add, so the stock journal says "somebody fixed the number" instead of
quietly filling the household's consumption history with corrections dressed as
meals.
### Two invariants everything else rests on
**Stock is counted in individual units, never in packages.** A twelve-pack of eggs is
booked in as twelve. "How many eggs do we have" is the question people actually ask,
and it makes booking out three of them arithmetic instead of a fractions-of-a-pack
problem. The vision model's `units_per_package` is a multiplier applied once on the
confirm screen — where it is editable, and where the screen states the result ("Books
in 12 eggs") before anything is written. Nothing downstream stores it. Set
`GROCY_DEFAULT_QU_ID` to a *Piece*-like unit accordingly.
**The fold key is the brand-free product kind.** Twelve eggs of brand X and ten of
brand Y are twenty-two eggs. The vision model is asked for a brand-free `kind`
alongside the brand, `/confirm` files the Grocy product under a **product group**
named after that kind, and the folding is a read of that group — so the grouping is
visible and editable in Grocy's own UI instead of living in a second classification
scheme here. The per-brand rows are never summed away: they are what the edit screen
expands to, because "take ten off brand Y specifically" has to stay possible.
The folding itself is deliberately dumb (casefold, strip a trailing `(Brand)`,
collapse whitespace). The clever version is a synonym problem — *eggs* vs *egg* vs
*free-range eggs* — that nobody wants adjudicated by a kitchen display at 19:00. When
the model answers inconsistently you get two lines instead of one, which is visible
and fixable; a wrongly merged line is neither.
## Which fridge is it in
The household has more than one cold appliance, so "in the fridge" is not an answer.
Two mechanisms address that, and **only the first is authoritative**:
**1. Grocy locations — the record.** The confirm screen's placement choice resolves to
a real Grocy location (`PANTRY_LOCATION_*`, created by name) and rides along on the
stock add. `/inventory` reports it. `POST /transfer` moves an amount between
appliances, and the edit screen has a "Move to…" control per row. That transfer is the
part that makes locations worth recording at all: without it, a location decays into
"where it was when it was bought", which is worse than no answer because it is
confidently wrong.
**2. Doorway hints — an observation, never a fact.** A Zigbee contact sensor on each
appliance door drives a Home Assistant automation that POSTs to `/doorway-event`:
```yaml
# Home Assistant automation — one per appliance door.
trigger:
- platform: state
entity_id: binary_sensor.fridge_kitchen_door
to: "on"
action:
- service: rest_command.pantry_doorway_event
data: { appliance: "fridge-kitchen", state: "opened" }
```
`pantry-vision` answers **202 immediately** and does the work on its own thread — the
burst is up to three snapshots and a vision call each, and an appliance door is not
something to keep a home automation waiting on. It pulls frames from that appliance's
Frigate camera, stops at the first one that identifies something, and writes what it
saw to its own SQLite file with a timestamp and a confidence.
**Nothing in that path writes stock.** A camera at a door cannot tell in from out,
misses when two things are carried at once, and sees nothing behind an arm. So it
produces "camera last saw something like this at Freezer (loggia), 20 min ago — a
sighting, not a fact", which is a sentence a person can evaluate, shown next to the
location Grocy actually records. Acting on it is a tap on Move.
Configure it with `PANTRY_DOOR_APPLIANCES` (`id:Grocy location name:frigate_camera`,
camera optional) and `FRIGATE_URL` — the same Frigate `chores` already pulls snapshots
from. **An appliance with a door sensor and no camera is a legitimate configuration**
and is the recommended starting point: it still records that the door opened, which is
the half of this feature that pays for itself.
### What the camera loop actually does
`frontend/app.js` samples a 32×24 greyscale thumbnail of the video every 700 ms and
only spends an `/identify` call when the picture has **settled** (it stopped moving)
*and* **changed** since the last thing it identified. Both gates exist because vision
latency is this phase's known open risk: the frames worth spending it on are the ones
where somebody is holding something still, and the same tin must never be identified
twice because nobody moved. "Identify now" overrides both, for the shiny jar under a
downlight that never settles.
A model that is down or answering nonsense **stops the loop** rather than
re-photographing the counter at it; the unload screen drops you into the manual form
instead.
## A real network listener, unlike admin-canvas
`admin-canvas` deliberately has **no published port** — only Home Assistant, on the
@ -154,21 +49,6 @@ already applies to identity-merge confirmation (see the *Identity store* row in
`docs/project-plan.md` §2) — a wrong camera guess costs one tap to fix, not a wrong
fact silently written into the household's inventory.
The book-out screens follow the same rule in the other direction, and it costs a tap
there too: recognition never consumes anything by itself, an ambiguous brand is asked
about rather than picked, and "throw away" states the amount before it goes. A camera
that quietly books out the wrong yoghurt produces an inventory nobody trusts, and an
inventory nobody trusts is worth exactly as much as no inventory.
One place where that guardrail is deliberately looser: the vision model is asked to
read a **printed best-before date** off the packaging when one is legible, and that
date lands pre-filled in the confirm screen's date field. The screen always says which
it is showing — *"Date read off the packaging — check it"* versus *"Estimated from
the category"* — and the server throws out anything more than a year in the past or
ten years out, because a misread label (small print, dot-matrix ink, curved packaging)
is the single most likely failure of this feature and that is what it looks like when
it happens.
## Configure
```sh
@ -204,40 +84,17 @@ responsive — this needs to be measured on real hardware, not assumed.
## Grocy API assumptions — unverified against a real instance
`server.py`'s Grocy calls (`_find_or_create_product`, `_add_to_stock`,
`_consume_stock`, `_set_stock_amount`, `_stock_items` and the `/recipes` proxy) are
written against Grocy's *documented* API shape, not checked against a running
instance. In particular:
`server.py`'s Grocy calls (`_find_or_create_product`, `_add_to_stock`, the `/inventory`
and `/recipes` proxies) are written against Grocy's *documented* API shape, not
checked against a running instance. In particular:
- Whether `GET /api/stock` rows carry a nested `product` object with a `name` field
by default, or need an explicit embed/expand parameter. `_stock_items` sidesteps
this by fetching `/api/objects/products` separately and joining on `product_id` —
it needs the group and location anyway, not just the name — and degrades to
`Product #<id>` if both are missing, rather than dropping the row.
by default, or need an explicit embed/expand parameter — `_handle_inventory`
degrades to `Product #<id>` if not, rather than dropping the row, but that's a
fallback, not a fix.
- Whether `POST /api/objects/products` with just
`name`/`location_id`/`qu_id_purchase`/`qu_id_stock` is actually enough to create a
minimal product on your Grocy version, or whether it requires more fields.
- Whether `POST /api/objects/product_groups` and `POST /api/objects/locations` accept
a bare `{"name": ...}`. Both are best-effort: a failure means the product lands
without a group (so it folds on its name instead) or in `GROCY_DEFAULT_LOCATION_ID`
(so it is in the wrong place but still in stock). Neither costs you the book-in.
- Whether `POST /api/stock/products/{id}/consume` takes `spoiled` as a boolean, and
what it returns when you try to consume more than is in stock — that refusal is
passed through to the screen verbatim, on the assumption that it is a real answer
("we have fewer than you think") rather than a transport failure.
- Whether `POST /api/stock/products/{id}/inventory` takes `new_amount` as the absolute
new figure and whether it *requires* `best_before_date` when the figure goes up. The
edit screen currently sends the amount alone.
- Whether `POST /api/stock/products/{id}/transfer` takes `location_id_from` /
`location_id_to`, and — the subtle one — **which location Grocy considers a stock
entry to be in.** `/transfer` uses the *product's* location as the source, which is
what `/confirm` set. Grocy can also hold one product's stock across several
locations at once, per entry; if your instance does that, the source this sends will
sometimes be wrong and the transfer will fail rather than move the wrong thing.
Reading `/api/stock/products/{id}/entries` is the fix if it comes up.
- Whether Frigate's `/api/<camera>/latest.jpg` returns a usable still for a camera that
is idle — `doorway.py` and `chores/check.py` both assume it does, neither has called
it.
- Whether the Recipes feature (`GET /api/objects/recipes`,
`GET /api/recipes/{id}/fulfillment`) needs to be explicitly enabled/populated
before it returns anything meaningful — `_handle_recipes` degrades to
@ -258,34 +115,7 @@ and its own README. It builds two containers: `pantry-vision` (this API) and
1. All of the Grocy API assumptions above.
2. Real-world vision-model accuracy and latency for grocery items — untested with
any actual model or camera. The unload loop is where latency bites hardest: it is
built to feel like a queue of items rather than a queue of round trips, and if an
`/identify` takes 20 seconds it will not feel like either.
3. **Whether the model returns a stable `kind` for the same product across scans.**
This is the assumption the whole brand-folding rests on, and the one most likely
to disappoint quietly: scan the same carton of eggs five times and see whether it
says "eggs" five times. If it drifts, the fix is a fixed `kind` vocabulary in the
prompt, not more clever folding on this side.
4. **Whether printed best-before dates are read correctly or confidently invented.**
The prompt says not to guess and the server bounds the range, but neither can catch
a plausible wrong date. Check a handful against the actual packets before trusting
the pre-filled field.
5. **That a twelve-pack really does arrive as twelve.** `units_per_package` is the
one model answer that gets multiplied rather than displayed, and the failure is
silent in both directions — twelve eggs booked in as one, or one jar booked in as
twelve.
6. **Whether a doorway camera can identify anything at all.** This is the assumption
the whole `/doorway-event` path rests on, and it is not the same task as the
kitchen display's: an item in a moving hand, at ~1.5 m, in whatever light the room
has, versus one held still against a plain background 30 cm from a webcam.
`docs/fridge-item-location.md` says to test it with the kitchen's existing camera
before buying a second one, and that advice applies to this code too — if the
answer is no, the door sensors are still worth having and this half simply stays
switched off (`PANTRY_DOOR_APPLIANCES` entries without a camera).
7. **Whether the hints are read as hints.** The wording on screen ("a sighting, not a
fact") is doing real work; if in practice people treat a sighting as the answer and
stop checking, that is a design failure this code cannot detect and the feature
should be turned off rather than tuned.
any actual model or camera.
3. Whether Ollama's `/api/generate` `images` field is still the right call shape for
whichever vision model you pick — some multimodal models are only exposed through
Ollama's newer `/api/chat` with a `images` field per-message instead; this was

286
pantry-vision/doorway.py
View File

@ -1,286 +0,0 @@
"""Door-sensor-triggered appliance cameras — the "which fridge is it in" half of
pantry-vision, from docs/fridge-item-location.md.
A Zigbee contact sensor on each cold appliance fires an HA automation, which POSTs
`/doorway-event` here. This module then pulls a snapshot (or a short burst) from that
appliance's camera via Frigate, runs it through the same vision identification the
kitchen display uses, and records what it saw as a **hint**.
WHY THIS IS A HINT AND NOT A FACT
---------------------------------
Read `docs/fridge-item-location.md` before changing anything here; the short version:
- **A camera at the door can only ever see the doorway.** It does not know whether the
item was going in or coming out, it misses when two things are carried at once, and
it sees nothing at all when an arm is in the way. Anything that presented its output
as the truth about where food is would be wrong several times a week, silently.
- So a hint is written with a timestamp, an appliance, and a confidence, and it is
shown as "last seen going past the loggia fridge, Tue 18:42" — a sentence a person
can evaluate. **Nothing here writes stock.** Moving an item between appliances is
`/transfer`, which a person taps, exactly like every other write in this service.
- Which means a wrong hint costs a glance. That is the whole design budget.
WHY A SEPARATE SQLITE FILE, WHEN "GROCY IS THE SYSTEM OF RECORD"
----------------------------------------------------------------
Grocy owns *stock*: what exists, how much, until when, and — via locations — where it
is meant to be. It has nowhere to put "a camera thinks it saw something like this go
past that door 40 seconds ago, and might be wrong". That is not inventory, it is
observation, and it has a retention life measured in days. Keeping it out of Grocy is
what stops a guess from ever being mistaken for stock — the same separation
digest-engine draws between its archive and its output.
The file is best-effort in the same way `archive.py` is over in digest-engine: a
corrupt or unwritable database logs a warning and the identification still happens.
No hint is worth failing a request over.
"""
from __future__ import annotations
import logging
import os
import sqlite3
import time
import urllib.error
import urllib.request
from datetime import datetime, timedelta, timezone
from pathlib import Path
LOG = logging.getLogger("pantry-vision.doorway")
DB_PATH = Path(os.environ.get("PANTRY_HINTS_DB_PATH", "/data/pantry-hints.db"))
# Same Frigate this project's `chores` already pulls snapshots from, same env name.
# Unset means the door-event endpoint accepts the event and records the opening
# without a picture — which is still worth having, see the module docstring.
FRIGATE_URL = os.environ.get("FRIGATE_URL", "").rstrip("/")
FRIGATE_TIMEOUT = float(os.environ.get("FRIGATE_TIMEOUT", "15"))
# How many frames to take per door event, and how far apart. An item crosses the
# doorway in about a second, so one frame is a coin toss and ten is a queue at the
# vision model. Three is a compromise, and the burst stops early on the first frame
# that actually identifies something.
BURST_FRAMES = int(os.environ.get("PANTRY_DOORWAY_BURST", "3"))
BURST_INTERVAL = float(os.environ.get("PANTRY_DOORWAY_BURST_INTERVAL", "0.7"))
# Hints are worthless once they are old — "last seen a month ago" tells you nothing a
# person didn't already know. Pruned on write, so the file cannot grow without bound.
HINT_RETENTION_DAYS = int(os.environ.get("PANTRY_HINT_RETENTION_DAYS", "30"))
def _now() -> str:
return datetime.now(timezone.utc).isoformat().replace("+00:00", "Z")
def appliances() -> dict[str, dict]:
"""PANTRY_DOOR_APPLIANCES: "id:Grocy location name:frigate_camera, ..."
The Grocy location name is the same string `PANTRY_LOCATION_*` uses, because the
point of the whole feature is to answer *which appliance*, and "which appliance"
has to mean the same thing here as it does on the confirm screen. The camera is
optional: an appliance with a door sensor and no camera still records openings,
which is the half of this feature that pays for itself.
"""
raw = os.environ.get("PANTRY_DOOR_APPLIANCES", "").strip()
result: dict[str, dict] = {}
for entry in raw.split(","):
entry = entry.strip()
if not entry:
continue
parts = [p.strip() for p in entry.split(":")]
if len(parts) < 2 or not parts[0] or not parts[1]:
LOG.warning("pantry-vision: ignoring malformed PANTRY_DOOR_APPLIANCES entry %r", entry)
continue
result[parts[0]] = {
"id": parts[0],
"location_name": parts[1],
"camera": parts[2] if len(parts) > 2 and parts[2] else "",
}
return result
# --- the hint store ---------------------------------------------------------------
def _db() -> sqlite3.Connection | None:
try:
DB_PATH.parent.mkdir(parents=True, exist_ok=True)
conn = sqlite3.connect(DB_PATH, timeout=5)
conn.row_factory = sqlite3.Row
conn.execute(
"""
CREATE TABLE IF NOT EXISTS hints (
id INTEGER PRIMARY KEY,
appliance TEXT NOT NULL,
location_name TEXT NOT NULL,
door_state TEXT NOT NULL,
product_id INTEGER,
kind TEXT,
name TEXT,
confidence TEXT,
identified INTEGER NOT NULL DEFAULT 0,
created_at TEXT NOT NULL
)
"""
)
conn.execute("CREATE INDEX IF NOT EXISTS hints_by_product ON hints (product_id, created_at)")
return conn
except (sqlite3.Error, OSError):
# OSError as well as sqlite3.Error, and this is not belt-and-braces: mkdir on
# an unmounted or read-only /data raises PermissionError, which is an OSError
# and not a database error at all. Without it, a deployment that forgot the
# bind mount would take down /inventory — the one screen that has nothing to
# do with hints — instead of quietly having no sightings to show.
LOG.warning("pantry-vision: hint database unusable at %s", DB_PATH, exc_info=True)
return None
def record_hint(appliance: dict, door_state: str, proposal: dict | None, product_id: int | None) -> None:
conn = _db()
if conn is None:
return
try:
with conn:
conn.execute(
"INSERT INTO hints (appliance, location_name, door_state, product_id, kind, name, "
"confidence, identified, created_at) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)",
(
appliance["id"],
appliance["location_name"],
door_state,
product_id,
(proposal or {}).get("kind") or "",
(proposal or {}).get("name") or "",
(proposal or {}).get("confidence") or "",
1 if proposal else 0,
_now(),
),
)
cutoff = (datetime.now(timezone.utc) - timedelta(days=HINT_RETENTION_DAYS)).isoformat()
conn.execute("DELETE FROM hints WHERE created_at < ?", (cutoff.replace("+00:00", "Z"),))
except sqlite3.Error:
LOG.warning("pantry-vision: could not write a doorway hint", exc_info=True)
finally:
conn.close()
def recent_hints(limit: int = 50) -> list[dict]:
conn = _db()
if conn is None:
return []
try:
rows = conn.execute(
"SELECT appliance, location_name, door_state, product_id, kind, name, confidence, "
"identified, created_at FROM hints ORDER BY created_at DESC LIMIT ?",
(max(1, min(500, limit)),),
).fetchall()
return [dict(row) for row in rows]
except sqlite3.Error:
LOG.warning("pantry-vision: could not read doorway hints", exc_info=True)
return []
finally:
conn.close()
def last_seen_by_product() -> dict[int, dict]:
"""{product_id: the most recent identified hint for it}.
Only identified hints: a door opening with nothing recognised in it says something
about the door, not about any particular jar, and attaching it to an item would be
inventing the very link this module refuses to invent.
"""
conn = _db()
if conn is None:
return {}
try:
# SQLite's documented bare-column rule: with MAX() in the select list, the
# other columns come from the row that matched it. That is what makes this one
# query instead of one per product.
rows = conn.execute(
"SELECT product_id, appliance, location_name, confidence, MAX(created_at) AS created_at "
"FROM hints WHERE product_id IS NOT NULL AND identified = 1 GROUP BY product_id"
).fetchall()
return {int(row["product_id"]): dict(row) for row in rows}
except sqlite3.Error:
LOG.warning("pantry-vision: could not read last-seen hints", exc_info=True)
return {}
finally:
conn.close()
# --- the camera -------------------------------------------------------------------
def _snapshot(camera: str) -> bytes | None:
"""One frame from Frigate, or None. Same endpoint shape `chores/check.py` uses.
VERIFY against a real Frigate: `/api/<camera>/latest.jpg` is its documented
always-available snapshot path, but nothing in this project has called it yet.
"""
if not (FRIGATE_URL and camera):
return None
try:
with urllib.request.urlopen(f"{FRIGATE_URL}/api/{camera}/latest.jpg", timeout=FRIGATE_TIMEOUT) as resp:
return resp.read()
except (urllib.error.URLError, urllib.error.HTTPError, TimeoutError, OSError):
LOG.warning("pantry-vision: could not fetch a snapshot from camera %r", camera, exc_info=True)
return None
def handle_event(appliance: dict, door_state: str, identify, match) -> dict:
"""Take a burst, identify the first frame that shows something, record the hint.
Runs off the request thread (see server.py) because a burst is up to three vision
calls and Home Assistant's `rest_command` should not be sitting on a socket for
however long the LLM host takes. `identify` and `match` are passed in rather than
imported so this module never has to know about Grocy — it deals in doors,
cameras and hints.
"""
camera = appliance.get("camera") or ""
if not camera:
record_hint(appliance, door_state, None, None)
LOG.info("pantry-vision: %s %s (no camera configured)", appliance["id"], door_state)
return {"appliance": appliance["id"], "identified": False, "reason": "no camera configured"}
proposal = None
for attempt in range(max(1, BURST_FRAMES)):
if attempt:
time.sleep(BURST_INTERVAL)
image = _snapshot(camera)
if image is None:
break
candidate = identify(image)
# `present: false` is the model saying the frame holds no grocery item — an
# arm, a closed door, an empty kitchen. That is the expected answer for most
# frames of most openings, and it is not a failure.
if candidate.get("present") and not candidate.get("degraded"):
proposal = candidate
break
if proposal is None:
record_hint(appliance, door_state, None, None)
LOG.info("pantry-vision: %s %s — nothing recognised in %d frame(s)", appliance["id"], door_state, BURST_FRAMES)
return {"appliance": appliance["id"], "identified": False, "reason": "nothing recognised"}
# An exact stock match makes the hint attachable to a row the household can act
# on. Without one it is still worth recording — "something like eggs went past the
# loggia freezer" is a useful thing to have seen, and it is exactly what a person
# would search for after failing to find eggs.
product_id = None
try:
matches = match(proposal.get("kind", ""), proposal.get("name", ""))
exact = [m for m in matches if m.get("exact")]
if exact:
product_id = int(exact[0]["product_id"])
elif len(matches) == 1:
product_id = int(matches[0]["product_id"])
except Exception:
LOG.warning("pantry-vision: could not match a doorway identification to stock", exc_info=True)
record_hint(appliance, door_state, proposal, product_id)
LOG.info(
"pantry-vision: %s %s — saw %r (confidence %s, stock row %s)",
appliance["id"], door_state, proposal.get("name"), proposal.get("confidence"), product_id,
)
return {
"appliance": appliance["id"],
"identified": True,
"name": proposal.get("name"),
"confidence": proposal.get("confidence"),
"product_id": product_id,
}

820
pantry-vision/frontend/app.js
View File

@ -6,17 +6,6 @@
// kitchen-display kiosk's own launch command, NOT hardcoded here — this file is a
// generic static asset with no secret in it, served read-only by pantry-web to
// whatever device points a browser at it.
//
// FOUR FLOWS, ONE CAMERA, ONE RULE
// --------------------------------
// Unload (book in), Consume (book out), Expired (throw away) and Edit (correct) are
// the four ways stock moves, and the first three are camera-first: hold the thing up,
// the model says what it is, a person confirms, it is written. The rule that shapes
// every one of them is server.py's — the camera proposes, the person disposes.
// Nothing here calls a write endpoint without a tap in between, including the flows
// where that costs an extra tap, because a camera that quietly books out the wrong
// brand of yoghurt produces an inventory nobody trusts, and an inventory nobody
// trusts is the same as no inventory.
"use strict";
const params = new URLSearchParams(location.search);
@ -43,101 +32,45 @@ function api(path, options) {
});
}
function postJson(path, body) {
return api(path, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify(body),
});
}
const $ = (id) => document.getElementById(id);
// --- Navigation ----------------------------------------------------------------
// Three tabs (Home / Inventory / Recipes) plus four flow screens reached from Home.
// "scan" is kept as an alias for the unload flow: it is the fragment
// kitchen-display-agent's "Show scan" MQTT button has been publishing since Phase 17,
// and a button in Home Assistant that stops working is a worse outcome than an
// old name living on here.
const SCREENS = ["home", "unload", "consume", "expired", "edit", "inventory", "recipes"];
const TABS = ["home", "inventory", "recipes"];
let currentScreen = "home";
function show(name) {
if (name === "scan") name = "unload";
if (!SCREENS.includes(name)) name = "home";
// Leaving a flow always releases the camera and cancels its loop: the kiosk has one
// webcam and a scan loop left running behind another screen would keep firing
// /identify at the LLM host with nobody watching the answers.
if (currentScreen !== name) stopScanner();
currentScreen = name;
document.querySelectorAll(".panel").forEach((p) => p.classList.toggle("active", p.id === name));
document.querySelectorAll(".tab").forEach((t) => t.classList.toggle("active", t.dataset.tab === name));
// A flow screen is not a tab; keep Home lit while one is open so the tab bar never
// shows nothing selected.
if (!TABS.includes(name)) document.querySelector('.tab[data-tab="home"]').classList.add("active");
// --- Tabs --------------------------------------------------------------------
const tabs = document.querySelectorAll(".tab");
const panels = document.querySelectorAll(".panel");
function activateTab(name) {
tabs.forEach((t) => t.classList.toggle("active", t.dataset.tab === name));
panels.forEach((p) => p.classList.toggle("active", p.id === name));
if (name === "inventory") loadInventory();
if (name === "recipes") loadRecipes();
if (name === "unload") startUnload();
if (name === "consume") startConsume();
if (name === "expired") startExpired();
if (name === "edit") loadEditInventory();
if (name === "scan") resetScan();
}
document.querySelectorAll(".tab").forEach((t) => t.addEventListener("click", () => show(t.dataset.tab)));
document.querySelectorAll("[data-goto]").forEach((b) => b.addEventListener("click", () => show(b.dataset.goto)));
tabs.forEach((t) => t.addEventListener("click", () => activateTab(t.dataset.tab)));
function toast(message, kind) {
const el = $("toast");
el.textContent = message;
el.className = kind || "";
el.hidden = false;
clearTimeout(toast._timer);
toast._timer = setTimeout(() => {
el.hidden = true;
}, 2600);
// A "Show <tab>" MQTT command from touchpanel-style HA control arrives as a URL
// fragment reload (hosts/kitchen-display/agent kills and relaunches Chromium at
// index.html#recipes, same pattern as the thin client's digest-browser) — honour it
// on load, same as any manual tap.
if (location.hash) {
const initial = location.hash.slice(1);
if (["scan", "inventory", "recipes"].includes(initial)) activateTab(initial);
}
// --- The camera ----------------------------------------------------------------
// One <video>, one MediaStream, moved between flows. The auto-scan loop samples a
// tiny greyscale thumbnail of each frame and only spends an /identify call when the
// picture has (a) settled and (b) actually changed since the last thing it
// identified. Both gates exist for the same reason: the vision model's latency is the
// known open risk in this phase (project-plan.md open decision #18), so the frames
// worth spending it on are the ones where somebody is holding something still, and
// the same tin should never be identified twice because nobody moved.
const SAMPLE_INTERVAL_MS = 700;
const STILL_THRESHOLD = 6; // mean per-pixel difference below which the frame is "settled"
const CHANGED_THRESHOLD = 12; // ...and above which it is a different item from the last one
const video = $("camera-preview");
const scanner = $("scanner");
const scanStatus = $("scan-status");
const cameraError = $("camera-error");
const captureNowBtn = $("capture-now");
const frameSample = $("frame-sample");
const capturedFrame = $("captured-frame");
// --- Scan ----------------------------------------------------------------------
const video = document.getElementById("camera-preview");
const captureBtn = document.getElementById("capture-btn");
const cameraError = document.getElementById("camera-error");
const scanCamera = document.getElementById("scan-camera");
const scanResult = document.getElementById("scan-result");
const capturedFrame = document.getElementById("captured-frame");
const confirmForm = document.getElementById("confirm-form");
const confirmStatus = document.getElementById("confirm-status");
const retakeBtn = document.getElementById("retake-btn");
let stream = null;
let scanTimer = null;
let scanBusy = false;
let previousSample = null;
let acceptedSample = null;
let onIdentified = null;
function mountScanner(screenId) {
const slot = document.querySelector(`#${screenId} .scanner-slot`);
if (slot && scanner.parentElement !== slot) slot.appendChild(scanner);
scanner.hidden = false;
}
function startCamera() {
if (stream) return Promise.resolve();
return navigator.mediaDevices
if (stream) return;
navigator.mediaDevices
.getUserMedia({ video: { facingMode: "environment" }, audio: false })
.then((s) => {
stream = s;
@ -147,148 +80,79 @@ function startCamera() {
.catch((err) => {
cameraError.textContent = `Camera unavailable: ${err.message}. See hosts/kitchen-display/README.md.`;
cameraError.hidden = false;
throw err;
});
}
function stopScanner() {
clearInterval(scanTimer);
scanTimer = null;
scanBusy = false;
onIdentified = null;
previousSample = null;
acceptedSample = null;
scanner.hidden = true;
if (stream) {
stream.getTracks().forEach((t) => t.stop());
stream = null;
video.srcObject = null;
}
function resetScan() {
scanResult.hidden = true;
scanCamera.hidden = false;
confirmStatus.textContent = "";
startCamera();
}
/** A 32x24 greyscale thumbnail of the current frame, as a plain array. */
function sampleFrame() {
if (!video.videoWidth) return null;
frameSample.width = 32;
frameSample.height = 24;
const ctx = frameSample.getContext("2d", { willReadFrequently: true });
ctx.drawImage(video, 0, 0, 32, 24);
const data = ctx.getImageData(0, 0, 32, 24).data;
const grey = new Array(32 * 24);
for (let i = 0; i < grey.length; i++) {
const p = i * 4;
grey[i] = (data[p] * 299 + data[p + 1] * 587 + data[p + 2] * 114) / 1000;
}
return grey;
}
function frameDelta(a, b) {
if (!a || !b) return Infinity;
let total = 0;
for (let i = 0; i < a.length; i++) total += Math.abs(a[i] - b[i]);
return total / a.length;
}
/**
* Run the camera on `screenId` and call `handler(proposal)` the first time it sees
* something. The loop stops itself on a hit; the flow calls resumeScan() when the
* person is done with that item and ready for the next one.
*/
function startScan(screenId, handler, statusText) {
onIdentified = handler;
mountScanner(screenId);
scanStatus.textContent = "Starting camera…";
startCamera()
.then(() => {
scanStatus.textContent = statusText || "Hold an item up to the camera…";
previousSample = null;
clearInterval(scanTimer);
scanTimer = setInterval(tick, SAMPLE_INTERVAL_MS);
})
.catch(() => {
scanStatus.textContent = "";
});
}
function resumeScan(statusText) {
if (!stream || !onIdentified) return;
scanner.hidden = false;
scanStatus.textContent = statusText || "Ready for the next one…";
previousSample = null;
clearInterval(scanTimer);
scanTimer = setInterval(tick, SAMPLE_INTERVAL_MS);
}
function pauseScan() {
clearInterval(scanTimer);
scanTimer = null;
}
function tick() {
if (scanBusy) return;
const sample = sampleFrame();
if (!sample) return;
const settled = frameDelta(sample, previousSample) < STILL_THRESHOLD;
const changed = frameDelta(sample, acceptedSample) > CHANGED_THRESHOLD;
previousSample = sample;
if (!settled) return;
if (!changed) {
scanStatus.textContent = "Waiting for the next item…";
return;
}
identifyNow(sample);
}
/** Capture at full resolution and ask the server what it is. */
function identifyNow(sample) {
if (scanBusy) return;
scanBusy = true;
scanStatus.textContent = "Identifying…";
captureBtn.addEventListener("click", () => {
if (!stream) return;
capturedFrame.width = video.videoWidth;
capturedFrame.height = video.videoHeight;
capturedFrame.getContext("2d").drawImage(video, 0, 0);
scanCamera.hidden = true;
scanResult.hidden = false;
confirmStatus.textContent = "Identifying…";
document.getElementById("f-confidence").textContent = "";
capturedFrame.toBlob(
(blob) => {
api("/identify", { method: "POST", body: blob, headers: { "Content-Type": "image/jpeg" } })
.then((proposal) => {
scanBusy = false;
if (!proposal.present) {
scanStatus.textContent = "Nothing recognised — hold the item closer.";
return;
}
acceptedSample = sample || sampleFrame();
pauseScan();
if (onIdentified) onIdentified(proposal);
document.getElementById("f-name").value = proposal.name || "";
document.getElementById("f-category").value = proposal.category || "other";
const days = Number.isFinite(proposal.estimated_shelf_life_days)
? proposal.estimated_shelf_life_days
: 7;
const due = new Date();
due.setDate(due.getDate() + days);
document.getElementById("f-date").value = due.toISOString().slice(0, 10);
document.getElementById("f-confidence").textContent =
`Model confidence: ${proposal.confidence || "unknown"}` +
(proposal.note ? ` — ${proposal.note}` : "") +
". Review before confirming.";
confirmStatus.textContent = "";
})
.catch((err) => {
scanBusy = false;
pauseScan();
scanStatus.textContent = `Could not identify: ${err.message}`;
// A dead vision model must not turn into a loop that keeps photographing
// the counter at it. The flow decides what to offer instead — for unload,
// the manual form; for the others, a message and a way out.
if (onIdentified) onIdentified({ present: true, degraded: true, error: err.message });
confirmStatus.textContent = `Could not identify: ${err.message}. Fill in manually.`;
});
},
"image/jpeg",
0.85
);
}
// "Identify now" is the escape hatch for the stillness gate: a shiny jar under a
// kitchen downlight can flicker enough to never settle, and standing there waving it
// is not an acceptable answer.
captureNowBtn.addEventListener("click", () => {
if (stream && !scanBusy) identifyNow(null);
});
// --- Shared item helpers -------------------------------------------------------
function escapeHtml(s) {
return String(s).replace(/[&<>"']/g, (c) => ({ "&": "&amp;", "<": "&lt;", ">": "&gt;", '"': "&quot;", "'": "&#39;" }[c]));
}
retakeBtn.addEventListener("click", resetScan);
confirmForm.addEventListener("submit", (event) => {
event.preventDefault();
confirmStatus.textContent = "Adding…";
api("/confirm", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
name: document.getElementById("f-name").value.trim(),
category: document.getElementById("f-category").value,
best_before_date: document.getElementById("f-date").value,
quantity: Number(document.getElementById("f-quantity").value) || 1,
}),
})
.then(() => {
confirmStatus.textContent = "Added. Put it away!";
setTimeout(resetScan, 1500);
})
.catch((err) => {
confirmStatus.textContent = `Could not add: ${err.message}`;
});
});
// --- Inventory -------------------------------------------------------------------
function urgencyClass(daysLeft) {
if (daysLeft === null || daysLeft === undefined) return "";
if (daysLeft < 0) return "urgent-expired";
@ -297,508 +161,30 @@ function urgencyClass(daysLeft) {
return "";
}
function dueLabel(days) {
if (days === null || days === undefined) return "no date";
if (days < 0) return `expired ${-days}d ago`;
if (days === 0) return "expires today";
return `${days}d left`;
}
function amountOf(row) {
const n = Number(row.amount);
return Number.isFinite(n) ? n : 0;
}
/** "12 × Eggs (Brand X)" the way every list in here writes it. */
function rowLabel(row) {
return `${amountOf(row)} × ${row.name}`;
}
// --- Unload groceries (book in) ------------------------------------------------
const confirmForm = $("confirm-form");
let unloadAdded = 0;
function startUnload() {
unloadAdded = 0;
$("unload-count").textContent = "";
confirmForm.hidden = true;
startScan("unload", onUnloadProposal, "Hold the first item up to the camera…");
}
function onUnloadProposal(proposal) {
scanner.hidden = false;
confirmForm.hidden = false;
$("confirm-status").textContent = "";
const degraded = !!proposal.degraded;
$("f-description").textContent = degraded
? "Could not identify this one — fill it in by hand."
: proposal.description || proposal.name || "";
$("f-name").value = proposal.name || "";
$("f-category").value = proposal.category || "other";
$("f-placement").value = proposal.recommended_placement || "cupboard";
$("f-date").value = proposal.best_before_date || new Date().toISOString().slice(0, 10);
$("f-date-source").textContent =
proposal.best_before_source === "label"
? "Date read off the packaging — check it."
: "Estimated from the category, not read off the packaging.";
$("f-packages").value = 1;
$("f-units").value = proposal.units_per_package || 1;
confirmForm.dataset.unitName = proposal.unit_name || "piece";
confirmForm.dataset.kind = proposal.kind || "";
confirmForm.dataset.brand = proposal.brand || "";
updateTotalLine();
// "You already have ten of these" at the only moment it can still change what
// somebody does — while the shopping is still on the counter.
const matches = proposal.stock_matches || [];
const held = matches.reduce((sum, m) => sum + amountOf(m), 0);
$("f-have").textContent = held
? `Already in stock: ${held} (${matches.length} ${matches.length === 1 ? "entry" : "entries"}).`
: proposal.stock_matches_unavailable
? "Could not check what is already in stock — Grocy did not answer."
: "";
$("f-confidence").textContent = degraded
? proposal.error || proposal.note || ""
: `Model confidence: ${proposal.confidence || "unknown"}. Review before confirming.`;
}
function updateTotalLine() {
const packs = Number($("f-packages").value) || 1;
const per = Number($("f-units").value) || 1;
const unit = confirmForm.dataset.unitName || "piece";
const total = packs * per;
$("f-total").textContent =
per > 1
? `Books in ${total} ${unit}${total === 1 ? "" : "s"} (${packs} × ${per}).`
: `Books in ${total} ${unit}${total === 1 ? "" : "s"}.`;
}
$("f-packages").addEventListener("input", updateTotalLine);
$("f-units").addEventListener("input", updateTotalLine);
$("skip-btn").addEventListener("click", () => {
confirmForm.hidden = true;
resumeScan("Skipped. Hold up the next item…");
});
confirmForm.addEventListener("submit", (event) => {
event.preventDefault();
$("confirm-status").textContent = "Adding…";
postJson("/confirm", {
name: $("f-name").value.trim(),
kind: confirmForm.dataset.kind || "",
brand: confirmForm.dataset.brand || "",
category: $("f-category").value,
placement: $("f-placement").value,
best_before_date: $("f-date").value,
quantity: Number($("f-packages").value) || 1,
units_per_package: Number($("f-units").value) || 1,
})
.then((result) => {
unloadAdded += 1;
$("unload-count").textContent = `${unloadAdded} booked in`;
$("confirm-status").textContent = "";
confirmForm.hidden = true;
toast(`Added ${result.amount} × ${$("f-name").value.trim()} — put it away!`);
resumeScan("Hold up the next item…");
})
.catch((err) => {
$("confirm-status").textContent = `Could not add: ${err.message}`;
});
});
// --- Consume article (book out) ------------------------------------------------
let consumeChoice = null;
let consumeUnitsPerPackage = 1;
function startConsume() {
$("consume-result").hidden = true;
$("consume-status").textContent = "";
startScan("consume", onConsumeProposal, "Hold up what you are about to use…");
}
function onConsumeProposal(proposal) {
$("consume-result").hidden = false;
$("consume-amount").hidden = true;
$("consume-status").textContent = "";
consumeUnitsPerPackage = proposal.units_per_package || 1;
if (proposal.degraded) {
$("consume-identified").textContent = "Could not identify that.";
$("consume-candidates").innerHTML =
'<p class="hint">Try again, or correct the amount by hand on the Edit inventory screen.</p>';
addRescanButton();
return;
}
const matches = proposal.stock_matches || [];
$("consume-identified").textContent = `Looks like: ${proposal.name}`;
if (!matches.length) {
$("consume-candidates").innerHTML =
'<p class="hint">Nothing matching that is booked in, so there is nothing to book out. ' +
"If it should be in stock, add it on the Unload screen first.</p>";
addRescanButton();
return;
}
// One unambiguous match goes straight to the amount step; several mean the camera
// knows the kind but not the brand, and picking one at random here would be exactly
// the silent wrong write this whole service is arranged to avoid.
const exact = matches.filter((m) => m.exact);
if (exact.length === 1) {
chooseConsumeRow(exact[0]);
return;
}
if (matches.length === 1) {
chooseConsumeRow(matches[0]);
return;
}
$("consume-candidates").innerHTML =
'<p class="hint">Which one? (Same kind, different brands or dates.)</p>' +
matches
.map(
(m, i) => `<button class="card choice" data-index="${i}">
<span class="card-name">${escapeHtml(rowLabel(m))}</span>
<span class="card-due">${escapeHtml(dueLabel(m.days_left))}</span>
</button>`
)
.join("");
$("consume-candidates")
.querySelectorAll("button.choice")
.forEach((btn) => btn.addEventListener("click", () => chooseConsumeRow(matches[Number(btn.dataset.index)])));
}
function addRescanButton() {
const btn = document.createElement("button");
btn.className = "big-btn secondary";
btn.textContent = "Scan again";
btn.addEventListener("click", () => {
$("consume-result").hidden = true;
resumeScan("Hold up what you are about to use…");
});
$("consume-candidates").appendChild(btn);
}
function chooseConsumeRow(row) {
consumeChoice = row;
$("consume-candidates").innerHTML = "";
$("consume-amount").hidden = false;
$("consume-chosen").textContent = `${row.name} — ${amountOf(row)} in stock, ${dueLabel(row.days_left)}`;
// "If multipack, ask how many": in stock terms a multipack is simply a row holding
// more than one unit, since everything was booked in as individual units. The quick
// buttons are the answers people actually give — one, the whole pack, or all of it.
const held = amountOf(row);
const quick = [1];
if (consumeUnitsPerPackage > 1 && consumeUnitsPerPackage <= held) quick.push(consumeUnitsPerPackage);
if (held > 1 && !quick.includes(held)) quick.push(held);
$("consume-qty").value = 1;
$("consume-qty").max = held || 1;
$("consume-quick").innerHTML = quick
.map(
(n) =>
`<button type="button" class="quick-btn" data-amount="${n}">${
n === held ? `all ${n}` : n === consumeUnitsPerPackage && n > 1 ? `whole pack (${n})` : n
}</button>`
)
.join("");
$("consume-quick")
.querySelectorAll(".quick-btn")
.forEach((btn) =>
btn.addEventListener("click", () => {
$("consume-qty").value = btn.dataset.amount;
})
);
}
$("consume-cancel").addEventListener("click", () => {
consumeChoice = null;
$("consume-result").hidden = true;
resumeScan("Hold up what you are about to use…");
});
$("consume-confirm").addEventListener("click", () => {
if (!consumeChoice) return;
const amount = Number($("consume-qty").value);
if (!(amount > 0)) {
$("consume-status").textContent = "Enter how many.";
return;
}
$("consume-status").textContent = "Booking out…";
postJson("/consume", { product_id: consumeChoice.product_id, amount, spoiled: false })
.then(() => {
toast(`Booked out ${amount} × ${consumeChoice.name}`);
consumeChoice = null;
$("consume-result").hidden = true;
$("consume-status").textContent = "";
resumeScan("Hold up the next thing…");
})
.catch((err) => {
$("consume-status").textContent = `Could not book out: ${err.message}`;
});
});
// --- Expired foods -------------------------------------------------------------
let expiredRows = [];
function startExpired() {
scanner.hidden = true;
$("expired-status").textContent = "";
loadExpired();
}
function loadExpired() {
const el = $("expired-list");
api("/expired")
.then((data) => {
expiredRows = data.items || [];
if (!expiredRows.length) {
el.innerHTML = '<p class="hint">Nothing has expired. </p>';
return;
}
el.innerHTML = expiredRows
.map(
(row, i) => `<div class="card urgent-expired">
<span class="card-name">${escapeHtml(rowLabel(row))}</span>
<span class="card-due">${escapeHtml(dueLabel(row.days_left))}</span>
<button class="row-btn" data-index="${i}">Throw away</button>
</div>`
)
.join("");
el.querySelectorAll(".row-btn").forEach((btn) =>
btn.addEventListener("click", () => throwAway(expiredRows[Number(btn.dataset.index)]))
);
})
.catch((err) => {
el.innerHTML = `<p class="error">Could not load expired foods: ${escapeHtml(err.message)}</p>`;
});
}
// Scanning is the intended way to clear this list — you are standing at the bin with
// the thing in your hand — and the per-row button is the fallback for when the label
// is unreadable or the food no longer looks like itself.
$("expired-scan-btn").addEventListener("click", () => {
$("expired-status").textContent = "";
startScan("expired", onExpiredProposal, "Hold up what you are throwing away…");
});
function onExpiredProposal(proposal) {
if (proposal.degraded) {
$("expired-status").textContent = "Could not identify that — use the Throw away button on the row instead.";
resumeScan("Try another item…");
return;
}
const matches = (proposal.stock_matches || []).filter((m) =>
expiredRows.some((row) => row.product_id === m.product_id)
);
if (!matches.length) {
$("expired-status").textContent = `${proposal.name} is not on the expired list — nothing removed.`;
resumeScan("Hold up the next one…");
return;
}
throwAway(matches[0]);
}
function throwAway(row) {
if (!row) return;
const amount = amountOf(row) || 1;
// One tap between the camera and a write, same as everywhere else here — with the
// amount stated, because "throw away" on a row holding twelve means all twelve.
if (!window.confirm(`Throw away ${amount} × ${row.name}?`)) {
resumeScan("Hold up the next one…");
return;
}
$("expired-status").textContent = "Removing…";
postJson("/consume", { product_id: row.product_id, amount, spoiled: true })
.then(() => {
toast(`Binned ${amount} × ${row.name}`);
$("expired-status").textContent = "";
loadExpired();
resumeScan("Hold up the next one…");
})
.catch((err) => {
$("expired-status").textContent = `Could not remove: ${err.message}`;
});
}
// --- Edit inventory ------------------------------------------------------------
// The only screen that shows the folded view and its per-brand rows together: "22
// eggs" is the number the household thinks in, and "10 of brand Y" is the number you
// need to be able to correct exactly. Hence a <details> per kind rather than two
// separate screens.
let knownLocations = [];
function loadEditInventory() {
const el = $("edit-list");
$("edit-status").textContent = "";
api("/inventory")
.then((data) => {
knownLocations = data.locations || [];
const groups = data.groups || [];
if (!groups.length) {
el.innerHTML = '<p class="hint">Nothing booked in yet.</p>';
return;
}
el.innerHTML = groups.map(renderEditGroup).join("");
wireEditRows(el);
})
.catch((err) => {
el.innerHTML = `<p class="error">Could not load inventory: ${escapeHtml(err.message)}</p>`;
});
}
function renderEditGroup(group) {
const brands = group.brand_count > 1 ? ` · ${group.brand_count} brands` : "";
return `<details class="group ${urgencyClass(group.soonest_days_left)}">
<summary>
<span class="card-name">${escapeHtml(group.display_name || group.kind)}</span>
<span class="card-amount">${escapeHtml(String(group.total_amount))}${escapeHtml(brands)}</span>
<span class="card-due">${escapeHtml(dueLabel(group.soonest_days_left))}</span>
</summary>
${(group.entries || []).map(renderEditRow).join("")}
</details>`;
}
function renderEditRow(row) {
return `<div class="edit-row" data-product="${row.product_id}">
<span class="card-name">${escapeHtml(row.brand || row.name)}</span>
<span class="card-due">${escapeHtml(dueLabel(row.days_left))}</span>
<button class="step-btn" data-step="-1">−</button>
<input class="amount-input" type="number" min="0" step="1" value="${escapeHtml(String(amountOf(row)))}">
<button class="step-btn" data-step="1">+</button>
<button class="row-btn save-btn">Save</button>
</div>
${renderWhereRow(row)}`;
}
// Where a thing is, in two registers that are deliberately not merged: the location
// Grocy records (which somebody chose, and which "Move" changes) and the doorway
// camera's last sighting (which nobody chose, and which is only ever a hint — see
// docs/fridge-item-location.md). Showing the sighting as a sentence with a time in it
// is what keeps it readable as a clue rather than as a fact.
function renderWhereRow(row) {
if (!knownLocations.length && !row.last_seen) return "";
const seen = row.last_seen
? `Camera last saw something like this at ${escapeHtml(row.last_seen.location_name)}, ${escapeHtml(
relativeTime(row.last_seen.created_at)
)} — a sighting, not a fact.`
: "";
const options = knownLocations
.map(
(loc) =>
`<option value="${loc.id}"${loc.id === row.location_id ? " selected" : ""}>${escapeHtml(loc.name)}</option>`
)
.join("");
return `<div class="where-row" data-product="${row.product_id}">
<span class="hint">${seen}</span>
${
knownLocations.length
? `<label class="where-move">Move to
<select class="where-select">${options}</select>
</label>
<button class="row-btn move-btn">Move</button>`
: ""
}
</div>`;
}
function relativeTime(iso) {
const then = Date.parse(iso || "");
if (!Number.isFinite(then)) return "at an unknown time";
const minutes = Math.round((Date.now() - then) / 60000);
if (minutes < 1) return "just now";
if (minutes < 60) return `${minutes} min ago`;
const hours = Math.round(minutes / 60);
if (hours < 24) return `${hours}h ago`;
return `${Math.round(hours / 24)}d ago`;
}
function wireEditRows(root) {
root.querySelectorAll(".edit-row").forEach((rowEl) => {
const input = rowEl.querySelector(".amount-input");
rowEl.querySelectorAll(".step-btn").forEach((btn) =>
btn.addEventListener("click", () => {
const next = (Number(input.value) || 0) + Number(btn.dataset.step);
input.value = Math.max(0, next);
// +/- write immediately: a step button that needs a second tap on Save is a
// step button people will forget to save. The freeform field waits for Save,
// because it is mid-typing until then.
saveEditRow(rowEl);
})
);
rowEl.querySelector(".save-btn").addEventListener("click", () => saveEditRow(rowEl));
});
root.querySelectorAll(".where-row").forEach((whereEl) => {
const moveBtn = whereEl.querySelector(".move-btn");
if (!moveBtn) return;
moveBtn.addEventListener("click", () => {
const productId = Number(whereEl.dataset.product);
const rowEl = root.querySelector(`.edit-row[data-product="${productId}"]`);
const amount = Number(rowEl && rowEl.querySelector(".amount-input").value);
if (!(amount > 0)) {
$("edit-status").textContent = "Nothing to move — the amount is zero.";
return;
}
$("edit-status").textContent = "Moving…";
postJson("/transfer", {
product_id: productId,
amount,
to_location_id: Number(whereEl.querySelector(".where-select").value),
})
.then((result) => {
$("edit-status").textContent = "";
toast(result.unchanged ? "Already there" : "Moved");
loadEditInventory();
})
.catch((err) => {
$("edit-status").textContent = `Could not move: ${err.message}`;
});
});
});
}
function saveEditRow(rowEl) {
const productId = Number(rowEl.dataset.product);
const amount = Number(rowEl.querySelector(".amount-input").value);
if (!(amount >= 0)) {
$("edit-status").textContent = "Amount has to be zero or more.";
return;
}
$("edit-status").textContent = "Saving…";
postJson("/adjust", { product_id: productId, amount })
.then(() => {
$("edit-status").textContent = "";
toast("Corrected");
loadEditInventory();
})
.catch((err) => {
$("edit-status").textContent = `Could not save: ${err.message}`;
});
}
// --- Inventory (read-only) -----------------------------------------------------
function loadInventory() {
const el = $("inventory-list");
const el = document.getElementById("inventory-list");
api("/inventory")
.then((data) => {
const groups = data.groups || [];
if (!groups.length) {
el.innerHTML = '<p class="hint">Nothing in stock yet — unload some groceries!</p>';
const items = data.items || [];
if (!items.length) {
el.innerHTML = '<p class="hint">Nothing in stock yet — scan something!</p>';
return;
}
el.innerHTML = groups
.map((group) => {
const brands = group.brand_count > 1 ? `${group.brand_count} brands` : "";
return `<div class="card ${urgencyClass(group.soonest_days_left)}">
<span class="card-name">${escapeHtml(group.display_name || group.kind)}</span>
<span class="card-amount">×${escapeHtml(String(group.total_amount))} ${escapeHtml(brands)}</span>
<span class="card-due">${escapeHtml(dueLabel(group.soonest_days_left))}</span>
el.innerHTML = items
.map((item) => {
const days = item.days_left;
const label =
days === null || days === undefined
? "no date"
: days < 0
? `expired ${-days}d ago`
: days === 0
? "expires today"
: `${days}d left`;
return `<div class="card ${urgencyClass(days)}">
<span class="card-name">${escapeHtml(item.name)}</span>
<span class="card-amount">×${escapeHtml(String(item.amount ?? ""))}</span>
<span class="card-due">${escapeHtml(label)}</span>
</div>`;
})
.join("");
@ -810,7 +196,7 @@ function loadInventory() {
// --- Recipes -----------------------------------------------------------------
function loadRecipes() {
const el = $("recipes-list");
const el = document.getElementById("recipes-list");
api("/recipes")
.then((data) => {
const recipes = data.recipes || [];
@ -833,7 +219,9 @@ function loadRecipes() {
});
}
// A "Show <screen>" MQTT command from HA arrives as a URL fragment reload
// (kitchen-display-agent kills and relaunches Chromium at index.html#inventory, same
// pattern as the thin client's digest-browser) — honour it on load, same as any tap.
show(location.hash ? location.hash.slice(1) : "home");
function escapeHtml(s) {
return String(s).replace(/[&<>"']/g, (c) => ({ "&": "&amp;", "<": "&lt;", ">": "&gt;", '"': "&quot;", "'": "&#39;" }[c]));
}
// Start on whatever tab is active (default: scan).
resetScan();

129
pantry-vision/frontend/index.html
View File

@ -14,53 +14,25 @@
as URL query params from the kitchen-display kiosk's own launch command
(hosts/kitchen-display/configs/sway/pantry-kiosk), never baked into this file, so
this stays a plain static asset with no secret in git — see app.js's top comment.
Four stock movements, four screens, all reached from Home and all camera-first
except the last: Unload groceries (book in), Consume article (book out), List
expired foods (throw away), Edit inventory (the keyboard fallback for when the
camera got it wrong). #scanner below is ONE camera element, moved into whichever
flow is running — see app.js's mountScanner(); a second getUserMedia stream on a
single-webcam kiosk is a black rectangle, not a second camera.
-->
<nav id="tabs">
<button class="tab active" data-tab="home">🏠<span>Home</span></button>
<button class="tab active" data-tab="scan">📷<span>Scan</span></button>
<button class="tab" data-tab="inventory">🧊<span>Inventory</span></button>
<button class="tab" data-tab="recipes">🍳<span>Recipes</span></button>
</nav>
<main>
<section id="home" class="panel active">
<div class="action-grid">
<button class="action-btn" data-goto="unload">🛍️<span>Unload groceries</span></button>
<button class="action-btn" data-goto="consume">🍽️<span>Consume article</span></button>
<button class="action-btn" data-goto="expired">🗑️<span>List expired foods</span></button>
<button class="action-btn" data-goto="edit">✏️<span>Edit inventory</span></button>
<section id="scan" class="panel active">
<div id="scan-camera">
<video id="camera-preview" autoplay playsinline muted></video>
<button id="capture-btn" class="big-btn">Capture</button>
<p id="camera-error" class="error" hidden></p>
</div>
</section>
<!-- The camera, mounted into whichever flow is active. -->
<div id="scanner" hidden>
<video id="camera-preview" autoplay playsinline muted></video>
<p id="scan-status" class="hint">Starting camera…</p>
<p id="camera-error" class="error" hidden></p>
<button id="capture-now" class="big-btn secondary">Identify now</button>
<canvas id="captured-frame" hidden></canvas>
<canvas id="frame-sample" hidden></canvas>
</div>
<!-- ---------------------------------------------------------------- Unload -->
<section id="unload" class="panel flow">
<header class="flow-head">
<button class="back-btn" data-goto="home">‹ Home</button>
<h1>Unload groceries</h1>
<span id="unload-count" class="flow-badge"></span>
</header>
<div class="scanner-slot"></div>
<form id="confirm-form" hidden>
<p id="f-description" class="lead"></p>
<label>Article <input id="f-name" type="text" required></label>
<div class="field-row">
<div id="scan-result" hidden>
<canvas id="captured-frame"></canvas>
<form id="confirm-form">
<label>Name <input id="f-name" type="text" required></label>
<label>Category
<select id="f-category">
<option value="produce">Produce</option>
@ -73,83 +45,18 @@
<option value="other">Other</option>
</select>
</label>
<label>Put it in
<select id="f-placement">
<option value="fridge">🧊 Fridge</option>
<option value="freezer">❄️ Freezer</option>
<option value="cupboard">🚪 Cupboard</option>
<option value="counter">🧺 Counter</option>
</select>
</label>
</div>
<label>Best before <input id="f-date" type="date" required></label>
<p id="f-date-source" class="hint"></p>
<div class="field-row">
<label>Packs <input id="f-packages" type="number" min="1" step="1" value="1"></label>
<label>Per pack <input id="f-units" type="number" min="1" step="1" value="1"></label>
</div>
<p id="f-total" class="total-line"></p>
<p id="f-have" class="hint"></p>
<p id="f-confidence" class="hint"></p>
<div class="form-actions">
<button type="button" id="skip-btn" class="big-btn secondary">Skip</button>
<button type="submit" id="confirm-btn" class="big-btn primary">Confirm &amp; next</button>
</div>
<p id="confirm-status" class="hint"></p>
</form>
</section>
<!-- --------------------------------------------------------------- Consume -->
<section id="consume" class="panel flow">
<header class="flow-head">
<button class="back-btn" data-goto="home">‹ Home</button>
<h1>Consume article</h1>
</header>
<div class="scanner-slot"></div>
<div id="consume-result" hidden>
<p id="consume-identified" class="lead"></p>
<div id="consume-candidates" class="card-list"></div>
<div id="consume-amount" hidden>
<p id="consume-chosen" class="lead"></p>
<div id="consume-quick" class="quick-row"></div>
<label>How many? <input id="consume-qty" type="number" min="1" step="1" value="1"></label>
<label>Best before <input id="f-date" type="date" required></label>
<label>Quantity <input id="f-quantity" type="number" min="1" step="1" value="1"></label>
<p id="f-confidence" class="hint"></p>
<div class="form-actions">
<button type="button" id="consume-cancel" class="big-btn secondary">Cancel</button>
<button type="button" id="consume-confirm" class="big-btn primary">Book out</button>
<button type="button" id="retake-btn" class="big-btn secondary">Retake</button>
<button type="submit" id="confirm-btn" class="big-btn primary">Confirm &amp; add</button>
</div>
</div>
<p id="consume-status" class="hint"></p>
<p id="confirm-status" class="hint"></p>
</form>
</div>
</section>
<!-- --------------------------------------------------------------- Expired -->
<section id="expired" class="panel flow">
<header class="flow-head">
<button class="back-btn" data-goto="home">‹ Home</button>
<h1>Expired foods</h1>
<button id="expired-scan-btn" class="flow-action">📷 Scan to remove</button>
</header>
<div class="scanner-slot"></div>
<p id="expired-status" class="hint"></p>
<div id="expired-list" class="card-list"><p class="hint">Loading…</p></div>
</section>
<!-- ------------------------------------------------------------------ Edit -->
<section id="edit" class="panel flow">
<header class="flow-head">
<button class="back-btn" data-goto="home">‹ Home</button>
<h1>Edit inventory</h1>
</header>
<p class="hint">
Everything booked in, same kind added up across brands. Open a line to correct one
brand exactly.
</p>
<p id="edit-status" class="hint"></p>
<div id="edit-list" class="card-list"><p class="hint">Loading…</p></div>
</section>
<!-- ------------------------------------------------------------- Reference -->
<section id="inventory" class="panel">
<div id="inventory-list" class="card-list">
<p class="hint">Loading…</p>
@ -163,8 +70,6 @@
</section>
</main>
<p id="toast" hidden></p>
<script src="app.js"></script>
</body>
</html>

Some files were not shown because too many files have changed in this diff Show More