11 KiB

Raw Blame History

Two shared endpoint surfaces: the now-playing visualiser and the 3D floorplan

Design notes for two things that must look the same on every screen in the house — a thin client on a TV, the Loggia all-in-one, the kitchen panel, the door panel — and which therefore belong in a shared, vendored SDK rather than being written four times.

Status: both are built — render/media-visualiser/ and render/floorplan-3d/. One thing changed in the building, and it is called out in section 2: the 3D view is canvas 2D, not three.js. The scene turned out to be prisms on a plane, which an isometric projection draws in ~200 lines without the megabyte, without a build step, and with predictable performance on the small panels. Everything else below was implemented as specified.

1. The now-playing visualiser

Wanted: a circular CAVA-style spectrum behind every playing-status screen, coloured from the album art, with lyrics scrolling under the cover when they exist.

The audio problem, which is the whole problem

CAVA reads an audio stream. The endpoints do not all have one:

Endpoint	Is the audio local?	Can it get a spectrum?
Thin client running mpv/spotifyd	yes	Yes — real FFT of the actual output
Audio endpoint (Spotify Connect box)	yes, but it is headless	Yes, but there is no screen to draw on
Kitchen / door panel showing what the living room is playing	no	No. There is no audio here at all

So a design that requires real audio analysis works on one endpoint and silently degrades to a dead circle on the others — the worst outcome, because the screen looks broken rather than looking deliberate.

Two-tier, declared up front:

Reactive tier — where audio is local, CAVA (or a WebAudio AnalyserNode when the player is in the browser) drives the bars. cava has a raw output mode writing plain numbers to a FIFO, which a small agent can publish over MQTT; that is the least fragile path on a machine already running an agent.
Synthetic tier — everywhere else, the ring is animated from track position and tempo, not from silence. It breathes with the beat rather than pretending to analyse it. Nobody watching a kitchen panel from across the room can tell, and it never looks broken.

Do not let the synthetic tier claim to be the reactive one anywhere in the UI. It is a mood light, and the moment somebody believes it is a spectrum they will report it as a bug every time it does not match a bass drop.

Colours from the album art

Pull 3–5 dominant colours from the cover, client-side, at load:

Draw the cover into a small offscreen canvas (64×64 is plenty — this is a palette, not a photograph).
Bucket pixels in RGB space, take the top buckets by population.
Reject near-greys and near-blacks before ranking. Album art is full of them, and a palette extracted naively from a dark cover gives you four indistinguishable dark greys and a visualiser that looks switched off.
Force a minimum contrast against the background. Lift the chosen colours in lightness until they clear the panel behind them; a visualiser you cannot see is the same as no visualiser, and this is the step that gets skipped.

No library needed — that is about forty lines of canvas work, and it keeps the "vendored, dependency-free" property the canvas SDKs already have.

Lyrics

Source: whatever the player exposes. Music Assistant and MPD both surface synced-lyrics fields when the provider has them; .lrc sidecar files are the other common case for local libraries.
Synced (LRC) vs plain: with timestamps, highlight the current line and scroll it to centre. Without them, do not fake it — scroll slowly, or just show the text. A plain-lyrics block auto-scrolled at a guessed rate is wrong within ten seconds and stays wrong.
Absent is the normal case. Most tracks in most libraries have no lyrics. The layout must be designed for "no lyrics" as the default state, with lyrics as the addition — not a gap where they would go.
Never fetch lyrics from the internet at render time. If lyrics are worth having, they are worth caching where the track is.

Where it lives

A new render/media-visualiser/ in the shared SDK style: one JS file, one CSS file, vendored into each host that needs it, exactly as digest-canvas-sdk and canvas-sdk already are. Inputs are a normalised now-playing object ({title, artist, album, art_url, position_ms, duration_ms, tempo?, lyrics?}) and an optional spectrum feed. Every endpoint already receives now-playing over MQTT; nothing new has to be plumbed for the synthetic tier.

2. The 3D floorplan

Wanted: the Pebble app's presence view, in 3D, on any endpoint — rooms, and who is in them, with profile pictures.

It is the same data, and that is the point

GET /floorplan/presence already returns everything: room polygons normalised 0–1, each room's occupants, each occupant's color, initial and has_photo, plus unplaced for people who are home but not locatable. The Pebble app and this render the same JSON at different fidelities — which is the reason to build the second one at all. If they diverge, one of them is lying.

Extrusion, not modelling

Do not author a 3D model. Take the existing 2D polygons and extrude them to a wall height, which is one THREE.ExtrudeGeometry per room and needs no new data beyond a single wall_height_m per level. A hand-authored model would be prettier, immediately stale the first time a room is redrawn, and unmaintainable by the person who drew the plan in a 2D editor.

Camera: fixed isometric by default, drag to orbit, no free-fly. An orbit camera on a wall panel is something people knock out of alignment and cannot get back.

The occupant markers

This is where 3D earns itself, because the Pebble's constraint — 18×18 px, no room for a face — is gone:

A billboarded disc above each occupied room, always facing the camera (never a flat sprite lying on the floor, which is unreadable at a glance).
Profile picture inside the disc, ringed in the person's colour, falling back to the initial on the colour when has_photo is false. The ring matters even with a photo: it is what ties this marker to the same person's marker on the watch and in the admin panel.
Occupied rooms lit, empty rooms dark, the same rule the watchapp uses, and for the same reason: state should read before detail. In 3D that is an emissive floor material rather than a fill colour, and the third state — drawn but never reported by HA — stays visually distinct, because rendering "no data" as "empty" is a quiet lie on any screen.
unplaced people get a shelf, not a hidden list: a strip along the bottom showing everyone who is home but unlocatable. They are the people you are most often looking for.

Exact positions within a room

Room-level presence answers which room. Exact positions need a sensor that reports coordinates, and BLE is not that sensor: RSSI-to-distance is noisy enough that trilateration in a house lands in the wrong room often enough to be useless.

The realistic source is mmWave radar (LD2450-class, ~€15–25 per room, ESPHome native), which tracks up to three moving targets and publishes each one's x/y in millimetres. What it cannot do is say who — it sees a moving blob.

So position and identity come from different sensors, and the whole design is in how they are combined:

Signal	Source	Gives
who	BLE / Bermuda	a person, resolved to a room
where	mmWave	a coordinate, with no name

The fusion rule is deliberately timid (floorplan_presence() in identity):

exactly one occupant in the room and exactly one target in the room → that target is that person. Anything else → the targets stay anonymous and the occupants stay unpositioned.

Two people in a room produce two blobs that cannot be told apart. Guessing which is which would put a name on the wrong person, and a presence display that does that occasionally is worse than one that never tries — you cannot tell the wrong answers from the right ones. position_ambiguous says out loud when this has happened.

What the plan needs before any of it works, both drawn by a human because nothing can infer them:

metres_wide per level — the real-world width of the 0–1 extent. Without it a reading in metres cannot become a point on the plan, and the API reports that rather than guessing a scale.
A sensor placement per radar — where it is on the plan and which way it faces. A wrong rotation mirrors every target it reports, which is the single most likely way to get positions that look plausible and are wrong.

Rendering follows the same honesty: a named person at a fused position gets their marker with a footprint dot on the floor (without it, a marker floated above the floor reads as being further back in the room); an unattributed target gets a hollow dashed puck with a question mark — no colour, no initial, because every visual language this view has for a person is reserved for people it can name. A radar target lights the room even with nobody named, because somebody is in there.

The honest constraint — and how it was resolved

The plan was to vendor three.js and call it a deliberate break with the dependency-free rule. Building it made the cheaper answer obvious and it was taken instead: canvas 2D with an isometric projection and painter's-algorithm sorting.

The scene is prisms standing on a plane — no lighting model worth the name, no textures, no physics, no camera motion beyond an orbit. That is ~200 lines of canvas 2D, it redraws in well under a millisecond on the small panels (where a WebGL context on integrated graphics is a much less predictable proposition), and the frontend stays at zero dependencies with nothing to keep patched.

This is not a compromise version of the three.js plan; it is the smaller correct tool for this specific scene. If the view ever grows real lighting or an imported model, three.js becomes right and floorplan3d.js becomes the fallback for the weak panels.

And the same caveat as the watchapp, which no amount of rendering fixes: this is only as right as room-level presence is, which has never been measured in this house. A beautiful 3D house with everybody sitting in the unplaced shelf is a beautiful 3D house that tells you nothing. Test that first — the admin panel's floorplan tab with Live ticked answers it for free.

Build order

The visualiser's synthetic tier, on one endpoint. It needs no new data and no audio plumbing, and it is what tells you whether the effect is worth the reactive tier's complexity.
Album-art palette extraction, which is where the look actually comes from.
Lyrics, synced-only at first.
The reactive tier, on the thin clients that have local audio.
The 3D floorplan — built. It turned out not to be a dependency addition at all (see above), so the sequencing argument that put it last no longer applies. The other reason still does, in full: it is only as right as room-level presence, which has never been measured here. Test that before believing the picture.

11 KiB Raw Blame History Unescape Escape