SmartestHome/docs/endpoint-surfaces.md

# Two shared endpoint surfaces: the now-playing visualiser and the 3D floorplan

Design notes for two things that must look the same on every screen in the house — a
thin client on a TV, the Loggia all-in-one, the kitchen panel, the door panel — and
which therefore belong in a **shared, vendored SDK** rather than being written four
times.

> **Status: both are built** — `render/media-visualiser/` and `render/floorplan-3d/`.
> One thing changed in the building, and it is called out in section 2: the 3D view is
> **canvas 2D, not three.js**. The scene turned out to be prisms on a plane, which an
> isometric projection draws in ~200 lines without the megabyte, without a build step,
> and with predictable performance on the small panels. Everything else below was
> implemented as specified.

---

## 1. The now-playing visualiser

Wanted: a circular CAVA-style spectrum behind every playing-status screen, coloured
from the album art, with lyrics scrolling under the cover when they exist.

### The audio problem, which is the whole problem

CAVA reads an audio stream. The endpoints do not all have one:

| Endpoint | Is the audio local? | Can it get a spectrum? |
|---|---|---|
| Thin client running mpv/spotifyd | **yes** | Yes — real FFT of the actual output |
| Audio endpoint (Spotify Connect box) | yes, but it is headless | Yes, but there is no screen to draw on |
| Kitchen / door panel showing *what the living room is playing* | **no** | **No. There is no audio here at all** |

So a design that requires real audio analysis works on one endpoint and silently
degrades to a dead circle on the others — the worst outcome, because the screen looks
broken rather than looking deliberate.

**Two-tier, declared up front:**

- **Reactive tier** — where audio is local, CAVA (or a WebAudio `AnalyserNode` when the
  player is in the browser) drives the bars. `cava` has a `raw` output mode writing
  plain numbers to a FIFO, which a small agent can publish over MQTT; that is the least
  fragile path on a machine already running an agent.
- **Synthetic tier** — everywhere else, the ring is animated from **track position and
  tempo**, not from silence. It breathes with the beat rather than pretending to
  analyse it. Nobody watching a kitchen panel from across the room can tell, and it
  never looks broken.

Do not let the synthetic tier claim to be the reactive one anywhere in the UI. It is a
mood light, and the moment somebody believes it is a spectrum they will report it as a
bug every time it does not match a bass drop.

### Colours from the album art

Pull 3–5 dominant colours from the cover, client-side, at load:

1. Draw the cover into a small offscreen canvas (64×64 is plenty — this is a palette,
   not a photograph).
2. Bucket pixels in RGB space, take the top buckets by population.
3. **Reject near-greys and near-blacks** before ranking. Album art is full of them, and
   a palette extracted naively from a dark cover gives you four indistinguishable dark
   greys and a visualiser that looks switched off.
4. **Force a minimum contrast against the background.** Lift the chosen colours in
   lightness until they clear the panel behind them; a visualiser you cannot see is
   the same as no visualiser, and this is the step that gets skipped.

No library needed — that is about forty lines of canvas work, and it keeps the
"vendored, dependency-free" property the canvas SDKs already have.

### Lyrics

- **Source**: whatever the player exposes. Music Assistant and MPD both surface
  synced-lyrics fields when the provider has them; `.lrc` sidecar files are the other
  common case for local libraries.
- **Synced (LRC) vs plain**: with timestamps, highlight the current line and scroll it
  to centre. Without them, do not fake it — scroll slowly, or just show the text. A
  plain-lyrics block auto-scrolled at a guessed rate is wrong within ten seconds and
  stays wrong.
- **Absent is the normal case.** Most tracks in most libraries have no lyrics. The
  layout must be designed for "no lyrics" as the default state, with lyrics as the
  addition — not a gap where they would go.
- Never fetch lyrics from the internet at render time. If lyrics are worth having,
  they are worth caching where the track is.

### Where it lives

A new `render/media-visualiser/` in the shared SDK style: one JS file, one CSS file,
vendored into each host that needs it, exactly as `digest-canvas-sdk` and
`canvas-sdk` already are. Inputs are a normalised now-playing object
(`{title, artist, album, art_url, position_ms, duration_ms, tempo?, lyrics?}`) and an
optional spectrum feed. Every endpoint already receives now-playing over MQTT; nothing
new has to be plumbed for the synthetic tier.

---

## 2. The 3D floorplan

Wanted: the Pebble app's presence view, in 3D, on any endpoint — rooms, and who is in
them, with profile pictures.

### It is the same data, and that is the point

`GET /floorplan/presence` already returns everything: room polygons normalised 0–1,
each room's occupants, each occupant's `color`, `initial` and `has_photo`, plus
`unplaced` for people who are home but not locatable. The Pebble app and this render
the same JSON at different fidelities — which is the reason to build the second one at
all. If they diverge, one of them is lying.

### Extrusion, not modelling

Do not author a 3D model. Take the existing 2D polygons and **extrude them to a wall
height**, which is one `THREE.ExtrudeGeometry` per room and needs no new data beyond a
single `wall_height_m` per level. A hand-authored model would be prettier, immediately
stale the first time a room is redrawn, and unmaintainable by the person who drew the
plan in a 2D editor.

Camera: fixed isometric by default, drag to orbit, no free-fly. An orbit camera on a
wall panel is something people knock out of alignment and cannot get back.

### The occupant markers

This is where 3D earns itself, because the Pebble's constraint — 18×18 px, no room for
a face — is gone:

- A **billboarded disc** above each occupied room, always facing the camera (never a
  flat sprite lying on the floor, which is unreadable at a glance).
- **Profile picture inside the disc**, ringed in the person's colour, falling back to
  the initial on the colour when `has_photo` is false. The ring matters even with a
  photo: it is what ties this marker to the same person's marker on the watch and in
  the admin panel.
- **Occupied rooms lit, empty rooms dark**, the same rule the watchapp uses, and for
  the same reason: state should read before detail. In 3D that is an emissive floor
  material rather than a fill colour, and the third state — *drawn but never reported
  by HA* — stays visually distinct, because rendering "no data" as "empty" is a quiet
  lie on any screen.
- **`unplaced` people get a shelf**, not a hidden list: a strip along the bottom
  showing everyone who is home but unlocatable. They are the people you are most often
  looking for.

### Exact positions within a room

Room-level presence answers *which room*. Exact positions need a sensor that reports
**coordinates**, and BLE is not that sensor: RSSI-to-distance is noisy enough that
trilateration in a house lands in the wrong room often enough to be useless.

The realistic source is **mmWave radar** (LD2450-class, ~€15–25 per room, ESPHome
native), which tracks up to three moving targets and publishes each one's x/y in
millimetres. What it cannot do is say **who** — it sees a moving blob.

So position and identity come from different sensors, and the whole design is in how
they are combined:

| Signal | Source | Gives |
|---|---|---|
| who | BLE / Bermuda | a person, resolved to a room |
| where | mmWave | a coordinate, with no name |

**The fusion rule is deliberately timid** (`floorplan_presence()` in `identity`):

> exactly one occupant in the room **and** exactly one target in the room → that target
> is that person. Anything else → the targets stay anonymous and the occupants stay
> unpositioned.

Two people in a room produce two blobs that cannot be told apart. Guessing which is
which would put a name on the wrong person, and a presence display that does that
occasionally is worse than one that never tries — you cannot tell the wrong answers
from the right ones. `position_ambiguous` says out loud when this has happened.

What the plan needs before any of it works, both drawn by a human because nothing can
infer them:

- **`metres_wide` per level** — the real-world width of the 0–1 extent. Without it a
  reading in metres cannot become a point on the plan, and the API reports that rather
  than guessing a scale.
- **A sensor placement per radar** — where it is on the plan and *which way it faces*.
  A wrong rotation mirrors every target it reports, which is the single most likely way
  to get positions that look plausible and are wrong.

Rendering follows the same honesty: a named person at a fused position gets their
marker with a footprint dot on the floor (without it, a marker floated above the floor
reads as being further back in the room); an unattributed target gets a **hollow dashed
puck with a question mark** — no colour, no initial, because every visual language this
view has for a person is reserved for people it can name. A radar target lights the
room even with nobody named, because somebody *is* in there.

### The honest constraint — and how it was resolved

The plan was to vendor three.js and call it a deliberate break with the
dependency-free rule. Building it made the cheaper answer obvious and it was taken
instead: **canvas 2D with an isometric projection and painter's-algorithm sorting.**

The scene is prisms standing on a plane — no lighting model worth the name, no
textures, no physics, no camera motion beyond an orbit. That is ~200 lines of canvas
2D, it redraws in well under a millisecond on the small panels (where a WebGL context
on integrated graphics is a much less predictable proposition), and the frontend stays
at zero dependencies with nothing to keep patched.

This is not a compromise version of the three.js plan; it is the smaller correct tool
for this specific scene. If the view ever grows real lighting or an imported model,
three.js becomes right and `floorplan3d.js` becomes the fallback for the weak panels.

And the same caveat as the watchapp, which no amount of rendering fixes: **this is only
as right as room-level presence is**, which has never been measured in this house. A
beautiful 3D house with everybody sitting in the `unplaced` shelf is a beautiful 3D
house that tells you nothing. Test that first — the admin panel's floorplan tab with
**Live** ticked answers it for free.

---

## Build order

1. **The visualiser's synthetic tier**, on one endpoint. It needs no new data and no
   audio plumbing, and it is what tells you whether the effect is worth the reactive
   tier's complexity.
2. **Album-art palette extraction**, which is where the look actually comes from.
3. **Lyrics**, synced-only at first.
4. **The reactive tier**, on the thin clients that have local audio.
5. **The 3D floorplan** — built. It turned out not to be a dependency addition at all
   (see above), so the sequencing argument that put it last no longer applies. The
   *other* reason still does, in full: **it is only as right as room-level presence,
   which has never been measured here.** Test that before believing the picture.