19 KiB

Raw Permalink Blame History

A workshop/office assistant: what's worth building

Feasibility note. The ask: identify laptops and computers by camera, identify components (mainboards especially), pull up disassembly guides, research part specs automatically, and hold a conversation about planning a project — all scoped per room.

The one finding that reorganises the whole thing: stop trying to recognise the object, and read the label on it. Everything below follows from that.

Why "identify it by shape" is the wrong problem

Ask a local 7–13B vision model what laptop it is looking at and you will get a confident, wrong answer, because:

Laptops are visually identical by design. A closed ThinkPad T480 and a T490 are the same black rectangle. So are half of Dell's Latitude line. There is no visual feature to learn, and a model that claims otherwise is pattern-matching on the wallpaper.
Mainboards are worse, not better. Board layout varies more than a laptop shell does, but the discriminating detail — where the fourth M.2 slot is, which VRM configuration — is exactly what a photo at workbench distance and a general-purpose VLM cannot resolve.
Getting it wrong is expensive here, unlike in the pantry. A wrong yoghurt costs a tap. A disassembly guide for the wrong laptop revision costs a broken clip or a ribbon cable, and a wrong pinout costs the board.

Meanwhile, every one of these objects carries its own identity in printed text:

Object	Where its identity actually is
Laptop	Service tag / serial sticker on the base — and on Dell/Lenovo/HP that tag resolves to the exact factory configuration
Mainboard	Model silkscreened on the PCB (`PRIME B450M-A`), usually near the RAM slots or the PCIe bracket
GPU / PSU / drives	Model and part number on the label
ICs	Top-marking, when it hasn't been sanded

So the pipeline is OCR-first: photo → detect text regions → read them → match against a parts database. The VLM's job is the small, tractable one it is good at — "what kind of thing is this, and where on it is the label" — and the identification comes from characters, not from vibes. This also degrades honestly: an unreadable label produces "I can't read it, hold it closer" rather than a plausible wrong model.

What stays hard, permanently: an unmarked part. A capacitor, a sanded IC, an anonymous barrel connector. Nothing here will identify those, and the assistant should say so rather than guess.

The four capabilities, ranked by whether they're worth it

1. Conversational project planning — build this first

The cheapest and the most useful, and the only one with no perception problem in it. It is a text LLM with a notebook: what am I building, what have I got, what's the next step, what did I decide last Tuesday and why. The value is entirely in the notebook being persistent and per-project, not in the model being clever.

This is also the piece that makes the others worth having: an identified mainboard is only useful if there is somewhere to put it ("this is for the NAS build").

Shape: a small service in the pattern this repo already uses four times over (identity, chores, pantry-vision) — SQLite, an HTTP API, bearer token, a static frontend on a display. Projects, notes, parts, photos, decisions.

2. Part-spec research — build it, with one hard rule

Every spec must be quoted from a fetched document, with its URL, and never generated from the model's memory. This is the same rule digest-engine's political prompt enforces on claims about the world and pantry-vision enforces on stock writes, and here it is a safety property rather than a quality one: a hallucinated TDP wastes an afternoon, a hallucinated pinout destroys hardware.

Practically: fetch the vendor page or datasheet PDF, cache it locally, extract, and show the extract next to its source. If nothing was fetched, the answer is "I couldn't find a datasheet", not a paragraph of plausible numbers.

This is the first component in this project that deliberately reaches the open internet for content, which is worth stating plainly given the local-first doctrine everything else follows. It is outbound-only, it does not require anything to be exposed, and it should run against an explicit allowlist of sources (vendor domains, iFixit, datasheet archives) with everything it fetches cached on disk — so the same board is looked up once, not once per question.

3. Disassembly guides — mostly a link, and that's fine

iFixit is the corpus, and it has a public API. The honest scope is: resolve the identified model to a guide, show the steps and images on the workshop display, cache what you fetch. Do not rewrite the guide through an LLM — the steps are the thing, and an LLM paraphrase of "disconnect the battery before removing the board" is strictly worse than the sentence itself.

Vendor service manuals (Lenovo's HMM PDFs, Dell's service manuals) cover what iFixit doesn't, and are the reason the cache is worth having: they are large PDFs you want locally, once.

4. Camera identification — build it last, and OCR-first

By the time the three above exist, this is a convenience: it saves typing a model number. Useful convenience — hands are usually dirty or full — but it is the piece with the perception risk, and it should not be the first thing attempted.

The interaction that actually works: hold the label to the camera, get the text back with a confidence, confirm before anything acts on it — the identical propose-never-auto-commit rule pantry-vision runs on. Scanning a barcode or QR code where one exists (most service tags have one) is strictly better than OCR and should be tried first.

Per-room scoping, which the plumbing now supports

"Make this a per-room system" splits into two questions, and the boring one is now answered:

Which devices are in which room — done. Every kiosk and audio endpoint declares a room (an HA area_id) in CoreSystemConfig.json; it is baked into the agent and published as suggested_area, so HA files each device in the right area by itself. See docs/rooms-and-endpoints.md. The workshop is a room like any other: give it a display and it declares room: workshop.

Which assistant is active in which room — the design question, and the answer that fits this project is: the room selects the toolset, not the personality. A voice request in the workshop can reach the parts database and the project notebook; the same words in the kitchen reach pantry-vision and the recipes. That is a routing table from area_id to capability set, sitting in front of HA Assist, and it is small.

Two reasons to do it that way rather than one assistant with everything:

Ambiguity collapses. "What have I got?" means stock in the kitchen and parts in the workshop, and no amount of prompt engineering makes one agent reliably guess which room the words came from — but identity's /speaker already answers "who spoke in this area", and the area is right there in the request.
The dangerous tools stay where they belong. Nothing in the kitchen should be able to open a disassembly guide, and nothing in the workshop needs to book out groceries.

The assistant needs somewhere to put things: fetched datasheets, extracted specs, generated diagrams, photos of the board, project notes, a scratch directory per project. Making that a plain directory on the container host, exported over SMB, is right for a reason worth stating: it means every artefact is a file you can open from a laptop with no API, no export step, and no dependence on this project still existing in two years. The assistant's output should outlive the assistant.

/opt/smart-home/workshop/
  projects/<slug>/
    notes/          markdown the assistant and you both write
    datasheets/     everything fetched, cached, named by part
    diagrams/       generated SVGs (see the next section)
    photos/         what the camera captured
    scratch/        the assistant's working directory

This project already runs Samba — gallery-smb, from setup-container-host.sh. Two things follow, and the second is a real trap:

Reuse the container, add a share. Samba serves many shares on one port; a second Samba container would fight the first for 445, which is already claimed (ports.gallery_smb). That is the same class of collision already recorded as open decision #31 for Music Assistant, and it is avoidable here by construction: one smb container, a gallery share and a workshop share, separate accounts.
The gallery share is read only = yes. This one cannot be, and that is the entire security difference between them. A writable share is fine; a writable share that an LLM writes into deserves three limits, none of which are exotic:
- Its own volume, not a subdirectory of anything else. The blast radius of a bad path is then "the workshop workspace", which is recoverable.
- Its own account, not the gallery's. Guest access off, same as the gallery.
- No execution. Nothing in the workspace is ever run by anything on the host — it is a place for documents, and the moment it becomes a place for scripts it is a different security question than this note answers.

Keep the assistant's state — projects, parts, decisions — in SQLite as with every other service here, and the artefacts on the share. The database is the index; the share is the filing cabinet. Mixing them (blobs in SQLite, or state in files) gets you the worst of both: a database you cannot browse and files nothing can query.

Technical display widgets, and the one decision that makes them cheap

The thin-client canvas SDK today has stat, chart, image, video, and plain markdown-ish text. The wanted additions — schematics, board plans, code-flow and data-structure diagrams, breadboard layouts, 3D — look like six new widgets. They are mostly one:

Render to SVG on the container host; show it in the canvas. One new svg window kind, plus renderers server-side.

That keeps the SDK's dependency-free, no-build-step property (the thing that makes it maintainable), puts heavy tooling on the machine that already has heavy tooling, and means a diagram is a file in the workspace share as well as something on a screen. Client-side rendering libraries would put a megabyte of JavaScript on a kiosk to produce a picture the server could have produced once and cached.

Widget	How	Honest difficulty
Circuit diagrams	`netlistsvg` (or KiCad's own SVG export) from a netlist the assistant produces	Medium — see the notation note below, which is the whole job
Board plans	KiCad `.kicad_pcb`/Gerber → SVG, server-side, pre-rendered	Medium, and only for boards you have files for. There is no path from a photo of a board to a layout drawing
Code-flow diagrams	Graphviz `dot` → SVG	Easy. Graphviz is small, deterministic, and an LLM writes `dot` reliably
Data-structure diagrams	Graphviz (records/HTML-like labels) → SVG	Easy, same pipeline
Assembly guides	The guide's own step images in sequence — see the iFixit section	Easy in 2D
Breadboard diagrams	Fritzing-style, and Fritzing's value is its part library, which is not callable	Hard, and least worth it — a photo of the actual breadboard is usually better and always more honest
3D models	glTF + a vendored three.js on the kiosk	The only one that genuinely needs client-side code — see below

EU/IEC notation is a symbol-library decision, not a rendering one

Rectangular resistors, not zigzags. This matters more than it sounds: most schematic tooling defaults to ANSI/US symbols, and a renderer that cannot be given a symbol set will quietly produce American schematics forever.

So the selection criterion for the schematic renderer is "can I supply my own symbol library?" before anything about output quality. netlistsvg takes custom skins, which is the concrete lever — an IEC skin, written once, vendored in the repo like every other asset here. KiCad can be configured the same way. Anything that hardcodes its symbols is disqualified regardless of how good its output looks, because the first zigzag resistor is the last time anyone trusts the diagram.

The same applies downstream: state the convention in the prompt that generates the netlist, and check it on the way out. A diagram in the wrong notation is not a style complaint — it is a diagram that reads wrong to the person holding the soldering iron.

The holographic look is a variable override, not a new stylesheet

The canvas SDK is already a holo aesthetic — glow text, glowing panel edges, a pulse animation, a dark blue base — and, importantly, it is already parameterised. Every colour in glow.css comes from custom properties on :root (--admin-accent, --admin-glow-color, --admin-edge, --admin-panel, --admin-bg).

So "make the workshop look holographic in purple/magenta" is one small override file loaded after the SDK, not a fork of it:

/* workshop-theme.css — loaded after glow.css. Overrides only. */
:root {
  --admin-bg:     #0a0510;             /* near-black, violet-biased */
  --admin-panel:  rgba(28, 12, 44, 0.82);
  --admin-edge:   rgba(200, 120, 255, 0.32);
  --admin-accent: #c084fc;             /* the purple everything glows in */
  --admin-text:   #eadcff;
  --admin-muted:  #a98fc4;
  --admin-glow-color: #ff3ec8;         /* magenta glow against a purple accent */
}

Two magenta-on-purple choices worth making deliberately: the accent (borders, labels, chrome) reads better as the lighter purple, and the glow as the magenta — glow bleeds and saturates, so the more aggressive colour belongs to the effect rather than to the thing being read. Keep --admin-good/--admin-warn as they are; green and amber mean something, and recolouring them into the theme would cost that meaning.

Do not tint the technical drawings. This is the one rule that keeps the look from becoming a liability: a schematic, a board plan or a pinout table needs contrast and neutral colour far more than it needs to match the frame — trace colours mean things, and a magenta glow over a resistor value is how you misread it. So:

Holo treatment on the chrome: window frames, titles, edges, the pulse, the background.
The drawing surface stays neutral — a light panel for schematics (they are drawn for paper and read best that way), or high-contrast line art on near-black. The frame around it can glow all it likes.
Anything conveying state by colour — a red LED in a diagram, an error, a live measurement — keeps its own colour. The theme owns the furniture, not the data.

The same override trick works if a future room wants its own palette; the SDK never needed to know about themes for this to be possible, which is why it costs one file.

3D: worth doing, but not first, and not with a live viewer at first

3D is the one item that cannot be an SVG. Two paths:

Pre-rendered orbit frames — render N views server-side, ship them as an image sequence, scrub with a drag. Cheap, works on every surface including the ones with no GPU, and covers "let me see what this connector looks like from behind", which is most of the actual need.
A real viewer — vendored three.js plus glTF. It is roughly a megabyte of JavaScript on a kiosk that currently ships none, which is a real break with the SDK's "vendored, dependency-free" rule and should be a deliberate decision rather than a drift into one.

Start with the frames. Move to a viewer if and only if manipulating the model turns out to be what people do, rather than glancing at it.

3D assembly guides — exploded views with per-step visibility — are the most speculative item in this entire note. They need a model with assembly structure, which almost nothing you download has, and authoring one per project is more work than the repair. Park it.

Hardware

The workshop needs what the kitchen already has: a screen you can touch with one knuckle, a camera, a microphone. hosts/kitchen-display/ is exactly this shape already — a Sway kiosk, a webcam, an agent, one workspace — so a workshop display is that image with a different frontend, not a new class of device.

The camera wants to be better than the kitchen's, though, and differently: reading a service tag or a PCB silkscreen is a macro problem. Autofocus at 10–20 cm and enough resolution to resolve 2 mm text matter far more than field of view. A cheap USB microscope/inspection camera is a better fit than a webcam, and is worth pricing before assuming the C920 answer carries over.

The compute problem, again

One RTX 3060, already carrying Assist, four digests a day, pantry-vision and chores. Adding OCR is fine (it is small, and CPU OCR is viable). Adding a second large model that has to be resident is not — and "the workshop assistant felt slow because someone asked for a digest" is exactly the contention already flagged as open decision #4. Design for one model swapped predictably, not two resident.

Recommended order

The project notebook + conversational planning, and the workspace share. No camera, no internet, no new hardware. Useful on day one, and it is where everything else lands — including every artefact the later steps produce.
Spec research with mandatory citation, against an allowlist, cached on disk.
The svg window kind + a Graphviz renderer. One kind and one small tool buys code-flow and data-structure diagrams immediately, and is the pipeline the schematic and board-plan renderers then plug into.
Guide lookup, by typed model number.
Schematics with an IEC symbol skin — the notation work is the work.
Camera identification, barcode first and OCR second, proposing rather than acting.
3D, as pre-rendered orbit frames. A live viewer only if manipulation turns out to be what people actually do.

The purple/magenta holo theme is not in that order because it is not a step: it is one override file and can land whenever.

Steps 1–3 need no new hardware at all — a browser on any existing thin client reaches them. That matters: it means the whole idea can be proved before buying a workshop display, and if the conversation-and-notebook half turns out to be the only part anyone uses, that is a complete and useful outcome rather than a failure.

What is unverified here

Everything past step 1. Specifically: whether iFixit's API terms permit caching what this would cache; whether a local VLM can read a PCB silkscreen at all under workshop lighting (testable today with any webcam and a spare board — do this before buying an inspection camera); whether service-tag lookups work without a vendor account; and whether the OCR-first pipeline holds up on the labels that are scratched, curved, or under a warranty sticker, which in a workshop is most of them.

On the display side specifically: whether netlistsvg's skin mechanism really can express a full IEC symbol set (the notation requirement rests entirely on that, and it is worth a two-hour spike before committing to the tool); whether an LLM produces netlists that are correct rather than merely well-formed, which is a different and much harder property; and how big a vendored three.js actually is against the kiosk's budget before deciding 3D is affordable.

19 KiB Raw Permalink Blame History Unescape Escape