129 lines
7.1 KiB
Markdown
129 lines
7.1 KiB
Markdown
# pantry-vision
|
|
|
|
The kitchen display's backend, from [Phase 17 of the project plan](../docs/project-plan.md).
|
|
|
|
The workflow this exists for: **come home, put down the shopping bag, hold one item
|
|
up to the kitchen display's camera, the system proposes what it is and roughly how
|
|
long it keeps, you confirm (editing anything it got wrong), put it away.** The same
|
|
display then shows the resulting inventory ordered by what expires soonest, and
|
|
Grocy's recipes, on request.
|
|
|
|
- **`pantry-vision`** (this directory) — a small always-on Python HTTP service.
|
|
`POST /identify` (a photo → a proposal via an Ollama vision model), `POST /confirm`
|
|
(a human-reviewed proposal → written into Grocy stock), `GET /inventory`,
|
|
`GET /recipes`, and `GET /shopping-list` (all three proxy Grocy, reshaped for a
|
|
frontend). All five endpoints are bearer-token gated. `/shopping-list` is also
|
|
consumed by `hosts/door-panel/`'s dashboard (Phase 18, "groceries running low") —
|
|
it's a thin reshape of Grocy's own `/api/stock/volatile` `missing_products`, not
|
|
new inventory logic.
|
|
- **`frontend/`** — the static single-page app the kitchen display's kiosk browser
|
|
loads: Scan / Inventory / Recipes, vanilla JS, no build step, no framework — same
|
|
"vendored, dependency-free" choice as the digest/admin canvas SDKs. Served
|
|
read-only by a `pantry-web` nginx container (`setup-container-host.sh`), the same
|
|
role `digest-web`/`admin-web` already play for their own hosts.
|
|
- **`hosts/kitchen-display/`** — the touch kiosk image that runs the frontend. See
|
|
that directory's own README for the device side of this.
|
|
|
|
## A real network listener, unlike admin-canvas
|
|
|
|
`admin-canvas` deliberately has **no published port** — only Home Assistant, on the
|
|
same compose network, ever calls it. `pantry-vision` is different on purpose: the
|
|
kitchen display is a separate physical device on the LAN and has to reach this
|
|
service directly (there is no HA-mediation step between "hold item up to camera" and
|
|
"get an identification back" — that has to be fast and synchronous). So
|
|
`PANTRY_VISION_PORT` **is** published, and every request — including the two GETs —
|
|
requires the bearer token, as the actual boundary instead of network placement.
|
|
|
|
The same token has to be baked into the kitchen display's own build config
|
|
(`hosts/kitchen-display/scripts/build-kitchen-display-iso.sh`), not just Home
|
|
Assistant's — see that host's README.
|
|
|
|
## `/identify` never writes anything by itself
|
|
|
|
This is the one guardrail that matters most in this whole phase. The vision model's
|
|
guess — name, category, how many days until it likely goes bad — is a **proposal**,
|
|
shown on screen for the person to review and edit before anything is confirmed. Only
|
|
`/confirm`, a separate call the frontend makes after the person taps "Confirm & add,"
|
|
ever writes to Grocy. This is the same "propose, never auto-commit" rule this project
|
|
already applies to identity-merge confirmation (see the *Identity store* row in
|
|
`docs/project-plan.md` §2) — a wrong camera guess costs one tap to fix, not a wrong
|
|
fact silently written into the household's inventory.
|
|
|
|
## Configure
|
|
|
|
```sh
|
|
cp pantry-vision/pantry-vision.env.example /opt/smart-home/pantry-vision/pantry-vision.env
|
|
openssl rand -hex 32 # put the result in PANTRY_VISION_TOKEN
|
|
chmod 600 /opt/smart-home/pantry-vision/pantry-vision.env
|
|
$EDITOR /opt/smart-home/pantry-vision/pantry-vision.env
|
|
```
|
|
|
|
You also need, inside Grocy's own UI (it's already running as the always-on `grocy`
|
|
container regardless of this phase): **Settings → Manage API keys** for
|
|
`GROCY_API_KEY`, and to confirm **Settings → Locations / Quantity units** actually
|
|
match `GROCY_DEFAULT_LOCATION_ID`/`GROCY_DEFAULT_QU_ID` (fresh-install defaults, not
|
|
guaranteed to match a Grocy that's already been customised).
|
|
|
|
## Pick a vision model
|
|
|
|
`OLLAMA_VISION_MODEL` defaults to `llava`, but **nothing here has confirmed that name
|
|
against a real pull** — pick a vision-capable model (`llava`, `qwen2.5vl`, or whatever
|
|
your LLM host's GPU/CPU tier can run at acceptable latency for someone standing at
|
|
the counter holding a can of beans) and:
|
|
|
|
```sh
|
|
ollama pull llava # on the LLM host, or whichever model you picked
|
|
```
|
|
|
|
Plain text models (`qwen2.5:14b-instruct`, used elsewhere in this project for
|
|
digest/Assist) **cannot see images at all** — pointing `OLLAMA_VISION_MODEL` at one
|
|
of those will not error clearly, it will just produce a useless/hallucinated
|
|
response for every photo. Latency is unmeasured; a vision pass on a CPU-only LLM
|
|
host could easily be too slow for a "hold item up to camera" interaction to feel
|
|
responsive — this needs to be measured on real hardware, not assumed.
|
|
|
|
## Grocy API assumptions — unverified against a real instance
|
|
|
|
`server.py`'s Grocy calls (`_find_or_create_product`, `_add_to_stock`, the `/inventory`
|
|
and `/recipes` proxies) are written against Grocy's *documented* API shape, not
|
|
checked against a running instance. In particular:
|
|
|
|
- Whether `GET /api/stock` rows carry a nested `product` object with a `name` field
|
|
by default, or need an explicit embed/expand parameter — `_handle_inventory`
|
|
degrades to `Product #<id>` if not, rather than dropping the row, but that's a
|
|
fallback, not a fix.
|
|
- Whether `POST /api/objects/products` with just
|
|
`name`/`location_id`/`qu_id_purchase`/`qu_id_stock` is actually enough to create a
|
|
minimal product on your Grocy version, or whether it requires more fields.
|
|
- Whether the Recipes feature (`GET /api/objects/recipes`,
|
|
`GET /api/recipes/{id}/fulfillment`) needs to be explicitly enabled/populated
|
|
before it returns anything meaningful — `_handle_recipes` degrades to
|
|
`"fulfilled": null` per-recipe on any failure rather than breaking the whole list.
|
|
|
|
Grocy exposes a live OpenAPI spec at `http://<grocy-host>:9283/api/openapi/specification`
|
|
once it's running — read that against a real instance before trusting any of the
|
|
above, and adjust `server.py` if the shapes differ.
|
|
|
|
## Deploy
|
|
|
|
Wired into `hosts/container-host/scripts/setup-container-host.sh` behind
|
|
`ENABLE_PANTRY_VISION` (off by default) — see that script's `# CONFIGURATION` block
|
|
and its own README. It builds two containers: `pantry-vision` (this API) and
|
|
`pantry-web` (nginx, serves `frontend/` read-only).
|
|
|
|
## Manual verification still outstanding
|
|
|
|
1. All of the Grocy API assumptions above.
|
|
2. Real-world vision-model accuracy and latency for grocery items — untested with
|
|
any actual model or camera.
|
|
3. Whether Ollama's `/api/generate` `images` field is still the right call shape for
|
|
whichever vision model you pick — some multimodal models are only exposed through
|
|
Ollama's newer `/api/chat` with a `images` field per-message instead; this was
|
|
written against `/api/generate`'s documented multimodal support and not run
|
|
against a real model.
|
|
4. CORS: `_respond`'s blanket `Access-Control-Allow-Origin: *` is fine for a
|
|
LAN-only, bearer-token-gated service with no cookies, but hasn't been checked
|
|
against a real browser's preflight behavior for the raw-image-bytes `POST
|
|
/identify` call in particular (some browsers preflight non-simple `Content-Type`s
|
|
like `image/jpeg` — `do_OPTIONS` is written to handle that but is untested).
|