# pantry-vision The kitchen display's backend, from [Phase 17 of the project plan](../docs/project-plan.md). The workflow this exists for: **come home, put down the shopping bag, hold one item up to the kitchen display's camera, the system proposes what it is and roughly how long it keeps, you confirm (editing anything it got wrong), put it away.** The same display then shows the resulting inventory ordered by what expires soonest, and Grocy's recipes, on request. - **`pantry-vision`** (this directory) — a small always-on Python HTTP service. `POST /identify` (a photo → a proposal via an Ollama vision model), the writes `POST /confirm` / `POST /consume` / `POST /adjust` / `POST /transfer`, the door-sensor hook `POST /doorway-event`, and the reads `GET /inventory`, `GET /expired`, `GET /doorway-events`, `GET /recipes`, `GET /shopping-list` (which proxy Grocy, reshaped for a frontend). Every endpoint is bearer-token gated. `/shopping-list` is also consumed by `hosts/door-panel/`'s dashboard (Phase 18, "groceries running low") — it's a thin reshape of Grocy's own `/api/stock/volatile` `missing_products`, not new inventory logic. - **`doorway.py`** — the appliance-door half: a contact sensor fires, a camera burst is pulled from Frigate, and what it recognised is recorded as a **hint**. See "Which fridge is it in" below, and `docs/fridge-item-location.md` for the argument. - **`frontend/`** — the static single-page app the kitchen display's kiosk browser loads: the four stock-movement screens below, plus Inventory and Recipes. Vanilla JS, no build step, no framework — same "vendored, dependency-free" choice as the digest/admin canvas SDKs. Served read-only by a `pantry-web` nginx container (`setup-container-host.sh`), the same role `digest-web`/`admin-web` already play for their own hosts. - **`hosts/kitchen-display/`** — the touch kiosk image that runs the frontend. See that directory's own README for the device side of this. ## The four ways stock moves Stock only ever changes in four ways, so the display has four buttons, and three of them are camera-first. | Screen | What it does | Grocy call underneath | | --- | --- | --- | | **Unload groceries** | The camera runs in a loop. Hold up an item, get a stats card — description, best-before, where to put it, and how many individual things are in the pack — confirm, and it moves straight on to the next item without you touching the screen again. | `stock/products/{id}/add` | | **Consume article** | Hold up what you are about to eat. If the same kind is in stock under more than one brand, it asks which; if the row holds more than one unit, it asks how many. | `stock/products/{id}/consume` | | **List expired foods** | Everything already past its date. Clear a line by scanning the item you are about to bin (or with the row's own button when the label is unreadable). Booked out as **spoiled**, not eaten — Grocy keeps those apart, and that is the only way you will ever find out what the household keeps buying and throwing away. | `stock/products/{id}/consume` with `spoiled` | | **Edit inventory** | The keyboard-and-buttons fallback: every article, `−`/`+`, and a freeform amount. This is the screen that does *not* use the camera, on purpose — it is where you go when the camera got something wrong. | `stock/products/{id}/inventory` | Corrections go through Grocy's *inventory-correction* endpoint rather than consume/add, so the stock journal says "somebody fixed the number" instead of quietly filling the household's consumption history with corrections dressed as meals. ### Two invariants everything else rests on **Stock is counted in individual units, never in packages.** A twelve-pack of eggs is booked in as twelve. "How many eggs do we have" is the question people actually ask, and it makes booking out three of them arithmetic instead of a fractions-of-a-pack problem. The vision model's `units_per_package` is a multiplier applied once on the confirm screen — where it is editable, and where the screen states the result ("Books in 12 eggs") before anything is written. Nothing downstream stores it. Set `GROCY_DEFAULT_QU_ID` to a *Piece*-like unit accordingly. **The fold key is the brand-free product kind.** Twelve eggs of brand X and ten of brand Y are twenty-two eggs. The vision model is asked for a brand-free `kind` alongside the brand, `/confirm` files the Grocy product under a **product group** named after that kind, and the folding is a read of that group — so the grouping is visible and editable in Grocy's own UI instead of living in a second classification scheme here. The per-brand rows are never summed away: they are what the edit screen expands to, because "take ten off brand Y specifically" has to stay possible. The folding itself is deliberately dumb (casefold, strip a trailing `(Brand)`, collapse whitespace). The clever version is a synonym problem — *eggs* vs *egg* vs *free-range eggs* — that nobody wants adjudicated by a kitchen display at 19:00. When the model answers inconsistently you get two lines instead of one, which is visible and fixable; a wrongly merged line is neither. ## Which fridge is it in The household has more than one cold appliance, so "in the fridge" is not an answer. Two mechanisms address that, and **only the first is authoritative**: **1. Grocy locations — the record.** The confirm screen's placement choice resolves to a real Grocy location (`PANTRY_LOCATION_*`, created by name) and rides along on the stock add. `/inventory` reports it. `POST /transfer` moves an amount between appliances, and the edit screen has a "Move to…" control per row. That transfer is the part that makes locations worth recording at all: without it, a location decays into "where it was when it was bought", which is worse than no answer because it is confidently wrong. **2. Doorway hints — an observation, never a fact.** A Zigbee contact sensor on each appliance door drives a Home Assistant automation that POSTs to `/doorway-event`: ```yaml # Home Assistant automation — one per appliance door. trigger: - platform: state entity_id: binary_sensor.fridge_kitchen_door to: "on" action: - service: rest_command.pantry_doorway_event data: { appliance: "fridge-kitchen", state: "opened" } ``` `pantry-vision` answers **202 immediately** and does the work on its own thread — the burst is up to three snapshots and a vision call each, and an appliance door is not something to keep a home automation waiting on. It pulls frames from that appliance's Frigate camera, stops at the first one that identifies something, and writes what it saw to its own SQLite file with a timestamp and a confidence. **Nothing in that path writes stock.** A camera at a door cannot tell in from out, misses when two things are carried at once, and sees nothing behind an arm. So it produces "camera last saw something like this at Freezer (loggia), 20 min ago — a sighting, not a fact", which is a sentence a person can evaluate, shown next to the location Grocy actually records. Acting on it is a tap on Move. Configure it with `PANTRY_DOOR_APPLIANCES` (`id:Grocy location name:frigate_camera`, camera optional) and `FRIGATE_URL` — the same Frigate `chores` already pulls snapshots from. **An appliance with a door sensor and no camera is a legitimate configuration** and is the recommended starting point: it still records that the door opened, which is the half of this feature that pays for itself. ### What the camera loop actually does `frontend/app.js` samples a 32×24 greyscale thumbnail of the video every 700 ms and only spends an `/identify` call when the picture has **settled** (it stopped moving) *and* **changed** since the last thing it identified. Both gates exist because vision latency is this phase's known open risk: the frames worth spending it on are the ones where somebody is holding something still, and the same tin must never be identified twice because nobody moved. "Identify now" overrides both, for the shiny jar under a downlight that never settles. A model that is down or answering nonsense **stops the loop** rather than re-photographing the counter at it; the unload screen drops you into the manual form instead. ## A real network listener, unlike admin-canvas `admin-canvas` deliberately has **no published port** — only Home Assistant, on the same compose network, ever calls it. `pantry-vision` is different on purpose: the kitchen display is a separate physical device on the LAN and has to reach this service directly (there is no HA-mediation step between "hold item up to camera" and "get an identification back" — that has to be fast and synchronous). So `PANTRY_VISION_PORT` **is** published, and every request — including the two GETs — requires the bearer token, as the actual boundary instead of network placement. The same token has to be baked into the kitchen display's own build config (`tools/build-kitchen-display-iso.sh`), not just Home Assistant's — see that host's README. ## `/identify` never writes anything by itself This is the one guardrail that matters most in this whole phase. The vision model's guess — name, category, how many days until it likely goes bad — is a **proposal**, shown on screen for the person to review and edit before anything is confirmed. Only `/confirm`, a separate call the frontend makes after the person taps "Confirm & add," ever writes to Grocy. This is the same "propose, never auto-commit" rule this project already applies to identity-merge confirmation (see the *Identity store* row in `docs/project-plan.md` §2) — a wrong camera guess costs one tap to fix, not a wrong fact silently written into the household's inventory. The book-out screens follow the same rule in the other direction, and it costs a tap there too: recognition never consumes anything by itself, an ambiguous brand is asked about rather than picked, and "throw away" states the amount before it goes. A camera that quietly books out the wrong yoghurt produces an inventory nobody trusts, and an inventory nobody trusts is worth exactly as much as no inventory. One place where that guardrail is deliberately looser: the vision model is asked to read a **printed best-before date** off the packaging when one is legible, and that date lands pre-filled in the confirm screen's date field. The screen always says which it is showing — *"Date read off the packaging — check it"* versus *"Estimated from the category"* — and the server throws out anything more than a year in the past or ten years out, because a misread label (small print, dot-matrix ink, curved packaging) is the single most likely failure of this feature and that is what it looks like when it happens. ## Configure ```sh cp pantry-vision/pantry-vision.env.example /opt/smart-home/pantry-vision/pantry-vision.env openssl rand -hex 32 # put the result in PANTRY_VISION_TOKEN chmod 600 /opt/smart-home/pantry-vision/pantry-vision.env $EDITOR /opt/smart-home/pantry-vision/pantry-vision.env ``` You also need, inside Grocy's own UI (it's already running as the always-on `grocy` container regardless of this phase): **Settings → Manage API keys** for `GROCY_API_KEY`, and to confirm **Settings → Locations / Quantity units** actually match `GROCY_DEFAULT_LOCATION_ID`/`GROCY_DEFAULT_QU_ID` (fresh-install defaults, not guaranteed to match a Grocy that's already been customised). ## Pick a vision model `OLLAMA_VISION_MODEL` defaults to `llava`, but **nothing here has confirmed that name against a real pull** — pick a vision-capable model (`llava`, `qwen2.5vl`, or whatever your LLM host's GPU/CPU tier can run at acceptable latency for someone standing at the counter holding a can of beans) and: ```sh ollama pull llava # on the LLM host, or whichever model you picked ``` Plain text models (`qwen2.5:14b-instruct`, used elsewhere in this project for digest/Assist) **cannot see images at all** — pointing `OLLAMA_VISION_MODEL` at one of those will not error clearly, it will just produce a useless/hallucinated response for every photo. Latency is unmeasured; a vision pass on a CPU-only LLM host could easily be too slow for a "hold item up to camera" interaction to feel responsive — this needs to be measured on real hardware, not assumed. ## Grocy API assumptions — unverified against a real instance `server.py`'s Grocy calls (`_find_or_create_product`, `_add_to_stock`, `_consume_stock`, `_set_stock_amount`, `_stock_items` and the `/recipes` proxy) are written against Grocy's *documented* API shape, not checked against a running instance. In particular: - Whether `GET /api/stock` rows carry a nested `product` object with a `name` field by default, or need an explicit embed/expand parameter. `_stock_items` sidesteps this by fetching `/api/objects/products` separately and joining on `product_id` — it needs the group and location anyway, not just the name — and degrades to `Product #` if both are missing, rather than dropping the row. - Whether `POST /api/objects/products` with just `name`/`location_id`/`qu_id_purchase`/`qu_id_stock` is actually enough to create a minimal product on your Grocy version, or whether it requires more fields. - Whether `POST /api/objects/product_groups` and `POST /api/objects/locations` accept a bare `{"name": ...}`. Both are best-effort: a failure means the product lands without a group (so it folds on its name instead) or in `GROCY_DEFAULT_LOCATION_ID` (so it is in the wrong place but still in stock). Neither costs you the book-in. - Whether `POST /api/stock/products/{id}/consume` takes `spoiled` as a boolean, and what it returns when you try to consume more than is in stock — that refusal is passed through to the screen verbatim, on the assumption that it is a real answer ("we have fewer than you think") rather than a transport failure. - Whether `POST /api/stock/products/{id}/inventory` takes `new_amount` as the absolute new figure and whether it *requires* `best_before_date` when the figure goes up. The edit screen currently sends the amount alone. - Whether `POST /api/stock/products/{id}/transfer` takes `location_id_from` / `location_id_to`, and — the subtle one — **which location Grocy considers a stock entry to be in.** `/transfer` uses the *product's* location as the source, which is what `/confirm` set. Grocy can also hold one product's stock across several locations at once, per entry; if your instance does that, the source this sends will sometimes be wrong and the transfer will fail rather than move the wrong thing. Reading `/api/stock/products/{id}/entries` is the fix if it comes up. - Whether Frigate's `/api//latest.jpg` returns a usable still for a camera that is idle — `doorway.py` and `chores/check.py` both assume it does, neither has called it. - Whether the Recipes feature (`GET /api/objects/recipes`, `GET /api/recipes/{id}/fulfillment`) needs to be explicitly enabled/populated before it returns anything meaningful — `_handle_recipes` degrades to `"fulfilled": null` per-recipe on any failure rather than breaking the whole list. Grocy exposes a live OpenAPI spec at `http://:9283/api/openapi/specification` once it's running — read that against a real instance before trusting any of the above, and adjust `server.py` if the shapes differ. ## Deploy Wired into `tools/setup-container-host.sh` behind `ENABLE_PANTRY_VISION` (off by default) — see that script's `# CONFIGURATION` block and its own README. It builds two containers: `pantry-vision` (this API) and `pantry-web` (nginx, serves `frontend/` read-only). ## Manual verification still outstanding 1. All of the Grocy API assumptions above. 2. Real-world vision-model accuracy and latency for grocery items — untested with any actual model or camera. The unload loop is where latency bites hardest: it is built to feel like a queue of items rather than a queue of round trips, and if an `/identify` takes 20 seconds it will not feel like either. 3. **Whether the model returns a stable `kind` for the same product across scans.** This is the assumption the whole brand-folding rests on, and the one most likely to disappoint quietly: scan the same carton of eggs five times and see whether it says "eggs" five times. If it drifts, the fix is a fixed `kind` vocabulary in the prompt, not more clever folding on this side. 4. **Whether printed best-before dates are read correctly or confidently invented.** The prompt says not to guess and the server bounds the range, but neither can catch a plausible wrong date. Check a handful against the actual packets before trusting the pre-filled field. 5. **That a twelve-pack really does arrive as twelve.** `units_per_package` is the one model answer that gets multiplied rather than displayed, and the failure is silent in both directions — twelve eggs booked in as one, or one jar booked in as twelve. 6. **Whether a doorway camera can identify anything at all.** This is the assumption the whole `/doorway-event` path rests on, and it is not the same task as the kitchen display's: an item in a moving hand, at ~1.5 m, in whatever light the room has, versus one held still against a plain background 30 cm from a webcam. `docs/fridge-item-location.md` says to test it with the kitchen's existing camera before buying a second one, and that advice applies to this code too — if the answer is no, the door sensors are still worth having and this half simply stays switched off (`PANTRY_DOOR_APPLIANCES` entries without a camera). 7. **Whether the hints are read as hints.** The wording on screen ("a sighting, not a fact") is doing real work; if in practice people treat a sighting as the answer and stop checking, that is a design failure this code cannot detect and the feature should be turned off rather than tuned. 3. Whether Ollama's `/api/generate` `images` field is still the right call shape for whichever vision model you pick — some multimodal models are only exposed through Ollama's newer `/api/chat` with a `images` field per-message instead; this was written against `/api/generate`'s documented multimodal support and not run against a real model. 4. CORS: `_respond`'s blanket `Access-Control-Allow-Origin: *` is fine for a LAN-only, bearer-token-gated service with no cookies, but hasn't been checked against a real browser's preflight behavior for the raw-image-bytes `POST /identify` call in particular (some browsers preflight non-simple `Content-Type`s like `image/jpeg` — `do_OPTIONS` is written to handle that but is untested).