SmartestHome/pantry-vision/README.md

pantry-vision

The kitchen display's backend, from Phase 17 of the project plan.

The workflow this exists for: come home, put down the shopping bag, hold one item up to the kitchen display's camera, the system proposes what it is and roughly how long it keeps, you confirm (editing anything it got wrong), put it away. The same display then shows the resulting inventory ordered by what expires soonest, and Grocy's recipes, on request.

• pantry-vision (this directory) — a small always-on Python HTTP service. POST /identify (a photo → a proposal via an Ollama vision model), the writes POST /confirm / POST /consume / POST /adjust / POST /transfer, the door-sensor hook POST /doorway-event, and the reads GET /inventory, GET /expired, GET /doorway-events, GET /recipes, GET /shopping-list (which proxy Grocy, reshaped for a frontend). Every endpoint is bearer-token gated. /shopping-list is also consumed by hosts/door-panel/'s dashboard (Phase 18, "groceries running low") — it's a thin reshape of Grocy's own /api/stock/volatile missing_products, not new inventory logic.
• doorway.py — the appliance-door half: a contact sensor fires, a camera burst is pulled from Frigate, and what it recognised is recorded as a hint. See "Which fridge is it in" below, and docs/fridge-item-location.md for the argument.
• frontend/ — the static single-page app the kitchen display's kiosk browser loads: the four stock-movement screens below, plus Inventory and Recipes. Vanilla JS, no build step, no framework — same "vendored, dependency-free" choice as the digest/admin canvas SDKs. Served read-only by a pantry-web nginx container (setup-container-host.sh), the same role digest-web/admin-web already play for their own hosts.
• hosts/kitchen-display/ — the touch kiosk image that runs the frontend. See that directory's own README for the device side of this.

The four ways stock moves

Stock only ever changes in four ways, so the display has four buttons, and three of them are camera-first.

Screen	What it does	Grocy call underneath
Unload groceries	The camera runs in a loop. Hold up an item, get a stats card — description, best-before, where to put it, and how many individual things are in the pack — confirm, and it moves straight on to the next item without you touching the screen again.	stock/products/{id}/add
Consume article	Hold up what you are about to eat. If the same kind is in stock under more than one brand, it asks which; if the row holds more than one unit, it asks how many.	stock/products/{id}/consume
List expired foods	Everything already past its date. Clear a line by scanning the item you are about to bin (or with the row's own button when the label is unreadable). Booked out as spoiled, not eaten — Grocy keeps those apart, and that is the only way you will ever find out what the household keeps buying and throwing away.	stock/products/{id}/consume with spoiled
Edit inventory	The keyboard-and-buttons fallback: every article, −/+, and a freeform amount. This is the screen that does not use the camera, on purpose — it is where you go when the camera got something wrong.	stock/products/{id}/inventory

Corrections go through Grocy's inventory-correction endpoint rather than consume/add, so the stock journal says "somebody fixed the number" instead of quietly filling the household's consumption history with corrections dressed as meals.

Two invariants everything else rests on

Stock is counted in individual units, never in packages. A twelve-pack of eggs is booked in as twelve. "How many eggs do we have" is the question people actually ask, and it makes booking out three of them arithmetic instead of a fractions-of-a-pack problem. The vision model's units_per_package is a multiplier applied once on the confirm screen — where it is editable, and where the screen states the result ("Books in 12 eggs") before anything is written. Nothing downstream stores it. Set GROCY_DEFAULT_QU_ID to a Piece-like unit accordingly.

The fold key is the brand-free product kind. Twelve eggs of brand X and ten of brand Y are twenty-two eggs. The vision model is asked for a brand-free kind alongside the brand, /confirm files the Grocy product under a product group named after that kind, and the folding is a read of that group — so the grouping is visible and editable in Grocy's own UI instead of living in a second classification scheme here. The per-brand rows are never summed away: they are what the edit screen expands to, because "take ten off brand Y specifically" has to stay possible.

The folding itself is deliberately dumb (casefold, strip a trailing (Brand), collapse whitespace). The clever version is a synonym problem — eggs vs egg vs free-range eggs — that nobody wants adjudicated by a kitchen display at 19:00. When the model answers inconsistently you get two lines instead of one, which is visible and fixable; a wrongly merged line is neither.

Which fridge is it in

The household has more than one cold appliance, so "in the fridge" is not an answer. Two mechanisms address that, and only the first is authoritative:

1. Grocy locations — the record. The confirm screen's placement choice resolves to a real Grocy location (PANTRY_LOCATION_*, created by name) and rides along on the stock add. /inventory reports it. POST /transfer moves an amount between appliances, and the edit screen has a "Move to…" control per row. That transfer is the part that makes locations worth recording at all: without it, a location decays into "where it was when it was bought", which is worse than no answer because it is confidently wrong.

2. Doorway hints — an observation, never a fact. A Zigbee contact sensor on each appliance door drives a Home Assistant automation that POSTs to /doorway-event:

# Home Assistant automation — one per appliance door.
trigger:
  - platform: state
    entity_id: binary_sensor.fridge_kitchen_door
    to: "on"
action:
  - service: rest_command.pantry_doorway_event
    data: { appliance: "fridge-kitchen", state: "opened" }

pantry-vision answers 202 immediately and does the work on its own thread — the burst is up to three snapshots and a vision call each, and an appliance door is not something to keep a home automation waiting on. It pulls frames from that appliance's Frigate camera, stops at the first one that identifies something, and writes what it saw to its own SQLite file with a timestamp and a confidence.

Nothing in that path writes stock. A camera at a door cannot tell in from out, misses when two things are carried at once, and sees nothing behind an arm. So it produces "camera last saw something like this at Freezer (loggia), 20 min ago — a sighting, not a fact", which is a sentence a person can evaluate, shown next to the location Grocy actually records. Acting on it is a tap on Move.

Configure it with PANTRY_DOOR_APPLIANCES (id:Grocy location name:frigate_camera, camera optional) and FRIGATE_URL — the same Frigate chores already pulls snapshots from. An appliance with a door sensor and no camera is a legitimate configuration and is the recommended starting point: it still records that the door opened, which is the half of this feature that pays for itself.

What the camera loop actually does

frontend/app.js samples a 32×24 greyscale thumbnail of the video every 700 ms and only spends an /identify call when the picture has settled (it stopped moving) and changed since the last thing it identified. Both gates exist because vision latency is this phase's known open risk: the frames worth spending it on are the ones where somebody is holding something still, and the same tin must never be identified twice because nobody moved. "Identify now" overrides both, for the shiny jar under a downlight that never settles.

A model that is down or answering nonsense stops the loop rather than re-photographing the counter at it; the unload screen drops you into the manual form instead.

A real network listener, unlike admin-canvas

admin-canvas deliberately has no published port — only Home Assistant, on the same compose network, ever calls it. pantry-vision is different on purpose: the kitchen display is a separate physical device on the LAN and has to reach this service directly (there is no HA-mediation step between "hold item up to camera" and "get an identification back" — that has to be fast and synchronous). So PANTRY_VISION_PORT is published, and every request — including the two GETs — requires the bearer token, as the actual boundary instead of network placement.

The same token has to be baked into the kitchen display's own build config (tools/build-kitchen-display-iso.sh), not just Home Assistant's — see that host's README.

/identify never writes anything by itself

This is the one guardrail that matters most in this whole phase. The vision model's guess — name, category, how many days until it likely goes bad — is a proposal, shown on screen for the person to review and edit before anything is confirmed. Only /confirm, a separate call the frontend makes after the person taps "Confirm & add," ever writes to Grocy. This is the same "propose, never auto-commit" rule this project already applies to identity-merge confirmation (see the Identity store row in docs/project-plan.md §2) — a wrong camera guess costs one tap to fix, not a wrong fact silently written into the household's inventory.

The book-out screens follow the same rule in the other direction, and it costs a tap there too: recognition never consumes anything by itself, an ambiguous brand is asked about rather than picked, and "throw away" states the amount before it goes. A camera that quietly books out the wrong yoghurt produces an inventory nobody trusts, and an inventory nobody trusts is worth exactly as much as no inventory.

One place where that guardrail is deliberately looser: the vision model is asked to read a printed best-before date off the packaging when one is legible, and that date lands pre-filled in the confirm screen's date field. The screen always says which it is showing — "Date read off the packaging — check it" versus "Estimated from the category" — and the server throws out anything more than a year in the past or ten years out, because a misread label (small print, dot-matrix ink, curved packaging) is the single most likely failure of this feature and that is what it looks like when it happens.

Configure

cp pantry-vision/pantry-vision.env.example /opt/smart-home/pantry-vision/pantry-vision.env
openssl rand -hex 32   # put the result in PANTRY_VISION_TOKEN
chmod 600 /opt/smart-home/pantry-vision/pantry-vision.env
$EDITOR /opt/smart-home/pantry-vision/pantry-vision.env

You also need, inside Grocy's own UI (it's already running as the always-on grocy container regardless of this phase): Settings → Manage API keys for GROCY_API_KEY, and to confirm Settings → Locations / Quantity units actually match GROCY_DEFAULT_LOCATION_ID/GROCY_DEFAULT_QU_ID (fresh-install defaults, not guaranteed to match a Grocy that's already been customised).

Pick a vision model

OLLAMA_VISION_MODEL defaults to llava, but nothing here has confirmed that name against a real pull — pick a vision-capable model (llava, qwen2.5vl, or whatever your LLM host's GPU/CPU tier can run at acceptable latency for someone standing at the counter holding a can of beans) and:

ollama pull llava   # on the LLM host, or whichever model you picked

Plain text models (qwen2.5:14b-instruct, used elsewhere in this project for digest/Assist) cannot see images at all — pointing OLLAMA_VISION_MODEL at one of those will not error clearly, it will just produce a useless/hallucinated response for every photo. Latency is unmeasured; a vision pass on a CPU-only LLM host could easily be too slow for a "hold item up to camera" interaction to feel responsive — this needs to be measured on real hardware, not assumed.

Grocy API assumptions — unverified against a real instance

server.py's Grocy calls (_find_or_create_product, _add_to_stock, _consume_stock, _set_stock_amount, _stock_items and the /recipes proxy) are written against Grocy's documented API shape, not checked against a running instance. In particular:

• Whether GET /api/stock rows carry a nested product object with a name field by default, or need an explicit embed/expand parameter. _stock_items sidesteps this by fetching /api/objects/products separately and joining on product_id — it needs the group and location anyway, not just the name — and degrades to Product #<id> if both are missing, rather than dropping the row.
• Whether POST /api/objects/products with just name/location_id/qu_id_purchase/qu_id_stock is actually enough to create a minimal product on your Grocy version, or whether it requires more fields.
• Whether POST /api/objects/product_groups and POST /api/objects/locations accept a bare {"name": ...}. Both are best-effort: a failure means the product lands without a group (so it folds on its name instead) or in GROCY_DEFAULT_LOCATION_ID (so it is in the wrong place but still in stock). Neither costs you the book-in.
• Whether POST /api/stock/products/{id}/consume takes spoiled as a boolean, and what it returns when you try to consume more than is in stock — that refusal is passed through to the screen verbatim, on the assumption that it is a real answer ("we have fewer than you think") rather than a transport failure.
• Whether POST /api/stock/products/{id}/inventory takes new_amount as the absolute new figure and whether it requires best_before_date when the figure goes up. The edit screen currently sends the amount alone.
• Whether POST /api/stock/products/{id}/transfer takes location_id_from / location_id_to, and — the subtle one — which location Grocy considers a stock entry to be in. /transfer uses the product's location as the source, which is what /confirm set. Grocy can also hold one product's stock across several locations at once, per entry; if your instance does that, the source this sends will sometimes be wrong and the transfer will fail rather than move the wrong thing. Reading /api/stock/products/{id}/entries is the fix if it comes up.
• Whether Frigate's /api/<camera>/latest.jpg returns a usable still for a camera that is idle — doorway.py and chores/check.py both assume it does, neither has called it.
• Whether the Recipes feature (GET /api/objects/recipes, GET /api/recipes/{id}/fulfillment) needs to be explicitly enabled/populated before it returns anything meaningful — _handle_recipes degrades to "fulfilled": null per-recipe on any failure rather than breaking the whole list.

Grocy exposes a live OpenAPI spec at http://<grocy-host>:9283/api/openapi/specification once it's running — read that against a real instance before trusting any of the above, and adjust server.py if the shapes differ.

Deploy

Wired into tools/setup-container-host.sh behind ENABLE_PANTRY_VISION (off by default) — see that script's # CONFIGURATION block and its own README. It builds two containers: pantry-vision (this API) and pantry-web (nginx, serves frontend/ read-only).

Manual verification still outstanding

1. All of the Grocy API assumptions above.
2. Real-world vision-model accuracy and latency for grocery items — untested with any actual model or camera. The unload loop is where latency bites hardest: it is built to feel like a queue of items rather than a queue of round trips, and if an /identify takes 20 seconds it will not feel like either.
3. Whether the model returns a stable kind for the same product across scans. This is the assumption the whole brand-folding rests on, and the one most likely to disappoint quietly: scan the same carton of eggs five times and see whether it says "eggs" five times. If it drifts, the fix is a fixed kind vocabulary in the prompt, not more clever folding on this side.
4. Whether printed best-before dates are read correctly or confidently invented. The prompt says not to guess and the server bounds the range, but neither can catch a plausible wrong date. Check a handful against the actual packets before trusting the pre-filled field.
5. That a twelve-pack really does arrive as twelve. units_per_package is the one model answer that gets multiplied rather than displayed, and the failure is silent in both directions — twelve eggs booked in as one, or one jar booked in as twelve.
6. Whether a doorway camera can identify anything at all. This is the assumption the whole /doorway-event path rests on, and it is not the same task as the kitchen display's: an item in a moving hand, at ~1.5 m, in whatever light the room has, versus one held still against a plain background 30 cm from a webcam. docs/fridge-item-location.md says to test it with the kitchen's existing camera before buying a second one, and that advice applies to this code too — if the answer is no, the door sensors are still worth having and this half simply stays switched off (PANTRY_DOOR_APPLIANCES entries without a camera).
7. Whether the hints are read as hints. The wording on screen ("a sighting, not a fact") is doing real work; if in practice people treat a sighting as the answer and stop checking, that is a design failure this code cannot detect and the feature should be turned off rather than tuned.
8. Whether Ollama's /api/generate images field is still the right call shape for whichever vision model you pick — some multimodal models are only exposed through Ollama's newer /api/chat with a images field per-message instead; this was written against /api/generate's documented multimodal support and not run against a real model.
9. CORS: _respond's blanket Access-Control-Allow-Origin: * is fine for a LAN-only, bearer-token-gated service with no cookies, but hasn't been checked against a real browser's preflight behavior for the raw-image-bytes POST /identify call in particular (some browsers preflight non-simple Content-Types like image/jpeg — do_OPTIONS is written to handle that but is untested).