SmartestHome/pantry-vision/README.md

# pantry-vision

The kitchen display's backend, from [Phase 17 of the project plan](../docs/project-plan.md).

The workflow this exists for: **come home, put down the shopping bag, hold one item
up to the kitchen display's camera, the system proposes what it is and roughly how
long it keeps, you confirm (editing anything it got wrong), put it away.** The same
display then shows the resulting inventory ordered by what expires soonest, and
Grocy's recipes, on request.

- **`pantry-vision`** (this directory) — a small always-on Python HTTP service.
  `POST /identify` (a photo → a proposal via an Ollama vision model), the writes
  `POST /confirm` / `POST /consume` / `POST /adjust` / `POST /transfer`, the
  door-sensor hook `POST /doorway-event`, and the reads `GET /inventory`,
  `GET /expired`, `GET /doorway-events`, `GET /recipes`, `GET /shopping-list` (which
  proxy Grocy, reshaped for a frontend). Every endpoint is bearer-token gated.
  `/shopping-list` is also consumed by `hosts/door-panel/`'s dashboard (Phase 18,
  "groceries running low") — it's a thin reshape of Grocy's own
  `/api/stock/volatile` `missing_products`, not new inventory logic.
- **`doorway.py`** — the appliance-door half: a contact sensor fires, a camera burst
  is pulled from Frigate, and what it recognised is recorded as a **hint**. See "Which
  fridge is it in" below, and `docs/fridge-item-location.md` for the argument.
- **`frontend/`** — the static single-page app the kitchen display's kiosk browser
  loads: the four stock-movement screens below, plus Inventory and Recipes. Vanilla
  JS, no build step, no framework — same "vendored, dependency-free" choice as the
  digest/admin canvas SDKs. Served read-only by a `pantry-web` nginx container
  (`setup-container-host.sh`), the same role `digest-web`/`admin-web` already play for
  their own hosts.
- **`hosts/kitchen-display/`** — the touch kiosk image that runs the frontend. See
  that directory's own README for the device side of this.

## The four ways stock moves

Stock only ever changes in four ways, so the display has four buttons, and three of
them are camera-first.

| Screen | What it does | Grocy call underneath |
| --- | --- | --- |
| **Unload groceries** | The camera runs in a loop. Hold up an item, get a stats card — description, best-before, where to put it, and how many individual things are in the pack — confirm, and it moves straight on to the next item without you touching the screen again. | `stock/products/{id}/add` |
| **Consume article** | Hold up what you are about to eat. If the same kind is in stock under more than one brand, it asks which; if the row holds more than one unit, it asks how many. | `stock/products/{id}/consume` |
| **List expired foods** | Everything already past its date. Clear a line by scanning the item you are about to bin (or with the row's own button when the label is unreadable). Booked out as **spoiled**, not eaten — Grocy keeps those apart, and that is the only way you will ever find out what the household keeps buying and throwing away. | `stock/products/{id}/consume` with `spoiled` |
| **Edit inventory** | The keyboard-and-buttons fallback: every article, `−`/`+`, and a freeform amount. This is the screen that does *not* use the camera, on purpose — it is where you go when the camera got something wrong. | `stock/products/{id}/inventory` |

Corrections go through Grocy's *inventory-correction* endpoint rather than
consume/add, so the stock journal says "somebody fixed the number" instead of
quietly filling the household's consumption history with corrections dressed as
meals.

### Two invariants everything else rests on

**Stock is counted in individual units, never in packages.** A twelve-pack of eggs is
booked in as twelve. "How many eggs do we have" is the question people actually ask,
and it makes booking out three of them arithmetic instead of a fractions-of-a-pack
problem. The vision model's `units_per_package` is a multiplier applied once on the
confirm screen — where it is editable, and where the screen states the result ("Books
in 12 eggs") before anything is written. Nothing downstream stores it. Set
`GROCY_DEFAULT_QU_ID` to a *Piece*-like unit accordingly.

**The fold key is the brand-free product kind.** Twelve eggs of brand X and ten of
brand Y are twenty-two eggs. The vision model is asked for a brand-free `kind`
alongside the brand, `/confirm` files the Grocy product under a **product group**
named after that kind, and the folding is a read of that group — so the grouping is
visible and editable in Grocy's own UI instead of living in a second classification
scheme here. The per-brand rows are never summed away: they are what the edit screen
expands to, because "take ten off brand Y specifically" has to stay possible.

The folding itself is deliberately dumb (casefold, strip a trailing `(Brand)`,
collapse whitespace). The clever version is a synonym problem — *eggs* vs *egg* vs
*free-range eggs* — that nobody wants adjudicated by a kitchen display at 19:00. When
the model answers inconsistently you get two lines instead of one, which is visible
and fixable; a wrongly merged line is neither.

## Which fridge is it in

The household has more than one cold appliance, so "in the fridge" is not an answer.
Two mechanisms address that, and **only the first is authoritative**:

**1. Grocy locations — the record.** The confirm screen's placement choice resolves to
a real Grocy location (`PANTRY_LOCATION_*`, created by name) and rides along on the
stock add. `/inventory` reports it. `POST /transfer` moves an amount between
appliances, and the edit screen has a "Move to…" control per row. That transfer is the
part that makes locations worth recording at all: without it, a location decays into
"where it was when it was bought", which is worse than no answer because it is
confidently wrong.

**2. Doorway hints — an observation, never a fact.** A Zigbee contact sensor on each
appliance door drives a Home Assistant automation that POSTs to `/doorway-event`:

```yaml
# Home Assistant automation — one per appliance door.
trigger:
  - platform: state
    entity_id: binary_sensor.fridge_kitchen_door
    to: "on"
action:
  - service: rest_command.pantry_doorway_event
    data: { appliance: "fridge-kitchen", state: "opened" }
```

`pantry-vision` answers **202 immediately** and does the work on its own thread — the
burst is up to three snapshots and a vision call each, and an appliance door is not
something to keep a home automation waiting on. It pulls frames from that appliance's
Frigate camera, stops at the first one that identifies something, and writes what it
saw to its own SQLite file with a timestamp and a confidence.

**Nothing in that path writes stock.** A camera at a door cannot tell in from out,
misses when two things are carried at once, and sees nothing behind an arm. So it
produces "camera last saw something like this at Freezer (loggia), 20 min ago — a
sighting, not a fact", which is a sentence a person can evaluate, shown next to the
location Grocy actually records. Acting on it is a tap on Move.

Configure it with `PANTRY_DOOR_APPLIANCES` (`id:Grocy location name:frigate_camera`,
camera optional) and `FRIGATE_URL` — the same Frigate `chores` already pulls snapshots
from. **An appliance with a door sensor and no camera is a legitimate configuration**
and is the recommended starting point: it still records that the door opened, which is
the half of this feature that pays for itself.

### What the camera loop actually does

`frontend/app.js` samples a 32×24 greyscale thumbnail of the video every 700 ms and
only spends an `/identify` call when the picture has **settled** (it stopped moving)
*and* **changed** since the last thing it identified. Both gates exist because vision
latency is this phase's known open risk: the frames worth spending it on are the ones
where somebody is holding something still, and the same tin must never be identified
twice because nobody moved. "Identify now" overrides both, for the shiny jar under a
downlight that never settles.

A model that is down or answering nonsense **stops the loop** rather than
re-photographing the counter at it; the unload screen drops you into the manual form
instead.

## A real network listener, unlike admin-canvas

`admin-canvas` deliberately has **no published port** — only Home Assistant, on the
same compose network, ever calls it. `pantry-vision` is different on purpose: the
kitchen display is a separate physical device on the LAN and has to reach this
service directly (there is no HA-mediation step between "hold item up to camera" and
"get an identification back" — that has to be fast and synchronous). So
`PANTRY_VISION_PORT` **is** published, and every request — including the two GETs —
requires the bearer token, as the actual boundary instead of network placement.

The same token has to be baked into the kitchen display's own build config
(`tools/build-kitchen-display-iso.sh`), not just Home
Assistant's — see that host's README.

## `/identify` never writes anything by itself

This is the one guardrail that matters most in this whole phase. The vision model's
guess — name, category, how many days until it likely goes bad — is a **proposal**,
shown on screen for the person to review and edit before anything is confirmed. Only
`/confirm`, a separate call the frontend makes after the person taps "Confirm & add,"
ever writes to Grocy. This is the same "propose, never auto-commit" rule this project
already applies to identity-merge confirmation (see the *Identity store* row in
`docs/project-plan.md` §2) — a wrong camera guess costs one tap to fix, not a wrong
fact silently written into the household's inventory.

The book-out screens follow the same rule in the other direction, and it costs a tap
there too: recognition never consumes anything by itself, an ambiguous brand is asked
about rather than picked, and "throw away" states the amount before it goes. A camera
that quietly books out the wrong yoghurt produces an inventory nobody trusts, and an
inventory nobody trusts is worth exactly as much as no inventory.

One place where that guardrail is deliberately looser: the vision model is asked to
read a **printed best-before date** off the packaging when one is legible, and that
date lands pre-filled in the confirm screen's date field. The screen always says which
it is showing — *"Date read off the packaging — check it"* versus *"Estimated from
the category"* — and the server throws out anything more than a year in the past or
ten years out, because a misread label (small print, dot-matrix ink, curved packaging)
is the single most likely failure of this feature and that is what it looks like when
it happens.

## Configure

```sh
cp pantry-vision/pantry-vision.env.example /opt/smart-home/pantry-vision/pantry-vision.env
openssl rand -hex 32   # put the result in PANTRY_VISION_TOKEN
chmod 600 /opt/smart-home/pantry-vision/pantry-vision.env
$EDITOR /opt/smart-home/pantry-vision/pantry-vision.env
```

You also need, inside Grocy's own UI (it's already running as the always-on `grocy`
container regardless of this phase): **Settings → Manage API keys** for
`GROCY_API_KEY`, and to confirm **Settings → Locations / Quantity units** actually
match `GROCY_DEFAULT_LOCATION_ID`/`GROCY_DEFAULT_QU_ID` (fresh-install defaults, not
guaranteed to match a Grocy that's already been customised).

## Pick a vision model

`OLLAMA_VISION_MODEL` defaults to `llava`, but **nothing here has confirmed that name
against a real pull** — pick a vision-capable model (`llava`, `qwen2.5vl`, or whatever
your LLM host's GPU/CPU tier can run at acceptable latency for someone standing at
the counter holding a can of beans) and:

```sh
ollama pull llava   # on the LLM host, or whichever model you picked
```

Plain text models (`qwen2.5:14b-instruct`, used elsewhere in this project for
digest/Assist) **cannot see images at all** — pointing `OLLAMA_VISION_MODEL` at one
of those will not error clearly, it will just produce a useless/hallucinated
response for every photo. Latency is unmeasured; a vision pass on a CPU-only LLM
host could easily be too slow for a "hold item up to camera" interaction to feel
responsive — this needs to be measured on real hardware, not assumed.

## Grocy API assumptions — unverified against a real instance

`server.py`'s Grocy calls (`_find_or_create_product`, `_add_to_stock`,
`_consume_stock`, `_set_stock_amount`, `_stock_items` and the `/recipes` proxy) are
written against Grocy's *documented* API shape, not checked against a running
instance. In particular:

- Whether `GET /api/stock` rows carry a nested `product` object with a `name` field
  by default, or need an explicit embed/expand parameter. `_stock_items` sidesteps
  this by fetching `/api/objects/products` separately and joining on `product_id` —
  it needs the group and location anyway, not just the name — and degrades to
  `Product #<id>` if both are missing, rather than dropping the row.
- Whether `POST /api/objects/products` with just
  `name`/`location_id`/`qu_id_purchase`/`qu_id_stock` is actually enough to create a
  minimal product on your Grocy version, or whether it requires more fields.
- Whether `POST /api/objects/product_groups` and `POST /api/objects/locations` accept
  a bare `{"name": ...}`. Both are best-effort: a failure means the product lands
  without a group (so it folds on its name instead) or in `GROCY_DEFAULT_LOCATION_ID`
  (so it is in the wrong place but still in stock). Neither costs you the book-in.
- Whether `POST /api/stock/products/{id}/consume` takes `spoiled` as a boolean, and
  what it returns when you try to consume more than is in stock — that refusal is
  passed through to the screen verbatim, on the assumption that it is a real answer
  ("we have fewer than you think") rather than a transport failure.
- Whether `POST /api/stock/products/{id}/inventory` takes `new_amount` as the absolute
  new figure and whether it *requires* `best_before_date` when the figure goes up. The
  edit screen currently sends the amount alone.
- Whether `POST /api/stock/products/{id}/transfer` takes `location_id_from` /
  `location_id_to`, and — the subtle one — **which location Grocy considers a stock
  entry to be in.** `/transfer` uses the *product's* location as the source, which is
  what `/confirm` set. Grocy can also hold one product's stock across several
  locations at once, per entry; if your instance does that, the source this sends will
  sometimes be wrong and the transfer will fail rather than move the wrong thing.
  Reading `/api/stock/products/{id}/entries` is the fix if it comes up.
- Whether Frigate's `/api/<camera>/latest.jpg` returns a usable still for a camera that
  is idle — `doorway.py` and `chores/check.py` both assume it does, neither has called
  it.
- Whether the Recipes feature (`GET /api/objects/recipes`,
  `GET /api/recipes/{id}/fulfillment`) needs to be explicitly enabled/populated
  before it returns anything meaningful — `_handle_recipes` degrades to
  `"fulfilled": null` per-recipe on any failure rather than breaking the whole list.

Grocy exposes a live OpenAPI spec at `http://<grocy-host>:9283/api/openapi/specification`
once it's running — read that against a real instance before trusting any of the
above, and adjust `server.py` if the shapes differ.

## Deploy

Wired into `tools/setup-container-host.sh` behind
`ENABLE_PANTRY_VISION` (off by default) — see that script's `# CONFIGURATION` block
and its own README. It builds two containers: `pantry-vision` (this API) and
`pantry-web` (nginx, serves `frontend/` read-only).

## Manual verification still outstanding

1. All of the Grocy API assumptions above.
2. Real-world vision-model accuracy and latency for grocery items — untested with
   any actual model or camera. The unload loop is where latency bites hardest: it is
   built to feel like a queue of items rather than a queue of round trips, and if an
   `/identify` takes 20 seconds it will not feel like either.
3. **Whether the model returns a stable `kind` for the same product across scans.**
   This is the assumption the whole brand-folding rests on, and the one most likely
   to disappoint quietly: scan the same carton of eggs five times and see whether it
   says "eggs" five times. If it drifts, the fix is a fixed `kind` vocabulary in the
   prompt, not more clever folding on this side.
4. **Whether printed best-before dates are read correctly or confidently invented.**
   The prompt says not to guess and the server bounds the range, but neither can catch
   a plausible wrong date. Check a handful against the actual packets before trusting
   the pre-filled field.
5. **That a twelve-pack really does arrive as twelve.** `units_per_package` is the
   one model answer that gets multiplied rather than displayed, and the failure is
   silent in both directions — twelve eggs booked in as one, or one jar booked in as
   twelve.
6. **Whether a doorway camera can identify anything at all.** This is the assumption
   the whole `/doorway-event` path rests on, and it is not the same task as the
   kitchen display's: an item in a moving hand, at ~1.5 m, in whatever light the room
   has, versus one held still against a plain background 30 cm from a webcam.
   `docs/fridge-item-location.md` says to test it with the kitchen's existing camera
   before buying a second one, and that advice applies to this code too — if the
   answer is no, the door sensors are still worth having and this half simply stays
   switched off (`PANTRY_DOOR_APPLIANCES` entries without a camera).
7. **Whether the hints are read as hints.** The wording on screen ("a sighting, not a
   fact") is doing real work; if in practice people treat a sighting as the answer and
   stop checking, that is a design failure this code cannot detect and the feature
   should be turned off rather than tuned.
3. Whether Ollama's `/api/generate` `images` field is still the right call shape for
   whichever vision model you pick — some multimodal models are only exposed through
   Ollama's newer `/api/chat` with a `images` field per-message instead; this was
   written against `/api/generate`'s documented multimodal support and not run
   against a real model.
4. CORS: `_respond`'s blanket `Access-Control-Allow-Origin: *` is fine for a
   LAN-only, bearer-token-gated service with no cookies, but hasn't been checked
   against a real browser's preflight behavior for the raw-image-bytes `POST
   /identify` call in particular (some browsers preflight non-simple `Content-Type`s
   like `image/jpeg` — `do_OPTIONS` is written to handle that but is untested).