# pantry-vision The kitchen display's backend, from [Phase 17 of the project plan](../docs/project-plan.md). The workflow this exists for: **come home, put down the shopping bag, hold one item up to the kitchen display's camera, the system proposes what it is and roughly how long it keeps, you confirm (editing anything it got wrong), put it away.** The same display then shows the resulting inventory ordered by what expires soonest, and Grocy's recipes, on request. - **`pantry-vision`** (this directory) — a small always-on Python HTTP service. `POST /identify` (a photo → a proposal via an Ollama vision model), `POST /confirm` (a human-reviewed proposal → written into Grocy stock), `GET /inventory` and `GET /recipes` (proxy Grocy, reshaped for the frontend). All four endpoints are bearer-token gated. - **`frontend/`** — the static single-page app the kitchen display's kiosk browser loads: Scan / Inventory / Recipes, vanilla JS, no build step, no framework — same "vendored, dependency-free" choice as the digest/admin canvas SDKs. Served read-only by a `pantry-web` nginx container (`setup-container-host.sh`), the same role `digest-web`/`admin-web` already play for their own hosts. - **`hosts/kitchen-display/`** — the touch kiosk image that runs the frontend. See that directory's own README for the device side of this. ## A real network listener, unlike admin-canvas `admin-canvas` deliberately has **no published port** — only Home Assistant, on the same compose network, ever calls it. `pantry-vision` is different on purpose: the kitchen display is a separate physical device on the LAN and has to reach this service directly (there is no HA-mediation step between "hold item up to camera" and "get an identification back" — that has to be fast and synchronous). So `PANTRY_VISION_PORT` **is** published, and every request — including the two GETs — requires the bearer token, as the actual boundary instead of network placement. The same token has to be baked into the kitchen display's own build config (`hosts/kitchen-display/scripts/build-kitchen-display-iso.sh`), not just Home Assistant's — see that host's README. ## `/identify` never writes anything by itself This is the one guardrail that matters most in this whole phase. The vision model's guess — name, category, how many days until it likely goes bad — is a **proposal**, shown on screen for the person to review and edit before anything is confirmed. Only `/confirm`, a separate call the frontend makes after the person taps "Confirm & add," ever writes to Grocy. This is the same "propose, never auto-commit" rule this project already applies to identity-merge confirmation (see the *Identity store* row in `docs/project-plan.md` §2) — a wrong camera guess costs one tap to fix, not a wrong fact silently written into the household's inventory. ## Configure ```sh cp pantry-vision/pantry-vision.env.example /opt/smart-home/pantry-vision/pantry-vision.env openssl rand -hex 32 # put the result in PANTRY_VISION_TOKEN chmod 600 /opt/smart-home/pantry-vision/pantry-vision.env $EDITOR /opt/smart-home/pantry-vision/pantry-vision.env ``` You also need, inside Grocy's own UI (it's already running as the always-on `grocy` container regardless of this phase): **Settings → Manage API keys** for `GROCY_API_KEY`, and to confirm **Settings → Locations / Quantity units** actually match `GROCY_DEFAULT_LOCATION_ID`/`GROCY_DEFAULT_QU_ID` (fresh-install defaults, not guaranteed to match a Grocy that's already been customised). ## Pick a vision model `OLLAMA_VISION_MODEL` defaults to `llava`, but **nothing here has confirmed that name against a real pull** — pick a vision-capable model (`llava`, `qwen2.5vl`, or whatever your LLM host's GPU/CPU tier can run at acceptable latency for someone standing at the counter holding a can of beans) and: ```sh ollama pull llava # on the LLM host, or whichever model you picked ``` Plain text models (`qwen2.5:14b-instruct`, used elsewhere in this project for digest/Assist) **cannot see images at all** — pointing `OLLAMA_VISION_MODEL` at one of those will not error clearly, it will just produce a useless/hallucinated response for every photo. Latency is unmeasured; a vision pass on a CPU-only LLM host could easily be too slow for a "hold item up to camera" interaction to feel responsive — this needs to be measured on real hardware, not assumed. ## Grocy API assumptions — unverified against a real instance `server.py`'s Grocy calls (`_find_or_create_product`, `_add_to_stock`, the `/inventory` and `/recipes` proxies) are written against Grocy's *documented* API shape, not checked against a running instance. In particular: - Whether `GET /api/stock` rows carry a nested `product` object with a `name` field by default, or need an explicit embed/expand parameter — `_handle_inventory` degrades to `Product #` if not, rather than dropping the row, but that's a fallback, not a fix. - Whether `POST /api/objects/products` with just `name`/`location_id`/`qu_id_purchase`/`qu_id_stock` is actually enough to create a minimal product on your Grocy version, or whether it requires more fields. - Whether the Recipes feature (`GET /api/objects/recipes`, `GET /api/recipes/{id}/fulfillment`) needs to be explicitly enabled/populated before it returns anything meaningful — `_handle_recipes` degrades to `"fulfilled": null` per-recipe on any failure rather than breaking the whole list. Grocy exposes a live OpenAPI spec at `http://:9283/api/openapi/specification` once it's running — read that against a real instance before trusting any of the above, and adjust `server.py` if the shapes differ. ## Deploy Wired into `hosts/container-host/scripts/setup-container-host.sh` behind `ENABLE_PANTRY_VISION` (off by default) — see that script's `# CONFIGURATION` block and its own README. It builds two containers: `pantry-vision` (this API) and `pantry-web` (nginx, serves `frontend/` read-only). ## Manual verification still outstanding 1. All of the Grocy API assumptions above. 2. Real-world vision-model accuracy and latency for grocery items — untested with any actual model or camera. 3. Whether Ollama's `/api/generate` `images` field is still the right call shape for whichever vision model you pick — some multimodal models are only exposed through Ollama's newer `/api/chat` with a `images` field per-message instead; this was written against `/api/generate`'s documented multimodal support and not run against a real model. 4. CORS: `_respond`'s blanket `Access-Control-Allow-Origin: *` is fine for a LAN-only, bearer-token-gated service with no cookies, but hasn't been checked against a real browser's preflight behavior for the raw-image-bytes `POST /identify` call in particular (some browsers preflight non-simple `Content-Type`s like `image/jpeg` — `do_OPTIONS` is written to handle that but is untested).