SmartestHome/pantry-vision

pantry-vision

The kitchen display's backend, from Phase 17 of the project plan.

The workflow this exists for: come home, put down the shopping bag, hold one item up to the kitchen display's camera, the system proposes what it is and roughly how long it keeps, you confirm (editing anything it got wrong), put it away. The same display then shows the resulting inventory ordered by what expires soonest, and Grocy's recipes, on request.

• pantry-vision (this directory) — a small always-on Python HTTP service. POST /identify (a photo → a proposal via an Ollama vision model), POST /confirm (a human-reviewed proposal → written into Grocy stock), GET /inventory and GET /recipes (proxy Grocy, reshaped for the frontend). All four endpoints are bearer-token gated.
• frontend/ — the static single-page app the kitchen display's kiosk browser loads: Scan / Inventory / Recipes, vanilla JS, no build step, no framework — same "vendored, dependency-free" choice as the digest/admin canvas SDKs. Served read-only by a pantry-web nginx container (setup-container-host.sh), the same role digest-web/admin-web already play for their own hosts.
• hosts/kitchen-display/ — the touch kiosk image that runs the frontend. See that directory's own README for the device side of this.

A real network listener, unlike admin-canvas

admin-canvas deliberately has no published port — only Home Assistant, on the same compose network, ever calls it. pantry-vision is different on purpose: the kitchen display is a separate physical device on the LAN and has to reach this service directly (there is no HA-mediation step between "hold item up to camera" and "get an identification back" — that has to be fast and synchronous). So PANTRY_VISION_PORT is published, and every request — including the two GETs — requires the bearer token, as the actual boundary instead of network placement.

The same token has to be baked into the kitchen display's own build config (hosts/kitchen-display/scripts/build-kitchen-display-iso.sh), not just Home Assistant's — see that host's README.

/identify never writes anything by itself

This is the one guardrail that matters most in this whole phase. The vision model's guess — name, category, how many days until it likely goes bad — is a proposal, shown on screen for the person to review and edit before anything is confirmed. Only /confirm, a separate call the frontend makes after the person taps "Confirm & add," ever writes to Grocy. This is the same "propose, never auto-commit" rule this project already applies to identity-merge confirmation (see the Identity store row in docs/project-plan.md §2) — a wrong camera guess costs one tap to fix, not a wrong fact silently written into the household's inventory.

Configure

cp pantry-vision/pantry-vision.env.example /opt/smart-home/pantry-vision/pantry-vision.env
openssl rand -hex 32   # put the result in PANTRY_VISION_TOKEN
chmod 600 /opt/smart-home/pantry-vision/pantry-vision.env
$EDITOR /opt/smart-home/pantry-vision/pantry-vision.env

You also need, inside Grocy's own UI (it's already running as the always-on grocy container regardless of this phase): Settings → Manage API keys for GROCY_API_KEY, and to confirm Settings → Locations / Quantity units actually match GROCY_DEFAULT_LOCATION_ID/GROCY_DEFAULT_QU_ID (fresh-install defaults, not guaranteed to match a Grocy that's already been customised).

Pick a vision model

OLLAMA_VISION_MODEL defaults to llava, but nothing here has confirmed that name against a real pull — pick a vision-capable model (llava, qwen2.5vl, or whatever your LLM host's GPU/CPU tier can run at acceptable latency for someone standing at the counter holding a can of beans) and:

ollama pull llava   # on the LLM host, or whichever model you picked

Plain text models (qwen2.5:14b-instruct, used elsewhere in this project for digest/Assist) cannot see images at all — pointing OLLAMA_VISION_MODEL at one of those will not error clearly, it will just produce a useless/hallucinated response for every photo. Latency is unmeasured; a vision pass on a CPU-only LLM host could easily be too slow for a "hold item up to camera" interaction to feel responsive — this needs to be measured on real hardware, not assumed.

Grocy API assumptions — unverified against a real instance

server.py's Grocy calls (_find_or_create_product, _add_to_stock, the /inventory and /recipes proxies) are written against Grocy's documented API shape, not checked against a running instance. In particular:

• Whether GET /api/stock rows carry a nested product object with a name field by default, or need an explicit embed/expand parameter — _handle_inventory degrades to Product #<id> if not, rather than dropping the row, but that's a fallback, not a fix.
• Whether POST /api/objects/products with just name/location_id/qu_id_purchase/qu_id_stock is actually enough to create a minimal product on your Grocy version, or whether it requires more fields.
• Whether the Recipes feature (GET /api/objects/recipes, GET /api/recipes/{id}/fulfillment) needs to be explicitly enabled/populated before it returns anything meaningful — _handle_recipes degrades to "fulfilled": null per-recipe on any failure rather than breaking the whole list.

Grocy exposes a live OpenAPI spec at http://<grocy-host>:9283/api/openapi/specification once it's running — read that against a real instance before trusting any of the above, and adjust server.py if the shapes differ.

Deploy

Wired into hosts/container-host/scripts/setup-container-host.sh behind ENABLE_PANTRY_VISION (off by default) — see that script's # CONFIGURATION block and its own README. It builds two containers: pantry-vision (this API) and pantry-web (nginx, serves frontend/ read-only).

Manual verification still outstanding

1. All of the Grocy API assumptions above.
2. Real-world vision-model accuracy and latency for grocery items — untested with any actual model or camera.
3. Whether Ollama's /api/generate images field is still the right call shape for whichever vision model you pick — some multimodal models are only exposed through Ollama's newer /api/chat with a images field per-message instead; this was written against /api/generate's documented multimodal support and not run against a real model.
4. CORS: _respond's blanket Access-Control-Allow-Origin: * is fine for a LAN-only, bearer-token-gated service with no cookies, but hasn't been checked against a real browser's preflight behavior for the raw-image-bytes POST /identify call in particular (some browsers preflight non-simple Content-Types like image/jpeg — do_OPTIONS is written to handle that but is untested).