SmartestHome/pantry-vision/server.py

1204 lines
55 KiB
Python
Raw Blame History

This file contains ambiguous Unicode characters!

This file contains ambiguous Unicode characters that may be confused with others in your current locale. If your use case is intentional and legitimate, you can safely ignore this warning. Use the Escape button to highlight these characters.

"""pantry-vision — camera-driven grocery cataloguing for the kitchen display, from
docs/project-plan.md Phase 17.
The workflow this exists for: come home, put down the shopping bag, hold one item up
to the kitchen display's camera, the system proposes what it is and how long it keeps,
you confirm (editing anything it got wrong) before it's written anywhere, then put the
item away. The display also shows the resulting inventory sorted by what expires
soonest, and Grocy's recipes.
The camera is the input device for all four of the household's stock movements, and
each one is a screen in frontend/app.js:
- **Unload groceries** (book in) — the camera runs in a loop, proposes each item
with its placement and pack size, one confirm per item.
- **Consume article** (book out) — hold up what you are eating, say how many
individual units, it comes off stock.
- **List expired foods** — what is past its date, cleared by scanning the
item you are about to throw away (booked out as spoiled, not consumed).
- **Edit inventory** — the keyboard-and-buttons fallback for when the
camera got something wrong: every article, +/-, and a freeform amount.
SECURITY BOUNDARY, same shape as admin-canvas/server.py's, with one deliberate
difference: this service DOES get a published port (see the README and
setup-container-host.sh's PANTRY_VISION_PORT), because unlike admin-canvas — which is
only ever called by Home Assistant on the compose network — this one is called
directly by the kitchen-display kiosk, a separate physical device on the LAN. Every
request still requires the bearer token below; an unset token fails closed (rejects
everything), never "auth optional".
Endpoints, all token-gated:
- POST /identify raw image bytes -> Ollama vision model -> a *proposal*
- POST /confirm JSON, human-reviewed/edited -> adds to Grocy stock
- POST /consume JSON -> books an amount back out of Grocy stock
- POST /adjust JSON -> corrects an amount (the edit-inventory screen)
- POST /transfer JSON -> moves an amount from one appliance to another
- POST /doorway-event JSON, from an HA automation on a fridge door sensor ->
pulls a camera burst and records a *hint* (never stock)
- GET /inventory proxies Grocy stock, soonest-expiring first, plus the
brand-folded `groups` view
- GET /expired the subset of /inventory that is already past its date
- GET /doorway-events the recent hints, newest first
- GET /recipes proxies Grocy recipes (+ fulfillment, best-effort)
- GET /shopping-list Grocy's own "below minimum stock", for the door panel
/identify NEVER writes anything by itself — same "propose, never auto-commit" rule
this project already applies to identity-merge confirmation (see the Identity store
row in docs/project-plan.md §2): the vision model's guess is shown to the person
standing at the display, who can edit the name/category/date/location before /confirm
is ever called. A wrong guess costs a tap to fix, not a wrong fact silently written
into the household's inventory. That rule is why the book-out and expiry flows also
identify-then-confirm rather than consuming straight off a recognition: the camera
choosing *what* to remove from stock by itself is the same silent-wrong-write, one
direction over.
TWO INVARIANTS THE REST OF THIS FILE DEPENDS ON
-----------------------------------------------
1. **Stock is counted in individual units, never in packages.** A twelve-pack of eggs
is booked in as twelve, because "how many eggs do we have" is the question the
household actually asks, and because it makes booking out three of them
arithmetic instead of a fractions-of-a-pack problem. `units_per_package` from the
vision model is a multiplier applied once at /confirm; nothing downstream stores
or needs it. See _confirm_amount().
2. **The fold key is the brand-free product kind.** Twelve eggs of brand X and ten of
brand Y are twenty-two eggs, and Grocy already has the right place to record which
kind a product is — its product group. So /confirm files each product under a
product group named after its kind, and the folding in _fold_stock() is a read of
that, not a second classification scheme living here. The per-brand rows stay
underneath, because "remove exactly these ten" has to remain possible on the
edit-inventory screen. See _fold_key().
WHERE A THING IS, AND WHY THAT IS A HINT
-----------------------------------------
The household has more than one cold appliance, so "in the fridge" is not an answer.
Two mechanisms address that, and only the first of them is authoritative:
- **Grocy locations.** The confirm screen's placement choice resolves to a real Grocy
location and rides along on the stock add; /transfer moves an amount between them.
Both are human actions, both are what /inventory reports. This is the record.
- **Doorway hints** (doorway.py). A contact sensor on an appliance door triggers a
camera burst, and whatever the vision model recognises is written to a separate
SQLite file as an observation with a timestamp and a confidence — never to Grocy,
never to stock. A camera at a door cannot tell in from out, misses when two things
are carried at once, and sees nothing behind an arm. So it produces "last seen going
past the loggia freezer, Tue 18:42", which a person can evaluate, and never a claim
about where anything is. docs/fridge-item-location.md is the long form of this.
"""
from __future__ import annotations
import base64
import json
import logging
import os
import re
import sys
import threading
import urllib.error
import urllib.request
from datetime import date, datetime, timedelta
from http import HTTPStatus
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from urllib.parse import parse_qs, urlsplit
import doorway
LOG = logging.getLogger("pantry-vision")
TOKEN = os.environ.get("PANTRY_VISION_TOKEN", "")
OLLAMA_HOST = os.environ.get("OLLAMA_HOST", "http://llm-host:11434").rstrip("/")
# TBD/unverified — see README.md's "Pick a vision model" section. Not every Ollama
# model can see images; this must be one that can (llava, qwen2.5vl, etc.) and must
# actually be pulled on the LLM host (`ollama pull <name>`) before /identify works.
OLLAMA_VISION_MODEL = os.environ.get("OLLAMA_VISION_MODEL", "llava")
OLLAMA_TIMEOUT = float(os.environ.get("OLLAMA_TIMEOUT", "120"))
GROCY_URL = os.environ.get("GROCY_URL", "http://grocy:80").rstrip("/")
GROCY_API_KEY = os.environ.get("GROCY_API_KEY", "")
# Fresh-install defaults ("Default" location, "Piece" quantity unit) — VERIFY against
# your own Grocy instance (Settings -> Locations / Quantity units) and override if
# they differ. Same "don't build against a guess" flag as every other cross-service ID
# in this project (ADMIN_WEB_URL placeholders, weather_entity_id, etc.).
GROCY_DEFAULT_LOCATION_ID = os.environ.get("GROCY_DEFAULT_LOCATION_ID", "1")
GROCY_DEFAULT_QU_ID = os.environ.get("GROCY_DEFAULT_QU_ID", "2")
# The recommended-placement answer is a Grocy location, looked up (or created) by
# name, so "fridge" on the confirm screen and "Fridge" in Grocy's own UI are the same
# shelf. Override the names if your Grocy already calls them something else — a
# mismatch here silently creates a second, duplicate location rather than failing.
PLACEMENT_LOCATION_NAMES = {
"fridge": os.environ.get("PANTRY_LOCATION_FRIDGE", "Fridge"),
"freezer": os.environ.get("PANTRY_LOCATION_FREEZER", "Freezer"),
"cupboard": os.environ.get("PANTRY_LOCATION_CUPBOARD", "Cupboard"),
"counter": os.environ.get("PANTRY_LOCATION_COUNTER", "Counter"),
}
# Where the vision model's category lands when it does not answer the placement
# question itself, or answers it with something that is not one of the four above.
# Deliberately conservative: cold is a safe wrong answer for food, warm is not.
CATEGORY_PLACEMENT = {
"produce": "fridge",
"dairy": "fridge",
"meat": "fridge",
"frozen": "freezer",
"pantry": "cupboard",
"bakery": "cupboard",
"beverage": "cupboard",
"other": "cupboard",
}
MAX_IMAGE_BYTES = int(os.environ.get("PANTRY_VISION_MAX_IMAGE_MB", "15")) * 1024 * 1024
MAX_JSON_BYTES = 64 * 1024
# A pack size the model is confident about is useful; one it invented is a hundred
# phantom eggs in the inventory. Clamped, and always shown on the confirm screen as an
# editable number before it multiplies anything.
MAX_UNITS_PER_PACKAGE = int(os.environ.get("PANTRY_MAX_UNITS_PER_PACKAGE", "240"))
_IDENTIFY_PROMPT = """You are looking at a single grocery item held up to a kitchen \
camera. Identify it and answer ONLY with a JSON object, no other text, in exactly \
this shape:
{"present": true,
"kind": "<what the item IS, with no brand name in it: 'eggs', 'whole milk', 'penne', 'frozen peas'>",
"brand": "<the brand printed on the packaging, or an empty string if none is visible>",
"description": "<one short line a person would want to read back: variety, size, flavour>",
"category": "<produce|dairy|meat|frozen|pantry|bakery|beverage|other>",
"recommended_placement": "<fridge|freezer|cupboard|counter — where this should be put away>",
"estimated_shelf_life_days": <integer, typical days from today until it goes bad in that place>,
"printed_best_before": "<YYYY-MM-DD, ONLY if a best-before or use-by date is actually legible on the packaging, otherwise null>",
"units_per_package": <integer: how many individual items are inside. 12 for a box of 12 eggs, 6 for a six-pack, 1 for a single loose item or one bottle>,
"unit_name": "<what one of those individual items is called: 'egg', 'bottle', 'slice', 'piece'>",
"confidence": "<high|medium|low>"}
Two rules that matter more than the rest:
- If no grocery item is being held up to the camera — an empty counter, a hand, a \
person, a blurred or dark frame — answer exactly {"present": false} and nothing else.
- Never invent "printed_best_before". Fill it in only if you can genuinely read the \
date off the packaging in this image; otherwise null. A guessed date is worse than \
no date, because nobody will re-check it."""
_JSON_OBJECT_RE = re.compile(r"\{.*\}", re.DOTALL)
_ISO_DATE_RE = re.compile(r"^\d{4}-\d{2}-\d{2}$")
_BRAND_SUFFIX_RE = re.compile(r"\s*\([^()]*\)\s*$")
def _http_json(method: str, url: str, payload: dict | None = None, headers: dict | None = None,
timeout: float = 30.0) -> dict:
data = json.dumps(payload).encode("utf-8") if payload is not None else None
req = urllib.request.Request(url, data=data, method=method)
req.add_header("Content-Type", "application/json")
for key, value in (headers or {}).items():
req.add_header(key, value)
with urllib.request.urlopen(req, timeout=timeout) as resp:
body = resp.read()
return json.loads(body) if body else {}
def _grocy_headers() -> dict:
return {"GROCY-API-KEY": GROCY_API_KEY}
def _grocy_get(path: str):
return _http_json("GET", f"{GROCY_URL}{path}", headers=_grocy_headers(), timeout=15)
def _grocy_post(path: str, payload: dict):
return _http_json("POST", f"{GROCY_URL}{path}", payload=payload, headers=_grocy_headers(), timeout=15)
def _degraded_proposal(note: str, **extra) -> dict:
"""The proposal returned when the model could not be reached or could not be
parsed. `degraded` is the field the frontend's unload loop watches: a degraded
answer stops the loop and drops the person into the manual form, instead of
re-photographing the same tin at a model that is down.
"""
proposal = {
"present": True,
"degraded": True,
"kind": "Unknown item",
"brand": "",
"description": "",
"category": "other",
"recommended_placement": "cupboard",
"estimated_shelf_life_days": 7,
"printed_best_before": None,
"units_per_package": 1,
"unit_name": "piece",
"confidence": "low",
"note": note,
}
proposal.update(extra)
return proposal
def _identify_via_ollama(image_bytes: bytes) -> dict:
"""Calls the vision model and returns a best-effort proposal — degrades to a
low-confidence placeholder rather than raising, mirroring digest-engine's
llm_client._fallback_document: a broken/blank result is worse than one flagged
as unreliable that a human has to review anyway (which they always do — see the
module docstring, /identify never writes anything by itself).
"""
payload = {
"model": OLLAMA_VISION_MODEL,
"prompt": _IDENTIFY_PROMPT,
"images": [base64.b64encode(image_bytes).decode("ascii")],
"stream": False,
"options": {"temperature": 0.2},
}
try:
result = _http_json("POST", f"{OLLAMA_HOST}/api/generate", payload=payload, timeout=OLLAMA_TIMEOUT)
text = result.get("response", "")
except Exception:
LOG.warning("pantry-vision: Ollama call failed", exc_info=True)
return _degraded_proposal(
f"Could not reach the vision model at {OLLAMA_HOST} — fill this in manually."
)
match = _JSON_OBJECT_RE.search(text)
if match:
try:
return _normalise_proposal(json.loads(match.group(0)))
except ValueError:
pass
LOG.warning("pantry-vision: could not parse a JSON proposal out of the model's response")
return _degraded_proposal(
"The vision model's response wasn't valid JSON — fill this in manually.",
raw_response=text[:500],
)
def _coerce_int(value, default: int, minimum: int, maximum: int) -> int:
try:
number = int(float(value))
except (TypeError, ValueError):
return default
return max(minimum, min(maximum, number))
def _as_int(value) -> int | None:
"""A JSON field as an int, or None if it isn't one. Missing and malformed are the
same answer on purpose: every caller here has to handle "not usable" anyway, and
telling a null apart from the string "banana" has never changed what they do."""
try:
return int(value)
except (TypeError, ValueError):
return None
def _as_float(value) -> float | None:
"""Same, for amounts — with NaN and the infinities rejected, because both survive
float() and neither is a quantity anybody can correct from a touchscreen once it
is in Grocy's stock table."""
try:
number = float(value)
except (TypeError, ValueError):
return None
if number != number or number in (float("inf"), float("-inf")):
return None
return number
def _printed_date(value) -> str | None:
"""The best-before the model claims to have read off the packaging, or None.
Sanity-bounded on purpose: a misread label is the most likely single failure of
this whole feature (small print, dot-matrix ink, curved packaging), and a date in
1998 or 2071 is how that failure shows up. Out-of-range means "fall back to the
shelf-life estimate", never "write it and hope somebody notices".
"""
if not isinstance(value, str) or not _ISO_DATE_RE.match(value.strip()):
return None
try:
parsed = datetime.strptime(value.strip(), "%Y-%m-%d").date()
except ValueError:
return None
days_out = (parsed - date.today()).days
if days_out < -365 or days_out > 3650:
return None
return parsed.isoformat()
def _display_name(kind: str, brand: str) -> str:
"""The Grocy product name. Brand in parentheses after the kind, so the fold key
is recoverable from the name alone if the product group is ever missing — see
_fold_key(), which strips exactly this suffix.
"""
kind = kind.strip() or "Unknown item"
brand = brand.strip()
return f"{kind} ({brand})" if brand else kind
def _fold_key(name) -> str:
"""The brand-free key two products have in common when they are the same thing.
Deliberately dumb — casefold, strip a trailing "(Brand)", collapse whitespace —
because the smart version of this is a synonym problem nobody wants adjudicated
by a kitchen display at 19:00 ("eggs" vs "egg" vs "free-range eggs"). The vision
model is asked for a stable brand-free `kind` precisely so this does not have to
be clever; when it answers inconsistently the two entries simply show up as two
lines, which is a visible, fixable outcome rather than a silently merged one.
"""
return " ".join(_BRAND_SUFFIX_RE.sub("", str(name or "")).split()).casefold()
def _normalise_proposal(raw: dict) -> dict:
"""The model's JSON, made safe to put in front of a person and to arithmetic on.
Every field is defaulted and bounded here rather than in the handler, so the
frontend can render a proposal without checking for missing keys, and so a model
that answers half the schema still produces a usable confirm screen.
"""
if isinstance(raw, dict) and raw.get("present") is False:
return {"present": False}
raw = raw if isinstance(raw, dict) else {}
# `name` is accepted as an alias for `kind` because a model that ignores half the
# schema still tends to produce a name, and one usable field beats a blank form.
kind = str(raw.get("kind") or raw.get("name") or "Unknown item").strip()
brand = str(raw.get("brand") or "").strip()
category = str(raw.get("category") or "other").strip().lower()
if category not in CATEGORY_PLACEMENT:
category = "other"
placement = str(raw.get("recommended_placement") or "").strip().lower()
if placement not in PLACEMENT_LOCATION_NAMES:
placement = CATEGORY_PLACEMENT[category]
shelf_life = _coerce_int(raw.get("estimated_shelf_life_days"), default=7, minimum=0, maximum=3650)
printed = _printed_date(raw.get("printed_best_before"))
return {
"present": True,
"degraded": False,
"kind": kind,
"brand": brand,
"name": _display_name(kind, brand),
"description": str(raw.get("description") or "").strip(),
"category": category,
"recommended_placement": placement,
"estimated_shelf_life_days": shelf_life,
# What the confirm screen puts in its date field: the label if it could be
# read, otherwise today plus the model's guess at how long it keeps. Which of
# the two it was is carried in `best_before_source`, because "read off the
# packet" and "estimated from the category" deserve different amounts of
# trust from the person about to tap Confirm.
"best_before_date": printed or _date_in_days(shelf_life),
"best_before_source": "label" if printed else "estimate",
"printed_best_before": printed,
"units_per_package": _coerce_int(
raw.get("units_per_package"), default=1, minimum=1, maximum=MAX_UNITS_PER_PACKAGE
),
"unit_name": str(raw.get("unit_name") or "piece").strip() or "piece",
"confidence": str(raw.get("confidence") or "low").strip().lower(),
}
def _find_or_create_by_name(path: str, name: str, extra: dict | None = None) -> int | None:
"""Find an object of `path` whose name matches, or create it. None on any failure.
One helper for products, locations and product groups because Grocy's object API
is the same shape for all three. Returning None rather than raising is the point
for the two optional ones: a Grocy that will not let us create a "Freezer"
location must not cost the household the ability to book in a bag of peas — the
item lands in the default location instead, which is wrong-but-visible.
"""
needle = name.strip().casefold()
if not needle:
return None
try:
existing = _grocy_get(path)
for row in existing if isinstance(existing, list) else []:
if str(row.get("name", "")).strip().casefold() == needle:
return int(row["id"])
created = _grocy_post(path, {"name": name.strip(), **(extra or {})})
return int(created["created_object_id"])
except (urllib.error.URLError, urllib.error.HTTPError, KeyError, ValueError, TypeError):
LOG.warning("pantry-vision: could not find or create %s named %r", path, name, exc_info=True)
return None
def _location_id_for(placement: str) -> int:
"""The Grocy location id for fridge/freezer/cupboard/counter, or the configured
default. Never raises: an unplaceable item is still an item worth booking in.
"""
name = PLACEMENT_LOCATION_NAMES.get(str(placement).strip().lower())
if name:
found = _find_or_create_by_name("/api/objects/locations", name)
if found is not None:
return found
return int(GROCY_DEFAULT_LOCATION_ID)
def _find_or_create_product(name: str, kind: str = "", location_id: int | None = None) -> int:
"""Best-effort product lookup/create against Grocy's object API.
`kind` becomes the product's Grocy **product group** — that is where the
brand-folding in _fold_stock() reads from, and putting it in Grocy rather than in
a table of our own means the grouping is visible and editable in Grocy's own UI
(invariant 2 in the module docstring). A product that already exists is left
exactly as it is, group included: the household's own corrections in Grocy outrank
a vision model's guess on the next scan of the same tin.
VERIFY against a real Grocy instance — this assumes GET /api/objects/products
returns a flat list of {id, name, ...} and that POST to the same path with just
name/location_id/qu_id_purchase/qu_id_stock is enough to create a minimal product.
Grocy's actual required-field set depends on its own settings and has not been
checked against a live instance; see README.md's verification list.
"""
products = _grocy_get("/api/objects/products")
needle = name.strip().casefold()
for product in products if isinstance(products, list) else []:
if str(product.get("name", "")).strip().casefold() == needle:
return int(product["id"])
payload = {
"name": name,
"location_id": int(location_id if location_id is not None else GROCY_DEFAULT_LOCATION_ID),
"qu_id_purchase": int(GROCY_DEFAULT_QU_ID),
"qu_id_stock": int(GROCY_DEFAULT_QU_ID),
}
group_id = _find_or_create_by_name("/api/objects/product_groups", kind) if kind.strip() else None
if group_id is not None:
payload["product_group_id"] = group_id
created = _grocy_post("/api/objects/products", payload)
return int(created["created_object_id"])
def _add_to_stock(product_id: int, amount: float, best_before_date: str, location_id: int | None = None) -> dict:
payload = {
"amount": amount,
"best_before_date": best_before_date,
"transaction_type": "purchase",
}
if location_id is not None:
payload["location_id"] = int(location_id)
return _grocy_post(f"/api/stock/products/{product_id}/add", payload)
def _consume_stock(product_id: int, amount: float, spoiled: bool) -> dict:
"""Book an amount back out. `spoiled=True` is what the expired-foods screen sends:
Grocy tracks thrown-away separately from eaten, and that distinction is the only
way the household will ever find out what it keeps buying and binning.
"""
return _grocy_post(
f"/api/stock/products/{product_id}/consume",
{"amount": amount, "spoiled": bool(spoiled), "transaction_type": "consume"},
)
def _set_stock_amount(product_id: int, new_amount: float, best_before_date: str | None = None) -> dict:
"""The edit-inventory screen's write: Grocy's own inventory-correction endpoint.
Both directions go through here on purpose. A correction is not a purchase and not
a meal — it is somebody saying the number was wrong — and Grocy's stock journal
should say so, otherwise the consumption history that Grocy's own reports are
built on quietly fills up with corrections dressed as eating.
"""
payload: dict = {"new_amount": new_amount}
if best_before_date:
payload["best_before_date"] = best_before_date
return _grocy_post(f"/api/stock/products/{product_id}/inventory", payload)
def _transfer_stock(product_id: int, amount: float, location_id_from: int, location_id_to: int) -> dict:
"""Move an amount between appliances — the action a doorway hint exists to prompt.
This is the one write that makes multi-fridge locations worth recording at all:
without it, a location is "where it was when it was bought", which decays into a
confidently wrong answer within a fortnight. See docs/fridge-item-location.md.
"""
return _grocy_post(
f"/api/stock/products/{product_id}/transfer",
{"amount": amount, "location_id_from": int(location_id_from), "location_id_to": int(location_id_to)},
)
def _date_in_days(days: int) -> str:
return (date.today() + timedelta(days=max(0, int(days)))).isoformat()
def _days_left(best_before: str | None) -> int | None:
if not best_before:
return None
try:
due = datetime.strptime(best_before, "%Y-%m-%d").date()
except ValueError:
return None
return (due - date.today()).days
def _stock_items() -> list[dict]:
"""Grocy's stock as this service's own row shape, product metadata attached.
The product catalogue is fetched separately and joined here rather than trusted to
arrive nested inside the stock rows: whether `GET /api/stock` embeds a `product`
object at all varies by Grocy version (see README.md), and every screen added in
this phase needs the product's id, group and location — not just its name — to be
able to consume, fold or correct it. A failed catalogue read degrades to whatever
the stock rows themselves carry, same "degrade, don't blank" rule as everywhere
else here.
"""
stock = _grocy_get("/api/stock")
products: dict[int, dict] = {}
groups: dict[int, str] = {}
locations: dict[int, str] = {}
try:
for row in _grocy_get("/api/objects/products") or []:
products[int(row["id"])] = row
for row in _grocy_get("/api/objects/product_groups") or []:
groups[int(row["id"])] = str(row.get("name", ""))
for row in _grocy_get("/api/objects/locations") or []:
locations[int(row["id"])] = str(row.get("name", ""))
except (urllib.error.URLError, urllib.error.HTTPError, KeyError, ValueError, TypeError):
LOG.warning("pantry-vision: product catalogue unreadable, folding on names alone", exc_info=True)
# Attached, not joined in Grocy: see doorway.py on why an observation and a stock
# row are kept in different files. A hint that has fallen out of the retention
# window simply isn't here, which reads on screen as "no recent sighting".
last_seen = doorway.last_seen_by_product()
items = []
for row in stock if isinstance(stock, list) else []:
product_id = _as_int(row.get("product_id"))
if product_id is None:
continue
product = products.get(product_id) or row.get("product") or {}
name = product.get("name") or f"Product #{product_id}"
best_before = row.get("best_before_date")
group_id = _as_int(product.get("product_group_id"))
group_name = groups.get(group_id, "") if group_id is not None else ""
# No product group (a product created before this phase, or by hand in Grocy)
# means the name minus its brand suffix is the fold key. That is the whole
# reason _display_name() puts the brand in parentheses.
kind = group_name or _BRAND_SUFFIX_RE.sub("", str(name)).strip()
brand_match = _BRAND_SUFFIX_RE.search(str(name))
# The product's own default location — where /confirm put it. Grocy can also
# hold stock of one product in several locations at once (per stock entry);
# this is the product-level answer, which is the one the household set and the
# one /transfer moves. Anything finer would need /stock/products/{id}/entries
# and a screen to show it on, and neither exists yet.
location_id = None
for source in (row, product):
location_id = _as_int(source.get("location_id"))
if location_id is not None:
break
items.append(
{
"product_id": product_id,
"name": name,
"kind": kind,
"brand": brand_match.group(0).strip(" ()") if brand_match else "",
"amount": row.get("amount"),
"best_before_date": best_before,
"days_left": _days_left(best_before),
"location_id": location_id,
"location": locations.get(location_id, "") if location_id is not None else "",
"last_seen": last_seen.get(product_id),
}
)
items.sort(key=lambda i: (i["days_left"] is None, i["days_left"]))
return items
def _amount_of(item: dict) -> float:
try:
return float(item.get("amount") or 0)
except (TypeError, ValueError):
return 0.0
def _fold_stock(items: list[dict]) -> list[dict]:
"""Twelve eggs of brand X plus ten of brand Y as one line saying twenty-two.
The per-brand rows ride along in `entries` rather than being summed away, because
the edit-inventory screen has to be able to take ten off brand Y specifically.
Ordering matches /inventory's: soonest-expiring group first, and inside a group,
soonest-expiring brand first — the thing to use up leads in both directions.
"""
folded: dict[str, dict] = {}
for item in items:
key = _fold_key(item.get("kind") or item.get("name"))
group = folded.setdefault(
key,
{"kind": key, "display_name": item.get("kind") or item.get("name"), "total_amount": 0.0, "entries": []},
)
group["total_amount"] += _amount_of(item)
group["entries"].append(item)
groups = []
for group in folded.values():
days = [e["days_left"] for e in group["entries"] if e["days_left"] is not None]
group["soonest_days_left"] = min(days) if days else None
group["brand_count"] = len({e["brand"] for e in group["entries"]})
group["entries"].sort(key=lambda e: (e["days_left"] is None, e["days_left"]))
groups.append(group)
groups.sort(key=lambda g: (g["soonest_days_left"] is None, g["soonest_days_left"]))
return groups
def _stock_matches(items: list[dict], kind: str, name: str) -> list[dict]:
"""The stock rows a just-identified item could be, best candidate first.
Book-out and expiry-clearing both need this: the camera says "eggs", and the
question is which of the household's egg rows to take from. An exact name match
(kind *and* brand) is the confident case and is marked as such; everything else in
the same fold group is offered as a choice rather than picked silently, because
guessing the brand wrong here writes a wrong number into stock — the one thing
this service is built not to do.
"""
key = _fold_key(kind or name)
exact_needle = str(name or "").strip().casefold()
matches = []
for item in items:
is_exact = bool(exact_needle) and str(item["name"]).strip().casefold() == exact_needle
if is_exact or _fold_key(item.get("kind") or item.get("name")) == key:
matches.append({**item, "exact": is_exact})
matches.sort(key=lambda m: (not m["exact"], m["days_left"] is None, m["days_left"]))
return matches
def _locations() -> list[dict]:
"""Grocy's locations, for the edit screen's "move it to…" control. Best-effort:
an empty list just means that control isn't offered this load."""
try:
rows = _grocy_get("/api/objects/locations")
return [{"id": int(r["id"]), "name": str(r.get("name", ""))} for r in rows if r.get("id") is not None]
except (urllib.error.URLError, urllib.error.HTTPError, KeyError, ValueError, TypeError):
LOG.warning("pantry-vision: could not read Grocy locations", exc_info=True)
return []
def _stock_matches_for(kind: str, name: str) -> list[dict]:
"""_stock_matches against a fresh read of stock — what doorway.py is handed so it
never has to know Grocy exists."""
return _stock_matches(_stock_items(), kind, name)
class Handler(BaseHTTPRequestHandler):
server_version = "pantry-vision/1"
def log_message(self, format, *args): # noqa: A002
LOG.info("%s - %s", self.address_string(), format % args)
def _authorized(self) -> bool:
if not TOKEN:
return False
return self.headers.get("Authorization", "") == f"Bearer {TOKEN}"
def _respond(self, status: HTTPStatus, payload) -> None:
body = json.dumps(payload).encode("utf-8")
self.send_response(status)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(body)))
self.send_header("Access-Control-Allow-Origin", "*")
self.end_headers()
self.wfile.write(body)
def _read_body(self, max_bytes: int) -> bytes:
try:
length = int(self.headers.get("Content-Length") or 0)
except ValueError:
raise ValueError("missing or invalid Content-Length") from None
if length <= 0:
return b""
if length > max_bytes:
raise ValueError(f"body too large ({length} > {max_bytes} bytes)")
return self.rfile.read(length)
def do_OPTIONS(self): # noqa: N802
# The frontend (pantry-web, a separate origin/port) fetches this API directly
# from the browser, so a plain CORS preflight has to succeed.
self.send_response(HTTPStatus.NO_CONTENT)
self.send_header("Access-Control-Allow-Origin", "*")
self.send_header("Access-Control-Allow-Headers", "Authorization, Content-Type")
self.send_header("Access-Control-Allow-Methods", "GET, POST, OPTIONS")
self.end_headers()
def do_GET(self): # noqa: N802
if not self._authorized():
self._respond(HTTPStatus.UNAUTHORIZED, {"error": "missing or invalid bearer token"})
return
path = urlsplit(self.path).path
if path == "/inventory":
self._handle_inventory()
elif path == "/expired":
self._handle_expired()
elif path == "/doorway-events":
self._handle_doorway_events()
elif path == "/recipes":
self._handle_recipes()
elif path == "/shopping-list":
self._handle_shopping_list()
else:
self._respond(HTTPStatus.NOT_FOUND, {"error": "no such endpoint"})
def do_POST(self): # noqa: N802
if not self._authorized():
self._respond(HTTPStatus.UNAUTHORIZED, {"error": "missing or invalid bearer token"})
return
path = urlsplit(self.path).path
if path == "/identify":
self._handle_identify()
elif path == "/confirm":
self._handle_confirm()
elif path == "/consume":
self._handle_consume()
elif path == "/adjust":
self._handle_adjust()
elif path == "/transfer":
self._handle_transfer()
elif path == "/doorway-event":
self._handle_doorway_event()
else:
self._respond(HTTPStatus.NOT_FOUND, {"error": "no such endpoint"})
def _json_body(self) -> dict | None:
"""The request body as a dict, or None after having already answered 400."""
try:
raw = self._read_body(MAX_JSON_BYTES)
payload = json.loads(raw or b"{}")
except (ValueError, json.JSONDecodeError) as exc:
self._respond(HTTPStatus.BAD_REQUEST, {"error": f"bad request body: {exc}"})
return None
if not isinstance(payload, dict):
self._respond(HTTPStatus.BAD_REQUEST, {"error": "body must be a JSON object"})
return None
return payload
def _product_id_and_amount(self, payload: dict) -> tuple[int, float] | None:
"""The two fields every write endpoint needs, validated once.
`amount` must be positive and finite — see _as_float() for why the second half
of that matters.
"""
product_id = _as_int(payload.get("product_id"))
if product_id is None:
self._respond(HTTPStatus.BAD_REQUEST, {"error": "'product_id' is required"})
return None
amount = _as_float(payload.get("amount"))
if amount is None:
self._respond(HTTPStatus.BAD_REQUEST, {"error": "'amount' must be a number"})
return None
if amount <= 0:
self._respond(HTTPStatus.BAD_REQUEST, {"error": "'amount' must be greater than zero"})
return None
return product_id, amount
def _handle_identify(self) -> None:
try:
image = self._read_body(MAX_IMAGE_BYTES)
except ValueError as exc:
self._respond(HTTPStatus.REQUEST_ENTITY_TOO_LARGE, {"error": str(exc)})
return
if not image:
self._respond(HTTPStatus.BAD_REQUEST, {"error": "empty image body"})
return
proposal = _identify_via_ollama(image)
# What the household already has of this, attached to the identification
# itself: it is what the book-out and expiry screens select from, and on the
# book-in screen it is the answer to "did I already buy milk on Tuesday?"
# asked at the only moment it can still change what somebody does. Best-effort
# — an unreachable Grocy degrades the proposal to a scan-only one rather than
# failing the identification the person is standing there waiting for.
if proposal.get("present"):
try:
proposal["stock_matches"] = _stock_matches(
_stock_items(), proposal.get("kind", ""), proposal.get("name", "")
)
except (urllib.error.URLError, urllib.error.HTTPError, ValueError, TypeError):
LOG.warning("pantry-vision: could not read stock for match lookup", exc_info=True)
proposal["stock_matches"] = []
proposal["stock_matches_unavailable"] = True
self._respond(HTTPStatus.OK, proposal)
def _handle_confirm(self) -> None:
payload = self._json_body()
if payload is None:
return
kind = str(payload.get("kind") or "").strip()
brand = str(payload.get("brand") or "").strip()
# `name` stays the authoritative field (an older frontend sends only this one);
# kind/brand refine it when the newer confirm screen supplies them.
name = str(payload.get("name") or "").strip() or _display_name(kind, brand)
if not kind:
kind = _BRAND_SUFFIX_RE.sub("", name).strip()
best_before = str(payload.get("best_before_date", "")).strip()
# Invariant 1 (module docstring): stock is individual units. `quantity` is how
# many packages were scanned, `units_per_package` how many things are in one,
# and their product is what Grocy is told about. Both default to 1, so the
# pre-existing payload shape means exactly what it always meant.
packages = _coerce_int(payload.get("quantity"), default=1, minimum=1, maximum=10000)
units_per_package = _coerce_int(
payload.get("units_per_package"), default=1, minimum=1, maximum=MAX_UNITS_PER_PACKAGE
)
amount = packages * units_per_package
if not name:
self._respond(HTTPStatus.BAD_REQUEST, {"error": "'name' is required"})
return
if not re.match(r"^\d{4}-\d{2}-\d{2}$", best_before):
self._respond(HTTPStatus.BAD_REQUEST, {"error": "'best_before_date' must be YYYY-MM-DD"})
return
if not GROCY_API_KEY:
self._respond(HTTPStatus.SERVICE_UNAVAILABLE, {"error": "GROCY_API_KEY is not configured"})
return
try:
location_id = _location_id_for(payload.get("placement") or payload.get("recommended_placement") or "")
product_id = _find_or_create_product(name, kind=kind, location_id=location_id)
_add_to_stock(product_id, amount, best_before, location_id=location_id)
except (urllib.error.URLError, urllib.error.HTTPError, KeyError, ValueError) as exc:
LOG.warning("pantry-vision: Grocy write failed", exc_info=True)
self._respond(HTTPStatus.BAD_GATEWAY, {"error": f"Grocy write failed: {exc}"})
return
LOG.info(
"pantry-vision: added %s x%s (%s x %s per pack, best before %s) to Grocy",
name, amount, packages, units_per_package, best_before,
)
self._respond(HTTPStatus.OK, {"ok": True, "product_id": product_id, "amount": amount})
def _handle_consume(self) -> None:
"""Book out — the "Consume article" screen, and the expired list's Remove.
Takes a product_id rather than a name on purpose: by the time this is called,
either the camera matched a specific stock row and the person confirmed which
one, or they picked it off a list. Resolving a name to a product here would
put the guess back on the write side, where nobody sees it.
"""
payload = self._json_body()
if payload is None:
return
parsed = self._product_id_and_amount(payload)
if parsed is None:
return
product_id, amount = parsed
if not GROCY_API_KEY:
self._respond(HTTPStatus.SERVICE_UNAVAILABLE, {"error": "GROCY_API_KEY is not configured"})
return
spoiled = bool(payload.get("spoiled"))
try:
_consume_stock(product_id, amount, spoiled)
except (urllib.error.URLError, urllib.error.HTTPError, KeyError, ValueError) as exc:
# Grocy refuses to consume more than it holds, and that refusal arrives
# here as an HTTPError. It is a real answer to a real question ("we have
# fewer than you think"), so it is passed through as one rather than
# flattened into "something went wrong".
LOG.warning("pantry-vision: Grocy consume failed", exc_info=True)
self._respond(HTTPStatus.BAD_GATEWAY, {"error": f"Grocy consume failed: {exc}"})
return
LOG.info("pantry-vision: consumed %s of product %s (spoiled=%s)", amount, product_id, spoiled)
self._respond(HTTPStatus.OK, {"ok": True, "product_id": product_id, "amount": amount, "spoiled": spoiled})
def _handle_adjust(self) -> None:
"""The edit-inventory screen: +, , and the freeform amount, all one write.
`amount` here is the absolute new figure, not a delta, because that is what
Grocy's inventory endpoint takes and because a delta computed on the display
from a number it read a minute ago is a lost update waiting for two people in
a kitchen. The +/- buttons do their arithmetic against the row the person is
looking at and send the result.
"""
payload = self._json_body()
if payload is None:
return
product_id = _as_int(payload.get("product_id"))
if product_id is None:
self._respond(HTTPStatus.BAD_REQUEST, {"error": "'product_id' is required"})
return
# Zero is allowed here and nowhere else: "we have none of these" is a correction
# somebody makes on purpose, unlike consuming or transferring nothing.
new_amount = _as_float(payload.get("amount"))
if new_amount is None or new_amount < 0:
self._respond(HTTPStatus.BAD_REQUEST, {"error": "'amount' must be zero or greater"})
return
if not GROCY_API_KEY:
self._respond(HTTPStatus.SERVICE_UNAVAILABLE, {"error": "GROCY_API_KEY is not configured"})
return
best_before = str(payload.get("best_before_date") or "").strip()
if best_before and not re.match(r"^\d{4}-\d{2}-\d{2}$", best_before):
self._respond(HTTPStatus.BAD_REQUEST, {"error": "'best_before_date' must be YYYY-MM-DD"})
return
try:
_set_stock_amount(product_id, new_amount, best_before or None)
except (urllib.error.URLError, urllib.error.HTTPError, KeyError, ValueError) as exc:
LOG.warning("pantry-vision: Grocy inventory correction failed", exc_info=True)
self._respond(HTTPStatus.BAD_GATEWAY, {"error": f"Grocy correction failed: {exc}"})
return
LOG.info("pantry-vision: corrected product %s to %s", product_id, new_amount)
self._respond(HTTPStatus.OK, {"ok": True, "product_id": product_id, "amount": new_amount})
def _handle_transfer(self) -> None:
"""Move stock from one appliance to another — "I put the peas in the other
freezer". Takes the placement name the confirm screen uses, so the household
only ever names appliances one way.
"""
payload = self._json_body()
if payload is None:
return
parsed = self._product_id_and_amount(payload)
if parsed is None:
return
product_id, amount = parsed
if not GROCY_API_KEY:
self._respond(HTTPStatus.SERVICE_UNAVAILABLE, {"error": "GROCY_API_KEY is not configured"})
return
# Three ways to say where it went, in order of directness: a Grocy location id
# (what the edit screen's picker sends, since /inventory already handed it the
# list), a placement word (what the confirm screen speaks), or a location name.
to_placement = str(payload.get("to_placement") or "").strip()
to_location_name = str(payload.get("to_location") or "").strip()
location_to = _as_int(payload.get("to_location_id"))
if location_to is None:
if to_placement:
location_to = _location_id_for(to_placement)
elif to_location_name:
found = _find_or_create_by_name("/api/objects/locations", to_location_name)
if found is None:
self._respond(HTTPStatus.BAD_GATEWAY, {"error": f"could not resolve location {to_location_name!r}"})
return
location_to = found
else:
self._respond(
HTTPStatus.BAD_REQUEST,
{"error": "one of 'to_location_id', 'to_placement' or 'to_location' is required"},
)
return
location_from = _as_int(payload.get("from_location_id"))
if location_from is None:
# Not supplied: read the product's current location rather than guessing,
# because Grocy's transfer needs a real source and a wrong one moves stock
# that isn't there.
try:
for item in _stock_items():
if item["product_id"] == product_id:
location_from = item["location_id"]
break
except (urllib.error.URLError, urllib.error.HTTPError):
LOG.warning("pantry-vision: could not read stock to find the source location", exc_info=True)
if location_from is None:
self._respond(
HTTPStatus.BAD_REQUEST,
{"error": "could not work out where this is now — send 'from_location_id'"},
)
return
if int(location_from) == int(location_to):
self._respond(HTTPStatus.OK, {"ok": True, "product_id": product_id, "unchanged": True})
return
try:
_transfer_stock(product_id, amount, location_from, location_to)
except (urllib.error.URLError, urllib.error.HTTPError, KeyError, ValueError) as exc:
LOG.warning("pantry-vision: Grocy transfer failed", exc_info=True)
self._respond(HTTPStatus.BAD_GATEWAY, {"error": f"Grocy transfer failed: {exc}"})
return
LOG.info("pantry-vision: transferred %s of product %s to location %s", amount, product_id, location_to)
self._respond(
HTTPStatus.OK,
{"ok": True, "product_id": product_id, "amount": amount, "location_id": location_to},
)
def _handle_doorway_event(self) -> None:
"""A fridge/freezer door sensor fired. Answer immediately, look afterwards.
Home Assistant's automation is holding a `rest_command` open while this
returns, and the work behind it is up to three snapshots and a vision call
each. So the request is validated, acknowledged with 202, and the camera burst
runs on its own thread — an appliance door is not something to keep a home
automation waiting on, and there is nothing for HA to do with the answer
anyway. Nothing here writes stock; see doorway.py's docstring for why.
"""
payload = self._json_body()
if payload is None:
return
configured = doorway.appliances()
appliance_id = str(payload.get("appliance") or "").strip()
appliance = configured.get(appliance_id)
if appliance is None:
self._respond(
HTTPStatus.BAD_REQUEST,
{
"error": f"unknown appliance {appliance_id!r}",
"known": sorted(configured),
"hint": "set PANTRY_DOOR_APPLIANCES — see pantry-vision.env.example",
},
)
return
state = str(payload.get("state") or "opened").strip().lower()
if state not in ("opened", "closed"):
self._respond(HTTPStatus.BAD_REQUEST, {"error": "'state' must be 'opened' or 'closed'"})
return
def run():
try:
doorway.handle_event(appliance, state, _identify_via_ollama, _stock_matches_for)
except Exception:
LOG.warning("pantry-vision: doorway event handling failed", exc_info=True)
threading.Thread(target=run, name=f"doorway-{appliance_id}", daemon=True).start()
self._respond(HTTPStatus.ACCEPTED, {"ok": True, "appliance": appliance_id, "state": state})
def _handle_doorway_events(self) -> None:
try:
limit = int(parse_qs(urlsplit(self.path).query).get("limit", ["50"])[0])
except (TypeError, ValueError):
limit = 50
self._respond(
HTTPStatus.OK,
{"appliances": sorted(doorway.appliances()), "events": doorway.recent_hints(limit)},
)
def _handle_inventory(self) -> None:
try:
items = _stock_items()
except (urllib.error.URLError, urllib.error.HTTPError) as exc:
LOG.warning("pantry-vision: Grocy /api/stock unreachable", exc_info=True)
self._respond(HTTPStatus.BAD_GATEWAY, {"error": f"Grocy unreachable: {exc}"})
return
# Both views, always. `items` is the per-product list the inventory tab and
# the edit screen work from; `groups` is the same rows folded brand-free
# (invariant 2). One request rather than two because they are one read of
# Grocy either way, and because a display showing "22 eggs" that disagrees
# with its own expanded list is the failure mode worth designing out.
# `locations` rides along so the edit screen can offer "move it to…" without a
# second round trip.
self._respond(
HTTPStatus.OK,
{"items": items, "groups": _fold_stock(items), "locations": _locations()},
)
def _handle_expired(self) -> None:
"""Everything already past its best-before date, most overdue first.
Its own endpoint rather than a client-side filter of /inventory so that "what
counts as expired" has one definition — this is the list the household throws
food away from, and Home Assistant will want the same answer.
"""
try:
items = _stock_items()
except (urllib.error.URLError, urllib.error.HTTPError) as exc:
LOG.warning("pantry-vision: Grocy /api/stock unreachable", exc_info=True)
self._respond(HTTPStatus.BAD_GATEWAY, {"error": f"Grocy unreachable: {exc}"})
return
expired = [i for i in items if i["days_left"] is not None and i["days_left"] < 0]
expired.sort(key=lambda i: i["days_left"])
self._respond(HTTPStatus.OK, {"items": expired})
def _handle_recipes(self) -> None:
try:
recipes = _grocy_get("/api/objects/recipes")
except (urllib.error.URLError, urllib.error.HTTPError) as exc:
LOG.warning("pantry-vision: Grocy recipes unreachable", exc_info=True)
self._respond(HTTPStatus.BAD_GATEWAY, {"error": f"Grocy unreachable: {exc}"})
return
results = []
for recipe in recipes if isinstance(recipes, list) else []:
entry = {"id": recipe.get("id"), "name": recipe.get("name"), "fulfilled": None}
# Best-effort only: Grocy's fulfillment endpoint needs the recipes feature
# actually set up (ingredients entered per recipe) to mean anything, and
# its exact response shape is unverified — see README.md. A failure here
# degrades to "fulfilled: null" (rendered as "unknown" client-side), never
# a broken page.
try:
fulfillment = _grocy_get(f"/api/recipes/{entry['id']}/fulfillment")
entry["fulfilled"] = bool(fulfillment.get("need_fulfilled"))
except Exception:
pass
results.append(entry)
self._respond(HTTPStatus.OK, {"recipes": results})
def _handle_shopping_list(self) -> None:
""""Groceries running low" for hosts/door-panel/'s dashboard (Phase 18) —
distinct from /inventory's soonest-to-expire sort: this is about quantity
below Grocy's own per-product minimum stock amount, not expiry date. Grocy's
`/api/stock/volatile` endpoint already computes exactly this
(`missing_products`) natively, so this is a thin reshape, not new logic —
same "don't duplicate what Grocy already tracks" rule as /inventory and
/recipes.
"""
try:
volatile = _grocy_get("/api/stock/volatile?missing_days=0")
except (urllib.error.URLError, urllib.error.HTTPError) as exc:
LOG.warning("pantry-vision: Grocy /api/stock/volatile unreachable", exc_info=True)
self._respond(HTTPStatus.BAD_GATEWAY, {"error": f"Grocy unreachable: {exc}"})
return
# VERIFY: assumes `missing_products` entries carry "name" and
# "amount_missing" fields directly — Grocy's documented shape, not checked
# against a live instance. Falls back to "product_id" as the display name
# rather than dropping the row, same degrade rule as /inventory.
missing = volatile.get("missing_products") if isinstance(volatile, dict) else None
items = []
for row in missing if isinstance(missing, list) else []:
items.append(
{
"name": row.get("name") or f"Product #{row.get('product_id')}",
"amount_missing": row.get("amount_missing"),
}
)
self._respond(HTTPStatus.OK, {"items": items})
def main() -> int:
logging.basicConfig(
level=os.environ.get("LOG_LEVEL", "INFO").upper(),
format="%(asctime)s %(levelname)-7s %(name)s: %(message)s",
)
if not TOKEN:
LOG.error(
"PANTRY_VISION_TOKEN is not set — every request will be rejected until it is. "
"See pantry-vision.env.example."
)
if not GROCY_API_KEY:
LOG.warning("GROCY_API_KEY is not set — /confirm will fail until it is configured.")
port = int(os.environ.get("PANTRY_VISION_PORT", "8095"))
server = ThreadingHTTPServer(("0.0.0.0", port), Handler)
LOG.info("pantry-vision listening on :%d (Ollama: %s, Grocy: %s)", port, OLLAMA_HOST, GROCY_URL)
try:
server.serve_forever()
except KeyboardInterrupt:
pass
return 0
if __name__ == "__main__":
sys.exit(main())