SmartestHome/docs/fridge-item-location.md

# Knowing *where* a thing is: multiple fridges, shelves, and what it would cost

A feasibility note that has since grown a build. `pantry-vision` (Phase 17) knows
**what** the household has and **when** it expires. This asks what it would take to
also know **where** — which of several fridges, and how far inside one.

The recommendation at the bottom is deliberately smaller than the question, and the
reason is in the middle.

> **Status, 2026-08-10.** Steps 1 and 4 of the recommendation are now written:
> appliances as Grocy locations with a `POST /transfer` action and a "Move to…"
> control on the edit screen, and the door-sensor→camera→hint path
> (`pantry-vision/doorway.py`, `POST /doorway-event`). **Step 2 is hardware nobody has
> bought and step 3 is a month nobody has spent** — which means the camera half is
> written against an assumption that has never been tested: that a local vision model
> can identify an item in a moving hand at doorway distance. An appliance can be
> configured with a door sensor and no camera, and that remains the recommended way to
> start. Nothing about writing the code changes the order in which it is worth
> switching on.

## First, decide which question is actually being asked

These are three different problems with three different price tags, and they get
conflated constantly:

| Level | The question | Honest difficulty |
| --- | --- | --- |
| **L1 — which appliance** | "Is the mustard in the kitchen fridge or the loggia one?" | Solved today, no new hardware, one extra tap |
| **L2 — which zone** | "Door shelf, middle shelf, or the crisper drawer?" | Achievable at *hint* quality (60–80%), real hardware, real work |
| **L3 — exact position** | "Second from the left, behind the milk" | Not worth it. See "Occlusion" below |

The value is overwhelmingly at L1, and it is the level this household actually has a
problem at — the whole premise of the question is *multiple fridges*, because the
freezer compartment of the main one is too small. Somebody standing in the kitchen
wondering whether to walk to the loggia is asking an L1 question. L2 saves ten seconds
of looking; L3 saves none, because by the time you have the door open you can see.

**Do not build downward from L3.** Every camera-in-the-fridge design starts there,
because that is the impressive demo, and every one of them founders on the same rock.

## L1: available now, and mostly a data-model decision

`pantry-vision` already resolves the confirm screen's fridge/freezer/cupboard answer
to a **Grocy location**, created by name (`PLACEMENT_LOCATION_NAMES` in `server.py`,
`PANTRY_LOCATION_*` in the env file), and passes `location_id` on the stock add. Grocy
has modelled multiple locations natively since long before this project existed.

So L1 is not a vision problem at all. It is three small changes, and two and a half of
them are now written:

1. **The placement answer becomes the real appliances** — "Fridge (kitchen)", "Freezer
   (loggia)", "Cupboard" — named in `PANTRY_LOCATION_*` and resolved to Grocy
   locations. *Done, with a caveat*: the confirm screen still offers the four
   categories (fridge / freezer / cupboard / counter) and each maps to one location
   name, so a household with two fridges has to pick which one that word means and
   move the other by hand. Reading the appliance list straight off Grocy's locations
   is the obvious next step and has not been taken.
2. **`/inventory` carries the location through**, so every list says where. *Done* —
   `location_id`/`location` on every row, `locations` on the response.
3. **Moving something between fridges needs a "moved" action**, or the data rots
   within a fortnight. *Done* — `POST /transfer` onto Grocy's own transfer endpoint,
   with a picker on the edit screen.

Point 3 is the whole ballgame, and it is the thing every camera proposal below is
really trying to buy its way out of: **the expensive part is not learning where things
are, it is noticing when they move.**

## L2/L3: what the camera approaches actually run into

### Occlusion is the rock

A single camera sees the front row. Groceries are stored in rows. The mustard behind
the milk is invisible, and no model — local, cloud, or otherwise — recovers it from a
picture that does not contain it. Multi-camera stereo does not fix this either; it is
not a depth problem, it is a line-of-sight problem.

This is why L3 is off the table and why L2 is a *hint*, not an assertion. Any design
that needs the camera to enumerate a shelf's contents is buying an answer that is
silently incomplete, which for an inventory is the worst failure available: "we have
no mustard" from a system that simply could not see it produces a second jar of
mustard.

### The interior-camera problems, in the order they will bite

- **Power.** Fridges have no interior outlet. The three options are a battery (poor:
  cold cuts usable lithium capacity substantially, and this is a duty-cycled camera,
  not a doorbell), a flat ribbon cable through the door gasket (works — this is how
  retrofit fridge cams do it — but it is a modification to the seal of an appliance
  that may be rented), or drilling a grommet (irreversible).
- **The freezer is out of scope for interior cameras, full stop.** Off-the-shelf
  camera modules — ESP32-CAM, Pi camera, USB webcams — are specified to roughly 0 °C
  at best. −18 °C is outside every consumer part's rating: lubricants stiffen,
  electrolytics lose capacitance, and condensation cycling on every door open
  eventually gets inside the enclosure. Industrial cold-chain cameras exist and cost
  more than the freezer. **Given that the second appliance exists precisely because
  the built-in freezer is too small, this rules interior cameras out of the exact
  compartment the question started from.**
- **Condensation.** Every door opening puts warm humid air on a cold lens. It fogs,
  and it fogs for minutes — longer than the interaction. Mitigations (sealed housing
  with desiccant, conformal coating, a lens heater) all cost power, which brings back
  problem one.
- **Light.** The interior lamp is on only while the door is open, so the only usable
  frames are the ones during an opening — at which point the camera might as well be
  outside.

Every one of those bullets is a *class* of problem, not a part-selection detail. Taken
together they say: **do not put the camera inside.**

### The approach that survives all four: watch the doorway, not the shelf

Point a camera at the **plane of the door opening**, from outside, and trigger on a
door sensor. Then you are not parsing a packed shelf — you are watching one item, held
in one hand, unoccluded, at a known moment, against a known background, with the
kitchen's own lighting. That is the *same* recognition problem `/identify` already
solves, which means it reuses the prompt, the model, and the "propose, never
auto-commit" guardrail instead of needing a new perception stack.

What it buys: **transactions, not state.** "A jar of X went into the loggia fridge at
18:42." Transactions are what keep L1 from rotting (the problem above), and a
sequence of them gives L2 for free if the camera can also see *which shelf the hand
went to* — a much weaker inference than reading the shelf, and one that degrades
gracefully into "somewhere in this fridge".

What it costs in honesty: it misses. Two items at once, an arm across the lens, a foot
closing the door. So its output is a **hint with a timestamp** — "last seen going into
the loggia fridge, Tuesday 18:42" — never an assertion the UI treats as fact. That
framing is what makes a 70%-accurate system useful instead of infuriating, and it is
the same distinction `digest-engine` already draws between what a source *says* and
what is *so*.

This is what `pantry-vision/doorway.py` implements. An HA automation on the contact
sensor POSTs `/doorway-event`, the service answers 202 and takes a short burst off the
request thread (three frames, stopping at the first that recognises anything), and the
result lands in **its own SQLite file** — not in Grocy, because Grocy owns stock and
has nowhere to put "a camera thinks it saw something like this go past that door, and
might be wrong". Hints expire after 30 days, because "last seen a month ago" tells
nobody anything they didn't know. The edit screen shows the sighting next to the
location Grocy records, and acting on it is a tap on *Move*.

### Where the cameras go, concretely

- **One camera per appliance**, mounted above the door on the hinge side, ~1.8–2 m,
  angled down so its field of view is the door aperture and the ~40 cm in front of it.
  Not inside the door swing. `docs/components.md` already lists **2× spare webcam
  (destination TBD)** — this is a plausible destination for one of them, which makes
  the experiment nearly free.
- **A contact sensor on every fridge and freezer door.** Zigbee, on the coordinator
  this project already has. This is the single highest value-per-euro item in this
  whole note and it is worth doing *even if no camera is ever installed*: it is what
  triggers the capture, it is what tells you which appliance a transaction belongs to,
  and on its own it already supports "the loggia freezer has been open for four
  minutes" — which prevents more spoilage per euro than any amount of localisation.
- **No new inter-VLAN paths.** Both devices sit where the existing camera/IoT devices
  sit; the capture is pulled by the container host, same direction as everything else.

### Approaches ruled out, and why

- **RFID/NFC tags per item.** The only technology that genuinely answers L3. It also
  requires tagging every single item by hand, which is a *worse* manual step than the
  one tap L1 costs — and a fridge is a metal box full of water, which detunes UHF tags
  badly. Rules itself out twice.
- **Load cells under each shelf.** Excellent at "how much is left", blind to identity,
  and a per-shelf retrofit on appliances that may be rented.
- **BLE/UWB.** Nothing to attach a transmitter to. Not applicable to passive goods.

## The compute question nobody asks until it's too late

Reading a shelf is not the same task as reading one held-up jar: small objects,
partial labels, angles, occlusion. A 7–13B local VLM will be markedly worse at it than
at the current task, and the honest architecture is detect-then-crop — a small object
detector proposing boxes, the VLM reading each crop — which is a Frigate-class,
always-on GPU workload.

There is one RTX 3060 in this design, and contention on it is already an open
question (project plan open decision #4, `MAX_LOADED_MODELS=1`). Adding a
continuous detector to a GPU that also has to answer Assist within voice latency and
run four digests a day is a scheduling problem, not a spare-capacity problem. **The
doorway-camera approach avoids this too**: it is event-driven, a handful of inferences
per day, on the model that is already loaded.

## Recommendation

In order, stopping wherever the household stops caring:

1. **Make L1 real in software** — appliances as Grocy locations, location shown in
   every list, and a transfer action so moves get recorded. No hardware. This is the
   only step that is unambiguously worth doing, and it must come first regardless:
   without it there is nowhere to *put* a camera's answer. **Written.**
2. **Contact sensors on every fridge/freezer door.** Cheap, Zigbee, immediately useful
   on their own, and the prerequisite for everything below. See
   `docs/components.md`'s "Appliance door sensors" — ~€10–15 each, mounted on the
   *outside* face of the door, because a coin cell at −18 °C is a false economy.
   **Not bought.**
3. **Live with 1+2 for a month.** The real finding will be whether anyone ever wanted
   L2, or whether "which fridge" plus "the door's been open" was the whole need. This
   step is not padding — it is the only way to avoid building L2 for a question nobody
   turned out to be asking. **Not done, and writing step 4 early does not skip it.**
4. **Only then**, a doorway camera on one appliance, producing timestamped *hints*,
   reusing `/identify` unchanged. One appliance, not all of them, until it earns the
   second. **Written** (`pantry-vision/doorway.py`), **untested against a real camera,
   and configurable per appliance so it can stay off.**
5. **Never** an interior camera in the freezer, and probably not in the fridge either.

A related chore was added alongside this: `chores`' **`groceries_out_of_place`** watch
point, which notices food standing out on a counter and nudges whoever the camera last
saw — litter's twin, with a 30-minute fuse instead of four hours, because the failure
there is spoilage rather than untidiness. It is a *different* system looking at a
*different* surface for a *different* reason, and deliberately so: it knows food is
out, not what the food is, and it never touches Grocy. See `chores/README.md`.

## What in here is unverified

Everything with a number in it. Specifically: consumer camera modules' actual
low-temperature behaviour (the 0 °C figure is a datasheet-class generalisation, not a
part this project has picked or tested); how badly a lens really fogs on a door
opening in this kitchen's humidity; whether a local VLM can identify an item held in a
moving hand at doorway distance at all — that last one is the assumption step 4 lives
or dies on, and it is testable today with the kitchen display's existing camera and
about twenty minutes, long before anything is bought.

Related open decisions in `docs/project-plan.md`: #18 (no vision model picked,
benchmarked, or measured for latency) and #4 (GPU contention). Both apply here
unchanged, and neither gets easier with a second camera pointed at anything.