SmartestHome/docs/fridge-item-location.md

Knowing where a thing is: multiple fridges, shelves, and what it would cost

A feasibility note that has since grown a build. pantry-vision (Phase 17) knows what the household has and when it expires. This asks what it would take to also know where — which of several fridges, and how far inside one.

The recommendation at the bottom is deliberately smaller than the question, and the reason is in the middle.

Status, 2026-08-10. Steps 1 and 4 of the recommendation are now written: appliances as Grocy locations with a POST /transfer action and a "Move to…" control on the edit screen, and the door-sensor→camera→hint path (pantry-vision/doorway.py, POST /doorway-event). Step 2 is hardware nobody has bought and step 3 is a month nobody has spent — which means the camera half is written against an assumption that has never been tested: that a local vision model can identify an item in a moving hand at doorway distance. An appliance can be configured with a door sensor and no camera, and that remains the recommended way to start. Nothing about writing the code changes the order in which it is worth switching on.

First, decide which question is actually being asked

These are three different problems with three different price tags, and they get conflated constantly:

Level	The question	Honest difficulty
L1 — which appliance	"Is the mustard in the kitchen fridge or the loggia one?"	Solved today, no new hardware, one extra tap
L2 — which zone	"Door shelf, middle shelf, or the crisper drawer?"	Achievable at hint quality (60–80%), real hardware, real work
L3 — exact position	"Second from the left, behind the milk"	Not worth it. See "Occlusion" below

The value is overwhelmingly at L1, and it is the level this household actually has a problem at — the whole premise of the question is multiple fridges, because the freezer compartment of the main one is too small. Somebody standing in the kitchen wondering whether to walk to the loggia is asking an L1 question. L2 saves ten seconds of looking; L3 saves none, because by the time you have the door open you can see.

Do not build downward from L3. Every camera-in-the-fridge design starts there, because that is the impressive demo, and every one of them founders on the same rock.

L1: available now, and mostly a data-model decision

pantry-vision already resolves the confirm screen's fridge/freezer/cupboard answer to a Grocy location, created by name (PLACEMENT_LOCATION_NAMES in server.py, PANTRY_LOCATION_* in the env file), and passes location_id on the stock add. Grocy has modelled multiple locations natively since long before this project existed.

So L1 is not a vision problem at all. It is three small changes, and two and a half of them are now written:

1. The placement answer becomes the real appliances — "Fridge (kitchen)", "Freezer (loggia)", "Cupboard" — named in PANTRY_LOCATION_* and resolved to Grocy locations. Done, with a caveat: the confirm screen still offers the four categories (fridge / freezer / cupboard / counter) and each maps to one location name, so a household with two fridges has to pick which one that word means and move the other by hand. Reading the appliance list straight off Grocy's locations is the obvious next step and has not been taken.
2. /inventory carries the location through, so every list says where. Done — location_id/location on every row, locations on the response.
3. Moving something between fridges needs a "moved" action, or the data rots within a fortnight. Done — POST /transfer onto Grocy's own transfer endpoint, with a picker on the edit screen.

Point 3 is the whole ballgame, and it is the thing every camera proposal below is really trying to buy its way out of: the expensive part is not learning where things are, it is noticing when they move.

L2/L3: what the camera approaches actually run into

Occlusion is the rock

A single camera sees the front row. Groceries are stored in rows. The mustard behind the milk is invisible, and no model — local, cloud, or otherwise — recovers it from a picture that does not contain it. Multi-camera stereo does not fix this either; it is not a depth problem, it is a line-of-sight problem.

This is why L3 is off the table and why L2 is a hint, not an assertion. Any design that needs the camera to enumerate a shelf's contents is buying an answer that is silently incomplete, which for an inventory is the worst failure available: "we have no mustard" from a system that simply could not see it produces a second jar of mustard.

The interior-camera problems, in the order they will bite

• Power. Fridges have no interior outlet. The three options are a battery (poor: cold cuts usable lithium capacity substantially, and this is a duty-cycled camera, not a doorbell), a flat ribbon cable through the door gasket (works — this is how retrofit fridge cams do it — but it is a modification to the seal of an appliance that may be rented), or drilling a grommet (irreversible).
• The freezer is out of scope for interior cameras, full stop. Off-the-shelf camera modules — ESP32-CAM, Pi camera, USB webcams — are specified to roughly 0 °C at best. −18 °C is outside every consumer part's rating: lubricants stiffen, electrolytics lose capacitance, and condensation cycling on every door open eventually gets inside the enclosure. Industrial cold-chain cameras exist and cost more than the freezer. Given that the second appliance exists precisely because the built-in freezer is too small, this rules interior cameras out of the exact compartment the question started from.
• Condensation. Every door opening puts warm humid air on a cold lens. It fogs, and it fogs for minutes — longer than the interaction. Mitigations (sealed housing with desiccant, conformal coating, a lens heater) all cost power, which brings back problem one.
• Light. The interior lamp is on only while the door is open, so the only usable frames are the ones during an opening — at which point the camera might as well be outside.

Every one of those bullets is a class of problem, not a part-selection detail. Taken together they say: do not put the camera inside.

The approach that survives all four: watch the doorway, not the shelf

Point a camera at the plane of the door opening, from outside, and trigger on a door sensor. Then you are not parsing a packed shelf — you are watching one item, held in one hand, unoccluded, at a known moment, against a known background, with the kitchen's own lighting. That is the same recognition problem /identify already solves, which means it reuses the prompt, the model, and the "propose, never auto-commit" guardrail instead of needing a new perception stack.

What it buys: transactions, not state. "A jar of X went into the loggia fridge at 18:42." Transactions are what keep L1 from rotting (the problem above), and a sequence of them gives L2 for free if the camera can also see which shelf the hand went to — a much weaker inference than reading the shelf, and one that degrades gracefully into "somewhere in this fridge".

What it costs in honesty: it misses. Two items at once, an arm across the lens, a foot closing the door. So its output is a hint with a timestamp — "last seen going into the loggia fridge, Tuesday 18:42" — never an assertion the UI treats as fact. That framing is what makes a 70%-accurate system useful instead of infuriating, and it is the same distinction digest-engine already draws between what a source says and what is so.

This is what pantry-vision/doorway.py implements. An HA automation on the contact sensor POSTs /doorway-event, the service answers 202 and takes a short burst off the request thread (three frames, stopping at the first that recognises anything), and the result lands in its own SQLite file — not in Grocy, because Grocy owns stock and has nowhere to put "a camera thinks it saw something like this go past that door, and might be wrong". Hints expire after 30 days, because "last seen a month ago" tells nobody anything they didn't know. The edit screen shows the sighting next to the location Grocy records, and acting on it is a tap on Move.

Where the cameras go, concretely

• One camera per appliance, mounted above the door on the hinge side, ~1.8–2 m, angled down so its field of view is the door aperture and the ~40 cm in front of it. Not inside the door swing. docs/components.md already lists 2× spare webcam (destination TBD) — this is a plausible destination for one of them, which makes the experiment nearly free.
• A contact sensor on every fridge and freezer door. Zigbee, on the coordinator this project already has. This is the single highest value-per-euro item in this whole note and it is worth doing even if no camera is ever installed: it is what triggers the capture, it is what tells you which appliance a transaction belongs to, and on its own it already supports "the loggia freezer has been open for four minutes" — which prevents more spoilage per euro than any amount of localisation.
• No new inter-VLAN paths. Both devices sit where the existing camera/IoT devices sit; the capture is pulled by the container host, same direction as everything else.

Approaches ruled out, and why

• RFID/NFC tags per item. The only technology that genuinely answers L3. It also requires tagging every single item by hand, which is a worse manual step than the one tap L1 costs — and a fridge is a metal box full of water, which detunes UHF tags badly. Rules itself out twice.
• Load cells under each shelf. Excellent at "how much is left", blind to identity, and a per-shelf retrofit on appliances that may be rented.
• BLE/UWB. Nothing to attach a transmitter to. Not applicable to passive goods.

The compute question nobody asks until it's too late

Reading a shelf is not the same task as reading one held-up jar: small objects, partial labels, angles, occlusion. A 7–13B local VLM will be markedly worse at it than at the current task, and the honest architecture is detect-then-crop — a small object detector proposing boxes, the VLM reading each crop — which is a Frigate-class, always-on GPU workload.

There is one RTX 3060 in this design, and contention on it is already an open question (project plan open decision #4, MAX_LOADED_MODELS=1). Adding a continuous detector to a GPU that also has to answer Assist within voice latency and run four digests a day is a scheduling problem, not a spare-capacity problem. The doorway-camera approach avoids this too: it is event-driven, a handful of inferences per day, on the model that is already loaded.

Recommendation

In order, stopping wherever the household stops caring:

1. Make L1 real in software — appliances as Grocy locations, location shown in every list, and a transfer action so moves get recorded. No hardware. This is the only step that is unambiguously worth doing, and it must come first regardless: without it there is nowhere to put a camera's answer. Written.
2. Contact sensors on every fridge/freezer door. Cheap, Zigbee, immediately useful on their own, and the prerequisite for everything below. See docs/components.md's "Appliance door sensors" — ~€10–15 each, mounted on the outside face of the door, because a coin cell at −18 °C is a false economy. Not bought.
3. Live with 1+2 for a month. The real finding will be whether anyone ever wanted L2, or whether "which fridge" plus "the door's been open" was the whole need. This step is not padding — it is the only way to avoid building L2 for a question nobody turned out to be asking. Not done, and writing step 4 early does not skip it.
4. Only then, a doorway camera on one appliance, producing timestamped hints, reusing /identify unchanged. One appliance, not all of them, until it earns the second. Written (pantry-vision/doorway.py), untested against a real camera, and configurable per appliance so it can stay off.
5. Never an interior camera in the freezer, and probably not in the fridge either.

A related chore was added alongside this: chores' groceries_out_of_place watch point, which notices food standing out on a counter and nudges whoever the camera last saw — litter's twin, with a 30-minute fuse instead of four hours, because the failure there is spoilage rather than untidiness. It is a different system looking at a different surface for a different reason, and deliberately so: it knows food is out, not what the food is, and it never touches Grocy. See chores/README.md.

What in here is unverified

Everything with a number in it. Specifically: consumer camera modules' actual low-temperature behaviour (the 0 °C figure is a datasheet-class generalisation, not a part this project has picked or tested); how badly a lens really fogs on a door opening in this kitchen's humidity; whether a local VLM can identify an item held in a moving hand at doorway distance at all — that last one is the assumption step 4 lives or dies on, and it is testable today with the kitchen display's existing camera and about twenty minutes, long before anything is bought.

Related open decisions in docs/project-plan.md: #18 (no vision model picked, benchmarked, or measured for latency) and #4 (GPU contention). Both apply here unchanged, and neither gets easier with a second camera pointed at anything.