45 KiB

Raw Permalink Blame History

digest-engine

The quarter-daily LLM digest from Phase 12 of the project plan.

Four times a day it ingests mail, messages, news and financial data, sends the lot to the existing Ollama host for synthesis into up to four sections — personal (social), political (news), household and network — and writes a rendered digest that digest-web serves to two surfaces: the thin client's kiosk Firefox workspace (full view) and a Home Assistant Lovelace iframe card (compact view).

Which of the four a run actually generates is the household's choice, per person — see Who gets which digest below.

It is a oneshot, not a daemon: a systemd timer runs docker compose run --rm digest-engine, exactly like the restic backup job.

Everything here is read-only. No replies, no marking mail read or archived, no calendar or Grocy writes, no message-platform writes of any kind. There is no mutation path in this component by construction — that is a hard requirement from the plan, not a default.

Layout

run.py                        oneshot entrypoint
preferences.py                who wants which digest, read from identity
archive.py                    the long memory — every item and number, kept across runs
agenda.py                     a Tagesordnung PDF -> the meeting it belongs to
ingest/                       one module per source, each `fetch(lookback_hours) -> list[dict]`
  telegram_login.py           standalone one-time interactive login (run by hand)
synth/llm_client.py           Ollama client + the digest JSON schema
synth/prompts/                one prompt template per section
render/digest-canvas-sdk/     vendored, offline JS/CSS — globe, window chrome, glow, renderer
render/templates/             compact.html (HA iframe) and full.html (kiosk)
feeds/curated-feeds.opml      the news feed list — edit this
feeds/rci-social.json         the RCI/section social + podcast accounts — edit this
IDSconf.json.example          OPNsense IDS config template (real file gitignored)
whatsapp-bridge/              Node.js sidecar, opt-in, see the warning below
output/                       per-run artifacts (gitignored)
compose-fragment.yaml.txt     compose blocks to splice into setup-container-host.sh

Configure

cp digest-engine/digest-engine.env.example /opt/smart-home/digest/digest-engine.env
chmod 600 /opt/smart-home/digest/digest-engine.env
$EDITOR /opt/smart-home/digest/digest-engine.env

Every source is off by default. Turn on only what you have credentials for — a disabled or misconfigured source logs a warning and contributes nothing, and can never take the rest of the run down with it.

Then edit feeds/curated-feeds.opml: the mainstream outlets in it are a clearly-marked placeholder list, only the marxist.com feed is a deliberate choice (the political prompt uses it as its analytical basis).

Who gets which digest

Every person in identity has their own set of the four sections, ticked in the admin panel's person editor (Digests → Generate for this person). Nothing else about the run changes; what changes is how much of it happens at all.

A section nobody has ticked is never generated. No synthesis call, no counter-run call, and — because run.py's SECTION_SOURCES knows which sources feed which section — no ingestion either. Turn the political section off for the whole household and the run stops fetching news, financial data and flight/naval traffic entirely. The saving is real WAN egress, not just tokens.
Each surface then shows a person their own subset. The digest carries a people list (name, nickname, id, sections) that the renderer filters on when it is given ?person=.
The default is all four, so a household that never opens this panel gets exactly the digest it had before this setting existed.

The display half is a filter, not an access control. digest-web serves the whole output volume read-only to anything on the LAN, so an unticked section is off somebody's screen and out of their narration — it is not hidden from them. The half that genuinely does not exist is the half that was never generated.

?person= is a name, nickname or id, and it is only ever set by a caller that has already resolved who is asking: hosts/thin-client/agent/thinclient_agent/digest_canvas.py passes it from a Home-Assistant-resolved request, per the plan's Phase 11.8 rule that the personal section is never shown on a guess.

The canvas is voice-activated, and the person is recognised automatically. Nothing shows a digest because somebody walked past a screen. When a spoken "play my digest" fires, HA asks identity's GET /speaker?area=<area> who is in that room — BLE identity plus the most recent Frigate face sighting — and passes the answer through as ?person=. When it cannot tell, it says so and the canvas falls back to everything-but-personal, which is the same rule as no person at all. So on full.html (the kiosk):

URL	What renders
`full.html?person=Amir`	exactly the sections Amir ticked
`full.html?person=Nobody` (unknown to identity)	everything generated except personal
`full.html` (no person)	everything generated except personal
`full.html?person=Amir`, run generated while identity was down	everything generated, personal included — the person was still resolved, and the outage already cost that run its preferences

compact.html accepts ?person= too but defaults to showing everything generated, personal included: that card is embedded in one person's own HA dashboard, which is already a per-account surface rather than a screen in a hallway.

If IDENTITY_URL is blank, or identity is down, or the token is wrong, the run generates all four sections and writes an empty people list — see preferences.py for why a failed lookup fails towards more digest rather than less. The one case that is honoured rather than overridden is a household where everybody really has ticked everything off: that run generates nothing, and says so in its log.

One-time steps before the first real run

Both of these are interactive and must be done by hand, once. Scheduled runs never prompt for anything.

Telegram — creates the session file telegram_ingest.py then reuses non-interactively. You will be asked for your phone number, the code Telegram sends, and your 2FA password if the account has one:

docker compose run --rm --entrypoint python digest-engine ingest/telegram_login.py

WhatsApp (only if you have opted in) — start the bridge and scan the QR code it prints to its own logs with WhatsApp -> Linked devices -> Link a device:

docker compose up -d whatsapp-bridge
docker compose logs -f whatsapp-bridge

The session persists in the bridge's /data/.wwebjs_auth volume, so this is a one-time scan unless WhatsApp invalidates the link.

Run once, manually

docker compose run --rm digest-engine

Output lands in output/<run-timestamp>/:

context.json — the ingested context bundle, kept for the Phase 12 follow-up voice Q&A (a spoken follow-up re-queries Ollama against this rather than re-ingesting).
digest.json — the rendered digest, both detail levels, every section this run generated (sections_generated says which, and people who asked for what).

output/latest.json is rewritten with the same payload and output/latest re-pointed at the newest run directory, so digest-web always serves the current digest with no coordination with the scheduler.

WhatsApp — read this before enabling

There is no officially sanctioned way to read your own WhatsApp messages programmatically. whatsapp-bridge runs a real Chromium logged into web.whatsapp.com as a linked device, deliberately headful under Xvfb because WhatsApp's automation detection specifically fingerprints headless Chrome. That is a meaningful mitigation. It is not immunity: this is still automated use of a personal account and accounts do get banned for it, historically on a ~2–8 week timescale.

If you enable it:

Use a secondary, non-critical number, not your main one.
Accept that the number may be banned, and that this is the highest-risk of the four message platforms by a wide margin.
Keep ENABLE_WHATSAPP_INGEST=false if you are at all unsure. The rest of the digest works fine without it.

Manual verification still outstanding

None of this has been run against real credentials or real accounts. Before trusting a scheduled run, verify by hand:

Each source in isolation, e.g. docker compose run --rm --entrypoint python digest-engine -c "import logging,os; logging.basicConfig(level='INFO'); from ingest import news_rss; print(len(news_rss.fetch(6)))".
That the IMAP mailbox shows no newly-read messages after a run (the folder is opened readonly=True, but confirm it against your provider).
That the Ollama model actually honours format: "json" and the schema — check output/<run>/digest.json for "degraded": true, which marks a section that fell back to plain text.
That both templates render: open http://<host>:8091/full.html and http://<host>:8091/compact.html.
The malformed-output fallback, by hand-editing output/latest.json into invalid JSON and reloading — the page must show a <pre> dump, never a blank screen.
The feed URLs in curated-feeds.opml — several mainstream outlets have changed or restricted their public RSS.
That the Nextcloud app password works over CalDAV and that recurring events land on the right day (the expansion path above is the one most likely to differ between Nextcloud versions).
The evening recipe, with DIGEST_FORCE_EVENING=true on a manual run — then confirm in Grocy that nothing was added to its shopping list and no stock moved.
The unviewed-digest merge: run once, do NOT show it on a thin client, run again (or set DIGEST_LOOKBACK_HOURS/wait) — confirm the second run's context.json has "merged_unviewed_previous_run": true and its digest actually carries forward the first run's content. Then show a digest on a thin client and run a third time — confirm that one merges nothing.
The counter run, deliberately: hand-edit a generated document before it's written (or patch synth/counter_run.py to run against a document with a fabricated quote spliced in) and confirm it actually gets dropped, not waved through — then confirm a real, correctly-grounded piece of Marxist analysis (a genuine merger analyzed via Lenin's imperialism) is NOT flagged just for being theoretical rather than a bare fact.
The per-person section toggles end to end, which have only been exercised against the API: untick a section for everybody in identity's admin panel, run once by hand, and confirm the run log says it skipped both that section's synthesis and its sources' ingestion, that digest.json's sections_generated agrees, and that full.html?person=<name> shows what that person ticked and nothing else. Then stop the identity container and run again — that run must generate all four sections rather than none.
That full.html with no ?person= really does leave the personal section out. It is a behaviour change: before this setting existed, the kiosk showed everyone's personal section to whoever walked past, despite digest_canvas.py having always claimed otherwise.
The political section's new structure against a real model. The ingest half is verified — ingest/rci_social.py was run live against the committed feeds/rci-social.json on 2026-08-06 and returned real YouTube and podcast entries with durations — but no local model has yet been asked to produce the four question windows, per-marker summaries and sources arrays in one JSON document. Check on the first real run that a 14B model actually fills sources rather than dropping the field, that it does not put episodes in the analysis, and that the compact pass still fits the HA card now that the section has more to say.
The Telegram channel entry in feeds/rci-social.json. The channel name comes from marxist.com's own footer but was never fetched — reading it needs the Telethon session, which only exists on the real deployment. Also confirm it does not double up: if you have joined that channel, its posts arrive a second time through telegram_ingest.py into the personal section.
The archive against real runs. Its logic is covered (dedup across runs, per-item annotation, series ordering, IDS recurrence, exclusion, and a broken database not taking a run down), but no real deployment has accumulated days of history yet. Check after a week that first_seen_at looks right on a story you remember, that the political section quotes both figures and both dates when it claims a trend, and that the network section has started calling its familiar signatures background rather than news. Watch the file size too — 400 days of mail is the setting to revisit first.
A real Tagesordnung, end to end. The matcher and the PDF extraction are tested against a generated PDF; nobody has yet sent a real branch agenda through a real mailbox. Confirm the attachment actually spools, that the points come out of a real layout (multi-column or heavily styled agendas are where pypdf's extraction gets ragged), that a scanned one degrades to text_extracted: false rather than to nonsense, and that no household todo appears that the document did not actually ask for.
The WhatsApp document download, which is new behaviour in the bridge — message._data.filename is undocumented API and downloadMedia() has never been exercised here. Check a document arrives, lands in /data/whatsapp-bridge/documents, and that photos and video are still never downloaded.
The OPNsense credentials via CoreSystemConfig.json. The export → generated IDSconf.json → opnsense_ids.py chain is tested with synthetic values; confirm a real key pair authenticates against a real firewall, and that the scoped user really cannot do more than read alerts.
The bias/ownership prompting, adversarially. Feed a run a story covered by both RT and the BBC and confirm the section applies the ownership analysis to both rather than only to the one it is easier to be sceptical of, and that a Times of Israel claim about Palestinians never reaches the digest in the section's own voice.

The political section — sources, ownership, and the four questions

The political digest answers four questions, in this order, one window each (synth/prompts/political.md is written around them):

What is relevant for the communist and class struggle globally right now.
What matters for organising here — Austria, and Vorarlberg specifically. A closure in Dornbirn outranks a foreign cabinet reshuffle for this one.
What else is consequential in the mid-to-long term without being class struggle directly — rearmament, energy, epidemics, supply chains, repression.
What is happening inside the RCI, and what comrades elsewhere report. Its sources are the theory feeds, the organisation's own social output, and the reports that arrive in the user's own mail — the one input here no feed can supply.

Plus a watch-later window: new videos and podcast episodes from the organisation's channels, kept out of the analysis entirely because they are things to watch later, not evidence.

Every source is read against who owns it

feeds/curated-feeds.opml carries owner and bias on every feed, and ingest/news_rss.py puts both on every entry. The prompt reads each item against them. This is not a reliability score — it is the same materialist analysis the section applies to everything else, applied to the press:

"State-affiliated" names who signs the cheque, not a propaganda bucket that private Western outlets are exempt from. RT is the outlet of the Russian capitalist class and its state; the BBC is the state broadcaster of a NATO power. Both get the treatment, or neither does.
A private outlet is the organ of a fraction of capital — named when it explains the coverage (the Washington Post's owner is Amazon's owner when the story is warehouse labour).
The business press (FT, CNBC) is often the most candid source in the file: it briefs capital honestly because capital is the reader.
category="news_labour" is the workers' and movement press — closer to the shop floor, and not the RCI, so its reporting is used and its conclusions are not adopted.

Zionist media get zero trust (category="news_zionist", and anything whose bias says so). Their claims about Palestinians are never repeated as established fact and their language is never adopted; they are read as evidence about the Israeli state itself — what its ruling class admits, prepares for, or falls out over. Zero trust is not inversion: a denial is not proof, and the "No speculation" rule still binds. The Palestinian, anti-Zionist and independent outlets in the same group are what corroboration is checked against.

The shipped list is about three dozen feeds across those blocs, all fetched and confirmed live on 2026-08-06 except four marked VERIFY in the file. Keep NEWS_MAX_ENTRIES_PER_FEED low (5 by default) — it multiplies by the feed count into one prompt.

Summaries on the globe, sources folded underneath

Every globe marker carries a summary (2–4 sentences on what is happening there) and its own sources; any window can carry sources too. The renderer prints marker summaries under the globe — not as tooltips, since the globe rotates and a briefing you can only read while its marker faces you is not a briefing — and folds citations into a collapsed Sources (n) block that costs no screen space until tapped. Each citation carries the outlet and its ownership, so who paid for a claim sits next to the claim.

Two guards on that, in synth/counter_run.py: a source whose URL is not present in the run's context is dropped, and a real source carrying an invented quote keeps the citation and loses the quote. A plausible-looking URL is the easiest thing in this schema to fabricate and the hardest to eyeball.

ingest/rci_social.py reads the accounts listed in feeds/rci-social.json and tags them theory_social, kept apart from the written analysis (theory) because a meeting announcement is not an argument. Only platforms with a public, keyless, first-party read path are implemented, and that is a hard line:

YouTube — the channel's own Atom feed (feeds/videos.xml?channel_id=), no key, no quota. The RCI's and the Austrian section's channels are configured and verified.
RSS — the section's podcast (anchor.fm), a Mastodon account's .rss, any site feed.
Telegram — through the same Telethon session telegram_ingest.py already uses. A public channel is resolved by username and read without joining it; no session means those entries are skipped and the rest still run.

Instagram, Facebook, WhatsApp channels, TikTok and X are deliberately not built. None has a read path that is both keyless and inside its own terms: Instagram's Basic Display API was retired in 2024 and the Graph API only reads accounts you own; Facebook page RSS died in 2018; WhatsApp channels have no API at all; X's free tier reads essentially nothing and Nitter is gone. Reading them would mean scraping, which this component does not do. Instagram is the real gap — it is where the section posts most. Follow it on your phone; do not point this at a scraping proxy, which moves the terms-of-service problem onto a third party without removing it.

"Your digest is ready" — the ntfy push

Because the canvas only opens when it is asked for, the notification is the one thing that reaches you unprompted, and it exists to answer one question: is this run worth going and asking for, or does it keep until tonight?

notify.py posts to the self-hosted ntfy this stack already runs for chores and identity. On Android that lands on the phone and relays to a watch (a Pebble needs nothing else installed). It costs no extra LLM call: every section already produces a narration at compact detail — two to four sentences written to be read aloud, which is the register a notification wants — so the push is those narrations trimmed to a line each, plus the political to-do count and any withheld/unverified marks. Nothing is generated here, so the notification can never claim something the digest itself does not say.

The per-person settings decide the push, not just the canvas. Somebody who switched the political section off gets no political content on their phone — otherwise the setting would be a lie in the place it is most visible. People are grouped by their own ntfy topic (identity's notify_topic, the same one arrival notifications use), and each topic gets the union of what the people behind it asked for: a topic is its audience, so two people sharing one have already agreed to share what arrives on it. No personal topic falls back to NTFY_TOPIC; no preferences at all (identity down) sends one household message about everything generated. DIGEST_WEB_URL makes the push tappable, opening that person's own digest — a shared topic gets the unfiltered page, since it has no single owner.

Published as JSON to ntfy's root, not as text with Title: headers the way chores does. HTTP headers are latin-1, and this digest quotes news headlines and household names: an em dash or an umlaut in a title raises UnicodeEncodeError before the request is even sent. That failed on real content and passed every ASCII test until one was written for it.

ntfy stays LAN-only; away from home it is reached over the WireGuard split tunnel (docs/network-integration.md §2.2), never a port forward. A failed push logs a warning and nothing else — the digest is already on disk by then, and no notification is worth failing a run over.

The archive — the digest's long memory

archive.py keeps a SQLite database in the /data volume holding every item the digest has ingested and every number it has measured. A run without it sees six hours and nothing else, which makes the most valuable things this system could say impossible to say: "up from 5.1% in June", "this signature has fired every night for a week", "merchant traffic through this chokepoint has halved", "this story first appeared on Monday and has not moved".

Two tables, because there are two kinds of thing:

items — discrete things (articles, messages, videos, calendar entries), deduplicated on a fingerprint (the URL when there is one). An article seen in four runs is one row seen four times, which is what makes first_seen_at meaningful. Every entry the prompts see now carries first_seen_at and times_seen, so "new this run" and "the same story for four days" are distinguishable without the model inferring it.
observations — numbers that only mean anything as a series: FRED and Stooq readings, aircraft and vessel counts per region, IDS alerts per signature and per host. One row per measurement, so a trend is a query.

History enters each prompt as its own labelled block with its own timestamps — never merged into this run's entries, so last month's figure can't be mistaken for today's news. The trend rules changed with it: the political section may now describe something as rising or falling when the history block supports it, with both figures and their dates, and the network section leads with what is new and says plainly when a signature is background noise it has seen fifteen times. Without history, both revert to "one snapshot is not a trend".

This keeps your mail and messages on disk for the retention window (400 days by default), where previously only the last few runs' context.json did. Nothing leaves the host and the file sits beside the Telegram session, but it is a real change in how long personal content is kept — DIGEST_ARCHIVE_EXCLUDE_SOURCES takes a comma-separated list of ingest keys to keep out of it entirely, and the financial/traffic/IDS trends work regardless of what you exclude. ENABLE_DIGEST_ARCHIVE=false turns the whole thing off. A corrupt or unwritable database logs a warning and the run proceeds without memory.

Meeting agendas — the Tagesordnung finds its meeting

A branch sends the agenda for Thursday's meeting as a PDF on Monday, by mail or WhatsApp. agenda.py matches it to the calendar event it belongs to, so the digest says "branch meeting Thursday 19:00 — agenda TO_12.08.pdf from Anna", and then uses what is actually in it.

What it matches on, in order: a date in the filename, subject or the document's own heading against an event starting that day; words in common between the agenda and the event's summary; both, which is the confident case. Everything is labelled with its confidence and reason, so a wrong match is visible rather than asserted, and an agenda that matches nothing is reported as unattached rather than dropped.

TO is special-cased. It is the abbreviation everyone actually uses and also the commonest two-letter word in English, so it is matched only as a standalone uppercase token in a filename or subject — never in body prose. The long words (Tagesordnung, Traktanden, agenda, Einladung; AGENDA_KEYWORDS) match case-insensitively anywhere.

The document is read. pypdf extracts the text, the numbered and bulleted lines are pulled out mechanically as points — a list extracted by code is a list that cannot be invented — and both go into the context.

An agenda's contents belong to the political section, which is where a branch agenda's party work belongs, and which means they reach only the people who ticked the political digest. The split is deliberate and the prompts enforce it from both sides:

the household section says only that an agenda arrived, for which meeting, from whom, and whether it could be read at all. It is told not to list points or derive tasks even though the text is in front of it. Somebody with the household digest and not the political one sees a meeting with an agenda, not its contents.
the political section carries political-agenda (the meeting and its points as written) and political-todo, titled "Political todos" — what the reader actually has to do before those meetings, one line each, verb first, naming the meeting and quoting the line of the agenda or its covering message the task came from. A task assigned to someone else is listed as theirs, so the reader knows it is covered. If nothing is actually asked for, the window is omitted — inventing preparation nobody asked for is the one failure here a reader would act on. It stays its own window at compact too, since it is the part of the digest people act on rather than read.
the same agendas double as the section's sharpest relevance filter: a story touching an agenda point outranks a bigger story that doesn't, and says why — "on Thursday's agenda".

calendar is therefore in the political section's SECTION_SOURCES: not for the diary, but because it cannot match an agenda to a meeting it never fetched.

A scanned agenda yields nothing — it is a page of images and there is no OCR here. That case attaches with text_extracted: false, and both prompts are told they may name such a document but must not characterise it.

Attachments are spooled to /data/attachments (mail) and /data/whatsapp-bridge/documents (the bridge, documents only — never photos or video). That is a write to the digest's own volume, exactly as context.json already is; nothing is written back to any mailbox or chat.

Traffic data — what exists and what does not

ingest/flight_traffic.py and ingest/naval_traffic.py feed the political section as extra category-tagged evidence, alongside news and financial data. They are not a new digest section, and synth/prompts/political.md is explicitly told to drop them when they corroborate nothing. Both are off by default.

Air traffic — OpenSky Network, bounded boxes over configured regions. FlightRadar24 and ADS-B Exchange were not used: FR24 prohibits scraping and sells API access, and ADS-B Exchange ended its freemium RapidAPI tier on 2025-03-01 (paid from $10/month). Before enabling, read the licensing note in digest-engine.env.example — OpenSky's Terms of Use require a prior written agreement for use of the REST API "in any operational capacity", which arguably covers a timer-driven digest. Attribution to OpenSky is required.

Naval traffic — aisstream.io, a free keyed WebSocket stream, sampled briefly per run. Read what it is for before enabling it: AIS cannot show naval force posture. Warships sail with AIS off routinely, and the "military ops" AIS type code is self-declared. What it shows is merchant traffic through chokepoints, which is genuinely useful in the negative — shipping abandoning a route arrives as freight, insurance and fuel costs. AISHub was not used: it is still contribute-to-access and needs an AIS receiver this project does not have. MarineTraffic, VesselFinder and Spire are paid.

Military movement — deliberately not built as a data source. There is no free structured feed of military movements. ACLED is retrospective conflict-event data, needs registration, and its EULA forbids redistribution and non-transformative derivative works; UCDP is keyless but lags by roughly a month; everything real-time is either commercial or a person on social media. Rather than invent an integration, the military-movement signal comes from OSINT/defence outlets added to feeds/curated-feeds.opml under category="osint_military", which reuses the existing news ingestion with no new code. Verified live 2026-07-28; Liveuamap has no free RSS feed (/rss is a paid-API signup page) and ISW and Long War Journal both 403'd the check — re-test those from the real network.

Home network intrusion detection — what exists and what does not

ingest/opnsense_ids.py pulls a summary of the Suricata alerts your existing OPNsense firewall raised during the digest window, tagged "category": "network_security". Off by default (ENABLE_OPNSENSE_IDS_INGEST=false).

The credentials live in CoreSystemConfig.json — an opnsense block for the address and tuning, secrets.opnsense_api_key/opnsense_api_secret for the key pair OPNsense mints (System → Access → Users → API keys; only OPNsense can issue them, so generate-tokens.py deliberately never invents one). A build writes them into the container host's /data/IDSconf.json alongside every other generated config. Field-by-field documentation stays in IDSconf.json.example, which is also what setup-container-host.sh seeds when you install by hand instead of from an image. Setting the credentials does not enable the ingest: ENABLE_OPNSENSE_IDS_INGEST=true in digest-engine.env is still a separate, deliberate step.

It is its own section (synth/prompts/network.md), not a couple of lines inside the household one as it was originally built. The reason is the per-person setting above: somebody who wants the calendar and the shopping list but not a nightly intrusion-detection readout — or the reverse — can only say so if the two are generated separately. The instructions themselves moved across unchanged, including the one that matters most: alerts are signature matches, never a claim that a device is compromised, and the section may never call the network safe.

Suricata is core, not a plugin. There is no os-suricata to install — the IDS module ships in OPNsense core and lives at Services → Intrusion Detection. The os-intrusion-detection-content-* plugins are ruleset content only, and ET Open needs none of them. Suricata is nonetheless off on a stock install; enable it and download a ruleset first, or every run will report ids_status and no alerts, which is the honest answer and not a quiet network.

It is a pull, like every other source here. Two endpoints, both read-only in effect: GET /api/ids/service/status and POST /api/ids/service/query_alerts. The latter is a POST only because that is how OPNsense routes filtered queries — it runs queryAlertLog.py, which reads /var/log/suricata/eve.json backwards. No SSH or file access to the firewall is needed, and no push agent runs on it.

Three limits worth knowing before you read the output:

No server-side time filter. searchPhrase matches signature/action/src/dst text only. Rows come back newest-first, so the window is applied client-side by paging until a row falls out of it, bounded by max_alerts_scanned. When that bound is hit — or when the alert log rotated mid-window — the entry carries window_truncated: true and the prompt is told the counts are a lower bound.
No severity. OPNsense flattens each eve.json record to signature + SID + action before returning it, discarding alert.severity and alert.category. get_alert_info uses the same flattening, so it does not help. Alerts are ranked by frequency, and the household prompt is told it cannot see severity.
Page-level ACLs. The "Services: Intrusion Detection" privilege matches api/ids/*, which covers start/stop/reconfigure/drop-alert-log as well as the alert query. OPNsense has no narrower built-in privilege, so the read-only guarantee is enforced by this code (which calls two endpoints and no others) and not by the firewall. Give the API key its own user with that one privilege and nothing else, and treat it as a credential that could restart your IDS if it leaked — the container host and the firewall are on the same flat LAN, since no VLAN segmentation is implemented in this project yet.

Raw packet captures — deliberately not done here. OPNsense does expose Interfaces: Diagnostics: Packet Capture over the API (/api/diagnostics/packet_capture/{set,start,stop,remove}), but every one of those is a POST that writes a job file and spawns tcpdump on the firewall. Starting a capture is a write action on someone else's router and is barred by this component's read-only invariant. Downloading and parsing pcaps into the digest would also mean hand-rolling malware detection over raw packets, which is strictly worse than reading the verdicts of a maintained ruleset that already inspected the same traffic in real time.

If you still want a rotating raw capture for manual inspection, keep it on OPNsense, e.g. a tcpdump -G 600 -W 12 -w /var/log/captures/cap-%F-%H%M.pcap rotation driven from the firewall's own cron (this needs shell access on OPNsense — the GUI cron only schedules predefined configd actions — and enough disk for ten-minute captures of a live link, which is not small). Then put a one-line pointer in IDSconf.json's packet_capture_reference; it is echoed verbatim into the digest so the network section can say "raw captures are at X". The digest engine never downloads, stores or analyses them.

Household data — what exists and what does not

ingest/caldav.py and ingest/grocy.py are what the household section actually runs on. Both are off by default and both need one credential created by hand.

Calendar — Nextcloud over CalDAV (Phase 8), via the maintained caldav library rather than hand-written REPORT XML. Auth is a Nextcloud app password (Settings → Security → Devices & sessions → Create new app password), not the account password and not OAuth2 — mandatory once 2FA is on, since the DAV endpoints cannot prompt for a second factor, and revocable on its own regardless. Point CALDAV_URL at the DAV root (https://<host>/remote.php/dav) and the client discovers the principal's calendars from there.

The window is deliberately asymmetric — DIGEST_LOOKBACK_HOURS backwards, so this morning's appointment and anything still running are still visible, and CALDAV_LOOKAHEAD_HOURS (default 48) forwards, because a calendar is mostly useful in the future tense. Recurring events are requested expanded, so a weekly standup arrives as the occurrence in this window rather than as the master event with an RRULE; if a server rejects expansion outright, the search is retried without it and a recurring series shows up as its master event.

Kitchen inventory — Grocy (Phase 7), reached at http://grocy on the shared compose network (port 80 inside the container; the published 9283 is not involved). One endpoint does most of the work: GET /api/stock/volatile, which returns due_products, overdue_products, expired_products and missing_products directly. Watch the naming — Grocy renamed expiring_products → due_products in v3.0.0, so older third-party examples are wrong against a current install. Chores and batteries come from GET /api/chores and GET /api/batteries; both use 2999-12-31 23:59:59 as a "no schedule" sentinel, which is filtered out rather than reported as a due date.

Auth is a GROCY-API-KEY header, generated at Grocy → Settings → Manage API keys. A Grocy API key is not scoped: it carries that user's full read and write rights, so give this one its own Grocy user, and note that the read-only guarantee is enforced by ingest/grocy.py calling nothing but GETs — the module's docstring names every write endpoint it deliberately does not use — and not by Grocy.

No compose or systemd changes were needed for either. digest-engine and grocy are already on the same default compose network, so the container name resolves; Nextcloud is external and reached over its normal URL; and the credentials are ordinary env vars in the digest-engine.env the service already loads.

The evening recipe suggestion

On one run a day — DIGEST_EVENING_HOUR, default 18 — the household section also suggests a dish built around whatever Grocy says is about to go off, plus a shopping list for the ingredients that dish needs and the house does not have. The other three runs omit it entirely rather than padding it in.

It is a suggestion, and nothing else. Nothing is written to Grocy: no item is added to its shopping list, no stock is consumed, no order is placed anywhere. Grocy's API supports all of that with the same key and this component uses none of it, per the read-only invariant above. You read the list and go shopping.

Which run is "evening" is derived from the container's local wall clock, not passed in by the caller. The systemd unit runs a bare docker compose run --rm digest-engine with no arguments and a manual run is the same command, so an argument or a unit-specific env var would have to be threaded through both and would silently misbehave on a hand-run digest; the container already has the host's TZ and /etc/localtime, which is the same clock the timer's OnCalendar fires against. The run is attributed to the most recent DIGEST_SCHEDULE slot at or before now rather than to an exact hour match, because the timer is Persistent=true — a host asleep at 18:00 fires late, and an exact match would drop the feature on precisely the days the digest is late.

Set DIGEST_FORCE_EVENING=true for a one-off run to test it at any hour. Keep DIGEST_SCHEDULE in step with the variable of the same name in tools/setup-container-host.sh, which is what sets the timer.

Merging an unviewed digest into the next one

If nobody actually looked at a run before the next one was due, its content is folded into the new run instead of being silently thrown away — see viewed_tracker.py, run.py's should_merge()/previous_section_document(), and the merge instruction synth/llm_client.py adds to the prompt when it applies.

"Viewed" means a thin client actually displayed the full canvas — the "Show digest canvas" button or a voice-resolved "play my digest" request, both of which go through thinclient_agent/main.py's on_show_digest(), which publishes a retained {"viewed_at": ...} to smarthome/digest/viewed on the same Mosquitto broker everything else in this project already shares. The compact HA-dashboard iframe view does not count — it's a browser rendering a static page, with no path back to MQTT at all, so leaving it open on a phone can never mark a digest viewed.

Each run compares that timestamp against the previous run's generated_at (output/latest/digest.json). If the previous run is newer than the last time anything was viewed — or nothing has ever been marked viewed, or no previous run exists yet — this run proceeds exactly as before. Otherwise, each section's own previous content (from the full detail level, the richest version) is handed to that section's synthesis pass as previous_unviewed_digest, with an instruction to combine it with the new material into one digest rather than repeating or discarding either — nothing is dropped, but nothing doubles up either.

If MQTT is unreachable, paho-mqtt isn't installed, or the retained message can't be parsed, viewed_tracker.last_viewed_at() returns None, which is treated the same as "viewed" — the safer of the two wrong answers, since it costs at most one merge that should have happened, rather than gluing every future run onto the last forever. MQTT_VIEWED_WAIT_SECONDS (default 3) bounds how long a run will wait for that retained message before moving on, so a dead broker never stalls a digest run.

Not yet run against a real broker or a real thin client — the retained-message round trip, the on_show_digest publish, and a genuine multi-cycle unviewed→merged sequence are all still on the manual-verification list.

The counter run — a final filter against hallucination

Before anything is written to output/, every generated document is checked by a second, independent LLM call (synth/counter_run.py) against the exact same context it was generated from. This is the final filter the plan calls for against false or unsourced information reaching the digest — it is not a substitute for the "No speculation" instructions already in each prompt, it's the backstop for when those instructions don't work.

It checks, per window: is every quotation an actual excerpt of something in the context (not a plausible-sounding invention); is every figure, date, or name traceable to something in the context; does every named theoretical connection (Lenin's imperialism, Marx's labour theory of value, etc.) correspond to a real event the context actually describes that way; does every stated correlation between two data sources actually have both halves present, not one assumed.

This does not mean second-guessing the digest's Marxist framing itself. The counter run shares the same RCI-derived theoretical basis as the document it's checking (see synth/prompts/counter_run.md) — its job is to confirm the underlying facts are real and a theoretical reading of them is a genuine structural match, not to apply a bourgeois-neutral standard of "objectivity" that would flag correct class analysis as unverifiable "opinion." That would smuggle in a different politics than the one this digest is written from, which is exactly the kind of error this pass exists to prevent, not commit.

What happens to something it flags:

A specific window it can't ground is dropped; the rest of the document is kept.
Narration it can't ground is cleared to empty — better silent than a false claim read aloud by the TTS voice.
A quote the model itself claims is "found in context" is also checked mechanically (a plain substring search against the same context), and overridden if it isn't actually there — the one claim type this doesn't have to take the verifying call's own word for.
If every window in a document gets dropped, the whole section is replaced with an honest "withheld pending verification" placeholder rather than shown empty or not at all.
If the counter run can't run at all (Ollama unreachable a second time, an unparseable verdict), the original document is kept but marked unverified — not silently passed through unchecked, and not blanked either, since a transient failure in this pass specifically shouldn't cost as much as the whole digest being down.

This doubles the number of Ollama calls per run (four calls per generated section rather than two — so 16 with all four sections on, and fewer for every section the household has switched off) — against a local, self-hosted model with no per-token cost and nobody waiting on the latency, the same tradeoff synth/llm_client.py already makes for generating compact and full as separate passes rather than truncating one into the other. Set COUNTER_RUN_MODEL if you want verification done by a different (e.g. larger) model than the one that generated the digest.

Not yet run for real — whether the counter-run prompt actually catches a genuinely hallucinated quote, versus over-flagging real ones, needs checking against actual model output before this can be trusted as more than plausible-sounding on paper.

45 KiB Raw Permalink Blame History Unescape Escape