SmartestHome/digest-engine

digest-engine

The quarter-daily LLM digest from Phase 12 of the project plan.

Four times a day it ingests mail, messages, news and financial data, sends the lot to the existing Ollama host for synthesis into three sections — personal, political and household — and writes a rendered digest that digest-web serves to two surfaces: the thin client's kiosk Firefox workspace (full view) and a Home Assistant Lovelace iframe card (compact view).

It is a oneshot, not a daemon: a systemd timer runs docker compose run --rm digest-engine, exactly like the restic backup job.

Everything here is read-only. No replies, no marking mail read or archived, no calendar or Grocy writes, no message-platform writes of any kind. There is no mutation path in this component by construction — that is a hard requirement from the plan, not a default.

Layout

run.py                        oneshot entrypoint
ingest/                       one module per source, each `fetch(lookback_hours) -> list[dict]`
  telegram_login.py           standalone one-time interactive login (run by hand)
synth/llm_client.py           Ollama client + the digest JSON schema
synth/prompts/                one prompt template per section
render/digest-canvas-sdk/     vendored, offline JS/CSS — globe, window chrome, glow, renderer
render/templates/             compact.html (HA iframe) and full.html (kiosk)
feeds/curated-feeds.opml      the news feed list — edit this
IDSconf.json.example          OPNsense IDS config template (real file gitignored)
whatsapp-bridge/              Node.js sidecar, opt-in, see the warning below
output/                       per-run artifacts (gitignored)
compose-fragment.yaml.txt     compose blocks to splice into setup-container-host.sh

Configure

cp digest-engine/digest-engine.env.example /opt/smart-home/digest/digest-engine.env
chmod 600 /opt/smart-home/digest/digest-engine.env
$EDITOR /opt/smart-home/digest/digest-engine.env

Every source is off by default. Turn on only what you have credentials for — a disabled or misconfigured source logs a warning and contributes nothing, and can never take the rest of the run down with it.

Then edit feeds/curated-feeds.opml: the mainstream outlets in it are a clearly-marked placeholder list, only the marxist.com feed is a deliberate choice (the political prompt uses it as its analytical basis).

One-time steps before the first real run

Both of these are interactive and must be done by hand, once. Scheduled runs never prompt for anything.

Telegram — creates the session file telegram_ingest.py then reuses non-interactively. You will be asked for your phone number, the code Telegram sends, and your 2FA password if the account has one:

docker compose run --rm --entrypoint python digest-engine ingest/telegram_login.py

WhatsApp (only if you have opted in) — start the bridge and scan the QR code it prints to its own logs with WhatsApp -> Linked devices -> Link a device:

docker compose up -d whatsapp-bridge
docker compose logs -f whatsapp-bridge

The session persists in the bridge's /data/.wwebjs_auth volume, so this is a one-time scan unless WhatsApp invalidates the link.

Run once, manually

docker compose run --rm digest-engine

Output lands in output/<run-timestamp>/:

• context.json — the ingested context bundle, kept for the Phase 12 follow-up voice Q&A (a spoken follow-up re-queries Ollama against this rather than re-ingesting).
• digest.json — the rendered digest, both detail levels, all three sections.

output/latest.json is rewritten with the same payload and output/latest re-pointed at the newest run directory, so digest-web always serves the current digest with no coordination with the scheduler.

WhatsApp — read this before enabling

There is no officially sanctioned way to read your own WhatsApp messages programmatically. whatsapp-bridge runs a real Chromium logged into web.whatsapp.com as a linked device, deliberately headful under Xvfb because WhatsApp's automation detection specifically fingerprints headless Chrome. That is a meaningful mitigation. It is not immunity: this is still automated use of a personal account and accounts do get banned for it, historically on a ~2–8 week timescale.

If you enable it:

• Use a secondary, non-critical number, not your main one.
• Accept that the number may be banned, and that this is the highest-risk of the four message platforms by a wide margin.
• Keep ENABLE_WHATSAPP_INGEST=false if you are at all unsure. The other three sections work fine without it.

Manual verification still outstanding

None of this has been run against real credentials or real accounts. Before trusting a scheduled run, verify by hand:

1. Each source in isolation, e.g. docker compose run --rm --entrypoint python digest-engine -c "import logging,os; logging.basicConfig(level='INFO'); from ingest import news_rss; print(len(news_rss.fetch(6)))".
2. That the IMAP mailbox shows no newly-read messages after a run (the folder is opened readonly=True, but confirm it against your provider).
3. That the Ollama model actually honours format: "json" and the schema — check output/<run>/digest.json for "degraded": true, which marks a section that fell back to plain text.
4. That both templates render: open http://<host>:8091/full.html and http://<host>:8091/compact.html.
5. The malformed-output fallback, by hand-editing output/latest.json into invalid JSON and reloading — the page must show a <pre> dump, never a blank screen.
6. The feed URLs in curated-feeds.opml — several mainstream outlets have changed or restricted their public RSS.
7. That the Nextcloud app password works over CalDAV and that recurring events land on the right day (the expansion path above is the one most likely to differ between Nextcloud versions).
8. The evening recipe, with DIGEST_FORCE_EVENING=true on a manual run — then confirm in Grocy that nothing was added to its shopping list and no stock moved.
9. The unviewed-digest merge: run once, do NOT show it on a thin client, run again (or set DIGEST_LOOKBACK_HOURS/wait) — confirm the second run's context.json has "merged_unviewed_previous_run": true and its digest actually carries forward the first run's content. Then show a digest on a thin client and run a third time — confirm that one merges nothing.
10. The counter run, deliberately: hand-edit a generated document before it's written (or patch synth/counter_run.py to run against a document with a fabricated quote spliced in) and confirm it actually gets dropped, not waved through — then confirm a real, correctly-grounded piece of Marxist analysis (a genuine merger analyzed via Lenin's imperialism) is NOT flagged just for being theoretical rather than a bare fact.

Traffic data — what exists and what does not

ingest/flight_traffic.py and ingest/naval_traffic.py feed the political section as extra category-tagged evidence, alongside news and financial data. They are not a new digest section, and synth/prompts/political.md is explicitly told to drop them when they corroborate nothing. Both are off by default.

Air traffic — OpenSky Network, bounded boxes over configured regions. FlightRadar24 and ADS-B Exchange were not used: FR24 prohibits scraping and sells API access, and ADS-B Exchange ended its freemium RapidAPI tier on 2025-03-01 (paid from $10/month). Before enabling, read the licensing note in digest-engine.env.example — OpenSky's Terms of Use require a prior written agreement for use of the REST API "in any operational capacity", which arguably covers a timer-driven digest. Attribution to OpenSky is required.

Naval traffic — aisstream.io, a free keyed WebSocket stream, sampled briefly per run. Read what it is for before enabling it: AIS cannot show naval force posture. Warships sail with AIS off routinely, and the "military ops" AIS type code is self-declared. What it shows is merchant traffic through chokepoints, which is genuinely useful in the negative — shipping abandoning a route arrives as freight, insurance and fuel costs. AISHub was not used: it is still contribute-to-access and needs an AIS receiver this project does not have. MarineTraffic, VesselFinder and Spire are paid.

Military movement — deliberately not built as a data source. There is no free structured feed of military movements. ACLED is retrospective conflict-event data, needs registration, and its EULA forbids redistribution and non-transformative derivative works; UCDP is keyless but lags by roughly a month; everything real-time is either commercial or a person on social media. Rather than invent an integration, the military-movement signal comes from OSINT/defence outlets added to feeds/curated-feeds.opml under category="osint_military", which reuses the existing news ingestion with no new code. Verified live 2026-07-28; Liveuamap has no free RSS feed (/rss is a paid-API signup page) and ISW and Long War Journal both 403'd the check — re-test those from the real network.

Home network intrusion detection — what exists and what does not

ingest/opnsense_ids.py pulls a summary of the Suricata alerts your existing OPNsense firewall raised during the digest window and folds it into the household section as one short item, tagged "category": "network_security". Off by default (ENABLE_OPNSENSE_IDS_INGEST=false); configured by IDSconf.json (template: IDSconf.json.example, real file gitignored, same pattern as digest-engine.env).

Suricata is core, not a plugin. There is no os-suricata to install — the IDS module ships in OPNsense core and lives at Services → Intrusion Detection. The os-intrusion-detection-content-* plugins are ruleset content only, and ET Open needs none of them. Suricata is nonetheless off on a stock install; enable it and download a ruleset first, or every run will report ids_status and no alerts, which is the honest answer and not a quiet network.

It is a pull, like every other source here. Two endpoints, both read-only in effect: GET /api/ids/service/status and POST /api/ids/service/query_alerts. The latter is a POST only because that is how OPNsense routes filtered queries — it runs queryAlertLog.py, which reads /var/log/suricata/eve.json backwards. No SSH or file access to the firewall is needed, and no push agent runs on it.

Three limits worth knowing before you read the output:

• No server-side time filter. searchPhrase matches signature/action/src/dst text only. Rows come back newest-first, so the window is applied client-side by paging until a row falls out of it, bounded by max_alerts_scanned. When that bound is hit — or when the alert log rotated mid-window — the entry carries window_truncated: true and the prompt is told the counts are a lower bound.
• No severity. OPNsense flattens each eve.json record to signature + SID + action before returning it, discarding alert.severity and alert.category. get_alert_info uses the same flattening, so it does not help. Alerts are ranked by frequency, and the household prompt is told it cannot see severity.
• Page-level ACLs. The "Services: Intrusion Detection" privilege matches api/ids/*, which covers start/stop/reconfigure/drop-alert-log as well as the alert query. OPNsense has no narrower built-in privilege, so the read-only guarantee is enforced by this code (which calls two endpoints and no others) and not by the firewall. Give the API key its own user with that one privilege and nothing else, and treat it as a credential that could restart your IDS if it leaked — the container host and the firewall are on the same flat LAN, since no VLAN segmentation is implemented in this project yet.

Raw packet captures — deliberately not done here. OPNsense does expose Interfaces: Diagnostics: Packet Capture over the API (/api/diagnostics/packet_capture/{set,start,stop,remove}), but every one of those is a POST that writes a job file and spawns tcpdump on the firewall. Starting a capture is a write action on someone else's router and is barred by this component's read-only invariant. Downloading and parsing pcaps into the digest would also mean hand-rolling malware detection over raw packets, which is strictly worse than reading the verdicts of a maintained ruleset that already inspected the same traffic in real time.

If you still want a rotating raw capture for manual inspection, keep it on OPNsense, e.g. a tcpdump -G 600 -W 12 -w /var/log/captures/cap-%F-%H%M.pcap rotation driven from the firewall's own cron (this needs shell access on OPNsense — the GUI cron only schedules predefined configd actions — and enough disk for ten-minute captures of a live link, which is not small). Then put a one-line pointer in IDSconf.json's packet_capture_reference; it is echoed verbatim into the digest so the household section can say "raw captures are at X". The digest engine never downloads, stores or analyses them.

Household data — what exists and what does not

ingest/caldav.py and ingest/grocy.py are what the household section actually runs on. Both are off by default and both need one credential created by hand.

Calendar — Nextcloud over CalDAV (Phase 8), via the maintained caldav library rather than hand-written REPORT XML. Auth is a Nextcloud app password (Settings → Security → Devices & sessions → Create new app password), not the account password and not OAuth2 — mandatory once 2FA is on, since the DAV endpoints cannot prompt for a second factor, and revocable on its own regardless. Point CALDAV_URL at the DAV root (https://<host>/remote.php/dav) and the client discovers the principal's calendars from there.

The window is deliberately asymmetric — DIGEST_LOOKBACK_HOURS backwards, so this morning's appointment and anything still running are still visible, and CALDAV_LOOKAHEAD_HOURS (default 48) forwards, because a calendar is mostly useful in the future tense. Recurring events are requested expanded, so a weekly standup arrives as the occurrence in this window rather than as the master event with an RRULE; if a server rejects expansion outright, the search is retried without it and a recurring series shows up as its master event.

Kitchen inventory — Grocy (Phase 7), reached at http://grocy on the shared compose network (port 80 inside the container; the published 9283 is not involved). One endpoint does most of the work: GET /api/stock/volatile, which returns due_products, overdue_products, expired_products and missing_products directly. Watch the naming — Grocy renamed expiring_products → due_products in v3.0.0, so older third-party examples are wrong against a current install. Chores and batteries come from GET /api/chores and GET /api/batteries; both use 2999-12-31 23:59:59 as a "no schedule" sentinel, which is filtered out rather than reported as a due date.

Auth is a GROCY-API-KEY header, generated at Grocy → Settings → Manage API keys. A Grocy API key is not scoped: it carries that user's full read and write rights, so give this one its own Grocy user, and note that the read-only guarantee is enforced by ingest/grocy.py calling nothing but GETs — the module's docstring names every write endpoint it deliberately does not use — and not by Grocy.

No compose or systemd changes were needed for either. digest-engine and grocy are already on the same default compose network, so the container name resolves; Nextcloud is external and reached over its normal URL; and the credentials are ordinary env vars in the digest-engine.env the service already loads.

The evening recipe suggestion

On one run a day — DIGEST_EVENING_HOUR, default 18 — the household section also suggests a dish built around whatever Grocy says is about to go off, plus a shopping list for the ingredients that dish needs and the house does not have. The other three runs omit it entirely rather than padding it in.

It is a suggestion, and nothing else. Nothing is written to Grocy: no item is added to its shopping list, no stock is consumed, no order is placed anywhere. Grocy's API supports all of that with the same key and this component uses none of it, per the read-only invariant above. You read the list and go shopping.

Which run is "evening" is derived from the container's local wall clock, not passed in by the caller. The systemd unit runs a bare docker compose run --rm digest-engine with no arguments and a manual run is the same command, so an argument or a unit-specific env var would have to be threaded through both and would silently misbehave on a hand-run digest; the container already has the host's TZ and /etc/localtime, which is the same clock the timer's OnCalendar fires against. The run is attributed to the most recent DIGEST_SCHEDULE slot at or before now rather than to an exact hour match, because the timer is Persistent=true — a host asleep at 18:00 fires late, and an exact match would drop the feature on precisely the days the digest is late.

Set DIGEST_FORCE_EVENING=true for a one-off run to test it at any hour. Keep DIGEST_SCHEDULE in step with the variable of the same name in tools/setup-container-host.sh, which is what sets the timer.

Merging an unviewed digest into the next one

If nobody actually looked at a run before the next one was due, its content is folded into the new run instead of being silently thrown away — see viewed_tracker.py, run.py's should_merge()/previous_section_document(), and the merge instruction synth/llm_client.py adds to the prompt when it applies.

"Viewed" means a thin client actually displayed the full canvas — the "Show digest canvas" button or a voice-resolved "play my digest" request, both of which go through thinclient_agent/main.py's on_show_digest(), which publishes a retained {"viewed_at": ...} to smarthome/digest/viewed on the same Mosquitto broker everything else in this project already shares. The compact HA-dashboard iframe view does not count — it's a browser rendering a static page, with no path back to MQTT at all, so leaving it open on a phone can never mark a digest viewed.

Each run compares that timestamp against the previous run's generated_at (output/latest/digest.json). If the previous run is newer than the last time anything was viewed — or nothing has ever been marked viewed, or no previous run exists yet — this run proceeds exactly as before. Otherwise, each section's own previous content (from the full detail level, the richest version) is handed to that section's synthesis pass as previous_unviewed_digest, with an instruction to combine it with the new material into one digest rather than repeating or discarding either — nothing is dropped, but nothing doubles up either.

If MQTT is unreachable, paho-mqtt isn't installed, or the retained message can't be parsed, viewed_tracker.last_viewed_at() returns None, which is treated the same as "viewed" — the safer of the two wrong answers, since it costs at most one merge that should have happened, rather than gluing every future run onto the last forever. MQTT_VIEWED_WAIT_SECONDS (default 3) bounds how long a run will wait for that retained message before moving on, so a dead broker never stalls a digest run.

Not yet run against a real broker or a real thin client — the retained-message round trip, the on_show_digest publish, and a genuine multi-cycle unviewed→merged sequence are all still on the manual-verification list.

The counter run — a final filter against hallucination

Before anything is written to output/, every generated document is checked by a second, independent LLM call (synth/counter_run.py) against the exact same context it was generated from. This is the final filter the plan calls for against false or unsourced information reaching the digest — it is not a substitute for the "No speculation" instructions already in each prompt, it's the backstop for when those instructions don't work.

It checks, per window: is every quotation an actual excerpt of something in the context (not a plausible-sounding invention); is every figure, date, or name traceable to something in the context; does every named theoretical connection (Lenin's imperialism, Marx's labour theory of value, etc.) correspond to a real event the context actually describes that way; does every stated correlation between two data sources actually have both halves present, not one assumed.

This does not mean second-guessing the digest's Marxist framing itself. The counter run shares the same RCI-derived theoretical basis as the document it's checking (see synth/prompts/counter_run.md) — its job is to confirm the underlying facts are real and a theoretical reading of them is a genuine structural match, not to apply a bourgeois-neutral standard of "objectivity" that would flag correct class analysis as unverifiable "opinion." That would smuggle in a different politics than the one this digest is written from, which is exactly the kind of error this pass exists to prevent, not commit.

What happens to something it flags:

• A specific window it can't ground is dropped; the rest of the document is kept.
• Narration it can't ground is cleared to empty — better silent than a false claim read aloud by the TTS voice.
• A quote the model itself claims is "found in context" is also checked mechanically (a plain substring search against the same context), and overridden if it isn't actually there — the one claim type this doesn't have to take the verifying call's own word for.
• If every window in a document gets dropped, the whole section is replaced with an honest "withheld pending verification" placeholder rather than shown empty or not at all.
• If the counter run can't run at all (Ollama unreachable a second time, an unparseable verdict), the original document is kept but marked unverified — not silently passed through unchecked, and not blanked either, since a transient failure in this pass specifically shouldn't cost as much as the whole digest being down.

This doubles the number of Ollama calls per run (12 instead of 6) — against a local, self-hosted model with no per-token cost and nobody waiting on the latency, the same tradeoff synth/llm_client.py already makes for generating compact and full as separate passes rather than truncating one into the other. Set COUNTER_RUN_MODEL if you want verification done by a different (e.g. larger) model than the one that generated the digest.

Not yet run for real — whether the counter-run prompt actually catches a genuinely hallucinated quote, versus over-flagging real ones, needs checking against actual model output before this can be trusted as more than plausible-sounding on paper.