SmartestHome/digest-engine/README.md

# digest-engine

The quarter-daily LLM digest from [Phase 12 of the project plan](../docs/project-plan.md).

Four times a day it ingests mail, messages, news and financial data, sends the
lot to the existing Ollama host for synthesis into three sections — **personal**,
**political** and **household** — and writes a rendered digest that `digest-web`
serves to two surfaces: the thin client's kiosk Firefox workspace (full view) and
a Home Assistant Lovelace iframe card (compact view).

It is a **oneshot**, not a daemon: a systemd timer runs
`docker compose run --rm digest-engine`, exactly like the restic backup job.

**Everything here is read-only.** No replies, no marking mail read or archived, no
calendar or Grocy writes, no message-platform writes of any kind. There is no
mutation path in this component by construction — that is a hard requirement from
the plan, not a default.

## Layout

```
run.py                        oneshot entrypoint
ingest/                       one module per source, each `fetch(lookback_hours) -> list[dict]`
  telegram_login.py           standalone one-time interactive login (run by hand)
synth/llm_client.py           Ollama client + the digest JSON schema
synth/prompts/                one prompt template per section
render/digest-canvas-sdk/     vendored, offline JS/CSS — globe, window chrome, glow, renderer
render/templates/             compact.html (HA iframe) and full.html (kiosk)
feeds/curated-feeds.opml      the news feed list — edit this
IDSconf.json.example          OPNsense IDS config template (real file gitignored)
whatsapp-bridge/              Node.js sidecar, opt-in, see the warning below
output/                       per-run artifacts (gitignored)
compose-fragment.yaml.txt     compose blocks to splice into setup-container-host.sh
```

## Configure

```sh
cp digest-engine/digest-engine.env.example /opt/smart-home/digest/digest-engine.env
chmod 600 /opt/smart-home/digest/digest-engine.env
$EDITOR /opt/smart-home/digest/digest-engine.env
```

Every source is off by default. Turn on only what you have credentials for — a
disabled or misconfigured source logs a warning and contributes nothing, and can
never take the rest of the run down with it.

Then edit `feeds/curated-feeds.opml`: the mainstream outlets in it are a
clearly-marked placeholder list, only the `marxist.com` feed is a deliberate
choice (the political prompt uses it as its analytical basis).

## One-time steps before the first real run

Both of these are interactive and must be done by hand, once. Scheduled runs
never prompt for anything.

**Telegram** — creates the session file `telegram_ingest.py` then reuses
non-interactively. You will be asked for your phone number, the code Telegram
sends, and your 2FA password if the account has one:

```sh
docker compose run --rm --entrypoint python digest-engine ingest/telegram_login.py
```

**WhatsApp** (only if you have opted in) — start the bridge and scan the QR code
it prints to its own logs with **WhatsApp -> Linked devices -> Link a device**:

```sh
docker compose up -d whatsapp-bridge
docker compose logs -f whatsapp-bridge
```

The session persists in the bridge's `/data/.wwebjs_auth` volume, so this is a
one-time scan unless WhatsApp invalidates the link.

## Run once, manually

```sh
docker compose run --rm digest-engine
```

Output lands in `output/<run-timestamp>/`:

- `context.json` — the ingested context bundle, kept for the Phase 12 follow-up
  voice Q&A (a spoken follow-up re-queries Ollama against this rather than
  re-ingesting).
- `digest.json` — the rendered digest, both detail levels, all three sections.

`output/latest.json` is rewritten with the same payload and `output/latest`
re-pointed at the newest run directory, so `digest-web` always serves the current
digest with no coordination with the scheduler.

## WhatsApp — read this before enabling

There is **no officially sanctioned way** to read your own WhatsApp messages
programmatically. `whatsapp-bridge` runs a real Chromium logged into
web.whatsapp.com as a linked device, deliberately **headful** under Xvfb because
WhatsApp's automation detection specifically fingerprints headless Chrome. That
is a meaningful mitigation. It is **not** immunity: this is still automated use of
a personal account and accounts do get banned for it, historically on a ~2–8 week
timescale.

If you enable it:

- Use a **secondary, non-critical number**, not your main one.
- Accept that the number may be banned, and that this is the highest-risk of the
  four message platforms by a wide margin.
- Keep `ENABLE_WHATSAPP_INGEST=false` if you are at all unsure. The other three
  sections work fine without it.

## Manual verification still outstanding

None of this has been run against real credentials or real accounts. Before
trusting a scheduled run, verify by hand:

1. Each source in isolation, e.g.
   `docker compose run --rm --entrypoint python digest-engine -c "import logging,os; logging.basicConfig(level='INFO'); from ingest import news_rss; print(len(news_rss.fetch(6)))"`.
2. That the IMAP mailbox shows **no** newly-read messages after a run (the folder
   is opened `readonly=True`, but confirm it against your provider).
3. That the Ollama model actually honours `format: "json"` and the schema — check
   `output/<run>/digest.json` for `"degraded": true`, which marks a section that
   fell back to plain text.
4. That both templates render: open `http://<host>:8091/full.html` and
   `http://<host>:8091/compact.html`.
5. The malformed-output fallback, by hand-editing `output/latest.json` into
   invalid JSON and reloading — the page must show a `<pre>` dump, never a blank
   screen.
6. The feed URLs in `curated-feeds.opml` — several mainstream outlets have
   changed or restricted their public RSS.
7. That the Nextcloud app password works over CalDAV and that recurring events
   land on the right day (the expansion path above is the one most likely to
   differ between Nextcloud versions).
8. The evening recipe, with `DIGEST_FORCE_EVENING=true` on a manual run — then
   confirm in Grocy that **nothing** was added to its shopping list and no stock
   moved.
9. The unviewed-digest merge: run once, do NOT show it on a thin client, run again
   (or set `DIGEST_LOOKBACK_HOURS`/wait) — confirm the second run's `context.json`
   has `"merged_unviewed_previous_run": true` and its digest actually carries
   forward the first run's content. Then show a digest on a thin client and run a
   third time — confirm that one merges nothing.
10. The counter run, deliberately: hand-edit a generated document before it's
    written (or patch `synth/counter_run.py` to run against a document with a
    fabricated quote spliced in) and confirm it actually gets dropped, not waved
    through — then confirm a real, correctly-grounded piece of Marxist analysis
    (a genuine merger analyzed via Lenin's imperialism) is NOT flagged just for
    being theoretical rather than a bare fact.

## Traffic data — what exists and what does not

`ingest/flight_traffic.py` and `ingest/naval_traffic.py` feed the **political**
section as extra `category`-tagged evidence, alongside news and financial data.
They are not a new digest section, and `synth/prompts/political.md` is explicitly
told to drop them when they corroborate nothing. Both are off by default.

**Air traffic** — OpenSky Network, bounded boxes over configured regions.
FlightRadar24 and ADS-B Exchange were not used: FR24 prohibits scraping and sells
API access, and ADS-B Exchange ended its freemium RapidAPI tier on 2025-03-01
(paid from $10/month). Before enabling, read the licensing note in
`digest-engine.env.example` — OpenSky's Terms of Use require a prior written
agreement for use of the REST API "in any operational capacity", which arguably
covers a timer-driven digest. Attribution to OpenSky is required.

**Naval traffic** — aisstream.io, a free keyed WebSocket stream, sampled briefly
per run. Read what it is for before enabling it: **AIS cannot show naval force
posture.** Warships sail with AIS off routinely, and the "military ops" AIS type
code is self-declared. What it shows is *merchant* traffic through chokepoints,
which is genuinely useful in the negative — shipping abandoning a route arrives
as freight, insurance and fuel costs. AISHub was not used: it is still
contribute-to-access and needs an AIS receiver this project does not have.
MarineTraffic, VesselFinder and Spire are paid.

**Military movement — deliberately not built as a data source.** There is no free
structured feed of military movements. ACLED is retrospective conflict-*event*
data, needs registration, and its EULA forbids redistribution and non-transformative
derivative works; UCDP is keyless but lags by roughly a month; everything
real-time is either commercial or a person on social media. Rather than invent an
integration, the military-movement signal comes from OSINT/defence outlets added
to `feeds/curated-feeds.opml` under `category="osint_military"`, which reuses the
existing news ingestion with no new code. Verified live 2026-07-28; Liveuamap has
no free RSS feed (`/rss` is a paid-API signup page) and ISW and Long War Journal
both 403'd the check — re-test those from the real network.

## Home network intrusion detection — what exists and what does not

`ingest/opnsense_ids.py` pulls a summary of the Suricata alerts your **existing
OPNsense firewall** raised during the digest window and folds it into the
**household** section as one short item, tagged `"category":
"network_security"`. Off by default (`ENABLE_OPNSENSE_IDS_INGEST=false`);
configured by `IDSconf.json` (template: `IDSconf.json.example`, real file
gitignored, same pattern as `digest-engine.env`).

**Suricata is core, not a plugin.** There is no `os-suricata` to install — the
IDS module ships in OPNsense core and lives at Services → Intrusion Detection.
The `os-intrusion-detection-content-*` plugins are ruleset *content* only, and
ET Open needs none of them. Suricata is nonetheless **off** on a stock install;
enable it and download a ruleset first, or every run will report `ids_status`
and no alerts, which is the honest answer and not a quiet network.

**It is a pull, like every other source here.** Two endpoints, both read-only in
effect: `GET /api/ids/service/status` and `POST /api/ids/service/query_alerts`.
The latter is a POST only because that is how OPNsense routes filtered queries —
it runs `queryAlertLog.py`, which reads `/var/log/suricata/eve.json` backwards.
No SSH or file access to the firewall is needed, and no push agent runs on it.

Three limits worth knowing before you read the output:

- **No server-side time filter.** `searchPhrase` matches signature/action/src/dst
  text only. Rows come back newest-first, so the window is applied client-side by
  paging until a row falls out of it, bounded by `max_alerts_scanned`. When that
  bound is hit — or when the alert log rotated mid-window — the entry carries
  `window_truncated: true` and the prompt is told the counts are a lower bound.
- **No severity.** OPNsense flattens each eve.json record to signature + SID +
  action before returning it, discarding `alert.severity` and `alert.category`.
  `get_alert_info` uses the same flattening, so it does not help. Alerts are
  ranked by frequency, and the household prompt is told it cannot see severity.
- **Page-level ACLs.** The "Services: Intrusion Detection" privilege matches
  `api/ids/*`, which covers start/stop/reconfigure/drop-alert-log as well as the
  alert query. OPNsense has no narrower built-in privilege, so the read-only
  guarantee is enforced by this code (which calls two endpoints and no others)
  and not by the firewall. Give the API key its own user with that one privilege
  and nothing else, and treat it as a credential that could restart your IDS if
  it leaked — the container host and the firewall are on the same flat LAN, since
  no VLAN segmentation is implemented in this project yet.

**Raw packet captures — deliberately not done here.** OPNsense does expose
Interfaces: Diagnostics: Packet Capture over the API
(`/api/diagnostics/packet_capture/{set,start,stop,remove}`), but every one of
those is a POST that writes a job file and spawns `tcpdump` on the firewall.
Starting a capture is a write action on someone else's router and is barred by
this component's read-only invariant. Downloading and parsing pcaps into the
digest would also mean hand-rolling malware detection over raw packets, which is
strictly worse than reading the verdicts of a maintained ruleset that already
inspected the same traffic in real time.

If you still want a rotating raw capture for manual inspection, keep it **on
OPNsense**, e.g. a `tcpdump -G 600 -W 12 -w /var/log/captures/cap-%F-%H%M.pcap`
rotation driven from the firewall's own cron (this needs shell access on
OPNsense — the GUI cron only schedules predefined configd actions — and enough
disk for ten-minute captures of a live link, which is not small). Then put a
one-line pointer in `IDSconf.json`'s `packet_capture_reference`; it is echoed
verbatim into the digest so the household section can say "raw captures are at
X". The digest engine never downloads, stores or analyses them.

## Household data — what exists and what does not

`ingest/caldav.py` and `ingest/grocy.py` are what the household section actually
runs on. Both are off by default and both need one credential created by hand.

**Calendar** — Nextcloud over CalDAV (Phase 8), via the maintained `caldav`
library rather than hand-written REPORT XML. Auth is a **Nextcloud app
password** (Settings → Security → Devices & sessions → Create new app password),
not the account password and not OAuth2 — mandatory once 2FA is on, since the DAV
endpoints cannot prompt for a second factor, and revocable on its own regardless.
Point `CALDAV_URL` at the DAV root (`https://<host>/remote.php/dav`) and the
client discovers the principal's calendars from there.

The window is deliberately asymmetric — `DIGEST_LOOKBACK_HOURS` backwards, so
this morning's appointment and anything still running are still visible, and
`CALDAV_LOOKAHEAD_HOURS` (default 48) forwards, because a calendar is mostly
useful in the future tense. Recurring events are requested **expanded**, so a
weekly standup arrives as the occurrence in this window rather than as the master
event with an RRULE; if a server rejects expansion outright, the search is
retried without it and a recurring series shows up as its master event.

**Kitchen inventory** — Grocy (Phase 7), reached at `http://grocy` on the shared
compose network (port 80 inside the container; the published 9283 is not
involved). One endpoint does most of the work: `GET /api/stock/volatile`, which
returns `due_products`, `overdue_products`, `expired_products` and
`missing_products` directly. **Watch the naming** — Grocy renamed
`expiring_products` → `due_products` in v3.0.0, so older third-party examples are
wrong against a current install. Chores and batteries come from `GET /api/chores`
and `GET /api/batteries`; both use `2999-12-31 23:59:59` as a "no schedule"
sentinel, which is filtered out rather than reported as a due date.

Auth is a `GROCY-API-KEY` header, generated at Grocy → Settings → Manage API
keys. **A Grocy API key is not scoped**: it carries that user's full read *and*
write rights, so give this one its own Grocy user, and note that the read-only
guarantee is enforced by `ingest/grocy.py` calling nothing but GETs — the
module's docstring names every write endpoint it deliberately does not use — and
not by Grocy.

**No compose or systemd changes were needed for either.** digest-engine and
grocy are already on the same default compose network, so the container name
resolves; Nextcloud is external and reached over its normal URL; and the
credentials are ordinary env vars in the `digest-engine.env` the service already
loads.

## The evening recipe suggestion

On one run a day — `DIGEST_EVENING_HOUR`, default `18` — the household section
also suggests a dish built around whatever Grocy says is about to go off, plus a
shopping list for the ingredients that dish needs and the house does not have.
The other three runs omit it entirely rather than padding it in.

**It is a suggestion, and nothing else.** Nothing is written to Grocy: no item is
added to its shopping list, no stock is consumed, no order is placed anywhere.
Grocy's API supports all of that with the same key and this component uses none
of it, per the read-only invariant above. You read the list and go shopping.

Which run is "evening" is derived from the container's local wall clock, not
passed in by the caller. The systemd unit runs a bare
`docker compose run --rm digest-engine` with no arguments and a manual run is the
same command, so an argument or a unit-specific env var would have to be threaded
through both and would silently misbehave on a hand-run digest; the container
already has the host's `TZ` and `/etc/localtime`, which is the same clock the
timer's `OnCalendar` fires against. The run is attributed to the most recent
`DIGEST_SCHEDULE` slot at or before now rather than to an exact hour match,
because the timer is `Persistent=true` — a host asleep at 18:00 fires late, and
an exact match would drop the feature on precisely the days the digest is late.

Set `DIGEST_FORCE_EVENING=true` for a one-off run to test it at any hour.
Keep `DIGEST_SCHEDULE` in step with the variable of the same name in
`hosts/container-host/scripts/setup-container-host.sh`, which is what sets the
timer.

## Merging an unviewed digest into the next one

If nobody actually looked at a run before the next one was due, its content is
folded into the new run instead of being silently thrown away — see
`viewed_tracker.py`, `run.py`'s `should_merge()`/`previous_section_document()`, and
the merge instruction `synth/llm_client.py` adds to the prompt when it applies.

**"Viewed" means a thin client actually displayed the full canvas** — the "Show
digest canvas" button or a voice-resolved "play my digest" request, both of which go
through `thinclient_agent/main.py`'s `on_show_digest()`, which publishes a retained
`{"viewed_at": ...}` to `smarthome/digest/viewed` on the same Mosquitto broker
everything else in this project already shares. **The compact HA-dashboard iframe
view does not count** — it's a browser rendering a static page, with no path back to
MQTT at all, so leaving it open on a phone can never mark a digest viewed.

Each run compares that timestamp against the previous run's `generated_at`
(`output/latest/digest.json`). If the previous run is newer than the last time
anything was viewed — or nothing has ever been marked viewed, or no previous run
exists yet — this run proceeds exactly as before. Otherwise, each section's own
previous content (from the `full` detail level, the richest version) is handed to
that section's synthesis pass as `previous_unviewed_digest`, with an instruction to
combine it with the new material into one digest rather than repeating or discarding
either — nothing is dropped, but nothing doubles up either.

If MQTT is unreachable, `paho-mqtt` isn't installed, or the retained message can't be
parsed, `viewed_tracker.last_viewed_at()` returns `None`, which is treated the same
as "viewed" — the safer of the two wrong answers, since it costs at most one merge
that should have happened, rather than gluing every future run onto the last
forever. `MQTT_VIEWED_WAIT_SECONDS` (default 3) bounds how long a run will wait for
that retained message before moving on, so a dead broker never stalls a digest run.

**Not yet run against a real broker or a real thin client** — the retained-message
round trip, the `on_show_digest` publish, and a genuine multi-cycle unviewed→merged
sequence are all still on the manual-verification list.

## The counter run — a final filter against hallucination

Before anything is written to `output/`, every generated document is checked
by a second, independent LLM call (`synth/counter_run.py`) against the exact
same context it was generated from. This is the final filter the plan calls
for against false or unsourced information reaching the digest — it is not a
substitute for the "No speculation" instructions already in each prompt, it's
the backstop for when those instructions don't work.

It checks, per window: is every quotation an actual excerpt of something in
the context (not a plausible-sounding invention); is every figure, date, or
name traceable to something in the context; does every named theoretical
connection (Lenin's imperialism, Marx's labour theory of value, etc.)
correspond to a real event the context actually describes that way; does
every stated correlation between two data sources actually have both halves
present, not one assumed.

**This does not mean second-guessing the digest's Marxist framing itself.**
The counter run shares the same RCI-derived theoretical basis as the document
it's checking (see `synth/prompts/counter_run.md`) — its job is to confirm the
underlying facts are real and a theoretical reading of them is a genuine
structural match, not to apply a bourgeois-neutral standard of "objectivity"
that would flag correct class analysis as unverifiable "opinion." That would
smuggle in a different politics than the one this digest is written from,
which is exactly the kind of error this pass exists to prevent, not commit.

What happens to something it flags:

- A specific window it can't ground is dropped; the rest of the document is
  kept.
- Narration it can't ground is cleared to empty — better silent than a false
  claim read aloud by the TTS voice.
- A quote the model itself claims is "found in context" is also checked
  mechanically (a plain substring search against the same context), and
  overridden if it isn't actually there — the one claim type this doesn't
  have to take the verifying call's own word for.
- If every window in a document gets dropped, the whole section is replaced
  with an honest "withheld pending verification" placeholder rather than
  shown empty or not at all.
- If the counter run can't run at all (Ollama unreachable a second time, an
  unparseable verdict), the original document is kept but marked
  `unverified` — not silently passed through unchecked, and not blanked
  either, since a transient failure in this pass specifically shouldn't cost
  as much as the whole digest being down.

This doubles the number of Ollama calls per run (12 instead of 6) — against a
local, self-hosted model with no per-token cost and nobody waiting on the
latency, the same tradeoff `synth/llm_client.py` already makes for generating
`compact` and `full` as separate passes rather than truncating one into the
other. Set `COUNTER_RUN_MODEL` if you want verification done by a different
(e.g. larger) model than the one that generated the digest.

**Not yet run for real** — whether the counter-run prompt actually catches a
genuinely hallucinated quote, versus over-flagging real ones, needs checking
against actual model output before this can be trusted as more than
plausible-sounding on paper.