SmartestHome/digest-engine/README.md

# digest-engine

The quarter-daily LLM digest from [Phase 12 of the project plan](../docs/project-plan.md).

Four times a day it ingests mail, messages, news and financial data, sends the
lot to the existing Ollama host for synthesis into up to four sections —
**personal** (social), **political** (news), **household** and **network** — and
writes a rendered digest that `digest-web` serves to two surfaces: the thin
client's kiosk Firefox workspace (full view) and a Home Assistant Lovelace iframe
card (compact view).

Which of the four a run actually generates is the household's choice, per person
— see [Who gets which digest](#who-gets-which-digest) below.

It is a **oneshot**, not a daemon: a systemd timer runs
`docker compose run --rm digest-engine`, exactly like the restic backup job.

**Everything here is read-only.** No replies, no marking mail read or archived, no
calendar or Grocy writes, no message-platform writes of any kind. There is no
mutation path in this component by construction — that is a hard requirement from
the plan, not a default.

## Layout

```
run.py                        oneshot entrypoint
preferences.py                who wants which digest, read from identity
archive.py                    the long memory — every item and number, kept across runs
agenda.py                     a Tagesordnung PDF -> the meeting it belongs to
ingest/                       one module per source, each `fetch(lookback_hours) -> list[dict]`
  telegram_login.py           standalone one-time interactive login (run by hand)
synth/llm_client.py           Ollama client + the digest JSON schema
synth/prompts/                one prompt template per section
render/digest-canvas-sdk/     vendored, offline JS/CSS — globe, window chrome, glow, renderer
render/templates/             compact.html (HA iframe) and full.html (kiosk)
feeds/curated-feeds.opml      the news feed list — edit this
feeds/rci-social.json         the RCI/section social + podcast accounts — edit this
IDSconf.json.example          OPNsense IDS config template (real file gitignored)
whatsapp-bridge/              Node.js sidecar, opt-in, see the warning below
output/                       per-run artifacts (gitignored)
compose-fragment.yaml.txt     compose blocks to splice into setup-container-host.sh
```

## Configure

```sh
cp digest-engine/digest-engine.env.example /opt/smart-home/digest/digest-engine.env
chmod 600 /opt/smart-home/digest/digest-engine.env
$EDITOR /opt/smart-home/digest/digest-engine.env
```

Every source is off by default. Turn on only what you have credentials for — a
disabled or misconfigured source logs a warning and contributes nothing, and can
never take the rest of the run down with it.

Then edit `feeds/curated-feeds.opml`: the mainstream outlets in it are a
clearly-marked placeholder list, only the `marxist.com` feed is a deliberate
choice (the political prompt uses it as its analytical basis).

## Who gets which digest

Every person in `identity` has their own set of the four sections, ticked in the
admin panel's person editor (**Digests → Generate for this person**). Nothing
else about the run changes; what changes is how much of it happens at all.

- **A section nobody has ticked is never generated.** No synthesis call, no
  counter-run call, and — because `run.py`'s `SECTION_SOURCES` knows which
  sources feed which section — no ingestion either. Turn the political section
  off for the whole household and the run stops fetching news, financial data
  and flight/naval traffic entirely. The saving is real WAN egress, not just
  tokens.
- **Each surface then shows a person their own subset.** The digest carries a
  `people` list (name, nickname, id, sections) that the renderer filters on when
  it is given `?person=`.
- **The default is all four**, so a household that never opens this panel gets
  exactly the digest it had before this setting existed.

**The display half is a filter, not an access control.** `digest-web` serves the
whole output volume read-only to anything on the LAN, so an unticked section is
off somebody's screen and out of their narration — it is not hidden from them.
The half that genuinely does not exist is the half that was never generated.

`?person=` is a name, nickname or id, and it is only ever set by a caller that
has **already** resolved who is asking:
`hosts/thin-client/agent/thinclient_agent/digest_canvas.py` passes it from a
Home-Assistant-resolved request, per the plan's Phase 11.8 rule that the
personal section is never shown on a guess.

**The canvas is voice-activated, and the person is recognised automatically.**
Nothing shows a digest because somebody walked past a screen. When a spoken "play
my digest" fires, HA asks `identity`'s `GET /speaker?area=<area>` who is in that
room — BLE identity plus the most recent Frigate face sighting — and passes the
answer through as `?person=`. When it cannot tell, it says so and the canvas falls
back to everything-but-personal, which is the same rule as no person at all. So
on `full.html` (the kiosk):

| URL | What renders |
|---|---|
| `full.html?person=Amir` | exactly the sections Amir ticked |
| `full.html?person=Nobody` (unknown to identity) | everything generated **except** personal |
| `full.html` (no person) | everything generated **except** personal |
| `full.html?person=Amir`, run generated while identity was down | everything generated, personal included — the person was still resolved, and the outage already cost that run its preferences |

`compact.html` accepts `?person=` too but defaults to showing everything
generated, personal included: that card is embedded in one person's own HA
dashboard, which is already a per-account surface rather than a screen in a
hallway.

If `IDENTITY_URL` is blank, or identity is down, or the token is wrong, the run
generates **all four sections** and writes an empty `people` list — see
`preferences.py` for why a failed lookup fails towards more digest rather than
less. The one case that is honoured rather than overridden is a household where
everybody really has ticked everything off: that run generates nothing, and says
so in its log.

## One-time steps before the first real run

Both of these are interactive and must be done by hand, once. Scheduled runs
never prompt for anything.

**Telegram** — creates the session file `telegram_ingest.py` then reuses
non-interactively. You will be asked for your phone number, the code Telegram
sends, and your 2FA password if the account has one:

```sh
docker compose run --rm --entrypoint python digest-engine ingest/telegram_login.py
```

**WhatsApp** (only if you have opted in) — start the bridge and scan the QR code
it prints to its own logs with **WhatsApp -> Linked devices -> Link a device**:

```sh
docker compose up -d whatsapp-bridge
docker compose logs -f whatsapp-bridge
```

The session persists in the bridge's `/data/.wwebjs_auth` volume, so this is a
one-time scan unless WhatsApp invalidates the link.

## Run once, manually

```sh
docker compose run --rm digest-engine
```

Output lands in `output/<run-timestamp>/`:

- `context.json` — the ingested context bundle, kept for the Phase 12 follow-up
  voice Q&A (a spoken follow-up re-queries Ollama against this rather than
  re-ingesting).
- `digest.json` — the rendered digest, both detail levels, every section this run
  generated (`sections_generated` says which, and `people` who asked for what).

`output/latest.json` is rewritten with the same payload and `output/latest`
re-pointed at the newest run directory, so `digest-web` always serves the current
digest with no coordination with the scheduler.

## WhatsApp — read this before enabling

There is **no officially sanctioned way** to read your own WhatsApp messages
programmatically. `whatsapp-bridge` runs a real Chromium logged into
web.whatsapp.com as a linked device, deliberately **headful** under Xvfb because
WhatsApp's automation detection specifically fingerprints headless Chrome. That
is a meaningful mitigation. It is **not** immunity: this is still automated use of
a personal account and accounts do get banned for it, historically on a ~2–8 week
timescale.

If you enable it:

- Use a **secondary, non-critical number**, not your main one.
- Accept that the number may be banned, and that this is the highest-risk of the
  four message platforms by a wide margin.
- Keep `ENABLE_WHATSAPP_INGEST=false` if you are at all unsure. The rest of the
  digest works fine without it.

## Manual verification still outstanding

None of this has been run against real credentials or real accounts. Before
trusting a scheduled run, verify by hand:

1. Each source in isolation, e.g.
   `docker compose run --rm --entrypoint python digest-engine -c "import logging,os; logging.basicConfig(level='INFO'); from ingest import news_rss; print(len(news_rss.fetch(6)))"`.
2. That the IMAP mailbox shows **no** newly-read messages after a run (the folder
   is opened `readonly=True`, but confirm it against your provider).
3. That the Ollama model actually honours `format: "json"` and the schema — check
   `output/<run>/digest.json` for `"degraded": true`, which marks a section that
   fell back to plain text.
4. That both templates render: open `http://<host>:8091/full.html` and
   `http://<host>:8091/compact.html`.
5. The malformed-output fallback, by hand-editing `output/latest.json` into
   invalid JSON and reloading — the page must show a `<pre>` dump, never a blank
   screen.
6. The feed URLs in `curated-feeds.opml` — several mainstream outlets have
   changed or restricted their public RSS.
7. That the Nextcloud app password works over CalDAV and that recurring events
   land on the right day (the expansion path above is the one most likely to
   differ between Nextcloud versions).
8. The evening recipe, with `DIGEST_FORCE_EVENING=true` on a manual run — then
   confirm in Grocy that **nothing** was added to its shopping list and no stock
   moved.
9. The unviewed-digest merge: run once, do NOT show it on a thin client, run again
   (or set `DIGEST_LOOKBACK_HOURS`/wait) — confirm the second run's `context.json`
   has `"merged_unviewed_previous_run": true` and its digest actually carries
   forward the first run's content. Then show a digest on a thin client and run a
   third time — confirm that one merges nothing.
10. The counter run, deliberately: hand-edit a generated document before it's
    written (or patch `synth/counter_run.py` to run against a document with a
    fabricated quote spliced in) and confirm it actually gets dropped, not waved
    through — then confirm a real, correctly-grounded piece of Marxist analysis
    (a genuine merger analyzed via Lenin's imperialism) is NOT flagged just for
    being theoretical rather than a bare fact.
11. The per-person section toggles end to end, which have only been exercised
    against the API: untick a section for everybody in identity's admin panel,
    run once by hand, and confirm the run log says it skipped both that
    section's synthesis **and** its sources' ingestion, that `digest.json`'s
    `sections_generated` agrees, and that `full.html?person=<name>` shows what
    that person ticked and nothing else. Then stop the `identity` container and
    run again — that run must generate all four sections rather than none.
12. That `full.html` with **no** `?person=` really does leave the personal
    section out. It is a behaviour change: before this setting existed, the
    kiosk showed everyone's personal section to whoever walked past, despite
    `digest_canvas.py` having always claimed otherwise.
13. **The political section's new structure against a real model.** The ingest
    half is verified — `ingest/rci_social.py` was run live against the committed
    `feeds/rci-social.json` on 2026-08-06 and returned real YouTube and podcast
    entries with durations — but no local model has yet been asked to produce the
    four question windows, per-marker summaries and `sources` arrays in one JSON
    document. Check on the first real run that a 14B model actually fills
    `sources` rather than dropping the field, that it does not put episodes in
    the analysis, and that the compact pass still fits the HA card now that the
    section has more to say.
14. **The Telegram channel entry in `feeds/rci-social.json`.** The channel name
    comes from marxist.com's own footer but was never fetched — reading it needs
    the Telethon session, which only exists on the real deployment. Also confirm
    it does not double up: if you have *joined* that channel, its posts arrive a
    second time through `telegram_ingest.py` into the personal section.
15. **The archive against real runs.** Its logic is covered (dedup across runs,
    per-item annotation, series ordering, IDS recurrence, exclusion, and a broken
    database not taking a run down), but no real deployment has accumulated days
    of history yet. Check after a week that `first_seen_at` looks right on a story
    you remember, that the political section quotes both figures and both dates
    when it claims a trend, and that the network section has started calling its
    familiar signatures background rather than news. Watch the file size too —
    400 days of mail is the setting to revisit first.
16. **A real Tagesordnung, end to end.** The matcher and the PDF extraction are
    tested against a generated PDF; nobody has yet sent a real branch agenda
    through a real mailbox. Confirm the attachment actually spools, that the
    points come out of a real layout (multi-column or heavily styled agendas are
    where pypdf's extraction gets ragged), that a scanned one degrades to
    `text_extracted: false` rather than to nonsense, and that no household todo
    appears that the document did not actually ask for.
17. **The WhatsApp document download**, which is new behaviour in the bridge —
    `message._data.filename` is undocumented API and `downloadMedia()` has never
    been exercised here. Check a document arrives, lands in
    `/data/whatsapp-bridge/documents`, and that photos and video are still never
    downloaded.
18. **The OPNsense credentials via CoreSystemConfig.json.** The export → generated
    `IDSconf.json` → `opnsense_ids.py` chain is tested with synthetic values;
    confirm a real key pair authenticates against a real firewall, and that the
    scoped user really cannot do more than read alerts.
19. **The bias/ownership prompting, adversarially.** Feed a run a story covered
    by both RT and the BBC and confirm the section applies the ownership analysis
    to both rather than only to the one it is easier to be sceptical of, and that
    a Times of Israel claim about Palestinians never reaches the digest in the
    section's own voice.

## The political section — sources, ownership, and the four questions

The political digest answers four questions, in this order, one window each
(`synth/prompts/political.md` is written around them):

1. What is relevant for the communist and class struggle **globally** right now.
2. What matters for **organising here** — Austria, and Vorarlberg specifically.
   A closure in Dornbirn outranks a foreign cabinet reshuffle for this one.
3. What else is **consequential in the mid-to-long term** without being class
   struggle directly — rearmament, energy, epidemics, supply chains, repression.
4. What is happening **inside the RCI**, and what comrades elsewhere report. Its
   sources are the `theory` feeds, the organisation's own social output, and the
   reports that arrive in the user's own **mail** — the one input here no feed
   can supply.

Plus a **watch-later** window: new videos and podcast episodes from the
organisation's channels, kept out of the analysis entirely because they are
things to watch later, not evidence.

### Every source is read against who owns it

`feeds/curated-feeds.opml` carries `owner` and `bias` on every feed, and
`ingest/news_rss.py` puts both on every entry. The prompt reads each item
against them. **This is not a reliability score** — it is the same materialist
analysis the section applies to everything else, applied to the press:

- "State-affiliated" names who signs the cheque, not a propaganda bucket that
  private Western outlets are exempt from. RT is the outlet of the Russian
  capitalist class and its state; the BBC is the state broadcaster of a NATO
  power. Both get the treatment, or neither does.
- A private outlet is the organ of a fraction of capital — named when it
  explains the coverage (the Washington Post's owner is Amazon's owner when the
  story is warehouse labour).
- The business press (FT, CNBC) is often the most candid source in the file: it
  briefs capital honestly because capital is the reader.
- `category="news_labour"` is the workers' and movement press — closer to the
  shop floor, and not the RCI, so its reporting is used and its conclusions are
  not adopted.

**Zionist media get zero trust** (`category="news_zionist"`, and anything whose
`bias` says so). Their claims about Palestinians are never repeated as
established fact and their language is never adopted; they are read as evidence
about the Israeli state itself — what its ruling class admits, prepares for, or
falls out over. Zero trust is not inversion: a denial is not proof, and the "No
speculation" rule still binds. The Palestinian, anti-Zionist and independent
outlets in the same group are what corroboration is checked against.

The shipped list is about three dozen feeds across those blocs, all fetched and
confirmed live on 2026-08-06 except four marked `VERIFY` in the file. **Keep
`NEWS_MAX_ENTRIES_PER_FEED` low** (5 by default) — it multiplies by the feed
count into one prompt.

### Summaries on the globe, sources folded underneath

Every globe marker carries a `summary` (2–4 sentences on what is happening
there) and its own `sources`; any window can carry `sources` too. The renderer
prints marker summaries under the globe — not as tooltips, since the globe
rotates and a briefing you can only read while its marker faces you is not a
briefing — and folds citations into a collapsed `Sources (n)` block that costs
no screen space until tapped. Each citation carries the outlet **and its
ownership**, so who paid for a claim sits next to the claim.

Two guards on that, in `synth/counter_run.py`: a source whose URL is not
present in the run's context is **dropped**, and a real source carrying an
invented quote keeps the citation and loses the quote. A plausible-looking URL
is the easiest thing in this schema to fabricate and the hardest to eyeball.

### The organisation's own social media

`ingest/rci_social.py` reads the accounts listed in `feeds/rci-social.json` and
tags them `theory_social`, kept apart from the written analysis (`theory`)
because a meeting announcement is not an argument. Only platforms with a public,
keyless, first-party read path are implemented, and that is a hard line:

- **YouTube** — the channel's own Atom feed (`feeds/videos.xml?channel_id=`), no
  key, no quota. The RCI's and the Austrian section's channels are configured and
  verified.
- **RSS** — the section's podcast (`anchor.fm`), a Mastodon account's `.rss`, any
  site feed.
- **Telegram** — through the same Telethon session `telegram_ingest.py` already
  uses. A public channel is resolved by username and read **without joining it**;
  no session means those entries are skipped and the rest still run.

**Instagram, Facebook, WhatsApp channels, TikTok and X are deliberately not
built.** None has a read path that is both keyless and inside its own terms:
Instagram's Basic Display API was retired in 2024 and the Graph API only reads
accounts you own; Facebook page RSS died in 2018; WhatsApp channels have no API
at all; X's free tier reads essentially nothing and Nitter is gone. Reading them
would mean scraping, which this component does not do. Instagram is the real
gap — it is where the section posts most. Follow it on your phone; do not point
this at a scraping proxy, which moves the terms-of-service problem onto a third
party without removing it.

## "Your digest is ready" — the ntfy push

Because the canvas only opens when it is asked for, the notification is the one
thing that reaches you unprompted, and it exists to answer one question: **is this
run worth going and asking for, or does it keep until tonight?**

`notify.py` posts to the self-hosted ntfy this stack already runs for `chores` and
`identity`. On Android that lands on the phone and relays to a watch (a Pebble
needs nothing else installed). It costs **no extra LLM call**: every section
already produces a `narration` at `compact` detail — two to four sentences written
to be read aloud, which is the register a notification wants — so the push is
those narrations trimmed to a line each, plus the political to-do count and any
`withheld`/`unverified` marks. Nothing is generated here, so the notification can
never claim something the digest itself does not say.

**The per-person settings decide the push, not just the canvas.** Somebody who
switched the political section off gets no political content on their phone —
otherwise the setting would be a lie in the place it is most visible. People are
grouped by their own ntfy topic (identity's `notify_topic`, the same one arrival
notifications use), and each topic gets the union of what the people behind it
asked for: a topic *is* its audience, so two people sharing one have already
agreed to share what arrives on it. No personal topic falls back to `NTFY_TOPIC`;
no preferences at all (identity down) sends one household message about everything
generated. `DIGEST_WEB_URL` makes the push tappable, opening that person's own
digest — a shared topic gets the unfiltered page, since it has no single owner.

Published as **JSON to ntfy's root**, not as text with `Title:` headers the way
`chores` does. HTTP headers are latin-1, and this digest quotes news headlines and
household names: an em dash or an umlaut in a title raises `UnicodeEncodeError`
before the request is even sent. That failed on real content and passed every
ASCII test until one was written for it.

ntfy stays LAN-only; away from home it is reached over the WireGuard split tunnel
(`docs/network-integration.md` §2.2), never a port forward. A failed push logs a
warning and nothing else — the digest is already on disk by then, and no
notification is worth failing a run over.

## The archive — the digest's long memory

`archive.py` keeps a SQLite database in the `/data` volume holding every item the
digest has ingested and every number it has measured. A run without it sees six
hours and nothing else, which makes the most valuable things this system could
say impossible to say: *"up from 5.1% in June"*, *"this signature has fired every
night for a week"*, *"merchant traffic through this chokepoint has halved"*,
*"this story first appeared on Monday and has not moved"*.

Two tables, because there are two kinds of thing:

- **`items`** — discrete things (articles, messages, videos, calendar entries),
  deduplicated on a fingerprint (the URL when there is one). An article seen in
  four runs is one row seen four times, which is what makes `first_seen_at`
  meaningful. Every entry the prompts see now carries `first_seen_at` and
  `times_seen`, so "new this run" and "the same story for four days" are
  distinguishable without the model inferring it.
- **`observations`** — numbers that only mean anything as a series: FRED and
  Stooq readings, aircraft and vessel counts per region, IDS alerts per signature
  and per host. One row per measurement, so a trend is a query.

History enters each prompt as its own labelled block with its own timestamps —
never merged into this run's entries, so last month's figure can't be mistaken
for today's news. **The trend rules changed with it**: the political section may
now describe something as rising or falling *when the history block supports it,
with both figures and their dates*, and the network section leads with what is
new and says plainly when a signature is background noise it has seen fifteen
times. Without history, both revert to "one snapshot is not a trend".

**This keeps your mail and messages on disk for the retention window** (400 days
by default), where previously only the last few runs' `context.json` did. Nothing
leaves the host and the file sits beside the Telegram session, but it is a real
change in how long personal content is kept — `DIGEST_ARCHIVE_EXCLUDE_SOURCES`
takes a comma-separated list of ingest keys to keep out of it entirely, and the
financial/traffic/IDS trends work regardless of what you exclude.
`ENABLE_DIGEST_ARCHIVE=false` turns the whole thing off. A corrupt or unwritable
database logs a warning and the run proceeds without memory.

## Meeting agendas — the Tagesordnung finds its meeting

A branch sends the agenda for Thursday's meeting as a PDF on Monday, by mail or
WhatsApp. `agenda.py` matches it to the calendar event it belongs to, so the
digest says "branch meeting Thursday 19:00 — agenda `TO_12.08.pdf` from Anna",
and then uses what is actually in it.

**What it matches on**, in order: a date in the filename, subject or the
document's own heading against an event starting that day; words in common
between the agenda and the event's summary; both, which is the confident case.
Everything is labelled with its confidence and reason, so a wrong match is
visible rather than asserted, and an agenda that matches nothing is reported as
unattached rather than dropped.

**`TO` is special-cased.** It is the abbreviation everyone actually uses and also
the commonest two-letter word in English, so it is matched only as a standalone
uppercase token in a filename or subject — never in body prose. The long words
(Tagesordnung, Traktanden, agenda, Einladung; `AGENDA_KEYWORDS`) match
case-insensitively anywhere.

**The document is read.** `pypdf` extracts the text, the numbered and bulleted
lines are pulled out mechanically as `points` — a list extracted by code is a
list that cannot be invented — and both go into the context.

**An agenda's contents belong to the political section**, which is where a branch
agenda's party work belongs, and which means they reach only the people who
ticked the political digest. The split is deliberate and the prompts enforce it
from both sides:

- the **household** section says only *that* an agenda arrived, for which
  meeting, from whom, and whether it could be read at all. It is told not to list
  points or derive tasks even though the text is in front of it. Somebody with
  the household digest and not the political one sees a meeting with an agenda,
  not its contents.
- the **political** section carries `political-agenda` (the meeting and its
  points as written) and **`political-todo`, titled "Political todos"** — what
  the reader actually has to do before those meetings, one line each, verb first,
  naming the meeting and quoting the line of the agenda or its covering message
  the task came from. A task assigned to someone else is listed as theirs, so the
  reader knows it is covered. If nothing is actually asked for, the window is
  omitted — inventing preparation nobody asked for is the one failure here a
  reader would act on. It stays its own window at `compact` too, since it is the
  part of the digest people act on rather than read.
- the same agendas double as the section's sharpest relevance filter: a story
  touching an agenda point outranks a bigger story that doesn't, and says why —
  *"on Thursday's agenda"*.

`calendar` is therefore in the political section's `SECTION_SOURCES`: not for the
diary, but because it cannot match an agenda to a meeting it never fetched.

**A scanned agenda yields nothing** — it is a page of images and there is no OCR
here. That case attaches with `text_extracted: false`, and both prompts are told
they may name such a document but must not characterise it.

Attachments are spooled to `/data/attachments` (mail) and
`/data/whatsapp-bridge/documents` (the bridge, documents only — never photos or
video). That is a write to the digest's own volume, exactly as `context.json`
already is; nothing is written back to any mailbox or chat.

## Traffic data — what exists and what does not

`ingest/flight_traffic.py` and `ingest/naval_traffic.py` feed the **political**
section as extra `category`-tagged evidence, alongside news and financial data.
They are not a new digest section, and `synth/prompts/political.md` is explicitly
told to drop them when they corroborate nothing. Both are off by default.

**Air traffic** — OpenSky Network, bounded boxes over configured regions.
FlightRadar24 and ADS-B Exchange were not used: FR24 prohibits scraping and sells
API access, and ADS-B Exchange ended its freemium RapidAPI tier on 2025-03-01
(paid from $10/month). Before enabling, read the licensing note in
`digest-engine.env.example` — OpenSky's Terms of Use require a prior written
agreement for use of the REST API "in any operational capacity", which arguably
covers a timer-driven digest. Attribution to OpenSky is required.

**Naval traffic** — aisstream.io, a free keyed WebSocket stream, sampled briefly
per run. Read what it is for before enabling it: **AIS cannot show naval force
posture.** Warships sail with AIS off routinely, and the "military ops" AIS type
code is self-declared. What it shows is *merchant* traffic through chokepoints,
which is genuinely useful in the negative — shipping abandoning a route arrives
as freight, insurance and fuel costs. AISHub was not used: it is still
contribute-to-access and needs an AIS receiver this project does not have.
MarineTraffic, VesselFinder and Spire are paid.

**Military movement — deliberately not built as a data source.** There is no free
structured feed of military movements. ACLED is retrospective conflict-*event*
data, needs registration, and its EULA forbids redistribution and non-transformative
derivative works; UCDP is keyless but lags by roughly a month; everything
real-time is either commercial or a person on social media. Rather than invent an
integration, the military-movement signal comes from OSINT/defence outlets added
to `feeds/curated-feeds.opml` under `category="osint_military"`, which reuses the
existing news ingestion with no new code. Verified live 2026-07-28; Liveuamap has
no free RSS feed (`/rss` is a paid-API signup page) and ISW and Long War Journal
both 403'd the check — re-test those from the real network.

## Home network intrusion detection — what exists and what does not

`ingest/opnsense_ids.py` pulls a summary of the Suricata alerts your **existing
OPNsense firewall** raised during the digest window, tagged `"category":
"network_security"`. Off by default (`ENABLE_OPNSENSE_IDS_INGEST=false`).

**The credentials live in `CoreSystemConfig.json`** — an `opnsense` block for the
address and tuning, `secrets.opnsense_api_key`/`opnsense_api_secret` for the key
pair OPNsense mints (System → Access → Users → API keys; only OPNsense can issue
them, so `generate-tokens.py` deliberately never invents one). A build writes
them into the container host's `/data/IDSconf.json` alongside every other
generated config. Field-by-field documentation stays in
`IDSconf.json.example`, which is also what `setup-container-host.sh` seeds when
you install by hand instead of from an image. Setting the credentials does **not**
enable the ingest: `ENABLE_OPNSENSE_IDS_INGEST=true` in `digest-engine.env` is
still a separate, deliberate step.

It is **its own section** (`synth/prompts/network.md`), not a couple of lines
inside the household one as it was originally built. The reason is the
per-person setting above: somebody who wants the calendar and the shopping list
but not a nightly intrusion-detection readout — or the reverse — can only say so
if the two are generated separately. The instructions themselves moved across
unchanged, including the one that matters most: alerts are signature matches,
never a claim that a device is compromised, and the section may never call the
network safe.

**Suricata is core, not a plugin.** There is no `os-suricata` to install — the
IDS module ships in OPNsense core and lives at Services → Intrusion Detection.
The `os-intrusion-detection-content-*` plugins are ruleset *content* only, and
ET Open needs none of them. Suricata is nonetheless **off** on a stock install;
enable it and download a ruleset first, or every run will report `ids_status`
and no alerts, which is the honest answer and not a quiet network.

**It is a pull, like every other source here.** Two endpoints, both read-only in
effect: `GET /api/ids/service/status` and `POST /api/ids/service/query_alerts`.
The latter is a POST only because that is how OPNsense routes filtered queries —
it runs `queryAlertLog.py`, which reads `/var/log/suricata/eve.json` backwards.
No SSH or file access to the firewall is needed, and no push agent runs on it.

Three limits worth knowing before you read the output:

- **No server-side time filter.** `searchPhrase` matches signature/action/src/dst
  text only. Rows come back newest-first, so the window is applied client-side by
  paging until a row falls out of it, bounded by `max_alerts_scanned`. When that
  bound is hit — or when the alert log rotated mid-window — the entry carries
  `window_truncated: true` and the prompt is told the counts are a lower bound.
- **No severity.** OPNsense flattens each eve.json record to signature + SID +
  action before returning it, discarding `alert.severity` and `alert.category`.
  `get_alert_info` uses the same flattening, so it does not help. Alerts are
  ranked by frequency, and the household prompt is told it cannot see severity.
- **Page-level ACLs.** The "Services: Intrusion Detection" privilege matches
  `api/ids/*`, which covers start/stop/reconfigure/drop-alert-log as well as the
  alert query. OPNsense has no narrower built-in privilege, so the read-only
  guarantee is enforced by this code (which calls two endpoints and no others)
  and not by the firewall. Give the API key its own user with that one privilege
  and nothing else, and treat it as a credential that could restart your IDS if
  it leaked — the container host and the firewall are on the same flat LAN, since
  no VLAN segmentation is implemented in this project yet.

**Raw packet captures — deliberately not done here.** OPNsense does expose
Interfaces: Diagnostics: Packet Capture over the API
(`/api/diagnostics/packet_capture/{set,start,stop,remove}`), but every one of
those is a POST that writes a job file and spawns `tcpdump` on the firewall.
Starting a capture is a write action on someone else's router and is barred by
this component's read-only invariant. Downloading and parsing pcaps into the
digest would also mean hand-rolling malware detection over raw packets, which is
strictly worse than reading the verdicts of a maintained ruleset that already
inspected the same traffic in real time.

If you still want a rotating raw capture for manual inspection, keep it **on
OPNsense**, e.g. a `tcpdump -G 600 -W 12 -w /var/log/captures/cap-%F-%H%M.pcap`
rotation driven from the firewall's own cron (this needs shell access on
OPNsense — the GUI cron only schedules predefined configd actions — and enough
disk for ten-minute captures of a live link, which is not small). Then put a
one-line pointer in `IDSconf.json`'s `packet_capture_reference`; it is echoed
verbatim into the digest so the network section can say "raw captures are at
X". The digest engine never downloads, stores or analyses them.

## Household data — what exists and what does not

`ingest/caldav.py` and `ingest/grocy.py` are what the household section actually
runs on. Both are off by default and both need one credential created by hand.

**Calendar** — Nextcloud over CalDAV (Phase 8), via the maintained `caldav`
library rather than hand-written REPORT XML. Auth is a **Nextcloud app
password** (Settings → Security → Devices & sessions → Create new app password),
not the account password and not OAuth2 — mandatory once 2FA is on, since the DAV
endpoints cannot prompt for a second factor, and revocable on its own regardless.
Point `CALDAV_URL` at the DAV root (`https://<host>/remote.php/dav`) and the
client discovers the principal's calendars from there.

The window is deliberately asymmetric — `DIGEST_LOOKBACK_HOURS` backwards, so
this morning's appointment and anything still running are still visible, and
`CALDAV_LOOKAHEAD_HOURS` (default 48) forwards, because a calendar is mostly
useful in the future tense. Recurring events are requested **expanded**, so a
weekly standup arrives as the occurrence in this window rather than as the master
event with an RRULE; if a server rejects expansion outright, the search is
retried without it and a recurring series shows up as its master event.

**Kitchen inventory** — Grocy (Phase 7), reached at `http://grocy` on the shared
compose network (port 80 inside the container; the published 9283 is not
involved). One endpoint does most of the work: `GET /api/stock/volatile`, which
returns `due_products`, `overdue_products`, `expired_products` and
`missing_products` directly. **Watch the naming** — Grocy renamed
`expiring_products` → `due_products` in v3.0.0, so older third-party examples are
wrong against a current install. Chores and batteries come from `GET /api/chores`
and `GET /api/batteries`; both use `2999-12-31 23:59:59` as a "no schedule"
sentinel, which is filtered out rather than reported as a due date.

Auth is a `GROCY-API-KEY` header, generated at Grocy → Settings → Manage API
keys. **A Grocy API key is not scoped**: it carries that user's full read *and*
write rights, so give this one its own Grocy user, and note that the read-only
guarantee is enforced by `ingest/grocy.py` calling nothing but GETs — the
module's docstring names every write endpoint it deliberately does not use — and
not by Grocy.

**No compose or systemd changes were needed for either.** digest-engine and
grocy are already on the same default compose network, so the container name
resolves; Nextcloud is external and reached over its normal URL; and the
credentials are ordinary env vars in the `digest-engine.env` the service already
loads.

## The evening recipe suggestion

On one run a day — `DIGEST_EVENING_HOUR`, default `18` — the household section
also suggests a dish built around whatever Grocy says is about to go off, plus a
shopping list for the ingredients that dish needs and the house does not have.
The other three runs omit it entirely rather than padding it in.

**It is a suggestion, and nothing else.** Nothing is written to Grocy: no item is
added to its shopping list, no stock is consumed, no order is placed anywhere.
Grocy's API supports all of that with the same key and this component uses none
of it, per the read-only invariant above. You read the list and go shopping.

Which run is "evening" is derived from the container's local wall clock, not
passed in by the caller. The systemd unit runs a bare
`docker compose run --rm digest-engine` with no arguments and a manual run is the
same command, so an argument or a unit-specific env var would have to be threaded
through both and would silently misbehave on a hand-run digest; the container
already has the host's `TZ` and `/etc/localtime`, which is the same clock the
timer's `OnCalendar` fires against. The run is attributed to the most recent
`DIGEST_SCHEDULE` slot at or before now rather than to an exact hour match,
because the timer is `Persistent=true` — a host asleep at 18:00 fires late, and
an exact match would drop the feature on precisely the days the digest is late.

Set `DIGEST_FORCE_EVENING=true` for a one-off run to test it at any hour.
Keep `DIGEST_SCHEDULE` in step with the variable of the same name in
`tools/setup-container-host.sh`, which is what sets the
timer.

## Merging an unviewed digest into the next one

If nobody actually looked at a run before the next one was due, its content is
folded into the new run instead of being silently thrown away — see
`viewed_tracker.py`, `run.py`'s `should_merge()`/`previous_section_document()`, and
the merge instruction `synth/llm_client.py` adds to the prompt when it applies.

**"Viewed" means a thin client actually displayed the full canvas** — the "Show
digest canvas" button or a voice-resolved "play my digest" request, both of which go
through `thinclient_agent/main.py`'s `on_show_digest()`, which publishes a retained
`{"viewed_at": ...}` to `smarthome/digest/viewed` on the same Mosquitto broker
everything else in this project already shares. **The compact HA-dashboard iframe
view does not count** — it's a browser rendering a static page, with no path back to
MQTT at all, so leaving it open on a phone can never mark a digest viewed.

Each run compares that timestamp against the previous run's `generated_at`
(`output/latest/digest.json`). If the previous run is newer than the last time
anything was viewed — or nothing has ever been marked viewed, or no previous run
exists yet — this run proceeds exactly as before. Otherwise, each section's own
previous content (from the `full` detail level, the richest version) is handed to
that section's synthesis pass as `previous_unviewed_digest`, with an instruction to
combine it with the new material into one digest rather than repeating or discarding
either — nothing is dropped, but nothing doubles up either.

If MQTT is unreachable, `paho-mqtt` isn't installed, or the retained message can't be
parsed, `viewed_tracker.last_viewed_at()` returns `None`, which is treated the same
as "viewed" — the safer of the two wrong answers, since it costs at most one merge
that should have happened, rather than gluing every future run onto the last
forever. `MQTT_VIEWED_WAIT_SECONDS` (default 3) bounds how long a run will wait for
that retained message before moving on, so a dead broker never stalls a digest run.

**Not yet run against a real broker or a real thin client** — the retained-message
round trip, the `on_show_digest` publish, and a genuine multi-cycle unviewed→merged
sequence are all still on the manual-verification list.

## The counter run — a final filter against hallucination

Before anything is written to `output/`, every generated document is checked
by a second, independent LLM call (`synth/counter_run.py`) against the exact
same context it was generated from. This is the final filter the plan calls
for against false or unsourced information reaching the digest — it is not a
substitute for the "No speculation" instructions already in each prompt, it's
the backstop for when those instructions don't work.

It checks, per window: is every quotation an actual excerpt of something in
the context (not a plausible-sounding invention); is every figure, date, or
name traceable to something in the context; does every named theoretical
connection (Lenin's imperialism, Marx's labour theory of value, etc.)
correspond to a real event the context actually describes that way; does
every stated correlation between two data sources actually have both halves
present, not one assumed.

**This does not mean second-guessing the digest's Marxist framing itself.**
The counter run shares the same RCI-derived theoretical basis as the document
it's checking (see `synth/prompts/counter_run.md`) — its job is to confirm the
underlying facts are real and a theoretical reading of them is a genuine
structural match, not to apply a bourgeois-neutral standard of "objectivity"
that would flag correct class analysis as unverifiable "opinion." That would
smuggle in a different politics than the one this digest is written from,
which is exactly the kind of error this pass exists to prevent, not commit.

What happens to something it flags:

- A specific window it can't ground is dropped; the rest of the document is
  kept.
- Narration it can't ground is cleared to empty — better silent than a false
  claim read aloud by the TTS voice.
- A quote the model itself claims is "found in context" is also checked
  mechanically (a plain substring search against the same context), and
  overridden if it isn't actually there — the one claim type this doesn't
  have to take the verifying call's own word for.
- If every window in a document gets dropped, the whole section is replaced
  with an honest "withheld pending verification" placeholder rather than
  shown empty or not at all.
- If the counter run can't run at all (Ollama unreachable a second time, an
  unparseable verdict), the original document is kept but marked
  `unverified` — not silently passed through unchecked, and not blanked
  either, since a transient failure in this pass specifically shouldn't cost
  as much as the whole digest being down.

This doubles the number of Ollama calls per run (four calls per generated
section rather than two — so 16 with all four sections on, and fewer for every
section the household has switched off) — against a
local, self-hosted model with no per-token cost and nobody waiting on the
latency, the same tradeoff `synth/llm_client.py` already makes for generating
`compact` and `full` as separate passes rather than truncating one into the
other. Set `COUNTER_RUN_MODEL` if you want verification done by a different
(e.g. larger) model than the one that generated the digest.

**Not yet run for real** — whether the counter-run prompt actually catches a
genuinely hallucinated quote, versus over-flagging real ones, needs checking
against actual model output before this can be trusted as more than
plausible-sounding on paper.