The_miro
/
SmartestHome
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Per-person digests, RCI sources, agendas, archive and TV power 

digest-engine grows from a single household digest into four per-person
sections, and gains the memory and inputs to make them worth reading.

Per-person sections. identity owns a digest_sections column per person,
edited in the admin panel and read by digest-engine at the start of every
run (GET /digest-preferences). A run generates the union the household
asked for: a section nobody wants costs no LLM call and no ingestion of
its sources. Each surface then filters to the person HA resolved. The
display half is a filter, not an access control, and says so.

Network is its own section, split out of household so the two can be
wanted separately.

Political section rebuilt around four questions (global class struggle,
organising in Vorarlberg, mid-term consequences, the International and
comrades' reports). ~30 international feeds added, each carrying owner
and bias, with a symmetric ownership analysis rather than a
reliability ranking; Zionist outlets get an explicit zero-trust rule that
is not inversion. RCI social/podcast ingestion (YouTube Atom, podcast RSS,
public Telegram via the existing session) feeds a watch-later window.
Globe markers carry summaries with fold-out sources; counter_run drops
citations whose URLs are not in the context.

Meeting agendas: a Tagesordnung arriving by mail or WhatsApp is matched to
its calendar event, read with pypdf, and its points extracted
mechanically. The political section owns the contents and derives
"Political todos"; the household section is told only that an agenda
exists, enforced structurally.

The archive keeps every ingested item and measured number across runs, so
trends may finally be stated with figures and dates attached.

ntfy push after each run, assembled from existing narrations, gated by the
same per-person sections.

identity gains GET /speaker: automatic recognition for the voice path from
BLE plus recent face sightings. Unresolved means show less, never ask, and
nothing displays a digest because someone walked past a screen.

OPNsense credentials move into CoreSystemConfig.json; thin clients gain a
Display switch (HDMI-CEC, DPMS fallback) so an empty room stops powering a
TV.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NpKsMV1Q2EHJ5cZVoLvK9M
digest-per-person-and-agendas
Amir Alexander Abdelbaki 2026-08-06 13:30:47 +02:00
parent 00991b9864
commit c10d803a12
41 changed files with 4458 additions and 193 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

15
CoreSystemConfig.json.template
View File

@ -115,18 +115,31 @@
},
"secrets": {
"_comment": "Generate the tokens with: openssl rand -hex 32. Each is required only if the service that uses it is enabled above; the validator says which. ha_token is a Long-Lived Access Token from HA's own UI (profile -> Security) and cannot be generated ahead of time — leave it empty for the first build and re-run once HA is up.",
"_comment": "Generate the tokens with: openssl rand -hex 32. Each is required only if the service that uses it is enabled above; the validator says which. ha_token is a Long-Lived Access Token from HA's own UI (profile -> Security) and cannot be generated ahead of time — leave it empty for the first build and re-run once HA is up. The opnsense_api_* pair is the same kind of thing: only OPNsense can mint it (System -> Access -> Users -> API keys), so it is never generated here either.",
"identity_token": "",
"pantry_vision_token": "",
"transit_token": "",
"mqtt_username": "",
"mqtt_password": "",
"ha_token": "",
"opnsense_api_key": "",
"opnsense_api_secret": "",
"ssh_authorized_key": "",
"kiosk_password": "",
"admin_password_hash": ""
},
"opnsense": {
"_comment": "The household's existing OPNsense firewall. Used by digest-engine's network digest, which pulls a Suricata intrusion-detection summary from it (GET /api/ids/service/status and POST /api/ids/service/query_alerts — two read endpoints, nothing else, ever). A build writes this block plus the secrets above into the container host's IDSconf.json; the long-form documentation for every field lives in digest-engine/IDSconf.json.example. Leave base_url empty to skip that file entirely. TWO THINGS THIS DOES NOT DO: it does not enable Suricata (do that at Services -> Intrusion Detection on the firewall, then download a ruleset), and it does not enable the ingest (set ENABLE_OPNSENSE_IDS_INGEST=true in digest-engine.env). SCOPE THE API KEY: give it its own OPNsense user with only the 'Services: Intrusion Detection' privilege — that privilege still covers api/ids/* including start/stop, because OPNsense ACLs are page-level, so the read-only guarantee comes from digest-engine calling exactly two endpoints and not from the firewall enforcing it.",
"base_url": "",
"verify_tls": true,
"interfaces": [],
"max_alerts_scanned": 5000,
"top_signatures": 8,
"top_hosts": 5,
"packet_capture_reference": ""
},
"proxy": {
"_comment": "One HTTPS front door for this repo's own services (Caddy), plus an HTTP->HTTPS redirect. Not a household-wide gateway — Home Assistant, Grocy and friends keep their own ports. hostname is what the certificate is issued for and what you type in the browser; it must resolve to the container host (a DNS override on OPNsense, a hosts entry, or just use the IP with tls: internal). tls 'internal' makes Caddy run its own CA — no external dependency, but browsers show a warning until you install its root (tools/export-proxy-ca.sh). tls 'custom' uses cert_file/key_file, which is how you'd use a real cert obtained via a DNS-01 challenge without exposing anything. See proxy/README.md.",
"enabled": true,

393
digest-engine/README.md
View File

@ -3,10 +3,14 @@
The quarter-daily LLM digest from [Phase 12 of the project plan](../docs/project-plan.md).
Four times a day it ingests mail, messages, news and financial data, sends the
lot to the existing Ollama host for synthesis into three sections — **personal**,
**political** and **household** — and writes a rendered digest that `digest-web`
serves to two surfaces: the thin client's kiosk Firefox workspace (full view) and
a Home Assistant Lovelace iframe card (compact view).
lot to the existing Ollama host for synthesis into up to four sections —
**personal** (social), **political** (news), **household** and **network** — and
writes a rendered digest that `digest-web` serves to two surfaces: the thin
client's kiosk Firefox workspace (full view) and a Home Assistant Lovelace iframe
card (compact view).
Which of the four a run actually generates is the household's choice, per person
— see [Who gets which digest](#who-gets-which-digest) below.
It is a **oneshot**, not a daemon: a systemd timer runs
`docker compose run --rm digest-engine`, exactly like the restic backup job.
@ -20,6 +24,9 @@ the plan, not a default.
```
run.py oneshot entrypoint
preferences.py who wants which digest, read from identity
archive.py the long memory — every item and number, kept across runs
agenda.py a Tagesordnung PDF -> the meeting it belongs to
ingest/ one module per source, each `fetch(lookback_hours) -> list[dict]`
telegram_login.py standalone one-time interactive login (run by hand)
synth/llm_client.py Ollama client + the digest JSON schema
@ -27,6 +34,7 @@ synth/prompts/ one prompt template per section
render/digest-canvas-sdk/ vendored, offline JS/CSS — globe, window chrome, glow, renderer
render/templates/ compact.html (HA iframe) and full.html (kiosk)
feeds/curated-feeds.opml the news feed list — edit this
feeds/rci-social.json the RCI/section social + podcast accounts — edit this
IDSconf.json.example OPNsense IDS config template (real file gitignored)
whatsapp-bridge/ Node.js sidecar, opt-in, see the warning below
output/ per-run artifacts (gitignored)
@ -49,6 +57,62 @@ Then edit `feeds/curated-feeds.opml`: the mainstream outlets in it are a
clearly-marked placeholder list, only the `marxist.com` feed is a deliberate
choice (the political prompt uses it as its analytical basis).
## Who gets which digest
Every person in `identity` has their own set of the four sections, ticked in the
admin panel's person editor (**Digests → Generate for this person**). Nothing
else about the run changes; what changes is how much of it happens at all.
- **A section nobody has ticked is never generated.** No synthesis call, no
counter-run call, and — because `run.py`'s `SECTION_SOURCES` knows which
sources feed which section — no ingestion either. Turn the political section
off for the whole household and the run stops fetching news, financial data
and flight/naval traffic entirely. The saving is real WAN egress, not just
tokens.
- **Each surface then shows a person their own subset.** The digest carries a
`people` list (name, nickname, id, sections) that the renderer filters on when
it is given `?person=`.
- **The default is all four**, so a household that never opens this panel gets
exactly the digest it had before this setting existed.
**The display half is a filter, not an access control.** `digest-web` serves the
whole output volume read-only to anything on the LAN, so an unticked section is
off somebody's screen and out of their narration — it is not hidden from them.
The half that genuinely does not exist is the half that was never generated.
`?person=` is a name, nickname or id, and it is only ever set by a caller that
has **already** resolved who is asking:
`hosts/thin-client/agent/thinclient_agent/digest_canvas.py` passes it from a
Home-Assistant-resolved request, per the plan's Phase 11.8 rule that the
personal section is never shown on a guess.
**The canvas is voice-activated, and the person is recognised automatically.**
Nothing shows a digest because somebody walked past a screen. When a spoken "play
my digest" fires, HA asks `identity`'s `GET /speaker?area=<area>` who is in that
room — BLE identity plus the most recent Frigate face sighting — and passes the
answer through as `?person=`. When it cannot tell, it says so and the canvas falls
back to everything-but-personal, which is the same rule as no person at all. So
on `full.html` (the kiosk):
| URL | What renders |
|---|---|
| `full.html?person=Amir` | exactly the sections Amir ticked |
| `full.html?person=Nobody` (unknown to identity) | everything generated **except** personal |
| `full.html` (no person) | everything generated **except** personal |
| `full.html?person=Amir`, run generated while identity was down | everything generated, personal included — the person was still resolved, and the outage already cost that run its preferences |
`compact.html` accepts `?person=` too but defaults to showing everything
generated, personal included: that card is embedded in one person's own HA
dashboard, which is already a per-account surface rather than a screen in a
hallway.
If `IDENTITY_URL` is blank, or identity is down, or the token is wrong, the run
generates **all four sections** and writes an empty `people` list — see
`preferences.py` for why a failed lookup fails towards more digest rather than
less. The one case that is honoured rather than overridden is a household where
everybody really has ticked everything off: that run generates nothing, and says
so in its log.
## One-time steps before the first real run
Both of these are interactive and must be done by hand, once. Scheduled runs
@ -84,7 +148,8 @@ Output lands in `output/<run-timestamp>/`:
- `context.json` — the ingested context bundle, kept for the Phase 12 follow-up
voice Q&A (a spoken follow-up re-queries Ollama against this rather than
re-ingesting).
- `digest.json` — the rendered digest, both detail levels, all three sections.
- `digest.json` — the rendered digest, both detail levels, every section this run
generated (`sections_generated` says which, and `people` who asked for what).
`output/latest.json` is rewritten with the same payload and `output/latest`
re-pointed at the newest run directory, so `digest-web` always serves the current
@ -105,8 +170,8 @@ If you enable it:
- Use a **secondary, non-critical number**, not your main one.
- Accept that the number may be banned, and that this is the highest-risk of the
four message platforms by a wide margin.
- Keep `ENABLE_WHATSAPP_INGEST=false` if you are at all unsure. The other three
sections work fine without it.
- Keep `ENABLE_WHATSAPP_INGEST=false` if you are at all unsure. The rest of the
digest works fine without it.
## Manual verification still outstanding
@ -144,6 +209,287 @@ trusting a scheduled run, verify by hand:
through — then confirm a real, correctly-grounded piece of Marxist analysis
(a genuine merger analyzed via Lenin's imperialism) is NOT flagged just for
being theoretical rather than a bare fact.
11. The per-person section toggles end to end, which have only been exercised
against the API: untick a section for everybody in identity's admin panel,
run once by hand, and confirm the run log says it skipped both that
section's synthesis **and** its sources' ingestion, that `digest.json`'s
`sections_generated` agrees, and that `full.html?person=<name>` shows what
that person ticked and nothing else. Then stop the `identity` container and
run again — that run must generate all four sections rather than none.
12. That `full.html` with **no** `?person=` really does leave the personal
section out. It is a behaviour change: before this setting existed, the
kiosk showed everyone's personal section to whoever walked past, despite
`digest_canvas.py` having always claimed otherwise.
13. **The political section's new structure against a real model.** The ingest
half is verified — `ingest/rci_social.py` was run live against the committed
`feeds/rci-social.json` on 2026-08-06 and returned real YouTube and podcast
entries with durations — but no local model has yet been asked to produce the
four question windows, per-marker summaries and `sources` arrays in one JSON
document. Check on the first real run that a 14B model actually fills
`sources` rather than dropping the field, that it does not put episodes in
the analysis, and that the compact pass still fits the HA card now that the
section has more to say.
14. **The Telegram channel entry in `feeds/rci-social.json`.** The channel name
comes from marxist.com's own footer but was never fetched — reading it needs
the Telethon session, which only exists on the real deployment. Also confirm
it does not double up: if you have *joined* that channel, its posts arrive a
second time through `telegram_ingest.py` into the personal section.
15. **The archive against real runs.** Its logic is covered (dedup across runs,
per-item annotation, series ordering, IDS recurrence, exclusion, and a broken
database not taking a run down), but no real deployment has accumulated days
of history yet. Check after a week that `first_seen_at` looks right on a story
you remember, that the political section quotes both figures and both dates
when it claims a trend, and that the network section has started calling its
familiar signatures background rather than news. Watch the file size too —
400 days of mail is the setting to revisit first.
16. **A real Tagesordnung, end to end.** The matcher and the PDF extraction are
tested against a generated PDF; nobody has yet sent a real branch agenda
through a real mailbox. Confirm the attachment actually spools, that the
points come out of a real layout (multi-column or heavily styled agendas are
where pypdf's extraction gets ragged), that a scanned one degrades to
`text_extracted: false` rather than to nonsense, and that no household todo
appears that the document did not actually ask for.
17. **The WhatsApp document download**, which is new behaviour in the bridge —
`message._data.filename` is undocumented API and `downloadMedia()` has never
been exercised here. Check a document arrives, lands in
`/data/whatsapp-bridge/documents`, and that photos and video are still never
downloaded.
18. **The OPNsense credentials via CoreSystemConfig.json.** The export → generated
`IDSconf.json` → `opnsense_ids.py` chain is tested with synthetic values;
confirm a real key pair authenticates against a real firewall, and that the
scoped user really cannot do more than read alerts.
19. **The bias/ownership prompting, adversarially.** Feed a run a story covered
by both RT and the BBC and confirm the section applies the ownership analysis
to both rather than only to the one it is easier to be sceptical of, and that
a Times of Israel claim about Palestinians never reaches the digest in the
section's own voice.
## The political section — sources, ownership, and the four questions
The political digest answers four questions, in this order, one window each
(`synth/prompts/political.md` is written around them):
1. What is relevant for the communist and class struggle **globally** right now.
2. What matters for **organising here** — Austria, and Vorarlberg specifically.
A closure in Dornbirn outranks a foreign cabinet reshuffle for this one.
3. What else is **consequential in the mid-to-long term** without being class
struggle directly — rearmament, energy, epidemics, supply chains, repression.
4. What is happening **inside the RCI**, and what comrades elsewhere report. Its
sources are the `theory` feeds, the organisation's own social output, and the
reports that arrive in the user's own **mail** — the one input here no feed
can supply.
Plus a **watch-later** window: new videos and podcast episodes from the
organisation's channels, kept out of the analysis entirely because they are
things to watch later, not evidence.
### Every source is read against who owns it
`feeds/curated-feeds.opml` carries `owner` and `bias` on every feed, and
`ingest/news_rss.py` puts both on every entry. The prompt reads each item
against them. **This is not a reliability score** — it is the same materialist
analysis the section applies to everything else, applied to the press:
- "State-affiliated" names who signs the cheque, not a propaganda bucket that
private Western outlets are exempt from. RT is the outlet of the Russian
capitalist class and its state; the BBC is the state broadcaster of a NATO
power. Both get the treatment, or neither does.
- A private outlet is the organ of a fraction of capital — named when it
explains the coverage (the Washington Post's owner is Amazon's owner when the
story is warehouse labour).
- The business press (FT, CNBC) is often the most candid source in the file: it
briefs capital honestly because capital is the reader.
- `category="news_labour"` is the workers' and movement press — closer to the
shop floor, and not the RCI, so its reporting is used and its conclusions are
not adopted.
**Zionist media get zero trust** (`category="news_zionist"`, and anything whose
`bias` says so). Their claims about Palestinians are never repeated as
established fact and their language is never adopted; they are read as evidence
about the Israeli state itself — what its ruling class admits, prepares for, or
falls out over. Zero trust is not inversion: a denial is not proof, and the "No
speculation" rule still binds. The Palestinian, anti-Zionist and independent
outlets in the same group are what corroboration is checked against.
The shipped list is about three dozen feeds across those blocs, all fetched and
confirmed live on 2026-08-06 except four marked `VERIFY` in the file. **Keep
`NEWS_MAX_ENTRIES_PER_FEED` low** (5 by default) — it multiplies by the feed
count into one prompt.
### Summaries on the globe, sources folded underneath
Every globe marker carries a `summary` (2–4 sentences on what is happening
there) and its own `sources`; any window can carry `sources` too. The renderer
prints marker summaries under the globe — not as tooltips, since the globe
rotates and a briefing you can only read while its marker faces you is not a
briefing — and folds citations into a collapsed `Sources (n)` block that costs
no screen space until tapped. Each citation carries the outlet **and its
ownership**, so who paid for a claim sits next to the claim.
Two guards on that, in `synth/counter_run.py`: a source whose URL is not
present in the run's context is **dropped**, and a real source carrying an
invented quote keeps the citation and loses the quote. A plausible-looking URL
is the easiest thing in this schema to fabricate and the hardest to eyeball.
### The organisation's own social media
`ingest/rci_social.py` reads the accounts listed in `feeds/rci-social.json` and
tags them `theory_social`, kept apart from the written analysis (`theory`)
because a meeting announcement is not an argument. Only platforms with a public,
keyless, first-party read path are implemented, and that is a hard line:
- **YouTube** — the channel's own Atom feed (`feeds/videos.xml?channel_id=`), no
key, no quota. The RCI's and the Austrian section's channels are configured and
verified.
- **RSS** — the section's podcast (`anchor.fm`), a Mastodon account's `.rss`, any
site feed.
- **Telegram** — through the same Telethon session `telegram_ingest.py` already
uses. A public channel is resolved by username and read **without joining it**;
no session means those entries are skipped and the rest still run.
**Instagram, Facebook, WhatsApp channels, TikTok and X are deliberately not
built.** None has a read path that is both keyless and inside its own terms:
Instagram's Basic Display API was retired in 2024 and the Graph API only reads
accounts you own; Facebook page RSS died in 2018; WhatsApp channels have no API
at all; X's free tier reads essentially nothing and Nitter is gone. Reading them
would mean scraping, which this component does not do. Instagram is the real
gap — it is where the section posts most. Follow it on your phone; do not point
this at a scraping proxy, which moves the terms-of-service problem onto a third
party without removing it.
## "Your digest is ready" — the ntfy push
Because the canvas only opens when it is asked for, the notification is the one
thing that reaches you unprompted, and it exists to answer one question: **is this
run worth going and asking for, or does it keep until tonight?**
`notify.py` posts to the self-hosted ntfy this stack already runs for `chores` and
`identity`. On Android that lands on the phone and relays to a watch (a Pebble
needs nothing else installed). It costs **no extra LLM call**: every section
already produces a `narration` at `compact` detail — two to four sentences written
to be read aloud, which is the register a notification wants — so the push is
those narrations trimmed to a line each, plus the political to-do count and any
`withheld`/`unverified` marks. Nothing is generated here, so the notification can
never claim something the digest itself does not say.
**The per-person settings decide the push, not just the canvas.** Somebody who
switched the political section off gets no political content on their phone —
otherwise the setting would be a lie in the place it is most visible. People are
grouped by their own ntfy topic (identity's `notify_topic`, the same one arrival
notifications use), and each topic gets the union of what the people behind it
asked for: a topic *is* its audience, so two people sharing one have already
agreed to share what arrives on it. No personal topic falls back to `NTFY_TOPIC`;
no preferences at all (identity down) sends one household message about everything
generated. `DIGEST_WEB_URL` makes the push tappable, opening that person's own
digest — a shared topic gets the unfiltered page, since it has no single owner.
Published as **JSON to ntfy's root**, not as text with `Title:` headers the way
`chores` does. HTTP headers are latin-1, and this digest quotes news headlines and
household names: an em dash or an umlaut in a title raises `UnicodeEncodeError`
before the request is even sent. That failed on real content and passed every
ASCII test until one was written for it.
ntfy stays LAN-only; away from home it is reached over the WireGuard split tunnel
(`docs/network-integration.md` §2.2), never a port forward. A failed push logs a
warning and nothing else — the digest is already on disk by then, and no
notification is worth failing a run over.
## The archive — the digest's long memory
`archive.py` keeps a SQLite database in the `/data` volume holding every item the
digest has ingested and every number it has measured. A run without it sees six
hours and nothing else, which makes the most valuable things this system could
say impossible to say: *"up from 5.1% in June"*, *"this signature has fired every
night for a week"*, *"merchant traffic through this chokepoint has halved"*,
*"this story first appeared on Monday and has not moved"*.
Two tables, because there are two kinds of thing:
- **`items`** — discrete things (articles, messages, videos, calendar entries),
deduplicated on a fingerprint (the URL when there is one). An article seen in
four runs is one row seen four times, which is what makes `first_seen_at`
meaningful. Every entry the prompts see now carries `first_seen_at` and
`times_seen`, so "new this run" and "the same story for four days" are
distinguishable without the model inferring it.
- **`observations`** — numbers that only mean anything as a series: FRED and
Stooq readings, aircraft and vessel counts per region, IDS alerts per signature
and per host. One row per measurement, so a trend is a query.
History enters each prompt as its own labelled block with its own timestamps —
never merged into this run's entries, so last month's figure can't be mistaken
for today's news. **The trend rules changed with it**: the political section may
now describe something as rising or falling *when the history block supports it,
with both figures and their dates*, and the network section leads with what is
new and says plainly when a signature is background noise it has seen fifteen
times. Without history, both revert to "one snapshot is not a trend".
**This keeps your mail and messages on disk for the retention window** (400 days
by default), where previously only the last few runs' `context.json` did. Nothing
leaves the host and the file sits beside the Telegram session, but it is a real
change in how long personal content is kept — `DIGEST_ARCHIVE_EXCLUDE_SOURCES`
takes a comma-separated list of ingest keys to keep out of it entirely, and the
financial/traffic/IDS trends work regardless of what you exclude.
`ENABLE_DIGEST_ARCHIVE=false` turns the whole thing off. A corrupt or unwritable
database logs a warning and the run proceeds without memory.
## Meeting agendas — the Tagesordnung finds its meeting
A branch sends the agenda for Thursday's meeting as a PDF on Monday, by mail or
WhatsApp. `agenda.py` matches it to the calendar event it belongs to, so the
digest says "branch meeting Thursday 19:00 — agenda `TO_12.08.pdf` from Anna",
and then uses what is actually in it.
**What it matches on**, in order: a date in the filename, subject or the
document's own heading against an event starting that day; words in common
between the agenda and the event's summary; both, which is the confident case.
Everything is labelled with its confidence and reason, so a wrong match is
visible rather than asserted, and an agenda that matches nothing is reported as
unattached rather than dropped.
**`TO` is special-cased.** It is the abbreviation everyone actually uses and also
the commonest two-letter word in English, so it is matched only as a standalone
uppercase token in a filename or subject — never in body prose. The long words
(Tagesordnung, Traktanden, agenda, Einladung; `AGENDA_KEYWORDS`) match
case-insensitively anywhere.
**The document is read.** `pypdf` extracts the text, the numbered and bulleted
lines are pulled out mechanically as `points` — a list extracted by code is a
list that cannot be invented — and both go into the context.
**An agenda's contents belong to the political section**, which is where a branch
agenda's party work belongs, and which means they reach only the people who
ticked the political digest. The split is deliberate and the prompts enforce it
from both sides:
- the **household** section says only *that* an agenda arrived, for which
meeting, from whom, and whether it could be read at all. It is told not to list
points or derive tasks even though the text is in front of it. Somebody with
the household digest and not the political one sees a meeting with an agenda,
not its contents.
- the **political** section carries `political-agenda` (the meeting and its
points as written) and **`political-todo`, titled "Political todos"** — what
the reader actually has to do before those meetings, one line each, verb first,
naming the meeting and quoting the line of the agenda or its covering message
the task came from. A task assigned to someone else is listed as theirs, so the
reader knows it is covered. If nothing is actually asked for, the window is
omitted — inventing preparation nobody asked for is the one failure here a
reader would act on. It stays its own window at `compact` too, since it is the
part of the digest people act on rather than read.
- the same agendas double as the section's sharpest relevance filter: a story
touching an agenda point outranks a bigger story that doesn't, and says why —
*"on Thursday's agenda"*.
`calendar` is therefore in the political section's `SECTION_SOURCES`: not for the
diary, but because it cannot match an agenda to a meeting it never fetched.
**A scanned agenda yields nothing** — it is a page of images and there is no OCR
here. That case attaches with `text_extracted: false`, and both prompts are told
they may name such a document but must not characterise it.
Attachments are spooled to `/data/attachments` (mail) and
`/data/whatsapp-bridge/documents` (the bridge, documents only — never photos or
video). That is a write to the digest's own volume, exactly as `context.json`
already is; nothing is written back to any mailbox or chat.
## Traffic data — what exists and what does not
@ -183,11 +529,28 @@ both 403'd the check — re-test those from the real network.
## Home network intrusion detection — what exists and what does not
`ingest/opnsense_ids.py` pulls a summary of the Suricata alerts your **existing
OPNsense firewall** raised during the digest window and folds it into the
**household** section as one short item, tagged `"category":
"network_security"`. Off by default (`ENABLE_OPNSENSE_IDS_INGEST=false`);
configured by `IDSconf.json` (template: `IDSconf.json.example`, real file
gitignored, same pattern as `digest-engine.env`).
OPNsense firewall** raised during the digest window, tagged `"category":
"network_security"`. Off by default (`ENABLE_OPNSENSE_IDS_INGEST=false`).
**The credentials live in `CoreSystemConfig.json`** — an `opnsense` block for the
address and tuning, `secrets.opnsense_api_key`/`opnsense_api_secret` for the key
pair OPNsense mints (System → Access → Users → API keys; only OPNsense can issue
them, so `generate-tokens.py` deliberately never invents one). A build writes
them into the container host's `/data/IDSconf.json` alongside every other
generated config. Field-by-field documentation stays in
`IDSconf.json.example`, which is also what `setup-container-host.sh` seeds when
you install by hand instead of from an image. Setting the credentials does **not**
enable the ingest: `ENABLE_OPNSENSE_IDS_INGEST=true` in `digest-engine.env` is
still a separate, deliberate step.
It is **its own section** (`synth/prompts/network.md`), not a couple of lines
inside the household one as it was originally built. The reason is the
per-person setting above: somebody who wants the calendar and the shopping list
but not a nightly intrusion-detection readout — or the reverse — can only say so
if the two are generated separately. The instructions themselves moved across
unchanged, including the one that matters most: alerts are signature matches,
never a claim that a device is compromised, and the section may never call the
network safe.
**Suricata is core, not a plugin.** There is no `os-suricata` to install — the
IDS module ships in OPNsense core and lives at Services → Intrusion Detection.
@ -238,7 +601,7 @@ rotation driven from the firewall's own cron (this needs shell access on
OPNsense — the GUI cron only schedules predefined configd actions — and enough
disk for ten-minute captures of a live link, which is not small). Then put a
one-line pointer in `IDSconf.json`'s `packet_capture_reference`; it is echoed
verbatim into the digest so the household section can say "raw captures are at
verbatim into the digest so the network section can say "raw captures are at
X". The digest engine never downloads, stores or analyses them.
## Household data — what exists and what does not
@ -393,7 +756,9 @@ What happens to something it flags:
either, since a transient failure in this pass specifically shouldn't cost
as much as the whole digest being down.
This doubles the number of Ollama calls per run (12 instead of 6) — against a
This doubles the number of Ollama calls per run (four calls per generated
section rather than two — so 16 with all four sections on, and fewer for every
section the household has switched off) — against a
local, self-hosted model with no per-token cost and nobody waiting on the
latency, the same tradeoff `synth/llm_client.py` already makes for generating
`compact` and `full` as separate passes rather than truncating one into the

399
digest-engine/agenda.py Normal file
View File

@ -0,0 +1,399 @@
"""Meeting agendas — attaching a "Tagesordnung" to the meeting it belongs to.
An agenda arrives as a PDF in mail or WhatsApp, days before the meeting it is
for. The meeting is in the calendar. Nothing connected the two, so the digest
would report "a message from Anna with an attachment" in one section and "branch
meeting, Thursday 19:00" in another, and leave the reader to notice.
This module makes that one fact: a calendar event that has an agenda says so,
naming the file, who sent it and where it arrived.
READING THE DOCUMENT
--------------------
The text is extracted (pypdf for PDFs, straight decode for plain text) and put in
the context, so the digest can list what is actually on the agenda, pull out what
the reader has been asked to prepare, and — in the political section — let this
week's agenda points steer which news is worth featuring. That last one is the
real payoff: "my branch is discussing rent controls on Thursday" is exactly the
filter that makes a news digest useful rather than merely comprehensive.
Extraction is best-effort and honest about failing. A scanned agenda is a page of
images and yields nothing; there is no OCR here and adding one would be a
different project. When nothing could be read, the document still attaches to its
meeting with `text_extracted: false`, and the prompts are told that a document
without text is one they may name but must not characterise.
The bytes never leave the household: the file is spooled to the digest's own
`/data` volume by the ingest modules, read here, and the extracted text goes only
into the local Ollama prompt like everything else.
HOW A MATCH IS MADE, IN ORDER
-----------------------------
1. **A date in the filename or subject** ("TO_12.08.pdf", "Tagesordnung 12.8.",
"agenda 2026-08-12") against an event starting that day. This is the strong
signal and it is tried first.
2. **Words in common** between the agenda's own text and an event's summary
("OG-Treffen" in both), among events in the lookahead window.
3. Both, which is the confident case and is labelled as such.
Anything that matches nothing is returned as an unattached agenda rather than
dropped — "an agenda arrived and I could not tell which meeting it is for" is
useful to a reader, and silently swallowing it would be the one outcome that
makes this feature worse than not having it.
WHY "TO" IS NOT MATCHED LIKE THE OTHER WORDS
--------------------------------------------
`TO` is the abbreviation everybody actually uses, and it is also the most common
two-letter word in English mail. It is therefore matched only as a standalone
uppercase token, and only in a filename or subject — never in body text. The
long words (Tagesordnung, Traktanden, agenda) are matched case-insensitively
anywhere, because they cannot collide with anything.
"""
import logging
import os
import re
from datetime import datetime, timezone
from pathlib import Path
LOG = logging.getLogger(__name__)
# Sources whose messages can carry an agenda. Telegram and Signal are here because a
# branch that uses them will send agendas there too; they contribute nothing unless
# their ingestion is enabled and actually carries attachment metadata.
MESSAGE_SOURCES = ("email", "whatsapp", "telegram", "signal", "discord")
DEFAULT_KEYWORDS = "tagesordnung,traktanden,traktandenliste,agenda,einladung"
# Standalone uppercase abbreviations, matched only in filenames and subjects.
ABBREVIATION_RE = re.compile(r"(?:^|[\s_\-\[(])((?:TO|TOP)\d*)(?:[\s_\-.\])]|$)")
DOCUMENT_TYPES = (".pdf", ".doc", ".docx", ".odt", ".rtf")
# 12.08.2026 / 12.8. / 2026-08-12 / 12-08-2026
DATE_PATTERNS = (
re.compile(r"(?P<year>20\d{2})[-_.](?P<month>\d{1,2})[-_.](?P<day>\d{1,2})"),
re.compile(r"(?P<day>\d{1,2})[-_.](?P<month>\d{1,2})[-_.](?P<year>20\d{2})"),
re.compile(r"(?P<day>\d{1,2})[-_.](?P<month>\d{1,2})\.?(?![\d.])"),
)
# Words too common to prove two strings are about the same meeting.
STOPWORDS = {
"der", "die", "das", "und", "für", "fur", "mit", "von", "zum", "zur", "des", "dem",
"the", "and", "for", "with", "our", "next", "meeting", "termin", "treffen",
"tagesordnung", "agenda", "einladung", "traktanden", "traktandenliste", "pdf",
}
MIN_TOKEN_LENGTH = 4
# How much of a document reaches the prompt. An agenda is a page or two; anything
# past this is a bundle of minutes and attachments that would crowd out the run's
# actual material.
MAX_TEXT_CHARS = 6000
MAX_PDF_PAGES = 12
# How much of a document counts as its heading for matching purposes — the block at
# the top where an agenda names its meeting and its date.
HEADING_CHARS = 300
# How much of it goes to the political section, which has to fit a world's worth of
# news around it. Enough for a page of agenda plus its preamble.
BRIEF_TEXT_CHARS = 3000
# Lines that look like agenda points: "1. Bericht", "TOP 3 — Kasse", "- Anträge".
POINT_RE = re.compile(r"^\s*(?:TOP\s*)?(?:\d{1,2}[.)]|[-•*])\s+(?P<text>\S.{2,160})$", re.IGNORECASE)
def _extract_text(path):
"""The document's text, or "" if it cannot be read. Never raises.
pypdf is imported here rather than at module scope so a missing or broken
optional dependency costs the agenda text and nothing else — the same rule
every ingestion module follows for its own client library.
"""
if not path:
return ""
file_path = Path(path)
if not file_path.is_file():
LOG.info("agenda: %s is not readable, keeping the document by name only", path)
return ""
suffix = file_path.suffix.lower()
try:
if suffix in (".txt", ".md"):
return file_path.read_text(encoding="utf-8", errors="replace")[:MAX_TEXT_CHARS]
if suffix != ".pdf":
# .docx/.odt are zip containers and .doc is a binary format; parsing them
# would mean another dependency for a case nobody in this household has
# hit yet. Named, not read — which the prompts handle.
return ""
from pypdf import PdfReader
reader = PdfReader(str(file_path))
pages = []
for page in reader.pages[:MAX_PDF_PAGES]:
pages.append(page.extract_text() or "")
return "\n".join(pages)[:MAX_TEXT_CHARS]
except Exception:
LOG.warning("agenda: could not extract text from %s", path, exc_info=True)
return ""
def _agenda_points(text):
"""The numbered/bulleted lines of an agenda, in order.
Extracted here rather than left to the model because these are the one part of
the document that has a reliable shape, and a list pulled out mechanically is a
list that cannot be invented. The model still gets the full text; this is what
it can quote from without having to find structure in a wall of PDF extraction.
"""
points = []
for line in (text or "").splitlines():
match = POINT_RE.match(line.strip())
if not match:
continue
point = " ".join(match.group("text").split())
if point and point not in points:
points.append(point)
return points[:30]
def _keywords():
raw = os.environ.get("AGENDA_KEYWORDS", DEFAULT_KEYWORDS)
return [word.strip().lower() for word in raw.split(",") if word.strip()]
def _looks_like_document(name, content_type):
name = (name or "").lower()
content_type = (content_type or "").lower()
if any(name.endswith(suffix) for suffix in DOCUMENT_TYPES):
return True
return "pdf" in content_type or "document" in content_type or "msword" in content_type
def _agenda_label(text, keywords):
"""Why this looked like an agenda, or None. Returned as a string because it ends
up in the digest: "matched on 'Tagesordnung'" is checkable by a human, "true" is
not."""
if not text:
return None
lowered = text.lower()
for word in keywords:
if word in lowered:
return word
match = ABBREVIATION_RE.search(text)
return match.group(1) if match else None
def _parse_date(text, reference_year):
"""First date-looking thing in `text`, as (year, month, day). A bare "12.8." takes
the year from the run itself — an agenda written without a year is always for the
near future, never for two years ago."""
if not text:
return None
for pattern in DATE_PATTERNS:
match = pattern.search(text)
if not match:
continue
groups = match.groupdict()
try:
day = int(groups["day"])
month = int(groups["month"])
year = int(groups.get("year") or reference_year)
except (TypeError, ValueError):
continue
if 1 <= month <= 12 and 1 <= day <= 31:
return year, month, day
return None
def _tokens(text):
words = re.split(r"[^0-9A-Za-zÄÖÜäöüß]+", (text or "").lower())
return {
word for word in words
if len(word) >= MIN_TOKEN_LENGTH and word not in STOPWORDS and not word.isdigit()
}
def _event_date(event):
raw = str(event.get("start") or "")
if not raw:
return None
try:
parsed = datetime.fromisoformat(raw.replace("Z", "+00:00"))
except ValueError:
# All-day events arrive as a bare date.
try:
parsed = datetime.strptime(raw[:10], "%Y-%m-%d")
except ValueError:
return None
return parsed.year, parsed.month, parsed.day
def _find_agendas(collected, keywords):
"""Every agenda-looking document in this run's messages, with the evidence."""
found = []
for source in MESSAGE_SOURCES:
for message in collected.get(source) or []:
if not isinstance(message, dict):
continue
attachments = [a for a in (message.get("attachments") or []) if isinstance(a, dict)]
documents = [
a for a in attachments
if _looks_like_document(a.get("filename"), a.get("content_type"))
]
if not documents:
continue
# The subject/caption counts as a label too: plenty of people send
# "Tagesordnung für Donnerstag" with the file named scan_0001.pdf.
heading = " ".join(str(message.get(key) or "") for key in ("subject", "body", "chat"))
for document in documents:
filename = str(document.get("filename") or "")
label = _agenda_label(filename, keywords)
matched_in = "filename"
if not label:
# The abbreviation is only trusted in a subject line, never in body
# prose — see the module docstring.
label = _agenda_label(str(message.get("subject") or ""), keywords)
matched_in = "subject"
if not label:
lowered = heading.lower()
label = next((word for word in keywords if word in lowered), None)
matched_in = "message text"
if not label:
continue
text = _extract_text(document.get("path"))
found.append({
"filename": filename or "(unnamed attachment)",
"content_type": document.get("content_type"),
"source": source,
"from": message.get("from"),
"chat": message.get("chat"),
"subject": message.get("subject"),
# The covering note. Kept because a good half of what a branch
# actually asks people to do is written here rather than in the
# numbered agenda — "bringt bitte eure Beiträge mit".
"body": str(message.get("body") or "")[:400],
"received_at": message.get("timestamp"),
"matched_keyword": label,
"matched_in": matched_in,
# False for a scanned agenda, an unsupported format, or a file
# the ingest could not spool. The prompts treat that case as
# "name it, do not characterise it".
"text_extracted": bool(text.strip()),
"points": _agenda_points(text),
"text": text.strip(),
})
return found
def attach(collected, reference_year=None):
"""Attaches agendas to calendar events in place and returns the unmatched ones.
Mutating the calendar entries is deliberate: the household section already
renders those events, so an event that carries its own `agenda_documents` needs
no second list to be cross-referenced against.
"""
events = [event for event in (collected.get("calendar") or []) if isinstance(event, dict)]
agendas = _find_agendas(collected, _keywords())
if not agendas:
return []
if not events:
LOG.info("agenda: %d agenda-looking document(s) but no calendar events to match", len(agendas))
return agendas
reference_year = reference_year or datetime.now(timezone.utc).year
unmatched = []
for agenda in agendas:
# The document's own heading — its first few lines, where an agenda puts the
# name of the meeting and its date. Only the heading: matching against the
# whole text would find a word in common with almost any event in the
# calendar and turn a confident match into a coincidence.
heading = (agenda.get("text") or "")[:HEADING_CHARS]
date = (
_parse_date(agenda["filename"], reference_year)
or _parse_date(agenda.get("subject"), reference_year)
or _parse_date(heading, reference_year)
)
agenda_tokens = _tokens(f"{agenda['filename']} {agenda.get('subject') or ''} {heading}")
best = None
for event in events:
same_day = date is not None and _event_date(event) == date
shared = agenda_tokens & _tokens(event.get("summary"))
if same_day and shared:
confidence, why = "high", f"same date and shared wording ({', '.join(sorted(shared))})"
elif same_day:
confidence, why = "medium", "the date in the file name matches this event"
elif shared:
confidence, why = "low", f"shared wording ({', '.join(sorted(shared))})"
else:
continue
rank = {"high": 3, "medium": 2, "low": 1}[confidence]
if best is None or rank > best[0]:
best = (rank, event, confidence, why)
if best is None:
unmatched.append(agenda)
continue
_, event, confidence, why = best
attached = dict(agenda)
attached["match_confidence"] = confidence
attached["match_reason"] = why
event.setdefault("agenda_documents", []).append(attached)
LOG.info(
"agenda: attached %r to %r (%s: %s)",
agenda["filename"], event.get("summary"), confidence, why,
)
if unmatched:
LOG.info("agenda: %d agenda(s) matched no event", len(unmatched))
return unmatched
def _brief(document, meeting, starts_at):
return {
"meeting": meeting,
"starts_at": starts_at,
"agenda_file": document.get("filename"),
"from": document.get("from"),
"received_at": document.get("received_at"),
# What was sent alongside the file. Half of what a branch actually asks
# people to do arrives in the covering message ("bringt eure Beiträge mit"),
# not in the numbered agenda itself.
"covering_message": " ".join(
part for part in (document.get("subject"), document.get("body")) if part
)[:400] or None,
"points": document.get("points") or [],
"text": (document.get("text") or "")[:BRIEF_TEXT_CHARS],
"text_extracted": document.get("text_extracted", False),
}
def upcoming(collected, unattached=None):
"""The agendas, for the political section.
This carries the document's actual text, not just its topics, because the
political section is where the agenda's content lives: it lists the points and
derives the reader's own to-do list from them. That placement is deliberate —
a branch agenda is party work, so it reaches only the people who asked for the
political digest, and a household member who did not is told a meeting has an
agenda without being shown what is on it.
The text is capped harder than the household section's copy: this prompt has a
world's worth of news to fit alongside it.
"""
briefs = []
for event in collected.get("calendar") or []:
if not isinstance(event, dict):
continue
for document in event.get("agenda_documents") or []:
briefs.append(_brief(document, event.get("summary"), event.get("start")))
for document in unattached or []:
briefs.append(_brief(document, None, None))
return briefs

428
digest-engine/archive.py Normal file
View File

@ -0,0 +1,428 @@
"""The digest's long memory — every ingested item, kept across runs.
A run is otherwise amnesiac: it fetches the last six hours, writes a digest, and
forgets. That makes some of the most valuable things this system could say
impossible to say at all. "This signature has fired every night for a week",
"unemployment is up for the third month", "merchant traffic through this
chokepoint has halved since May", "this story first appeared four days ago and
has not moved" — none of them are visible in one window, and all of them are the
kind of finding the political and network prompts are otherwise told they may
not make (see the "one snapshot is not a trend" rules).
This module is that memory: a SQLite database in the same `/data` volume the
Telegram session and IDSconf.json already live in, written at the end of
ingestion and read back at the start of the next run.
TWO TABLES, BECAUSE THERE ARE TWO KINDS OF THING
------------------------------------------------
`items` — discrete things that happen once and are then referred to again: an
article, a message, a video, a calendar event. Deduplicated on a fingerprint, so
an article that appears in four consecutive runs is ONE row that has been seen
four times, not four rows. That is what makes `first_seen_at` meaningful, and
`first_seen_at` is what tells the prompt whether it is looking at a new
development or the same story it showed you yesterday.
`observations` — numbers that are re-measured every run and only mean anything
as a series: an unemployment rate, a share price, an aircraft count in a region,
the number of IDS alerts on a signature. Stored one row per measurement so a
trend is a query rather than a guess.
WHAT THIS IS NOT
----------------
It is not a cache — nothing reads it to avoid fetching. It is not a second
source of truth: every claim in a digest still has to trace to this run's
context, and history enters the prompt as its own clearly-labelled block with
its own dates attached, so "in the last 30 days" can never be presented as
something that happened today.
It is also not free of consequence: this file persists the household's mail and
messages for as long as its retention window, where before them only the last
few runs' `context.json` did. `DIGEST_ARCHIVE_EXCLUDE_SOURCES` exists for
exactly that reason, and README.md says so plainly rather than burying it.
Every operation here is best-effort. A corrupt or unwritable database logs a
warning and the run continues without history — a digest with no memory is worth
far more than no digest, the same rule every ingestion module follows.
"""
import hashlib
import json
import logging
import os
import sqlite3
from datetime import datetime, timedelta, timezone
from pathlib import Path
LOG = logging.getLogger(__name__)
DEFAULT_DB_PATH = "/data/digest-archive.db"
DEFAULT_RETENTION_DAYS = 400
# Sources whose items are discrete things worth deduplicating and dating. Anything
# not listed contributes observations only (see _record_observations) or nothing.
ITEM_SOURCES = (
"news", "rci_social", "email", "signal", "telegram", "discord", "whatsapp",
"calendar", "grocy",
)
# How much of an item's own text takes part in its fingerprint. Long enough that two
# genuinely different messages don't collide, short enough that an article whose
# summary gets re-edited between runs is still recognised as the same article.
FINGERPRINT_TEXT_CHARS = 200
MAX_SERIES_POINTS = 60
def _now_iso():
return datetime.now(timezone.utc).replace(microsecond=0).isoformat().replace("+00:00", "Z")
def _db_path():
return Path(os.environ.get("DIGEST_ARCHIVE_DB_PATH", DEFAULT_DB_PATH))
def _excluded_sources():
raw = os.environ.get("DIGEST_ARCHIVE_EXCLUDE_SOURCES", "")
return {part.strip() for part in raw.split(",") if part.strip()}
def _connect():
path = _db_path()
path.parent.mkdir(parents=True, exist_ok=True)
conn = sqlite3.connect(path, timeout=15)
conn.row_factory = sqlite3.Row
conn.executescript(
"""
CREATE TABLE IF NOT EXISTS items (
id INTEGER PRIMARY KEY,
-- What makes two sightings the same thing. See _fingerprint().
fingerprint TEXT NOT NULL UNIQUE,
source TEXT NOT NULL,
category TEXT,
-- The item's own timestamp (when it was published/sent), which is not the
-- same as when this system first saw it — a feed can be hours behind.
occurred_at TEXT,
first_seen_at TEXT NOT NULL,
last_seen_at TEXT NOT NULL,
times_seen INTEGER NOT NULL DEFAULT 1,
first_run_id TEXT NOT NULL,
last_run_id TEXT NOT NULL,
title TEXT,
url TEXT,
payload TEXT NOT NULL
);
CREATE INDEX IF NOT EXISTS items_source_seen ON items (source, last_seen_at);
CREATE INDEX IF NOT EXISTS items_occurred ON items (occurred_at);
CREATE TABLE IF NOT EXISTS observations (
id INTEGER PRIMARY KEY,
-- Stable machine key ("fred:UNRATE", "ids:signature:2013028"), plus a human
-- label kept alongside it so a series can be read without a lookup table.
series TEXT NOT NULL,
label TEXT,
metric TEXT NOT NULL,
value REAL NOT NULL,
observed_at TEXT NOT NULL,
run_id TEXT NOT NULL,
-- One measurement per series per metric per timestamp: a re-run of the same
-- hour must not double-count, and FRED re-serving last month's figure must
-- not look like a fresh reading.
UNIQUE (series, metric, observed_at)
);
CREATE INDEX IF NOT EXISTS observations_series ON observations (series, observed_at);
"""
)
return conn
def _fingerprint(source, item):
"""What makes two sightings of a thing the same thing.
A URL when there is one — that is what an article, a video or a post actually
is. Otherwise the source plus the item's own timestamp and the head of its
text, which is stable for a message (they don't get edited under you) and
stable enough for a calendar event.
"""
url = str(item.get("url") or item.get("link") or "").strip()
if url:
return hashlib.sha256(f"{source}\n{url}".encode("utf-8")).hexdigest()
text = " ".join(
str(item.get(key) or "")
for key in ("title", "summary", "subject", "body", "message", "name", "chat")
)[:FINGERPRINT_TEXT_CHARS]
stamp = str(item.get("timestamp") or item.get("start") or item.get("occurred_at") or "")
return hashlib.sha256(f"{source}\n{stamp}\n{text}".encode("utf-8")).hexdigest()
def _item_title(item):
for key in ("title", "subject", "summary", "name", "chat"):
value = item.get(key)
if value:
return str(value)[:300]
body = item.get("body") or item.get("message") or ""
return str(body)[:300] or None
def _record_items(conn, run_id, seen_at, collected, excluded):
"""Upserts this run's items and annotates them in place with what the archive
already knew. The annotation is the whole point: every entry the prompt sees
carries `first_seen_at` and `times_seen`, so "new this run" and "the same story
for four days" are distinguishable without the model having to infer it."""
known = 0
for source in ITEM_SOURCES:
if source in excluded:
continue
for item in collected.get(source) or []:
if not isinstance(item, dict):
continue
fingerprint = _fingerprint(source, item)
row = conn.execute(
"SELECT first_seen_at, times_seen FROM items WHERE fingerprint = ?",
(fingerprint,),
).fetchone()
if row:
times_seen = row["times_seen"] + 1
conn.execute(
"UPDATE items SET last_seen_at = ?, last_run_id = ?, times_seen = ? "
"WHERE fingerprint = ?",
(seen_at, run_id, times_seen, fingerprint),
)
item["first_seen_at"] = row["first_seen_at"]
item["times_seen"] = times_seen
known += 1
else:
conn.execute(
"INSERT INTO items (fingerprint, source, category, occurred_at, "
"first_seen_at, last_seen_at, times_seen, first_run_id, last_run_id, "
"title, url, payload) VALUES (?, ?, ?, ?, ?, ?, 1, ?, ?, ?, ?, ?)",
(
fingerprint,
source,
item.get("category"),
item.get("timestamp") or item.get("start"),
seen_at,
seen_at,
run_id,
run_id,
_item_title(item),
item.get("url") or item.get("link"),
json.dumps(item, ensure_ascii=False, default=str),
),
)
item["first_seen_at"] = seen_at
item["times_seen"] = 1
return known
def _observation_rows(collected):
"""The numeric series worth trending, flattened out of this run's summaries.
Each source's shape is different and each is handled explicitly rather than by
a generic "find the numbers" walk — a wrong series is worse than a missing one,
because it would be trended and reported with a straight face.
"""
rows = []
for entry in collected.get("financial") or []:
if not isinstance(entry, dict):
continue
if entry.get("source") == "fred" and entry.get("series_id"):
try:
rows.append((
f"fred:{entry['series_id']}", entry["series_id"], "value",
float(entry["latest_value"]), str(entry.get("latest_date") or ""),
))
except (TypeError, ValueError):
pass
elif entry.get("source") == "stooq" and entry.get("symbol"):
try:
rows.append((
f"stooq:{entry['symbol']}", entry["symbol"], "close",
float(entry["close"]), str(entry.get("date") or ""),
))
except (TypeError, ValueError):
pass
for entry in collected.get("flight_traffic") or []:
if not isinstance(entry, dict) or not entry.get("region"):
continue
observed = str(entry.get("observed_at") or "")
region = entry["region"]
rows.append((f"flight:{region}", region, "aircraft_total",
float(entry.get("aircraft_total") or 0), observed))
rows.append((f"flight:{region}", region, "military_callsign_matches",
float(entry.get("military_callsign_match_count") or 0), observed))
for entry in collected.get("naval_traffic") or []:
if not isinstance(entry, dict) or not entry.get("region"):
continue
region = entry["region"]
rows.append((f"naval:{region}", region, "distinct_vessels",
float(entry.get("distinct_vessels") or 0),
str(entry.get("observed_at") or "")))
for entry in collected.get("opnsense_ids") or []:
if not isinstance(entry, dict):
continue
observed = str(entry.get("observed_at") or "")
rows.append(("ids:total", "IDS alerts", "alerts",
float(entry.get("alert_count") or 0), observed))
for signature in entry.get("top_signatures") or []:
if not isinstance(signature, dict) or signature.get("sid") is None:
continue
rows.append((
f"ids:signature:{signature['sid']}",
str(signature.get("signature") or signature["sid"])[:300],
"alerts", float(signature.get("count") or 0), observed,
))
for host in entry.get("top_local_hosts") or []:
if isinstance(host, dict) and host.get("ip"):
rows.append((f"ids:host:{host['ip']}", host["ip"], "alerts",
float(host.get("alerts") or 0), observed))
return [row for row in rows if row[4]]
def _prune(conn, retention_days):
cutoff = (datetime.now(timezone.utc) - timedelta(days=retention_days)).replace(
microsecond=0).isoformat().replace("+00:00", "Z")
conn.execute("DELETE FROM items WHERE last_seen_at < ?", (cutoff,))
conn.execute("DELETE FROM observations WHERE observed_at < ?", (cutoff,))
def record(run_id, collected):
"""Writes this run into the archive and annotates `collected` in place with what
was already known. Returns False if the archive is unavailable, in which case the
run simply proceeds without memory."""
if os.environ.get("ENABLE_DIGEST_ARCHIVE", "true").strip().lower() != "true":
LOG.info("archive: disabled, this run will neither read nor write history")
return False
excluded = _excluded_sources()
if excluded:
LOG.info("archive: not storing %s", ", ".join(sorted(excluded)))
seen_at = _now_iso()
try:
retention_days = int(os.environ.get("DIGEST_ARCHIVE_RETENTION_DAYS", DEFAULT_RETENTION_DAYS))
except ValueError:
retention_days = DEFAULT_RETENTION_DAYS
try:
with _connect() as conn:
known = _record_items(conn, run_id, seen_at, collected, excluded)
observations = [
row for row in _observation_rows(collected)
if row[0].split(":", 1)[0] not in excluded
]
for series, label, metric, value, observed_at in observations:
conn.execute(
"INSERT OR IGNORE INTO observations "
"(series, label, metric, value, observed_at, run_id) VALUES (?, ?, ?, ?, ?, ?)",
(series, label, metric, value, observed_at, run_id),
)
_prune(conn, retention_days)
LOG.info(
"archive: recorded run %s (%d item(s) already known, %d observation(s))",
run_id, known, len(observations),
)
return True
except Exception:
LOG.warning("archive: could not record this run, continuing without it", exc_info=True)
return False
def _series_history(conn, prefixes, points=MAX_SERIES_POINTS):
series = {}
for prefix in prefixes:
rows = conn.execute(
"SELECT series, label, metric, value, observed_at FROM observations "
"WHERE series LIKE ? ORDER BY observed_at DESC LIMIT ?",
(f"{prefix}%", points * 12),
).fetchall()
for row in rows:
key = f"{row['series']}|{row['metric']}"
entry = series.setdefault(key, {
"series": row["series"], "label": row["label"],
"metric": row["metric"], "readings": [],
})
if len(entry["readings"]) < points:
entry["readings"].append({"at": row["observed_at"], "value": row["value"]})
# Oldest-first reads like a series rather than a stack.
for entry in series.values():
entry["readings"].reverse()
return sorted(series.values(), key=lambda entry: entry["series"])
def _span_days(conn):
row = conn.execute(
"SELECT MIN(first_seen_at) AS oldest FROM items"
).fetchone()
if not row or not row["oldest"]:
return 0
try:
oldest = datetime.fromisoformat(str(row["oldest"]).replace("Z", "+00:00"))
except ValueError:
return 0
return max(0, (datetime.now(timezone.utc) - oldest).days)
def history(sections):
"""The history blocks for the sections this run is generating.
Deliberately shaped per section rather than handed over whole: the network
prompt has no use for share prices, and every token of history in a prompt is a
token not spent on this run's actual material.
"""
if os.environ.get("ENABLE_DIGEST_ARCHIVE", "true").strip().lower() != "true":
return {}
try:
with _connect() as conn:
span = _span_days(conn)
blocks = {}
if "political" in sections:
blocks["political"] = {
"archive_span_days": span,
"note": (
"Earlier readings of the same series, oldest first, each with its "
"own timestamp. This is the only basis on which this section may "
"describe anything as rising, falling or unchanged."
),
"series": _series_history(conn, ("fred:", "stooq:", "flight:", "naval:")),
}
if "network" in sections:
# Recurrence is the whole question for an IDS: one alert is noise, the
# same signature every night on the same host is a fact about the house.
signatures = conn.execute(
"SELECT series, label, SUM(value) AS alerts, COUNT(*) AS runs_seen, "
"MIN(observed_at) AS first_seen, MAX(observed_at) AS last_seen "
"FROM observations WHERE series LIKE 'ids:signature:%' "
"GROUP BY series ORDER BY alerts DESC LIMIT 12"
).fetchall()
hosts = conn.execute(
"SELECT label AS host, SUM(value) AS alerts, COUNT(*) AS runs_seen, "
"MIN(observed_at) AS first_seen, MAX(observed_at) AS last_seen "
"FROM observations WHERE series LIKE 'ids:host:%' "
"GROUP BY series ORDER BY alerts DESC LIMIT 10"
).fetchall()
blocks["network"] = {
"archive_span_days": span,
"note": (
"Alert history across every run in the archive, not just this "
"window. `runs_seen` is how many digest runs this has appeared in "
"— that is what tells a one-off apart from something recurring."
),
"alert_totals": _series_history(conn, ("ids:total",), points=30),
"recurring_signatures": [dict(row) for row in signatures],
"recurring_hosts": [dict(row) for row in hosts],
}
return blocks
except Exception:
LOG.warning("archive: could not read history, continuing without it", exc_info=True)
return {}

91
digest-engine/digest-engine.env.example
View File

@ -32,6 +32,85 @@ DIGEST_EVENING_HOUR=18
# feature can be checked without waiting for 18:00. Leave false in production.
DIGEST_FORCE_EVENING=false
# ---------------------------------------------------------------------------
# Who wants which digest — read from identity (Phase 6) at the start of a run.
# Each person ticks their sections (network / household / social / political) in
# identity's admin panel; this run generates the union of what the household
# asked for, and a section nobody wants costs no LLM call and no ingestion.
#
# Leave IDENTITY_URL blank to skip the lookup entirely and always generate all
# four — which is also what happens if identity is unreachable or the token is
# wrong. Failing that way round is deliberate: one unreachable container must
# never silently cost the household its whole digest. See preferences.py.
#
# identity is on the same compose network, so the container name resolves; the
# port is `ports.identity` in CoreSystemConfig.json (8097 by default). The token
# is the SAME value as identity.env's IDENTITY_TOKEN (`secrets.identity_token`
# in CoreSystemConfig.json) — it is a shared secret between the two, exactly
# like the one the door panel and kitchen display are built with.
# ---------------------------------------------------------------------------
IDENTITY_URL=http://identity:8097
IDENTITY_TOKEN=
IDENTITY_TIMEOUT=10
# ---------------------------------------------------------------------------
# "Your digest is ready" — a push to the self-hosted ntfy this stack already runs
# for chores and identity (notify.py). Costs no extra LLM call: the summary is
# assembled from each section's own compact narration, plus the political to-do
# count and any withheld/unverified marks, so it can never claim something the
# digest itself doesn't say.
#
# WHO GETS WHAT: people are grouped by their own ntfy topic from identity (the
# same one arrival notifications use), and each topic gets only the sections the
# people behind it actually asked for — a person who switched the political
# section off gets no political content on their phone either. NTFY_TOPIC is the
# household fallback for anyone with no topic of their own, and the single
# destination when identity is unreachable.
#
# On Android the notification relays to a Pebble or any other watch with nothing
# else installed. ntfy stays LAN-only; away from home it is reached over the
# WireGuard split tunnel (docs/network-integration.md §2.2), never a port forward.
#
# DIGEST_WEB_URL makes the notification tappable, opening that person's own
# digest. Leave it blank for a push with no link.
# ---------------------------------------------------------------------------
ENABLE_DIGEST_NOTIFY=true
NTFY_URL=http://ntfy
NTFY_TOPIC=digest
DIGEST_WEB_URL=
# ---------------------------------------------------------------------------
# The archive — the digest's long memory (archive.py). Every ingested item and
# every measured number is kept in a SQLite database in the same /data volume,
# so a run can say "up from 5.1% in June", "this signature has fired every night
# for a week", or "this story first appeared on Monday" instead of only ever
# seeing the last six hours.
#
# READ THIS BEFORE LEAVING IT ON: it persists your mail, messages and calendar
# entries for the whole retention window, where previously only the last few
# runs' context.json did. Nothing leaves the host, and the file sits next to the
# Telegram session on the same volume — but it is a real change in how long
# personal content is kept. DIGEST_ARCHIVE_EXCLUDE_SOURCES takes a comma-
# separated list of ingest keys (email, signal, telegram, discord, whatsapp,
# news, rci_social, calendar, grocy) to keep out of it entirely; the trend
# features for financial, traffic and IDS data work regardless of what you
# exclude here.
# ---------------------------------------------------------------------------
ENABLE_DIGEST_ARCHIVE=true
DIGEST_ARCHIVE_DB_PATH=/data/digest-archive.db
DIGEST_ARCHIVE_RETENTION_DAYS=400
DIGEST_ARCHIVE_EXCLUDE_SOURCES=
# ---------------------------------------------------------------------------
# Meeting agendas (agenda.py) — a "Tagesordnung" PDF arriving by mail or
# WhatsApp is attached to the calendar event it belongs to. Comma-separated
# keywords matched case-insensitively in a file name, subject or message; the
# bare abbreviations TO and TOP are always recognised too, but only as
# standalone tokens in a file name or subject (matching "to" in body text would
# flag half your mail). The document itself is never opened — only its name.
# ---------------------------------------------------------------------------
AGENDA_KEYWORDS=tagesordnung,traktanden,traktandenliste,agenda,einladung
# ---------------------------------------------------------------------------
# LLM synthesis — the existing Phase 3 Ollama host
# ---------------------------------------------------------------------------
@ -78,6 +157,18 @@ ENABLE_DISCORD_INGEST=false
# even with the headful-Chromium mitigation. Use a secondary/non-critical number.
ENABLE_WHATSAPP_INGEST=false
ENABLE_NEWS_INGEST=true
# How many entries each feed in curated-feeds.opml may contribute to one prompt.
# That file ships around three dozen feeds, so this multiplies: 5 x 40 feeds is
# already a large context for a 14B model. Raise it only with a bigger model.
NEWS_MAX_ENTRIES_PER_FEED=5
# The RCI's and your own section's social/media output (YouTube, podcast RSS, a
# public Telegram channel), listed in feeds/rci-social.json — no credentials
# needed for the RSS-based ones. Feeds the political section as "theory_social",
# and new episodes become its watch-later window. Telegram entries reuse the
# session TELEGRAM_* below already sets up, and are skipped without it.
ENABLE_RCI_SOCIAL_INGEST=true
RCI_SOCIAL_CONF_PATH=/app/feeds/rci-social.json
RCI_SOCIAL_MAX_PER_SOURCE=10
ENABLE_FINANCIAL_INGEST=true
# Off by default on purpose — read the OpenSky terms-of-use note further down
# before turning this on, it is a licensing decision, not a technical one.

437
digest-engine/feeds/curated-feeds.opml
View File

@ -2,15 +2,33 @@
<!--
Curated feed list for digest-engine's news ingestion.
This file is USER-EDITABLE and is meant to be edited. Only the marxist.com feed
below is a confirmed, deliberate choice (it is the theoretical basis the
political prompt reasons from — see synth/prompts/political.md). The exact
mainstream-outlet selection was explicitly left to the user's judgment and is
still an open decision in docs/project-plan.md §4.
This file is USER-EDITABLE and is meant to be edited. Only the RCI feeds below
are a confirmed, deliberate choice (they are the theoretical basis the political
prompt reasons from — see synth/prompts/political.md). Everything else is a
starting spread, not a canon.
Every <outline> with an xmlUrl is fetched. The category attribute is passed
through to the LLM: "theory" marks a feed whose analysis frames the section,
"news" marks a feed treated as raw reporting.
Every <outline> with an xmlUrl is fetched. Three attributes are passed through
to the LLM: "category" ("theory" frames the section, "news" is raw reporting,
"osint_military" is force-movement reporting), plus "owner" and "bias".
OWNER AND BIAS ARE NOT A RELIABILITY SCORE. They record who pays for an outlet
and whose politics it carries, because no outlet in this file is neutral and
synth/prompts/political.md is told to read every story against its ownership —
a private Western paper is the organ of a fraction of capital exactly as a
state wire is the organ of a capitalist state. Fill them in for anything you
add; the prompt falls back to "ownership not recorded here" when they are
missing, which is a worse answer than one honest line.
HOW MANY FEEDS IS TOO MANY: every feed here contributes up to
NEWS_MAX_ENTRIES_PER_FEED entries to one prompt. This file holds around three
dozen feeds, so keep that variable low (5 is the shipped default) unless you
have moved the digest to a model with a much larger context window than the
14B one this project defaults to.
VERIFICATION: every feed below was fetched and confirmed returning a real
RSS/Atom/RDF document on 2026-08-06, except the four marked VERIFY, which
answered 401/403/404 to that check from this machine (bot-blocking or a moved
path — re-test from the container host's own network before trusting them).
-->
<opml version="2.0">
<head>
@ -19,148 +37,433 @@
<body>
<!--
RCI (Revolutionary Communist International, formerly the IMT) theoretical
sources. All three fetched and confirmed returning real RSS 2.0 feeds
(2026-07-28). The RCI has national sections in 70+ countries
(marxist.com/links.htm) — these three are a representative sample
(flagship + one European + one English-language section), not the full
list; add more from links.htm the same way if you want deeper coverage.
sources. The RCI has national sections in 70+ countries
(marxist.com/links.htm) — these three are flagship + the user's own Austrian
section + the English-language paper, not the full list.
The section's social-media and podcast output is NOT here: it is a separate
source with a separate config, feeds/rci-social.json, tagged
"theory_social" so the prompt can tell a video announcement from an
analytical article. marxist.com's own podcast episodes do arrive through
this feed, posted as articles prefixed "[Podcast]".
-->
<outline text="Theory" title="Theory">
<outline type="rss"
text="In Defence of Marxism (marxist.com)"
title="In Defence of Marxism (marxist.com)"
category="theory"
owner="Revolutionary Communist International"
bias="Marxist; the analytical basis this section reasons from"
xmlUrl="https://www.marxist.com/feed/rss"
htmlUrl="https://www.marxist.com/"/>
<outline type="rss"
text="Der Funke (RCI Austria)"
title="Der Funke (RCI Austria)"
category="theory"
owner="Revolutionäre Kommunistische Partei (RKP), Austrian section of the RCI"
bias="Marxist; the household's own section"
xmlUrl="https://www.derfunke.at/feed/rss"
htmlUrl="https://www.derfunke.at/"/>
<outline type="rss"
text="The Communist (RCI, English-language)"
title="The Communist (RCI, English-language)"
category="theory"
owner="Revolutionary Communist Party (Britain), RCI section"
bias="Marxist"
xmlUrl="https://communist.red/feed/"
htmlUrl="https://communist.red/"/>
</outline>
<!--
EDIT THIS — mainstream + state-affiliated outlets. All fetched and confirmed
returning real RSS feeds (2026-07-28) except Washington Post, which blocked
the verification fetch (HTTP 403 — plausible bot-blocking rather than a dead
feed, since third-party feed directories list it as live) and should be
re-checked once digest-engine actually runs from its real network.
RT (Russia) and Global Times (China) are state-affiliated outlets, not
independent press — political.md is told to weigh them as such rather than
as neutral reporting. vol.at and derstandard.at are Austrian mainstream
(regional and national respectively), included per the user's request
alongside the RCI/Der Funke Austrian theoretical source above.
Workers', labour and movement press. Not "theory" — these are reporting,
written from inside the movement rather than about it, and the political
prompt weighs them as such: closer to the shop floor than any outlet below,
and with their own political lines that are not the RCI's.
-->
<outline text="Mainstream news" title="Mainstream news">
<outline text="Workers' and movement press" title="Workers' and movement press">
<outline type="rss"
text="Peoples Dispatch"
title="Peoples Dispatch"
category="news_labour"
owner="Tricontinental-linked left movement media, non-commercial"
bias="Communist/anti-imperialist; reports movements and left governments of the global south that the wires ignore"
xmlUrl="https://peoplesdispatch.org/feed/"
htmlUrl="https://peoplesdispatch.org/"/>
<outline type="rss"
text="Morning Star"
title="Morning Star"
category="news_labour"
owner="People's Press Printing Society, a readers' co-operative; politically close to the Communist Party of Britain"
bias="British communist daily; trade-union movement coverage found nowhere else in this file"
xmlUrl="https://morningstaronline.co.uk/rss.xml"
htmlUrl="https://morningstaronline.co.uk/"/>
<outline type="rss"
text="Labor Notes"
title="Labor Notes"
category="news_labour"
owner="Labor Education and Research Project, non-profit"
bias="US rank-and-file labour; strikes and contract fights reported from the members' side, not the union bureaucracy's"
xmlUrl="https://labornotes.org/rss.xml"
htmlUrl="https://labornotes.org/"/>
<outline type="rss"
text="Jacobin"
title="Jacobin"
category="news_labour"
owner="Jacobin Foundation, subscriber-funded"
bias="US social-democratic left; reformist rather than revolutionary — useful reporting, a politics this section does not share"
xmlUrl="https://jacobin.com/feed"
htmlUrl="https://jacobin.com/"/>
</outline>
<!--
Palestine and the region. Grouped deliberately so the split in ownership is
visible in one place: the first four are Palestinian, anti-Zionist or
movement outlets; the last two are Israeli papers, kept because what the
Israeli ruling class tells itself is evidence — of its own intentions, never
of what happened to Palestinians. See the "Zionist media" section of
synth/prompts/political.md for the zero-trust rule that applies to them.
-->
<outline text="Palestine and the region" title="Palestine and the region">
<outline type="rss"
text="The Electronic Intifada"
title="The Electronic Intifada"
category="news"
owner="independent, reader-funded, Palestinian-led"
bias="Anti-Zionist, Palestinian standpoint; openly partisan and says so"
xmlUrl="https://electronicintifada.net/rss.xml"
htmlUrl="https://electronicintifada.net/"/>
<outline type="rss"
text="Mondoweiss"
title="Mondoweiss"
category="news"
owner="independent US non-profit, reader-funded"
bias="Anti-Zionist; US-focused reporting on Palestine and the lobby"
xmlUrl="https://mondoweiss.net/feed/"
htmlUrl="https://mondoweiss.net/"/>
<outline type="rss"
text="+972 Magazine"
title="+972 Magazine"
category="news"
owner="Israeli-Palestinian journalists' collective, non-profit"
bias="Anti-occupation Israeli/Palestinian left; frequently the first to document what Israeli outlets will not"
xmlUrl="https://www.972mag.com/feed/"
htmlUrl="https://www.972mag.com/"/>
<outline type="rss"
text="Middle East Monitor"
title="Middle East Monitor"
category="news"
owner="London-based non-profit; funding not publicly itemised, editorially pro-Palestinian"
bias="Pro-Palestinian regional coverage; Islamist-sympathetic on some questions"
xmlUrl="https://www.middleeastmonitor.com/feed/"
htmlUrl="https://www.middleeastmonitor.com/"/>
<outline type="rss"
text="Haaretz"
title="Haaretz"
category="news_zionist"
owner="Schocken family (Amos Schocken) and Leonid Nevzlin, Israeli private capital"
bias="ZIONIST — liberal-Zionist, i.e. it opposes the occupation's methods while taking the settler-colonial state as given. Its own investigative desk has repeatedly documented Israeli army killings, so it is the most useful of the Israeli papers; it is still an Israeli bourgeois paper and gets the zero-trust rule."
xmlUrl="https://www.haaretz.com/cmlink/1.4605102"
htmlUrl="https://www.haaretz.com/"/>
<outline type="rss"
text="The Times of Israel"
title="The Times of Israel"
category="news_zionist"
owner="privately held (Seth Klarman, US hedge-fund capital, principal backer)"
bias="ZIONIST AND APOLOGIST FOR THE GENOCIDE IN GAZA. Reproduces IDF statements as fact, launders massacres into 'strikes on militants', and treats Palestinian casualty figures as contested while treating Israeli ones as given. Zero trust on any claim about Palestinians. Read only as evidence of what the Israeli state is saying about itself."
xmlUrl="https://www.timesofisrael.com/feed/"
htmlUrl="https://www.timesofisrael.com/"/>
</outline>
<!--
Anglo-European private capital's press. The house organs of different
fractions of the ruling class — finance capital (FT), the boardroom (WSJ),
liberal internationalism (Guardian, NYT, Economist). The prompt is told to
read them as such, and specifically that the business press is often the
most candid source in this file: it briefs capital honestly because capital
is the reader.
-->
<outline text="Private capital's press" title="Private capital's press">
<outline type="rss"
text="Financial Times — home"
title="Financial Times — home"
category="news"
owner="Nikkei Inc. (Japan)"
bias="The paper of internationally-minded finance capital. Candid about crises and profit rates because its readers must act on them"
xmlUrl="https://www.ft.com/rss/home"
htmlUrl="https://www.ft.com/"/>
<outline type="rss"
text="The Economist — International"
title="The Economist — International"
category="news"
owner="Exor (Agnelli family) with Rothschild and Schroder family stakes"
bias="Classical liberalism written for the people who run states and firms; its 'we' is the ruling class"
xmlUrl="https://www.economist.com/international/rss.xml"
htmlUrl="https://www.economist.com/"/>
<outline type="rss"
text="The Wall Street Journal — World"
title="The Wall Street Journal — World"
category="news"
owner="News Corp (Murdoch family control)"
bias="US business conservatism; the editorial page is openly the employers' side, the reporting is aimed at investors"
xmlUrl="https://feeds.a.dj.com/rss/RSSWorldNews.xml"
htmlUrl="https://www.wsj.com/world"/>
<outline type="rss"
text="The New York Times — World"
title="The New York Times — World"
category="news"
owner="New York Times Company; Sulzberger family control through dual-class shares"
bias="US liberal establishment; reliably follows Washington's framing on any war the US is party to"
xmlUrl="https://rss.nytimes.com/services/xml/rss/nyt/World.xml"
htmlUrl="https://www.nytimes.com/section/world"/>
<outline type="rss"
text="CNBC — International"
title="CNBC — International"
category="news"
owner="Comcast / NBCUniversal"
bias="Markets-first business television; states plainly what capital expects to happen next"
xmlUrl="https://search.cnbc.com/rs/search/combinedcms/view.xml?partnerId=wrss01&amp;id=100727362"
htmlUrl="https://www.cnbc.com/world/"/>
<outline type="rss"
text="The Guardian — World"
title="The Guardian — World"
category="news"
owner="Scott Trust Ltd, a private trust; no shareholders, no proprietor"
bias="British liberal centre-left and Atlanticist; the least proprietor-driven of the big British papers, still firmly within the establishment"
xmlUrl="https://www.theguardian.com/world/rss"
htmlUrl="https://www.theguardian.com/world"/>
<outline type="rss"
text="Washington Post — World (VERIFY: 401/403 to our check)"
title="Washington Post — World"
category="news"
owner="Jeff Bezos, sole owner (Amazon)"
bias="US liberal establishment, owned outright by one of the largest capitalists alive — treat its coverage of Amazon, logistics and labour accordingly"
xmlUrl="https://feeds.washingtonpost.com/rss/world"
htmlUrl="https://www.washingtonpost.com/world/"/>
</outline>
<!--
State broadcasters and state wires — of every bloc, filed together on
purpose. A state broadcaster is the outlet of a capitalist state whether the
state is NATO's or its rivals'; the prompt is told that "state-affiliated"
names who signs the cheque, not a special propaganda bucket that Western
outlets are exempt from.
-->
<outline text="State broadcasters and wires" title="State broadcasters and wires">
<outline type="rss"
text="BBC News — World"
title="BBC News — World"
category="news"
category="news_state_affiliated"
owner="British state: royal-charter corporation, licence fee, board appointments via government"
bias="The state broadcaster of a NATO power. Its house style reads as neutrality; on any war Britain is party to it follows the Foreign Office's frame"
xmlUrl="http://feeds.bbci.co.uk/news/world/rss.xml"
htmlUrl="https://www.bbc.com/news/world"/>
<outline type="rss"
text="Deutsche Welle — Top stories"
title="Deutsche Welle — Top stories"
category="news_state_affiliated"
owner="German federal state broadcaster, funded from the federal budget"
bias="Germany's outward-facing outlet; German foreign-policy framing by design, and formally committed to a pro-Israel editorial line"
xmlUrl="https://rss.dw.com/rdf/rss-en-all"
htmlUrl="https://www.dw.com/en/"/>
<outline type="rss"
text="France 24 — English"
title="France 24 — English"
category="news_state_affiliated"
owner="France Médias Monde, French state-owned"
bias="French state's foreign-facing outlet; reliable French/EU establishment framing, and the Françafrique blind spot that comes with it"
xmlUrl="https://www.france24.com/en/rss"
htmlUrl="https://www.france24.com/en/"/>
<outline type="rss"
text="Al Jazeera English — All"
title="Al Jazeera English — All"
category="news"
category="news_state_affiliated"
owner="Qatari state (ruling Al Thani family)"
bias="Gulf monarchy's outlet. The best-resourced on-the-ground reporting from Palestine in this file, and near-silence on Qatar itself and on Gulf migrant labour — both facts are the same fact"
xmlUrl="https://www.aljazeera.com/xml/rss/all.xml"
htmlUrl="https://www.aljazeera.com/"/>
<outline type="rss"
text="Middle East Eye"
title="Middle East Eye"
category="news"
owner="privately held, London; funding not disclosed and widely reported as Qatar-aligned"
bias="Critical of Israeli, Gulf-monarchy and US policy in the region; its own funding is the thing it does not report on"
xmlUrl="https://www.middleeasteye.net/rss"
htmlUrl="https://www.middleeasteye.net/"/>
<outline type="rss"
text="The Guardian — World"
title="The Guardian — World"
category="news"
xmlUrl="https://www.theguardian.com/world/rss"
htmlUrl="https://www.theguardian.com/world"/>
text="RT (Russia)"
title="RT (Russia)"
category="news_state_affiliated"
owner="ANO TV-Novosti, funded by the Russian state"
bias="The outlet of the Russian capitalist class and its state. Not 'the other side' — Russian capital's own imperialism is the one thing it will not report. Its critiques of NATO can be materially accurate and are made in the interests of a rival bourgeoisie"
xmlUrl="https://www.rt.com/rss/"
htmlUrl="https://www.rt.com/"/>
<outline type="rss"
text="Deutsche Welle — Top stories"
title="Deutsche Welle — Top stories"
category="news"
xmlUrl="https://rss.dw.com/rdf/rss-en-all"
htmlUrl="https://www.dw.com/en/"/>
text="TASS (Russia)"
title="TASS (Russia)"
category="news_state_affiliated"
owner="Russian state news agency"
bias="The Russian state's wire — read for what Moscow is announcing, not for what happened"
xmlUrl="https://tass.com/rss/v2.xml"
htmlUrl="https://tass.com/"/>
<outline type="rss"
text="Washington Post — World (VERIFY: blocked our check, see comment above)"
title="Washington Post — World"
category="news"
xmlUrl="https://feeds.washingtonpost.com/rss/world"
htmlUrl="https://www.washingtonpost.com/world/"/>
text="Global Times (China)"
title="Global Times (China)"
category="news_state_affiliated"
owner="People's Daily, i.e. the Communist Party of China"
bias="Chinese state, nationalist register. China's own capital export and its treatment of Chinese workers are the subjects it does not cover"
xmlUrl="https://www.globaltimes.cn/rss/outbrain.xml"
htmlUrl="https://www.globaltimes.cn/"/>
<outline type="rss"
text="VOL.AT (Vorarlberg, Austria)"
title="VOL.AT (Vorarlberg, Austria)"
text="CGTN — World (China)"
title="CGTN — World (China)"
category="news_state_affiliated"
owner="China Media Group, Chinese state"
bias="Beijing's outward-facing broadcaster; same blind spots as Global Times, calmer register"
xmlUrl="https://www.cgtn.com/subscribe/rss/section/world.xml"
htmlUrl="https://www.cgtn.com/"/>
<outline type="rss"
text="Anadolu Agency — English (Turkey)"
title="Anadolu Agency — English (Turkey)"
category="news_state_affiliated"
owner="Turkish state agency"
bias="Ankara's wire. Useful on Palestine and on Turkey's neighbourhood, silent on Kurds and on repression at home"
xmlUrl="https://www.aa.com.tr/en/rss/default?cat=guncel"
htmlUrl="https://www.aa.com.tr/en"/>
</outline>
<!--
Asia, Africa and Latin America — mostly the local bourgeois press, included
because a digest assembled only from London, New York and Moscow gets the
world wrong. Their national capitalist classes have their own interests, and
those interests are visible in the copy.
-->
<outline text="Asia, Africa, Latin America" title="Asia, Africa, Latin America">
<outline type="rss"
text="South China Morning Post — World"
title="South China Morning Post — World"
category="news"
xmlUrl="https://www.vol.at/rss"
htmlUrl="https://www.vol.at/"/>
owner="Alibaba Group"
bias="Hong Kong paper owned by Chinese capital; better sourced on China than the Western wires, constrained on anything Beijing treats as settled"
xmlUrl="https://www.scmp.com/rss/91/feed"
htmlUrl="https://www.scmp.com/"/>
<outline type="rss"
text="Nikkei Asia"
title="Nikkei Asia"
category="news"
owner="Nikkei Inc. (which also owns the FT)"
bias="Japanese capital's view of Asian supply chains and rearmament; business-first"
xmlUrl="https://asia.nikkei.com/rss/feed/nar"
htmlUrl="https://asia.nikkei.com/"/>
<outline type="rss"
text="The Hindu — International"
title="The Hindu — International"
category="news"
owner="The Hindu Group (Kasturi and Sons), Indian private capital"
bias="Indian liberal, historically left-of-centre and unusually willing to cover farmers' and workers' agitations"
xmlUrl="https://www.thehindu.com/news/international/feeder/default.rss"
htmlUrl="https://www.thehindu.com/"/>
<outline type="rss"
text="AllAfrica — latest"
title="AllAfrica — latest"
category="news"
owner="AllAfrica Global Media; aggregates African outlets, each item's real publisher named in the item"
bias="Aggregator, so the bias is whichever paper filed it — check the item's own attribution before leaning on it"
xmlUrl="https://allafrica.com/tools/headlines/rdf/latest/headlines.rdf"
htmlUrl="https://allafrica.com/"/>
<outline type="rss"
text="Daily Maverick (South Africa)"
title="Daily Maverick (South Africa)"
category="news"
owner="independent South African publisher, reader- and sponsor-funded"
bias="South African liberal; strong on state capture and mining capital, hostile to the organised left"
xmlUrl="https://www.dailymaverick.co.za/dmrss/"
htmlUrl="https://www.dailymaverick.co.za/"/>
<outline type="rss"
text="MercoPress (South America)"
title="MercoPress (South America)"
category="news"
owner="Mercosur-region news agency, Montevideo"
bias="South American business and trade press; commodity prices, ports and trade blocs, from capital's side"
xmlUrl="https://en.mercopress.com/rss/"
htmlUrl="https://en.mercopress.com/"/>
<outline type="rss"
text="Buenos Aires Times (Argentina)"
title="Buenos Aires Times (Argentina)"
category="news"
owner="Perfil group, Argentine private media capital"
bias="Argentine liberal-business English edition; the place to watch austerity, the IMF and the response to both"
xmlUrl="https://www.batimes.com.ar/feed"
htmlUrl="https://www.batimes.com.ar/"/>
</outline>
<!--
Austrian press, for the household's own country. Regional and national
bourgeois papers — the local political weather the Austrian section is
organising in.
-->
<outline text="Austria" title="Austria">
<outline type="rss"
text="DER STANDARD"
title="DER STANDARD"
category="news"
owner="Bronner family with a Süddeutsche Zeitung stake"
bias="Austrian national liberal daily"
xmlUrl="https://www.derstandard.at/rss"
htmlUrl="https://www.derstandard.at/"/>
<outline type="rss"
text="RT (Russia — state-affiliated)"
title="RT (Russia — state-affiliated)"
category="news_state_affiliated"
xmlUrl="https://www.rt.com/rss/"
htmlUrl="https://www.rt.com/"/>
<outline type="rss"
text="Global Times (China — state-affiliated)"
title="Global Times (China — state-affiliated)"
category="news_state_affiliated"
xmlUrl="https://www.globaltimes.cn/rss/outbrain.xml"
htmlUrl="https://www.globaltimes.cn/"/>
text="VOL.AT (Vorarlberg)"
title="VOL.AT (Vorarlberg)"
category="news"
owner="Russmedia, Vorarlberg private media group"
bias="Regional bourgeois press; local employers, local politics, no national line to speak of"
xmlUrl="https://www.vol.at/rss"
htmlUrl="https://www.vol.at/"/>
</outline>
<!--
Military/OSINT reporting. There is no free structured API for military
movements — see digest-engine/README.md, "Military movement" — so this is
the honest substitute: outlets that do the tracking themselves, carried
through to the political prompt under category="osint_military" so it can
tell reporting on troop and fleet movements apart from general news.
through under category="osint_military" so the prompt can tell reporting on
troop and fleet movements apart from general news.
All four fetched and confirmed returning real RSS 2.0 (2026-07-28).
Deliberately NOT included: Liveuamap (liveuamap.com/rss is a paid-API
signup page, not a feed), ISW/understandingwar.org and Long War Journal
(both returned 403 to the verification fetch — like Washington Post above,
plausibly bot-blocking rather than dead; re-check from the real network and
add them here if they work). None of these outlets is independent of the
states it reports on — political.md's class analysis applies to them as
much as to anything else in this file.
Deliberately NOT included: Liveuamap (liveuamap.com/rss is a paid-API signup
page, not a feed), ISW/understandingwar.org and Long War Journal (both 403 to
the verification fetch — re-check from the real network and add them here if
they work). None of these outlets is independent of the states it reports on;
most are read by, and sell to, the arms industry.
-->
<outline text="Military / OSINT" title="Military / OSINT">
<outline type="rss"
text="Bellingcat"
title="Bellingcat"
category="osint_military"
owner="Dutch-registered foundation; funders have included the NED and European governments"
bias="Open-source investigation, genuinely rigorous method, and a target selection that tracks Western foreign-policy priorities"
xmlUrl="https://www.bellingcat.com/feed/"
htmlUrl="https://www.bellingcat.com/"/>
<outline type="rss"
text="Naval News"
title="Naval News"
category="osint_military"
owner="private defence-trade publisher"
bias="Trade press for naval procurement — accurate on hulls and deployments, reads arms spending as good news"
xmlUrl="https://www.navalnews.com/feed/"
htmlUrl="https://www.navalnews.com/"/>
<outline type="rss"
text="The War Zone"
title="The War Zone"
category="osint_military"
owner="Recurrent Ventures, US private media"
bias="Well-sourced US military reporting written for enthusiasts of it"
xmlUrl="https://www.twz.com/feed"
htmlUrl="https://www.twz.com/"/>
<outline type="rss"
text="Defense News"
title="Defense News"
category="osint_military"
owner="Sightline Media Group, US defence-trade publisher"
bias="The arms industry's own trade paper — contracts and budgets reported as business news, because that is what they are to its readers"
xmlUrl="https://www.defensenews.com/arc/outboundfeeds/rss/"
htmlUrl="https://www.defensenews.com/"/>
</outline>

95
digest-engine/feeds/rci-social.json Normal file
View File

@ -0,0 +1,95 @@
{
"_readme": [
"Social-media accounts read by digest-engine/ingest/rci_social.py and fed to the",
"political section as category=\"theory_social\". JSON has no comment syntax, so",
"the documentation lives in these underscore-prefixed keys; the module ignores",
"every key it does not know about.",
"",
"THIS FILE IS MEANT TO BE EDITED — same as feeds/curated-feeds.opml, and unlike",
"IDSconf.json it holds no credentials, so it is committed as-is. Set",
"ENABLE_RCI_SOCIAL_INGEST=true in digest-engine.env to switch it on.",
"",
"Per entry:",
" platform 'youtube' | 'rss' | 'telegram'",
" name what the digest calls the account",
" scope 'section' (your own national section) or 'international' (the RCI",
" as a whole). Passed to the prompt, which is told not to merge the",
" two voices — 'my branch is doing X' is not 'the International says X'.",
" enabled set false to keep an entry here without fetching it",
"",
" content_type 'episode' (something to watch or listen to later) or 'post'",
" (something said now). Episodes are collected into the political",
" section's own watch-later window instead of competing with the",
" analysis. Defaults to 'episode' for youtube, 'post' otherwise.",
"",
" youtube needs channel_id, the opaque UC... id — YouTube's feed endpoint does",
" not accept an @handle. Find it by opening the channel and reading",
" 'externalId' in View Source, or from a /channel/UC... URL.",
" rss needs url. Works for a Mastodon account (https://<instance>/@user.rss),",
" a podcast feed, or any site feed.",
" telegram needs channel, the public @username without the @. Read through the",
" SAME Telethon session ingest/telegram_ingest.py uses — nothing joins",
" or subscribes to the channel, and without that session these entries",
" are skipped while the rest still run.",
"",
"NOT SUPPORTED, ON PURPOSE: Instagram, Facebook, WhatsApp channels, TikTok and",
"X/Twitter. None has a keyless read path that stays inside its own terms, and",
"this component does not scrape. The module docstring says what each of them",
"would actually require. Instagram is the real gap — it is where the section",
"posts most, and reading it here would mean scraping it."
],
"sources": [
{
"_verified": "Feed fetched and confirmed live 2026-08-06 (channel 'RKP – Revolutionäre Kommunistische Partei').",
"platform": "youtube",
"name": "RKP Österreich (YouTube)",
"scope": "section",
"channel_id": "UCMXItw4ipjk2CigCHuXHqlw"
},
{
"_verified": "Feed fetched and confirmed live 2026-08-06 (channel 'Revolutionary Communist International').",
"platform": "youtube",
"name": "RCI (YouTube)",
"scope": "international",
"channel_id": "UCtO2zDQt-AhF6SwYbXDdFKA"
},
{
"_verified": [
"Linked from marxist.com's own footer (checked 2026-08-06). NOT fetched during",
"verification — reading it needs the Telegram session, which only exists on the",
"real deployment. If the channel name is wrong this entry logs a warning and",
"costs nothing else."
],
"platform": "telegram",
"name": "marxist.com (Telegram)",
"scope": "international",
"channel": "marxistcom"
},
{
"_verified": [
"Feed fetched and confirmed live 2026-08-06 ('Der Funke - die kommunistische",
"Stimme'). This is the RSS behind anchor.fm/derfunkeat — the section's own",
"podcast, and the audio half of the watch-later list."
],
"platform": "rss",
"name": "Der Funke — Podcast",
"scope": "section",
"content_type": "episode",
"url": "https://anchor.fm/s/1b913a00/podcast/rss"
},
{
"_example": [
"Template for a Mastodon (or any other) feed — off until you fill in a real",
"URL. marxist.com itself needs no entry here: its podcast episodes are posted",
"as articles prefixed '[Podcast]' and already arrive through",
"feeds/curated-feeds.opml's category=\"theory\" feed (checked 2026-08-06)."
],
"platform": "rss",
"name": "Example — a Mastodon account",
"scope": "section",
"url": "https://instance.example/@account.rss",
"enabled": false
}
]
}

78
digest-engine/ingest/email_imap.py
View File

@ -10,15 +10,32 @@ is the documented, supported path for exactly this case.
import email
import email.utils
import hashlib
import logging
import os
import re
from datetime import datetime, timedelta, timezone
from email.header import decode_header, make_header
from pathlib import Path
LOG = logging.getLogger(__name__)
MAX_BODY_CHARS = 2000
# Where document attachments are spooled for agenda.py to read. Inside the digest's
# own /data volume, alongside the Telegram session and the archive database.
DEFAULT_ATTACHMENT_DIR = "/data/attachments"
DOCUMENT_SUFFIXES = (".pdf", ".doc", ".docx", ".odt", ".rtf", ".txt", ".md")
DOCUMENT_TYPES = {
"application/pdf",
"application/msword",
"application/vnd.openxmlformats-officedocument.wordprocessingml.document",
"application/vnd.oasis.opendocument.text",
"application/rtf",
"text/plain",
}
def _decode(value):
if not value:
@ -45,6 +62,66 @@ def _body_text(message):
return payload.decode(message.get_content_charset() or "utf-8", "replace")
def _safe_name(filename, fallback):
"""A filename that cannot escape the spool directory. Takes the basename, keeps
only characters that are unambiguously safe, and prefixes a short hash of the
original so two "Tagesordnung.pdf"s from different senders can coexist."""
base = os.path.basename(filename or "").strip() or fallback
cleaned = re.sub(r"[^A-Za-z0-9._-]+", "_", base)[:80].lstrip(".") or fallback
digest = hashlib.sha256((filename or fallback).encode("utf-8")).hexdigest()[:8]
return f"{digest}-{cleaned}"
def _attachments(message, uid):
"""Attachment metadata, plus the file itself for documents.
Documents are written to DIGEST_ATTACHMENT_DIR so `agenda.py` can read a meeting
agenda's text out of them. Everything else (images, calendar invites, the usual
signature logos) is recorded by name and type only — this is a digest, not a
mail archive, and there is no reason to spool a 4 MB photo to disk.
This writes to the digest's own data volume, exactly as `context.json` already
does. It is not a write against the mailbox: the message is fetched read-only,
the folder stays unread, and nothing is flagged, moved or deleted.
"""
if not message.is_multipart():
return []
spool = Path(os.environ.get("DIGEST_ATTACHMENT_DIR", DEFAULT_ATTACHMENT_DIR))
max_bytes = int(os.environ.get("EMAIL_MAX_ATTACHMENT_MB", "10")) * 1024 * 1024
save_documents = os.environ.get("EMAIL_SAVE_DOCUMENTS", "true").strip().lower() == "true"
found = []
for index, part in enumerate(message.walk()):
filename = part.get_filename()
disposition = str(part.get("Content-Disposition", ""))
if not filename and "attachment" not in disposition:
continue
name = _decode(filename) if filename else ""
content_type = part.get_content_type()
entry = {"filename": name, "content_type": content_type, "path": None}
is_document = name.lower().endswith(DOCUMENT_SUFFIXES) or content_type in DOCUMENT_TYPES
if save_documents and is_document:
try:
payload = part.get_payload(decode=True) or b""
if len(payload) > max_bytes:
LOG.info("email: attachment %r is %d bytes, not spooling it", name, len(payload))
else:
spool.mkdir(parents=True, exist_ok=True)
target = spool / _safe_name(name, f"uid{uid}-part{index}")
target.write_bytes(payload)
entry["path"] = str(target)
entry["bytes"] = len(payload)
except Exception:
# A document that cannot be spooled is still worth reporting by name.
LOG.warning("email: could not save attachment %r", name, exc_info=True)
found.append(entry)
return found
def fetch(lookback_hours):
host = os.environ.get("EMAIL_IMAP_HOST", "").strip()
user = os.environ.get("EMAIL_USERNAME", "").strip()
@ -94,6 +171,7 @@ def fetch(lookback_hours):
"subject": _decode(parsed.get("Subject")),
"timestamp": sent_at.isoformat() if sent_at else None,
"body": _body_text(parsed).strip()[:MAX_BODY_CHARS],
"attachments": _attachments(parsed, uid),
}
)
except Exception:

18
digest-engine/ingest/news_rss.py
View File

@ -5,6 +5,13 @@ The feed list is user-editable at digest-engine/feeds/curated-feeds.opml; each
OPML with `category="theory"` and that category is carried through onto every
entry, because the political prompt treats it as the analytical basis rather than
as one more headline source.
Three OPML attributes are passed through onto every entry: `category`, `owner`
and `bias`. The last two are what let the political prompt read a story against
who paid for it instead of treating "the news" as a single undifferentiated
input — see the source-criticism section of synth/prompts/political.md. They are
free text and are never parsed here; this module carries them, it does not
interpret them.
"""
import calendar
@ -39,6 +46,15 @@ def _read_opml(path):
"url": url,
"title": outline.get("title") or outline.get("text") or url,
"category": outline.get("category") or "news",
# Who owns the outlet and where it sits politically, both free text,
# both carried onto every entry. The political prompt is told to read
# each item against them rather than against a "reliable/unreliable"
# ranking: no outlet in this file is neutral, and which fraction of
# capital (or which capitalist state) pays for one is a fact about the
# reporting, not a footnote. Missing attributes just arrive as null —
# the prompt handles that as "ownership not recorded here".
"owner": outline.get("owner"),
"bias": outline.get("bias"),
}
)
return feeds
@ -94,6 +110,8 @@ def fetch(lookback_hours):
"source": "news",
"feed": feed["title"],
"category": feed["category"],
"owner": feed["owner"],
"bias": feed["bias"],
"title": entry.get("title", "").strip(),
"link": entry.get("link", ""),
"timestamp": published.isoformat() if published else None,

285
digest-engine/ingest/rci_social.py Normal file
View File

@ -0,0 +1,285 @@
"""RCI social-media ingestion — the organisation's own public output.
Feeds the **political** section alongside the RCI/Der Funke RSS already in
feeds/curated-feeds.opml, tagged `"category": "theory_social"` so
synth/prompts/political.md can tell the section's agitational and organisational
posts ("come to this meeting", "watch this explainer") apart from its written
analysis, which stays tagged `"theory"`. Sources are listed in the committed,
user-editable feeds/rci-social.json.
WHAT IS REACHABLE WITHOUT SCRAPING, AND WHAT IS NOT
---------------------------------------------------
Only platforms with a public, keyless, first-party read path are implemented.
That is a hard line, not a to-do list:
- **YouTube** — `https://www.youtube.com/feeds/videos.xml?channel_id=UC...`
is YouTube's own Atom feed, no key and no quota. Verified live against both
configured channels 2026-08-06.
- **Any RSS/Atom feed** — a Mastodon account's `.rss`, a podcast feed, a
section's own site. Same `feedparser` this project already vendors for news.
- **Telegram** — read through the Telethon session `telegram_ingest.py`
already logs in with. A public channel resolves by username and its history
reads **without joining it**: no `JoinChannelRequest`, no
`send_read_acknowledge()`, nothing written anywhere. If that session does
not exist, these entries are skipped and the RSS ones still run.
Deliberately NOT built, because none of them has a read path that is both
keyless and within the platform's terms — and this component's whole premise is
that it never scrapes and never logs in as a person to a platform that forbids
it (the one exception, WhatsApp, is opt-in and carries its own ban warning in
README.md):
- **Instagram** (`@rkp_austria`, `@revcomintern`) — the Basic Display API was
retired in December 2024, and the Graph API only reads accounts you own,
through a reviewed Meta app. Reading somebody else's public account means
scraping, which is both against Instagram's terms and the fastest way to get
an account or an IP blocked. This is the biggest real gap in this module:
Instagram is where the section posts most.
- **Facebook** (`/derfunke.at`) — page RSS was killed in 2018; Page Public
Content Access needs Meta app review for a business use case this is not.
- **WhatsApp channel** — no API of any kind. The existing `whatsapp-bridge`
sidecar is a linked *personal* device and its library's channel support is
experimental; wiring it up would extend that sidecar's ban risk to a feature
that can be had by reading the website instead.
- **X/Twitter** (`@revcomintern`) — the free API tier is write-oriented and
reads essentially nothing; Nitter instances are gone.
If you want those, the honest answer is to follow the accounts on your phone.
Do not "solve" it here by pointing this module at a scraping proxy: that moves
the terms-of-service problem onto a third party without removing it.
"""
import html
import json
import logging
import os
import re
from datetime import datetime, timedelta, timezone
LOG = logging.getLogger(__name__)
DEFAULT_CONF_PATH = "/app/feeds/rci-social.json"
# YouTube's own feed endpoint. Documented and keyless, but it takes the opaque
# channel id (`UC...`) only — an `@handle` is not accepted, so the config file
# holds ids and says where to find them.
YOUTUBE_FEED_URL = "https://www.youtube.com/feeds/videos.xml?channel_id={channel_id}"
MAX_BODY_CHARS = 1200
DEFAULT_MAX_PER_SOURCE = 10
# What a source publishes, unless its config entry says otherwise. A YouTube channel
# is episodes by definition; a plain feed could be either (a podcast is episodes, a
# Mastodon account is posts), so it defaults to the less presumptuous of the two.
DEFAULT_CONTENT_TYPES = {"youtube": "episode", "rss": "post", "telegram": "post"}
_TAG_RE = re.compile(r"<[^>]+>")
# Deliberately duplicated from news_rss.py rather than shared: every module in
# this package stands on its own so that one source's breakage can never reach
# another's, and a shared helpers module would be a single import for all of them
# to fail on. Same trade counter_run.py makes against llm_client.py.
def _plain_text(markup):
return html.unescape(_TAG_RE.sub(" ", markup or "")).strip()
def _entry_time(entry):
parsed = entry.get("published_parsed") or entry.get("updated_parsed")
if not parsed:
return None
import calendar
return datetime.fromtimestamp(calendar.timegm(parsed), tz=timezone.utc)
def _load_sources(path):
try:
with open(path, encoding="utf-8") as handle:
config = json.load(handle)
except FileNotFoundError:
LOG.warning("rci_social: %s not found, skipping", path)
return []
except Exception:
LOG.warning("rci_social: %s could not be parsed, skipping", path, exc_info=True)
return []
sources = []
for entry in config.get("sources") or []:
if not isinstance(entry, dict) or entry.get("enabled") is False:
continue
platform = str(entry.get("platform") or "").strip().lower()
if platform not in ("rss", "youtube", "telegram"):
LOG.warning("rci_social: unknown platform %r, skipping that entry", platform)
continue
sources.append(
{
"platform": platform,
"name": str(entry.get("name") or "").strip() or platform,
# "episode" (something to watch or listen to later) vs "post"
# (something said now). The political prompt keeps episodes out of
# the analysis and puts them in their own watch-later window, so a
# 40-minute video never competes with a strike for space.
"content_type": str(entry.get("content_type") or DEFAULT_CONTENT_TYPES.get(platform, "post")).strip().lower(),
# `scope` is passed straight through to the prompt: "section" is the
# user's own national section, "international" the RCI as a whole.
# The two are not the same voice and the digest should not merge them.
"scope": str(entry.get("scope") or "section").strip().lower(),
"url": str(entry.get("url") or "").strip(),
"channel_id": str(entry.get("channel_id") or "").strip(),
"channel": str(entry.get("channel") or "").strip().lstrip("@"),
}
)
return sources
def _item(source, title, body, url, timestamp):
return {
"source": "rci_social",
# Not "theory": these are posts, not the written analysis the political
# prompt reasons from. See that prompt's "The organisation's own social
# media" section for the difference it is told to keep.
"category": "theory_social",
"platform": source["platform"],
"account": source["name"],
"scope": source["scope"],
"content_type": source["content_type"],
"title": title,
"url": url,
"timestamp": timestamp,
"body": (body or "")[:MAX_BODY_CHARS],
}
def _fetch_feed(source, since, max_per_source):
import feedparser
url = source["url"]
if source["platform"] == "youtube":
if not source["channel_id"]:
LOG.warning("rci_social: %s has no channel_id, skipping", source["name"])
return []
url = YOUTUBE_FEED_URL.format(channel_id=source["channel_id"])
if not url:
LOG.warning("rci_social: %s has no url, skipping", source["name"])
return []
parsed = feedparser.parse(url)
items = []
for entry in parsed.entries[:max_per_source]:
published = _entry_time(entry)
if published is None or published < since:
continue
item = _item(
source,
_plain_text(entry.get("title")),
_plain_text(entry.get("summary") or entry.get("description")),
entry.get("link") or url,
published.isoformat(),
)
# Podcast feeds carry <itunes:duration>; YouTube's Atom feed does not. Passed
# through when it exists because "is this 8 minutes or 90" is most of what
# decides whether something makes it onto a watch-later list.
duration = entry.get("itunes_duration")
if duration:
item["duration"] = str(duration)
items.append(item)
return items
async def _fetch_telegram_async(sources, since, api_id, api_hash, session_path, max_per_source):
from telethon import TelegramClient
items = []
client = TelegramClient(session_path, api_id, api_hash)
await client.connect()
try:
if not await client.is_user_authorized():
LOG.warning(
"rci_social: telegram session at %s is not authorized — run "
"`python ingest/telegram_login.py` once; skipping the telegram sources",
session_path,
)
return []
for source in sources:
try:
# Resolving a public channel by username is a read. Nothing here
# joins it, subscribes to it, or marks anything read — same
# invariant as telegram_ingest.py, which never calls
# send_read_acknowledge() either.
entity = await client.get_entity(source["channel"])
async for message in client.iter_messages(entity, limit=max_per_source):
if message.date is None or message.date < since:
break
if not message.message:
continue
items.append(
_item(
source,
"",
message.message,
f"https://t.me/{source['channel']}/{message.id}",
message.date.isoformat(),
)
)
except Exception:
LOG.warning(
"rci_social: could not read telegram channel %r, skipping",
source["channel"],
exc_info=True,
)
finally:
await client.disconnect()
return items
def _fetch_telegram(sources, since, max_per_source):
api_id = os.environ.get("TELEGRAM_API_ID", "").strip()
api_hash = os.environ.get("TELEGRAM_API_HASH", "").strip()
session_path = os.environ.get("TELEGRAM_SESSION_PATH", "/data/telegram.session")
# The same credentials and session file telegram_ingest.py uses. A household
# that never set Telegram up simply doesn't get these sources; it is not an
# error and must not cost the RSS ones.
if not (api_id and api_hash and os.path.exists(session_path)):
LOG.info("rci_social: no usable telegram session, skipping the telegram sources")
return []
import asyncio
return asyncio.run(
_fetch_telegram_async(sources, since, int(api_id), api_hash, session_path, max_per_source)
)
def fetch(lookback_hours):
sources = _load_sources(os.environ.get("RCI_SOCIAL_CONF_PATH", DEFAULT_CONF_PATH))
if not sources:
return []
max_per_source = int(os.environ.get("RCI_SOCIAL_MAX_PER_SOURCE", DEFAULT_MAX_PER_SOURCE))
since = datetime.now(timezone.utc) - timedelta(hours=float(lookback_hours))
items = []
for source in sources:
if source["platform"] == "telegram":
continue
try:
items.extend(_fetch_feed(source, since, max_per_source))
except Exception:
# One dead account never costs the others, exactly like a dead feed in
# news_rss.py — a section that renamed a channel shouldn't silence the rest.
LOG.warning("rci_social: %s failed, skipping it", source["name"], exc_info=True)
telegram_sources = [source for source in sources if source["platform"] == "telegram"]
if telegram_sources:
try:
items.extend(_fetch_telegram(telegram_sources, since, max_per_source))
except Exception:
LOG.warning("rci_social: telegram ingestion failed, skipping it", exc_info=True)
items.sort(key=lambda item: item["timestamp"], reverse=True)
LOG.info("rci_social: %d post(s) in the last %sh", len(items), lookback_hours)
return items

24
digest-engine/ingest/whatsapp_ingest.py
View File

@ -22,6 +22,7 @@ import json
import logging
import os
from datetime import datetime, timedelta, timezone
from pathlib import Path
LOG = logging.getLogger(__name__)
@ -46,6 +47,11 @@ def fetch(lookback_hours):
LOG.warning("whatsapp: could not claim %s for reading, skipping", messages_path, exc_info=True)
return []
# The bridge writes documents next to its message file, in its own /data. Derived
# from the message path rather than configured separately, so the two can never
# be pointed at different places by half-updating the environment.
documents_dir = Path(messages_path).parent / "documents"
since = datetime.now(timezone.utc) - timedelta(hours=lookback_hours)
messages = []
@ -70,6 +76,24 @@ def fetch(lookback_hours):
"chat": record.get("chat"),
"timestamp": sent_at.isoformat() if sent_at else None,
"body": (record.get("body") or "")[:MAX_BODY_CHARS],
# Shaped like email's `attachments` so agenda.py can treat
# both the same way. `path` is filled only for documents
# the bridge actually saved; photos and video are recorded
# by type and never downloaded. Older bridge lines predate
# these fields and simply carry no attachment.
"attachments": (
[{
"filename": record.get("filename") or "",
"content_type": record.get("mimetype") or record.get("type") or "",
"path": (
str(documents_dir / record["document_file"])
if record.get("document_file")
else None
),
}]
if record.get("has_media")
else []
),
}
)
except Exception:

249
digest-engine/notify.py Normal file
View File

@ -0,0 +1,249 @@
"""'Your digest is ready' — a push with enough of the digest to judge it by.
The digest is generated four times a day and shown only when somebody asks for it
— a spoken "play my digest", or the button in Home Assistant. Nothing displays it
because a person walked past a screen. That makes the notification the only thing
that reaches you unprompted, and its job is exactly one decision: is this run
worth going and asking for, or does it keep until tonight?
So it pushes a short summary to ntfy after every run, readable on a phone or a
wrist (Android relays notifications to a Pebble with nothing else installed),
without opening the canvas at all.
WHAT IT SENDS, AND WHY IT COSTS NO EXTRA LLM CALL
-------------------------------------------------
Every section already produces a `narration` at `compact` detail — two to four
sentences written to be read aloud, which is exactly the register a notification
wants. The summary is those narrations, trimmed, plus the few things that are
worth knowing before you open anything: how many political to-dos this run
produced, and whether any section was withheld or came back unverified. Nothing
here is generated: it is assembled from the documents the run already wrote, so a
notification can never claim something the digest itself does not say.
WHO GETS WHAT
-------------
The per-person section settings (see preferences.py) decide the content of the
push, not just the canvas. Somebody who has switched the political section off
does not get political content pushed to their phone — that would make the
setting a lie in the one place it is most visible.
People are grouped by their own ntfy topic (identity's `notify_topic`, the same
one arrival notifications use). Each topic gets the union of the sections wanted
by the people who share it: a topic IS its audience, so two people sharing one
have already agreed to share what arrives on it. Anyone without a topic of their
own falls back to `NTFY_TOPIC`, the household topic — and with no preferences
known at all (identity down, nobody registered), one message about everything
generated goes there.
NETWORK PLACEMENT
-----------------
This POSTs to the self-hosted ntfy already running in this compose stack for
`chores` and `identity` — container to container, never near the firewall. Getting
it onto a phone away from home is the WireGuard split tunnel described in
docs/network-integration.md §2.2, not a port forward. Nothing new is exposed.
Best-effort throughout: a failed push logs a warning. The digest is already
written by the time this runs, and no notification is worth failing a run over.
"""
import logging
import os
from urllib.parse import urlencode
import requests
LOG = logging.getLogger(__name__)
HTTP_TIMEOUT = 10
# How much of a section's narration survives into the push. ntfy shows a few lines
# before truncating, and a Pebble shows fewer still — the point is to be enough to
# judge by, not to be the digest.
MAX_SECTION_CHARS = 220
MAX_BODY_CHARS = 1400
SECTION_LABELS = {
"personal": "Social",
"political": "Political",
"household": "Household",
"network": "Network",
}
# ntfy renders these as icons next to the title.
SECTION_TAGS = {
"personal": "speech_balloon",
"political": "newspaper",
"household": "house",
"network": "shield",
}
def _trim(text, limit):
text = " ".join(str(text or "").split())
if len(text) <= limit:
return text
# Cut at a sentence end when there is one in reach, so the push doesn't stop
# mid-clause; otherwise at a word boundary.
window = text[:limit]
for stop in (". ", "! ", "? "):
cut = window.rfind(stop)
if cut > limit * 0.5:
return window[: cut + 1].strip()
cut = window.rfind(" ")
return (window[:cut] if cut > 0 else window).rstrip() + "…"
def _todo_count(document):
"""How many political to-dos this run produced, or None if there is no todo
window. Counted from the window's own content rather than inferred from prose —
a number in a notification has to be a number the digest actually contains."""
for window in document.get("windows") or []:
if str(window.get("id")) != "political-todo":
continue
content = window.get("content")
if isinstance(content, list):
return len(content)
# A todo window that came back as prose still means "there are some".
return 1 if str(content or "").strip() else 0
return None
def _section_lines(documents, sections):
"""One line per section, in the order the canvas lays them out."""
by_section = {
document.get("section"): document
for document in documents.get("compact") or []
if isinstance(document, dict)
}
lines = {}
for section in sections:
document = by_section.get(section)
if not document:
continue
label = SECTION_LABELS.get(section, section.title())
if document.get("withheld"):
lines[section] = f"{label}: withheld — nothing could be verified against its sources."
continue
text = _trim(document.get("narration"), MAX_SECTION_CHARS)
if not text:
# A section with no narration still ran; saying so is more useful than
# leaving a gap the reader has to interpret.
text = "generated, no narration."
if document.get("degraded"):
text = f"unavailable — {text}"
elif document.get("unverified"):
text = f"[unverified] {text}"
if section == "political":
todos = _todo_count(document)
if todos:
text = f"{todos} todo{'s' if todos != 1 else ''}. {text}"
lines[section] = f"{label}: {text}"
return lines
def _click_url(person_name):
"""Where the notification takes you when tapped: this person's own digest."""
base = os.environ.get("DIGEST_WEB_URL", "").strip().rstrip("/")
if not base:
return None
if not person_name:
return f"{base}/full.html"
return f"{base}/full.html?{urlencode({'person': person_name})}"
def _audiences(prefs, sections, default_topic):
"""topic -> (sections that topic may see, the names behind it).
Grouping rather than one message per person: without it a household where
nobody has set a personal topic would get one identical push per registered
person, four times a day, which is how a useful notification becomes one people
turn off.
"""
if prefs is None:
return {default_topic: (list(sections), [])} if default_topic else {}
grouped = {}
for person in prefs["people"]:
topic = (person.get("notify_topic") or default_topic or "").strip()
wanted = [section for section in sections if section in person["digest_sections"]]
if not topic:
LOG.info("notify: %s has no ntfy topic and no household default, skipping", person["name"])
continue
if not wanted:
continue
visible, names = grouped.setdefault(topic, (set(), []))
visible.update(wanted)
names.append(person["name"])
return {
topic: ([section for section in sections if section in visible], names)
for topic, (visible, names) in grouped.items()
}
def _post(base_url, topic, title, body, tags, click):
"""Published as JSON to ntfy's root rather than as text to /<topic>.
The header-based form (`Title:`, `Tags:`) is what `chores` uses and is fine for
its plain-English nudges, but HTTP headers are latin-1: a title or a name with
an em dash, an umlaut or a euro sign in it raises UnicodeEncodeError before the
request is even sent. This digest quotes news headlines and household member
names, so that would have failed on real content and worked on every test that
used ASCII. The JSON body is UTF-8 throughout.
"""
payload = {"topic": topic, "title": title, "message": body}
if tags:
payload["tags"] = tags
if click:
payload["click"] = click
response = requests.post(base_url, json=payload, timeout=HTTP_TIMEOUT)
response.raise_for_status()
def send(context, documents, sections, prefs):
"""Pushes one summary per audience. Never raises; returns how many went out."""
if os.environ.get("ENABLE_DIGEST_NOTIFY", "true").strip().lower() != "true":
return 0
base_url = os.environ.get("NTFY_URL", "").strip().rstrip("/")
if not base_url:
LOG.info("notify: NTFY_URL is not set, not pushing a digest notification")
return 0
default_topic = os.environ.get("NTFY_TOPIC", "").strip()
audiences = _audiences(prefs, sections, default_topic)
if not audiences:
LOG.info("notify: nobody to notify about this run")
return 0
lines = _section_lines(documents, sections)
slot = str(context.get("slot_hour", "")).zfill(2)
sent = 0
for topic, (visible, names) in audiences.items():
body = "\n".join(lines[section] for section in visible if section in lines)
if not body:
continue
if len(body) > MAX_BODY_CHARS:
body = body[:MAX_BODY_CHARS].rstrip() + "…"
# One name in the click-through: a shared topic has no single owner, so it
# gets the unfiltered page rather than an arbitrary person's.
click = _click_url(names[0] if len(names) == 1 else None)
labels = [SECTION_LABELS.get(section) or str(section) for section in visible]
title = f"Digest {slot}:00 — {', '.join(labels)}"
tags = [SECTION_TAGS[section] for section in visible if section in SECTION_TAGS]
try:
_post(base_url, topic, title, body, tags, click)
sent += 1
LOG.info("notify: pushed the digest summary to %r (%s)", topic, ", ".join(visible))
except Exception:
LOG.warning("notify: could not push to %r", topic, exc_info=True)
return sent

124
digest-engine/preferences.py Normal file
View File

@ -0,0 +1,124 @@
"""Who wants which digest — read from `identity` at the start of every run.
`identity` (Phase 6) is this project's source of truth for per-person household
facts: it already owns chore exemptions, reminder styles and arrival-notification
settings for exactly the same reason it owns this one. Nothing about a person's
preferences is stored here; this module is a read of `GET /digest-preferences`
and nothing else.
WHAT THE PREFERENCE ACTUALLY CONTROLS
-------------------------------------
Two different things, and they are worth keeping apart:
1. **What gets generated.** run.py generates the UNION of the sections the
household asked for. A section nobody has ticked is never sent to the LLM at
all — no synthesis call, no counter-run call, and it never reaches output/.
That is the sense in which this is a per-person "should this be generated for
me" toggle rather than a display setting bolted on at the end.
2. **What each surface shows.** Each person's own subset is written into the
digest alongside the documents, and the renderer filters to it. That half is
a **display filter, not an access control**: digest-web serves the whole
output volume read-only to anything on the LAN, so anyone who can open the
canvas can open the JSON behind it. Ticking a box off keeps a section off
someone's screen and out of their narration; it does not make it secret from
them. Say so plainly rather than implying a boundary this stack doesn't have.
FAILING SAFE MEANS GENERATING MORE, NOT LESS
--------------------------------------------
Every failure here — identity disabled, unreachable, a bad token, a malformed
response — returns None, which run.py treats as "no preferences known" and falls
back to generating every section, exactly as this component did before the
setting existed. The alternative reading of a failed lookup ("nobody asked for
anything, generate nothing") would let one unreachable container silently cost
the household its whole digest, which is far worse than one run that shows a
section someone had opted out of.
The same reasoning applies one level down: a household that runs `identity` but
has not registered anybody yet is treated as "no preferences known" too, because
an empty person table is the state of a fresh install, not a decision. Everyone
opting out of everything IS a decision, and it is honoured (loudly).
"""
import logging
import os
import requests
LOG = logging.getLogger(__name__)
DEFAULT_TIMEOUT = 10.0
def fetch(known_sections):
"""`{"people": [...], "wanted": [...]}` as identity reports it, or None.
`known_sections` is this component's own section list — anything identity names
that this version of digest-engine has no prompt for is dropped here rather than
handed to a synthesis pass that would fail on the missing file.
"""
base_url = os.environ.get("IDENTITY_URL", "").strip().rstrip("/")
token = os.environ.get("IDENTITY_TOKEN", "").strip()
if not base_url:
LOG.info("IDENTITY_URL is not set, generating every digest section")
return None
try:
response = requests.get(
f"{base_url}/digest-preferences",
headers={"Authorization": f"Bearer {token}"} if token else {},
timeout=float(os.environ.get("IDENTITY_TIMEOUT", DEFAULT_TIMEOUT)),
)
response.raise_for_status()
payload = response.json()
if not isinstance(payload, dict):
raise ValueError("response was not a JSON object")
except Exception:
LOG.warning(
"could not read digest preferences from %s, generating every section",
base_url,
exc_info=True,
)
return None
people = []
for entry in payload.get("people") or []:
if not isinstance(entry, dict) or not entry.get("name"):
continue
people.append(
{
"id": entry.get("id"),
"name": str(entry["name"]),
"nickname": entry.get("nickname"),
"digest_sections": [
section
for section in known_sections
if section in (entry.get("digest_sections") or [])
],
}
)
if not people:
LOG.info("identity knows no people yet, generating every digest section")
return None
wanted = [
section
for section in known_sections
if any(section in person["digest_sections"] for person in people)
]
LOG.info(
"digest preferences: %d person(s), generating %s",
len(people),
", ".join(wanted) or "nothing — everybody has opted out of every section",
)
return {"people": people, "wanted": wanted}
def sections_to_generate(prefs, known_sections):
"""The sections this run should actually synthesise. Everything, when nothing is
known about who wants what (see the module docstring on why that is the safe
direction); otherwise exactly the union the household asked for.
"""
if prefs is None:
return list(known_sections)
return list(prefs["wanted"])

84
digest-engine/render/digest-canvas-sdk/glow.css
View File

@ -159,6 +159,90 @@
align-items: center;
}
/* ---------------------------------------------------------------------------
* Sources — the fold-out citation block under a window or a globe marker's
* brief. Collapsed by default: the digest is read across a room, so the claim is
* what you see and the receipts are one tap away. <details> does the folding
* natively, so there is no JS and no state to keep in sync.
* --------------------------------------------------------------------------- */
.digest-sources {
margin-top: 0.8rem;
border-top: 1px solid var(--digest-edge);
padding-top: 0.5rem;
font-size: 0.82rem;
}
.digest-sources > summary {
cursor: pointer;
color: var(--digest-accent);
letter-spacing: 0.04em;
text-transform: uppercase;
font-size: 0.72rem;
/* A big enough hit area for a fingertip on the kitchen tablet. */
padding: 0.25rem 0;
list-style-position: outside;
}
.digest-sources[open] > summary { margin-bottom: 0.5rem; }
.digest-source-list {
margin: 0;
padding-left: 1.05rem;
color: var(--digest-muted);
}
.digest-source-list li { margin-bottom: 0.65rem; }
.digest-source-list li:last-child { margin-bottom: 0; }
.digest-source-head a {
color: var(--digest-text);
text-decoration: none;
border-bottom: 1px dotted var(--digest-edge);
word-break: break-word;
}
.digest-source-meta {
margin-top: 0.15rem;
font-size: 0.75rem;
opacity: 0.85;
}
.digest-source-quote {
margin: 0.35rem 0 0;
padding-left: 0.6rem;
border-left: 2px solid var(--digest-edge);
color: var(--digest-text);
font-style: italic;
}
/* Per-marker briefings under the globe. */
.digest-globe-briefs {
width: 100%;
margin-top: 0.9rem;
display: flex;
flex-direction: column;
gap: 0.85rem;
}
.digest-globe-brief {
border-left: 2px solid var(--digest-edge);
padding-left: 0.7rem;
}
.digest-brief-title {
margin: 0 0 0.25rem;
font-size: 0.9rem;
font-weight: 600;
letter-spacing: 0.01em;
}
.digest-brief-summary {
margin: 0;
font-size: 0.88rem;
color: var(--digest-text);
}
/* ---------------------------------------------------------------------------
* Globe
* --------------------------------------------------------------------------- */

129
digest-engine/render/digest-canvas-sdk/render.js
View File

@ -13,6 +13,17 @@
* - a globe with no markers -> the globe still draws, just empty
*
* A blank page is the one outcome that must never happen.
*
* PER-PERSON SECTIONS
* -------------------
* A run carries `people` — each household member's own set of digest sections, as
* identity's admin panel recorded it and run.py copied it in (see
* digest-engine/preferences.py). Given `options.person` — a name, nickname or id
* that Home Assistant already resolved — only that person's sections are drawn.
*
* This is a DISPLAY FILTER, NOT AN ACCESS CONTROL. digest-web serves this same JSON
* read-only to anything on the LAN, so a section left out here is off a screen, not
* out of anyone's reach. Do not describe it to a user as privacy.
*/
(function (global) {
@ -59,13 +70,105 @@
return [];
}
// The sections this viewer gets, or null for "nobody was resolved / nothing is
// known about them" — which the caller, not this function, decides what to do with.
function sectionsForPerson(parsed, person) {
if (!person) { return null; }
var wanted = String(person).trim().toLowerCase();
if (!wanted) { return null; }
var people = (parsed && parsed.people) || [];
for (var i = 0; i < people.length; i++) {
var entry = people[i] || {};
var matches =
String(entry.name || '').toLowerCase() === wanted ||
String(entry.nickname || '').toLowerCase() === wanted ||
String(entry.id) === wanted;
if (matches) { return entry.digest_sections || []; }
}
// Named somebody this digest has never heard of — treated exactly like naming
// nobody, never as "show them everything because they must be new".
return null;
}
function filterSections(sections, parsed, options) {
var person = String(options.person || '').trim();
var people = (parsed && parsed.people) || [];
// Somebody WAS resolved, but this run has no preferences for anyone — identity was
// unreachable when it generated (see preferences.py). That run already fell back to
// generating everything, and hiding a resolved person's own personal section on top
// of it would make one container being down cost them more than it has to. Same
// direction of failure at both ends: more digest, not less.
if (person && !people.length) { return sections; }
var allowed = sectionsForPerson(parsed, person);
if (allowed) {
return sections.filter(function (doc) {
return doc && allowed.indexOf(doc.section) !== -1;
});
}
// No resolved person. The personal section is somebody's mail and messages, and
// the plan's rule for the shared kiosk is that it is never shown on a guess — see
// thinclient_agent/digest_canvas.py, which passes `person` only when Home
// Assistant has already resolved exactly who asked.
if (options.requirePersonForPersonal) {
return sections.filter(function (doc) {
return !doc || doc.section !== 'personal';
});
}
return sections;
}
// A marker's own briefing: the summary of what is happening there, and its
// sources folded away underneath. Rendered below the globe rather than as a
// tooltip on the marker — the globe rotates, markers pass behind the limb, and a
// summary you can only read while its marker happens to be facing you is not a
// summary. The label ties the two together.
function renderMarkerBriefs(container, markers) {
var briefed = (markers || []).filter(function (marker) {
return marker && (marker.summary || (marker.sources && marker.sources.length));
});
if (!briefed.length) { return; }
var wrap = document.createElement('div');
wrap.className = 'digest-globe-briefs';
briefed.forEach(function (marker) {
var brief = document.createElement('div');
brief.className = 'digest-globe-brief';
var heading = document.createElement('h4');
heading.className = 'digest-brief-title';
// Carries the marker's own colour so the eye can pair a red hammer-and-sickle
// on the globe with the paragraph explaining it.
if (marker.color) { heading.style.color = marker.color; }
heading.textContent = marker.label || 'Marker';
brief.appendChild(heading);
if (marker.summary) {
var text = document.createElement('p');
text.className = 'digest-brief-summary';
text.textContent = marker.summary;
brief.appendChild(text);
}
var sources = global.DigestWindow.sources(marker.sources);
if (sources) { brief.appendChild(sources); }
wrap.appendChild(brief);
});
container.appendChild(wrap);
}
function renderGlobeWindow(container, win) {
var mount = document.createElement('div');
var el = global.DigestWindow.open({
title: win.title || 'Globe',
content: mount,
container: container,
variant: 'globe'
variant: 'globe',
sources: win.sources
});
var globe = new global.DigestGlobe(mount);
@ -90,6 +193,8 @@
mount.parentNode.appendChild(caption);
}
renderMarkerBriefs(mount.parentNode, win.globe_markers);
return el;
}
@ -104,6 +209,9 @@
title: win.title || '',
content: win.content,
container: container,
// Any window may carry citations, not just the globe — a "Highlights" list
// and a standalone analysis window both need somewhere to put their receipts.
sources: win.sources,
x: win.x,
y: win.y,
w: win.w,
@ -149,7 +257,24 @@
return;
}
sections.forEach(function (doc) {
var visible = filterSections(sections, parsed, options);
// Everything this run generated was filtered out — every section this viewer
// wanted is one nobody generated, or they asked for none at all. That is a
// legitimate outcome of the settings, not a failure, so it gets a plain window
// rather than the <pre> dump a malformed digest gets. Still never a blank page.
if (!visible.length) {
global.DigestWindow.open({
title: 'Nothing in this digest',
content:
'No digest section is switched on for this screen. Sections are chosen ' +
'per person in the identity admin panel.',
container: container
});
return;
}
visible.forEach(function (doc) {
try {
renderSection(container, doc);
} catch (err) {

82
digest-engine/render/digest-canvas-sdk/window-chrome.js
View File

@ -71,7 +71,83 @@
return body;
}
// Only http(s) becomes a clickable link. The content of every window here is
// model output, and `javascript:` in an href would be script execution handed to
// whatever the LLM emitted — the one place this renderer's degrade-don't-throw
// habit is not enough. Anything else is shown as plain text instead.
function safeHref(url) {
var text = String(url || '').trim();
return /^https?:\/\//i.test(text) ? text : null;
}
// The fold-out sources block. Collapsed by default because the digest is read
// across a room: the claim is what you see, the receipts are one tap away, and
// an open citation list would push the next window off the screen.
function renderSources(sources, label) {
var usable = (sources || []).filter(function (source) {
return source && (source.title || source.outlet || source.url || source.quote);
});
if (!usable.length) { return null; }
var details = document.createElement('details');
details.className = 'digest-sources';
var summary = document.createElement('summary');
summary.textContent = (label || 'Sources') + ' (' + usable.length + ')';
details.appendChild(summary);
var list = document.createElement('ul');
list.className = 'digest-source-list';
usable.forEach(function (source) {
var li = document.createElement('li');
var head = document.createElement('div');
head.className = 'digest-source-head';
var href = safeHref(source.url);
var titleText = source.title || source.url || source.outlet;
if (href) {
var link = document.createElement('a');
link.href = href;
link.textContent = titleText;
link.rel = 'noopener noreferrer';
head.appendChild(link);
} else {
head.appendChild(document.createTextNode(titleText));
}
li.appendChild(head);
// Outlet and ownership sit with the citation rather than in the prose: who
// published a claim is part of reading it, and it belongs next to the claim.
var attribution = [source.outlet, source.owner, source.bias]
.filter(function (part) { return part; })
.join(' · ');
if (attribution) {
var meta = document.createElement('div');
meta.className = 'digest-source-meta';
meta.textContent = attribution;
li.appendChild(meta);
}
if (source.quote) {
var quote = document.createElement('blockquote');
quote.className = 'digest-source-quote';
quote.textContent = source.quote;
li.appendChild(quote);
}
list.appendChild(li);
});
details.appendChild(list);
return details;
}
var DigestWindow = {
// Exposed so render.js can attach the same block under a globe marker's brief,
// where there is no window of its own to hang it on.
sources: renderSources,
open: function (options) {
options = options || {};
@ -98,7 +174,11 @@
bar.appendChild(title);
el.appendChild(bar);
el.appendChild(renderContent(options.content));
var body = renderContent(options.content);
var sources = renderSources(options.sources);
if (sources) { body.appendChild(sources); }
el.appendChild(body);
if (typeof options.x === 'number' && typeof options.y === 'number') {
el.classList.add('digest-window-positioned');

9
digest-engine/render/templates/compact.html
View File

@ -54,8 +54,15 @@
var canvas = document.getElementById('canvas');
// Optional ?person=<name|nickname|id> — put it in the Lovelace card's URL to get
// that person's own digest sections (chosen in identity's admin panel). Unlike
// full.html this defaults to showing everything that was generated, personal
// section included: this card is embedded in somebody's own HA dashboard, which is
// already a per-account surface, not a kiosk in a hallway that anyone walks past.
var person = new URLSearchParams(location.search).get('person') || '';
function load() {
DigestRender.load(canvas, DIGEST_URL, { detailLevel: 'compact' });
DigestRender.load(canvas, DIGEST_URL, { detailLevel: 'compact', person: person });
}
load();

14
digest-engine/render/templates/full.html
View File

@ -61,8 +61,20 @@
var canvas = document.getElementById('canvas');
// ?person=<name|nickname|id>, set by thinclient_agent/digest_canvas.py from a
// request Home Assistant had ALREADY resolved — this page never works out who is
// standing in front of it. With it, only that person's own digest sections are
// drawn (they pick them in identity's admin panel); without it, the personal
// section is left out rather than shown to whoever happens to walk past, which is
// the plan's Phase 11.8 rule: never guess whose personal section this is.
var person = new URLSearchParams(location.search).get('person') || '';
function load() {
DigestRender.load(canvas, DIGEST_URL, { detailLevel: 'full' });
DigestRender.load(canvas, DIGEST_URL, {
detailLevel: 'full',
person: person,
requirePersonForPersonal: true
});
}
load();

1
digest-engine/requirements.txt
View File

@ -7,3 +7,4 @@ websocket-client>=1.7
caldav>=2.0
icalendar>=5.0
paho-mqtt>=1.6
pypdf>=4.0

198
digest-engine/run.py
View File

@ -10,10 +10,19 @@ Every ingestion source is independently toggleable and independently fallible. A
source that is disabled, misconfigured, or simply broken (a stale Telegram
session, a signal-cli container that is down) logs a warning and contributes an
empty list — it must never take down the rest of the run, because a digest with
two of three sections is worth far more than no digest at all.
most of its sections is worth far more than no digest at all.
Everything here is read-only. See docs/project-plan.md Phase 12 step 8.
WHICH DIGESTS GET GENERATED
---------------------------
Not necessarily all four. Each person in `identity` has a per-person set of the
sections they want (network, household, personal/social, political/news), and
this run generates the union of what the household asked for — a section nobody
wants costs no LLM call and never reaches output/. See preferences.py, including
why every failure in that lookup means "generate everything" rather than
"generate nothing".
Before anything is written to output/, every generated document passes through
synth/counter_run.py — a second LLM call that checks the document against the
same context it was generated from and drops anything that doesn't trace back
@ -55,11 +64,16 @@ from ingest import (
naval_traffic,
news_rss,
opnsense_ids,
rci_social,
signal_ingest,
telegram_ingest,
whatsapp_ingest,
)
from synth import counter_run, llm_client
import agenda
import archive
import notify
import preferences
import viewed_tracker
LOG = logging.getLogger("digest")
@ -72,6 +86,7 @@ SOURCES = (
("ENABLE_DISCORD_INGEST", "discord", discord_ingest),
("ENABLE_WHATSAPP_INGEST", "whatsapp", whatsapp_ingest),
("ENABLE_NEWS_INGEST", "news", news_rss),
("ENABLE_RCI_SOCIAL_INGEST", "rci_social", rci_social),
("ENABLE_FINANCIAL_INGEST", "financial", financial),
("ENABLE_FLIGHT_TRAFFIC_INGEST", "flight_traffic", flight_traffic),
("ENABLE_NAVAL_TRAFFIC_INGEST", "naval_traffic", naval_traffic),
@ -80,8 +95,22 @@ SOURCES = (
("ENABLE_GROCY_INGEST", "grocy", grocy),
)
PERSONAL_SOURCES = ("email", "signal", "telegram", "discord", "whatsapp")
POLITICAL_SOURCES = ("news", "financial", "email", "flight_traffic", "naval_traffic")
# Which ingested sources feed which section. Two jobs: it is what the personal
# context's `messages` map is built from, and it is how collect() knows that a source
# whose only consumers are switched off for this run needs no fetching at all — a
# digest section nobody asked for should cost neither an LLM call nor the WAN round
# trips its sources would have made. `email` appears twice on purpose: mail is both
# personal correspondence and, occasionally, politically relevant evidence.
SECTION_SOURCES = {
"personal": ("email", "signal", "telegram", "discord", "whatsapp"),
# `calendar` is here for the agendas, not for the diary: the political section
# owns the branch agenda's contents and the to-do list that comes out of it (see
# agenda.py), and it cannot match an agenda to a meeting it has never fetched.
"political": ("news", "rci_social", "financial", "email", "flight_traffic",
"naval_traffic", "calendar"),
"household": ("calendar", "grocy"),
"network": ("opnsense_ids",),
}
DEFAULT_SCHEDULE = "00,06,12,18"
DEFAULT_EVENING_HOUR = 18
@ -133,9 +162,21 @@ def resolve_slot():
return current, evening_hour, is_evening
def collect(lookback_hours):
def collect(lookback_hours, sections=None):
"""`sections` is the set of digest sections this run is generating; a source no
enabled section reads is skipped entirely rather than fetched and then dropped.
None means "generate everything", i.e. fetch everything that is switched on.
"""
needed = None
if sections is not None:
needed = {source for section in sections for source in SECTION_SOURCES.get(section, ())}
collected = {}
for toggle, key, module in SOURCES:
if needed is not None and key not in needed:
LOG.info("no enabled digest section reads %s, skipping its ingestion", key)
collected[key] = []
continue
if not env_flag(toggle):
LOG.info("%s is not enabled, skipping %s ingestion", toggle, key)
collected[key] = []
@ -185,7 +226,38 @@ def previous_section_document(previous_run, section):
return None
def build_section_contexts(collected, lookback_hours, is_evening_run, previous_run=None):
def household_calendar(events):
"""The calendar as the household section sees it: an event that has an agenda says
so, without carrying what the agenda says.
The points and the text go to the political section only (see agenda.upcoming()),
because a branch agenda is party work and reaches the people who asked for that
digest. Enforcing it here rather than only in the prompt means a household-only
run never has the contents in front of it to leak in the first place — a rule the
model cannot break is worth more than one it is told.
"""
trimmed = []
for event in events:
documents = event.get("agenda_documents") if isinstance(event, dict) else None
if not documents:
trimmed.append(event)
continue
copy = dict(event)
copy["agenda_documents"] = [
{
key: document.get(key)
for key in ("filename", "from", "source", "received_at",
"match_confidence", "match_reason", "text_extracted")
}
for document in documents
]
trimmed.append(copy)
return trimmed
def build_section_contexts(collected, lookback_hours, is_evening_run, previous_run=None,
sections=None, section_history=None, unattached_agendas=None,
upcoming_agendas=None):
# The full pantry is only ever needed to work out what the evening recipe still
# requires. On the other three runs it is a few hundred lines of prompt that
# buys nothing, so it is dropped rather than sent and then ignored.
@ -198,35 +270,75 @@ def build_section_contexts(collected, lookback_hours, is_evening_run, previous_r
contexts = {
"personal": {
"lookback_hours": lookback_hours,
"messages": {key: collected.get(key, []) for key in PERSONAL_SOURCES},
"messages": {key: collected.get(key, []) for key in SECTION_SOURCES["personal"]},
},
"political": {
"lookback_hours": lookback_hours,
"news": collected.get("news", []),
# The organisation's own posts, kept in their own key rather than folded
# into `news`: they are the section's voice, not reporting, and the prompt
# is told to read them differently. See ingest/rci_social.py.
"rci_social": collected.get("rci_social", []),
"financial": collected.get("financial", []),
"mail": collected.get("email", []),
# Extra evidence for the existing political synthesis, not a fourth
# window type — see the traffic-data section of synth/prompts/political.md.
"flight_traffic": collected.get("flight_traffic", []),
"naval_traffic": collected.get("naval_traffic", []),
# What the household is about to sit down and discuss, from the agendas
# that arrived by mail or messenger. Steers curation: a story about the
# subject of Thursday's branch meeting is worth more than one that isn't.
# Topics only — the full agenda text stays in the household section.
"upcoming_agendas": upcoming_agendas or [],
},
# Both keys stay present even when their source is off or broken, so the
# prompt sees the shape it is promised and says "nothing scheduled" instead
# of hallucinating an event.
"household": {
"lookback_hours": lookback_hours,
"calendar": collected.get("calendar", []),
"calendar": household_calendar(collected.get("calendar", [])),
"grocy": grocy_entries,
# Drives the evening-only recipe/shopping-list section of
# synth/prompts/household.md. Nothing is ever written back to Grocy.
"is_evening_run": is_evening_run,
# Home network status sits with the household, not the political
# section — it is a "something in this house needs your attention"
# item. See the network-security section of synth/prompts/household.md.
"network_security": collected.get("opnsense_ids", []),
# Agendas that arrived by mail or message and could not be tied to any
# event — reported rather than dropped, because "an agenda came and I
# can't tell which meeting it's for" is worth a line. Reduced to the same
# fact-level fields as the attached ones: what arrived and from whom, not
# what it says. See household_calendar() above.
"unattached_agendas": [
{
key: document.get(key)
for key in ("filename", "from", "source", "received_at", "text_extracted")
}
for document in (unattached_agendas or [])
],
},
}
# Its own section rather than a couple of lines inside the household one: a
# household member who wants the calendar but not a nightly intrusion-detection
# readout (or the reverse) can only say so if the two are separately generated.
# See synth/prompts/network.md.
contexts["network"] = {
"lookback_hours": lookback_hours,
"network_security": collected.get("opnsense_ids", []),
}
# A section nobody asked for is dropped here, before its context is ever built into
# a prompt — see preferences.py. Done by subtraction from the full set rather than
# by building each context conditionally, so adding a section later can't silently
# forget this filter.
if sections is not None:
contexts = {name: context for name, context in contexts.items() if name in sections}
# Earlier readings from the archive, as their own clearly-labelled block with its
# own timestamps — never merged into this run's entries, so a prompt can't mistake
# last month's figure for something that happened in this window. See archive.py.
for section, block in (section_history or {}).items():
if section in contexts:
contexts[section]["history"] = block
if previous_run is not None:
for section, context in contexts.items():
doc = previous_section_document(previous_run, section)
@ -240,7 +352,7 @@ def build_section_contexts(collected, lookback_hours, is_evening_run, previous_r
return contexts
def write_output(output_dir, run_id, context, documents):
def write_output(output_dir, run_id, context, documents, sections, people):
run_dir = output_dir / run_id
run_dir.mkdir(parents=True, exist_ok=True)
@ -254,6 +366,16 @@ def write_output(output_dir, run_id, context, documents):
"run_id": run_id,
"generated_at": context["generated_at"],
"sections": documents,
# What this run actually generated, so a surface can tell "nobody asked for
# this section" apart from "it was generated and came back empty".
"sections_generated": sections,
# Who wants which of them, carried alongside the documents so the renderer can
# filter to the person Home Assistant resolved without a second fetch (and
# without digest-web needing to talk to identity at all). Empty when
# preferences could not be read — see preferences.py. THIS IS A DISPLAY
# FILTER, NOT AN ACCESS CONTROL: this whole file is served read-only to the
# LAN, so it hides a section from a screen, not from a person.
"people": people,
}
payload = json.dumps(digest, indent=2, ensure_ascii=False, default=str)
(run_dir / "digest.json").write_text(payload, encoding="utf-8")
@ -289,7 +411,33 @@ def main():
LOG.info("digest run %s starting (lookback %sh)", run_id, lookback_hours)
current_slot, evening_hour, is_evening_run = resolve_slot()
collected = collect(lookback_hours)
# Read before ingestion so a section nobody wants costs nothing at all — not an
# LLM call, and not the WAN round trips its sources would have made either.
prefs = preferences.fetch(llm_client.SECTIONS)
sections = preferences.sections_to_generate(prefs, llm_client.SECTIONS)
if not sections:
LOG.warning(
"every household member has opted out of every digest section — "
"generating nothing, which is what was asked for"
)
collected = collect(lookback_hours, sections)
# The long memory: this run goes into the archive, and every item it just
# collected comes back annotated with when this system first saw it. History for
# the sections being generated is read afterwards, so a series includes the
# reading taken a moment ago rather than stopping at the previous run.
archive.record(run_id, collected)
section_history = archive.history(sections)
# A "Tagesordnung" PDF that arrived by mail or WhatsApp belongs to the meeting it
# is for, not to a list of attachments. Done before the contexts are built so the
# calendar entries the household section sees already carry theirs. Only works
# when the message sources are being fetched at all — see SECTION_SOURCES: a run
# generating only the household section deliberately does not go and read the
# household's mail.
unattached_agendas = agenda.attach(collected)
previous_run = load_previous_run(output_dir)
previous_viewed_at = viewed_tracker.last_viewed_at()
@ -312,26 +460,44 @@ def main():
"item_counts": {key: len(items) for key, items in collected.items()},
"sources": collected,
"merged_unviewed_previous_run": merge_previous,
# Kept in the persisted context (and so in the Phase 12 follow-up voice Q&A's
# view of this run) because "why is there no political section today" is
# answered by this and by nothing else in the bundle.
"sections_generated": sections,
"digest_preferences_known": prefs is not None,
}
section_contexts = build_section_contexts(
collected, lookback_hours, is_evening_run,
previous_run=previous_run if merge_previous else None,
sections=sections,
section_history=section_history,
unattached_agendas=unattached_agendas,
upcoming_agendas=agenda.upcoming(collected, unattached_agendas),
)
documents = llm_client.generate_all(section_contexts)
documents = llm_client.generate_all(section_contexts, sections=sections)
# The final filter, per docs/project-plan.md: a second, independent pass over
# each document against the same context it was generated from, before anything
# is written to output/. See synth/counter_run.py for what it catches and how it
# fails safe.
documents = counter_run.verify_all(documents, section_contexts)
run_dir = write_output(output_dir, run_id, context, documents)
run_dir = write_output(
output_dir, run_id, context, documents,
sections=sections,
people=prefs["people"] if prefs else [],
)
# After the digest is on disk, never before: a push is a promise that there is
# something to open. Best-effort — see notify.py.
notify.send(context, documents, sections, prefs)
LOG.info(
"digest run %s complete: %s -> %s",
"digest run %s complete: %s -> %s (sections: %s)",
run_id,
context["item_counts"],
run_dir,
", ".join(sections) or "none",
)
return 0

48
digest-engine/synth/counter_run.py
View File

@ -116,6 +116,45 @@ def _window_haystack(window):
return _normalize(content or "")
def _ground_sources(container, context_blob):
"""Strips citations that don't trace back to the context, in place.
A `sources` block is the strongest claim the digest makes — a URL and a quote
together assert "this exists and says this" — and it is also the easiest thing
for a model to compose out of thin air, because a plausible URL looks exactly
like a real one. Both halves are checked mechanically here rather than being
left to the verifying call's own judgement, for the same reason the quote check
below exists: substring presence in the context is a fact, not an opinion, and
no second opinion improves on it.
A source with a fabricated URL is dropped whole. A real source carrying an
invented quote keeps the source and loses the quote — the citation is still
true, only the excerpt was not.
"""
sources = container.get("sources")
if not isinstance(sources, list):
return
kept = []
for source in sources:
if not isinstance(source, dict):
continue
url = str(source.get("url") or "").strip()
if url and _normalize(url) not in context_blob:
LOG.warning("counter-run: dropping a source whose URL is not in the context: %r", url[:120])
continue
quote = str(source.get("quote") or "").strip()
if quote and _normalize(quote) not in context_blob:
LOG.warning("counter-run: clearing a source quote not found in the context: %r", quote[:80])
source = {key: value for key, value in source.items() if key != "quote"}
kept.append(source)
if kept:
container["sources"] = kept
else:
container.pop("sources", None)
def verify_document(document, context):
"""Returns a possibly-filtered copy of `document`. Never raises."""
if document.get("degraded"):
@ -199,6 +238,15 @@ def verify_document(document, context):
if not kept:
return _withheld_document(document)
# Citations are checked after the window verdicts, not before: there is no point
# grounding the sources of a window that is about to be dropped whole. Markers
# carry their own sources (the globe briefs), so they are walked too.
for window in kept:
_ground_sources(window, context_blob)
for marker in window.get("globe_markers") or []:
if isinstance(marker, dict):
_ground_sources(marker, context_blob)
result = dict(document)
result["windows"] = kept
if not bool(verdict.get("narration_grounded", True)):

98
digest-engine/synth/llm_client.py
View File

@ -13,13 +13,23 @@ keep the two in sync when changing either.
{
"generated_at": "2026-07-28T12:00:00Z",
"detail_level": "compact" | "full",
"section": "personal" | "political" | "household",
"section": "personal" | "political" | "household" | "network",
"windows": [
{
"id": "string, unique within this section",
"title": "string",
"kind": "text" | "list" | "globe",
"content": "markdown-ish string for kind=text, or an array of strings for kind=list",
"sources": [
{
"title": "headline or page title",
"outlet": "who published it",
"owner": "who owns that outlet, as the context gives it",
"bias": "that outlet's politics, as the context gives it",
"url": "https://...",
"quote": "verbatim excerpt"
}
],
"globe_markers": [
{
"lat": 0.0,
@ -27,7 +37,9 @@ keep the two in sync when changing either.
"label": "string",
"icon": "star|hammer-sickle|default",
"color": "#hex",
"glow": true
"glow": true,
"summary": "what is happening at this location",
"sources": [ ... same shape as above ... ]
}
]
}
@ -38,18 +50,32 @@ keep the two in sync when changing either.
`globe_markers` is only present (and non-empty) on `kind: "globe"` windows, which
in practice only the political section produces.
`sources` is optional everywhere and may appear on any window and on any marker.
The renderer folds it into a collapsed "Sources (n)" block — on screen the claim
is what you read and the citations are one tap underneath, so a section can be
fully sourced without turning the canvas into a bibliography. A marker's
`summary` is its briefing, rendered under the globe rather than as a tooltip,
because the globe rotates and a marker's own face is not always toward you.
WHICH SECTIONS RUN
------------------
`SECTIONS` is every section this component knows how to generate; which of them a
given run actually generates is decided by the household, not here — see
../preferences.py and generate_all()'s `sections` argument. A section nobody has
ticked in the identity admin panel costs no call at all.
DETAIL LEVELS
-------------
Each section is generated twice per run, once at `compact` and once at `full`
(6 calls total), rather than generating `full` once and truncating it client-side:
Each enabled section is generated twice per run, once at `compact` and once at
`full`, rather than generating `full` once and truncating it client-side:
truncation gives you the first N windows of a document written to be expansive,
so a "compact" window can still hold a 400-word blob that overflows the HA
Lovelace iframe card, whereas a second pass yields prose actually written to be
terse. The cost is 3 extra calls against a local, self-hosted Ollama on a batch
timer — no per-token bill and no latency anyone is waiting on — so correctness of
the compact rendering wins. Both passes reuse one ingestion pass and one assembled
context, which is what docs/project-plan.md means by "without needing two
independent generation passes".
terse. The cost is one extra call per enabled section against a local, self-hosted
Ollama on a batch timer — no per-token bill and no latency anyone is waiting on —
so correctness of the compact rendering wins. Both passes reuse one ingestion pass
and one assembled context, which is what docs/project-plan.md means by "without
needing two independent generation passes".
"""
import json
@ -62,7 +88,13 @@ import requests
LOG = logging.getLogger(__name__)
SECTIONS = ("personal", "political", "household")
# Every section this component can generate, in the order they are laid out on the
# canvas. `network` was carved out of the household section rather than added next to
# it: the household prompt was already carrying the firewall's IDS summary as "one
# small household item", and a household that wants to know about its calendar without
# a nightly intrusion-detection readout (or the reverse) had no way to say so while the
# two shared one document.
SECTIONS = ("personal", "political", "household", "network")
DETAIL_LEVELS = ("compact", "full")
PROMPT_DIR = Path(__file__).parent / "prompts"
@ -101,6 +133,27 @@ def _fallback_document(section, detail_level, text, title="Digest (plain text fa
}
def _coerce_sources(raw):
"""Normalises a window's or a marker's `sources` list, dropping anything that
isn't at least identifiable. Kept permissive on purpose: a citation with a
title and no URL is still a citation, and dropping it would make the digest
look better-sourced than it is by hiding the weak entries. What it is NOT
permissive about is shape — a string, or a dict of nothing, is not a source.
"""
sources = []
for entry in raw or []:
if not isinstance(entry, dict):
continue
source = {
key: str(entry.get(key)).strip()
for key in ("title", "outlet", "owner", "bias", "url", "quote")
if entry.get(key)
}
if source:
sources.append(source)
return sources
def _coerce_document(raw, section, detail_level):
if not isinstance(raw, dict):
raise ValueError("model output was not a JSON object")
@ -116,14 +169,16 @@ def _coerce_document(raw, section, detail_level):
"kind": kind,
"content": window.get("content", ""),
}
sources = _coerce_sources(window.get("sources"))
if sources:
coerced["sources"] = sources
if kind == "globe":
markers = []
for marker in window.get("globe_markers") or []:
if not isinstance(marker, dict):
continue
try:
markers.append(
{
coerced_marker = {
"lat": float(marker.get("lat", 0.0)),
"lon": float(marker.get("lon", 0.0)),
"label": str(marker.get("label") or ""),
@ -131,9 +186,17 @@ def _coerce_document(raw, section, detail_level):
"color": str(marker.get("color") or "#8ab4ff"),
"glow": bool(marker.get("glow", False)),
}
)
except (TypeError, ValueError):
continue
# The marker's own briefing: what is happening there, and the
# citations behind it, rendered under the globe by render.js. Both
# optional — a marker with neither is still a marker.
if marker.get("summary"):
coerced_marker["summary"] = str(marker["summary"])
marker_sources = _coerce_sources(marker.get("sources"))
if marker_sources:
coerced_marker["sources"] = marker_sources
markers.append(coerced_marker)
coerced["globe_markers"] = markers
windows.append(coerced)
@ -232,10 +295,15 @@ def generate_section(section, detail_level, context):
return _fallback_document(section, detail_level, text.strip())
def generate_all(section_contexts):
def generate_all(section_contexts, sections=None):
"""`sections` is the subset the household actually asked for (../preferences.py);
None means all of them. Ordering always follows SECTIONS rather than the caller's
list, so the canvas doesn't reshuffle itself when somebody edits a checkbox.
"""
wanted = SECTIONS if sections is None else [s for s in SECTIONS if s in sections]
documents = {level: [] for level in DETAIL_LEVELS}
for detail_level in DETAIL_LEVELS:
for section in SECTIONS:
for section in wanted:
LOG.info("synth: generating %s/%s", section, detail_level)
documents[detail_level].append(
generate_section(section, detail_level, section_contexts.get(section, {}))

13
digest-engine/synth/prompts/counter_run.md
View File

@ -53,6 +53,19 @@ For each window in the document, check:
change at the same location, a market move and a news item). The context
must actually contain both halves of the correlation, not just one, with
the other inferred or assumed.
- **Citations.** A window (and a globe marker) may carry a `sources` list of
`{title, outlet, owner, bias, url, quote}` entries. Each one must correspond
to an actual entry in the context, and its `outlet`, `owner` and `bias` must
match what the context records for that entry rather than being characterised
from your own knowledge of the outlet. **Attributing a claim to a source that
does not contain it is worse than leaving it unsourced** — flag the window. A
fabricated URL is checked mechanically after this pass as well, but flag it if
you see one.
- **Attribution of hostile sources.** Where the document repeats a claim made by
an outlet whose `bias` marks it as state-affiliated or as Zionist, it must
present that claim as *that outlet's statement*, not as established fact. A
document that launders such a claim into its own voice is not grounded, even
when the outlet really did say it — flag the window and say so.
Err towards flagging. When genuinely unsure whether something is supported,
treat it as unsupported — the cost of over-filtering one border-line claim is

62
digest-engine/synth/prompts/household.md
View File

@ -3,10 +3,13 @@
You are the household section of a household digest that is generated four times
a day. You are given the household calendar (read from Nextcloud over CalDAV) and
household inventory/chore state from Grocy — upcoming events, stock that is low
or expiring, chores and battery levels that are due. Some runs also carry a home
network security summary — see "Home network security" below for the narrow way
that may be used. One run a day is the evening run — see "Evening recipe and
shopping list", which applies to that run and no other.
or expiring, chores and battery levels that are due. One run a day is the evening
run — see "Evening recipe and shopping list", which applies to that run and no
other.
The home network has its own section and is not your subject: say nothing about
the firewall, intrusion detection, or anything on the network, even if you think
it belongs here.
`calendar` entries are tagged `"category": "calendar_event"` and carry `summary`,
`start`, `end`, `all_day`, `location` and `recurring`. Times are UTC (`Z`) unless
@ -40,31 +43,36 @@ You are read-only. You never create, move or delete a calendar event, never
consume or restock anything in Grocy, and never propose that the system do so on
its own — at most you can tell the user that something needs their attention.
## Home network security
## Meeting agendas
Some runs carry a `network_security` entry tagged `"category":
"network_security"`: a summary of the Suricata intrusion-detection alerts the
household firewall raised during this digest window. Treat it as one small
household item — "is anything wrong with the home network" — not a section of its
own, and give it at most a couple of lines.
A calendar entry may carry `agenda_documents`: a Tagesordnung or similar that
arrived by mail or messenger and was matched to that meeting. Each one names the
`filename`, who it came `from`, which `source` it arrived through, when
(`received_at`), and how confident the match was (`match_confidence`,
`match_reason`).
- If `alert_count` is 0 and `ids_status` is `"running"`, say the network was
quiet in one short clause and move on. Do not pad it.
- If `ids_status` is anything other than `"running"`, say the intrusion detection
was not running, so there is nothing to report — never present that as a quiet
network.
- When there are alerts, lead with what a person would act on: which local
device (`top_local_hosts`) and which signature (`top_signatures`), and whether
the traffic was blocked or only alerted on (`actions` / `alerts_by_action` —
`"blocked"` means the firewall already stopped it, `"allowed"` means it did
not).
- Respect the `caveat` field. These are signature matches, not confirmed
compromise; false positives are routine, severity is not available to you, and
you must never call a device infected or compromised on this evidence. Say what
fired and let the user judge. Never state that the network is safe or clean.
- If `window_truncated` is true, say the counts are a lower bound.
- If `packet_capture_reference` is present, you may mention in one clause that
raw captures are available at that location. You have not read them.
**You report that an agenda exists. You do not report what is in it.** The
agenda's contents — its points, and the to-do list that comes out of them —
belong to the political section, because a branch agenda is party work and only
the people who asked for the political digest are shown it. That separation is
the point, not an oversight: do not list agenda points here, do not summarise the
document, and do not derive tasks from it, even though the text is in front of
you.
What to say here:
- One line with the event: "branch meeting Thursday 19:00 — agenda
`TO_12.08.pdf` arrived from Anna by mail on Monday". The meeting and the fact
that its agenda is here, nothing further.
- If `match_confidence` is `low`, say the agenda *appears* to belong to that
meeting and name the reason, so a wrong match is visible rather than asserted.
- If `text_extracted` is false, the document could not be read at all (a scan;
there is no OCR here). Say so in the same line — it is the one thing about an
agenda's contents worth reporting in this section, because it tells the reader
not to expect it elsewhere either.
- `unattached_agendas` in the context are agendas that matched no event: mention
them in one line each — what arrived and from whom — so the reader knows a
document is waiting for a meeting the calendar does not have.
## Evening recipe and shopping list

111
digest-engine/synth/prompts/network.md Normal file
View File

@ -0,0 +1,111 @@
# Home network digest
You are the home-network section of a household digest that is generated four
times a day. You are given a summary of the intrusion-detection alerts the
household's own OPNsense firewall (Suricata) raised during this digest window,
tagged `"category": "network_security"`.
Your job answers one question: **is anything wrong with the home network right
now?** This is read on a wall display and spoken aloud in a kitchen, by people
who are not on call and did not ask to become firewall analysts. Two or three
short windows at most, even at `detail_level: full`. If there is nothing to
report, say so in one line and stop — a quiet network is a one-line answer, not a
section to pad.
## Reading the context
- If `alert_count` is 0 and `ids_status` is `"running"`, say the network was
quiet during the window and stop there.
- If `ids_status` is anything other than `"running"`, say the intrusion detection
was **not running**, so there is nothing to report. Never present that as a
quiet network — it is the absence of an answer, not a good one.
- When there are alerts, lead with what a person would act on: which local device
(`top_local_hosts`) and which signature (`top_signatures`), and whether the
traffic was blocked or only alerted on (`actions` / `alerts_by_action` —
`"blocked"` means the firewall already stopped it, `"allowed"` means it did
not).
- If `window_truncated` is true, say the counts are a lower bound.
- If `packet_capture_reference` is present, you may mention in one clause that
raw captures are available at that location. You have not read them.
- If the context carries no `network_security` entry at all, say in one line that
no network data was collected this run. Do not infer that the network was
quiet, and do not invent an alert, a device, or a signature.
## History: one alert is noise, the same alert every night is a fact
The context may carry a `history` block from the digest's own archive of past
runs: `alert_totals` (the alert count of each earlier run), `recurring_signatures`
and `recurring_hosts`, each with `alerts` (the total across the archive),
`runs_seen` (how many runs it has appeared in), and `first_seen`/`last_seen`.
`archive_span_days` says how far back the archive actually goes.
This is the most useful thing in this section, because recurrence is what
separates background noise from something worth looking at:
- **Lead with what is new.** A signature firing for the first time — `runs_seen`
of 1, or a `first_seen` inside this window — is the item a person should read
first, even if a familiar signature fired more times.
- **Say plainly when something is routine.** A signature that has fired in
fifteen of the last twenty runs is background: name it in one clause as
ongoing, with its `first_seen` date, and do not present it as an event. A
household that gets told about the same alert four times a day stops reading
this section, and then it is worth nothing.
- **A host that has just started appearing is worth naming**, with the date it
first appeared. That is the shape of "something on this network changed".
- Compare this run's count against `alert_totals` only in figures you can point
at, and say how long the archive covers. A week of history does not support
"unusually high".
- If there is no `history` block, or `archive_span_days` is small, say nothing
about trends at all.
## What you must not claim
Respect the `caveat` field. These are **signature matches, not confirmed
compromise**; false positives are routine and severity is not available to you.
- Never call a device infected, compromised or breached on this evidence. Say
what fired, on which host, and let the reader judge.
- Never state that the network is safe, clean or secure. The most you can say is
that nothing fired during this window, which is a different claim.
- Never recommend that anything be blocked, disconnected, rebooted or
reconfigured automatically. You are read-only: this component cannot touch the
firewall, and it must not propose that the system act on its own. Telling a
person that something deserves their attention is the whole of what you may do.
## Output
Output **only** a single JSON object matching this schema — no prose before or
after it, no markdown code fence:
```json
{
"generated_at": "2026-07-28T12:00:00Z",
"detail_level": "compact" | "full",
"section": "network",
"windows": [
{
"id": "string, unique within this section",
"title": "string",
"kind": "text" | "list",
"content": "markdown-ish string for kind=text, or an array of strings for kind=list"
}
],
"narration": "a short plain-text script suitable for TTS narration of this section, 2-4 sentences"
}
```
Rules:
- `section` must be exactly `"network"`.
- `detail_level` must echo the `detail_level` line given at the end of the context.
- Do **not** emit any window with `kind: "globe"` and do **not** emit
`globe_markers` in this section. The globe belongs to the political section.
- `id` must be unique within this section, lowercase, hyphenated (e.g.
`network-alerts`, `network-status`).
- Alerts, signatures and hosts are enumerable — use `kind: "list"` with an array
of short strings, each leading with the host or the signature name.
- At `detail_level: compact`, keep the whole section to one window.
- `narration` is spoken aloud by a TTS voice, so no markdown, no URLs, no emoji.
Read out an IP address only if it is the point of the item.
- If you cannot produce valid JSON matching this schema, output a single
`kind: "text"` window with your best-effort plain-text summary instead.

355
digest-engine/synth/prompts/political.md
View File

@ -1,13 +1,54 @@
# Political digest
You are the political section of a household digest that is generated four times
a day. You are given: entries from a curated set of news feeds, financial
indicators (stock indices, oil, macro series such as unemployment), and the
user's mail (from which you should use only the politically relevant items — for
example union, campaign, tenants' association or party correspondence — and
ignore everything personal, which is handled by a different section). Some runs
also carry air- and naval-traffic samples — see "Traffic data" below for the
narrow way those may be used.
a day. You are given: entries from a curated set of news feeds from around the
world, the RCI's own publications and social-media output, financial indicators
(stock indices, oil, macro series such as unemployment), and the user's mail
(from which you should use only the politically relevant items — for example
union, campaign, tenants' association or party correspondence — and ignore
everything personal, which is handled by a different section). Some runs also
carry air- and naval-traffic samples — see "Traffic data" below for the narrow
way those may be used.
The reader is a communist, a member of the RCI's Austrian section, living in
Vorarlberg. Write for someone who already holds this politics and needs to be
oriented in the world this week — not for someone who needs to be convinced of
it.
## The four questions
Everything below serves four questions. They are the structure of the section,
not a checklist to append: decide what goes in the digest by asking which
question an item answers, and drop it if it answers none.
1. **What is relevant for the communist and class struggle globally right now?**
Strikes, organising, revolutionary situations, defeats and their causes, the
state of the workers' movement, and the economic conditions driving all of it.
2. **What matters for going out and organising here — Austria, and Vorarlberg
specifically?** Local and national disputes, plant closures and layoffs, rents
and prices, the far right's activity, unions and works councils, university
and school agitation, anything a person could act on in the next days. A small
Vorarlberg item outranks a large foreign one for this question: a closure in
Dornbirn is more use on a paper sale than a cabinet reshuffle abroad. The
Austrian feeds (`VOL.AT` for Vorarlberg, `DER STANDARD` nationally) are where
this mostly comes from; a national item counts if it lands locally.
3. **What else is consequential in the mid-to-long term, even when it is not
directly class struggle?** Wars and rearmament, climate and energy, epidemics,
supply chains and food, technological change, state repression and
surveillance law, migration regimes. Read them for what they will mean for
organising in a year or five, and say which — this is the question where a
slow-moving development beats a loud one.
4. **What has been happening inside the RCI, and what are comrades elsewhere
reporting?** Congresses, campaigns, splits and fusions, election results,
repression against sections, growth. The sources are the `theory` entries, the
organisation's own social output, and — importantly — the **reports and
bulletins in the user's own mail**: internal reports from comrades in other
sections arrive there, and they are the one source in this context that no
feed can supply. Treat a comrade's report as a report, not as an anonymous
claim: say which section or comrade it came from where the mail says so.
Each question gets its own window (see "Output"). A question with nothing worth
saying this run gets one honest line, never filler.
## What this section is, and is not
@ -72,15 +113,163 @@ speculation "No speculation" below rules out elsewhere; it's fine, and
expected, for an item to warrant analysis without a specific named theory
attached to it.
Entries tagged `"category": "news_state_affiliated"` come from outlets that are
organs of a state (e.g. Russian or Chinese state media), not independent press.
Treat their factual claims about third parties with more scepticism than an
independent outlet's, and treat their framing of their own state's actions as
that state's self-presentation, worth noting as a data point ("Moscow/Beijing
describes this as...") rather than reporting it as settled fact. Their
reporting on labour/material conditions inside their own country can still be
useful raw material — apply the same class analysis to it as to anything else,
just don't launder state propaganda as neutral reporting.
## The organisation's own social media
The context may carry a `rci_social` list: posts from the RCI's and the Austrian
section's own accounts, each tagged `"category": "theory_social"` and carrying
`account`, `platform`, `scope` and `content_type`.
- **`scope: "section"` is the reader's own organisation; `scope:
"international"` is the RCI as a whole. Do not merge the two voices** — "my
section is holding a meeting on Thursday" and "the International has published
a statement" are different facts and the reader acts on them differently.
- These are the organisation's **public voice**, not its analysis. A post
announcing a meeting, a demonstration, a campaign launch, a paper sale or a new
video is worth surfacing plainly, with its date and place if the post gives
them, because it is something the reader can actually turn up to.
- **Never use a post as evidence for a claim about the world.** A short
agitational post is advocacy; ground factual claims in the news entries and in
the written analysis tagged `"theory"`. If a post asserts something and nothing
else in the context supports it, say the section is saying it — do not restate
it as established.
- Do not pad. If nothing new was posted this run, say nothing about social media
at all.
## Watch later
Entries with `content_type: "episode"` — new videos and podcast episodes from the
organisation's channels — get their **own window**, and are kept out of the
analysis entirely. They are not news and they are not evidence; they are things
the reader might choose to watch or listen to later.
- Use one `kind: "list"` window with id `political-watch-later`, titled something
like "Watch later". One entry per episode: its title, which account or channel
it came from, and its `duration` when the context gives one.
- Say in a few words what it is about **only if the entry's own title or summary
says** — never guess at the content of a video from its title alone.
- Order newest first, cap it at about six entries, and omit the window entirely
when there are no new episodes this run. Never carry an episode over into one
of the four question windows or onto the globe.
- A `"theory"` article whose title is marked as a podcast (marxist.com prefixes
these with `[Podcast]`) belongs in this window too, not in the analysis.
## The agendas: what is being discussed, and what you have to do
The context may carry `upcoming_agendas`: meetings the reader is going to, each
with the agenda that was sent out — its `points` (the numbered items, extracted
from the document mechanically), its `text`, the `covering_message` it arrived
with, who it came `from`, and `text_extracted` saying whether any of it could be
read at all.
**This section is where an agenda's contents live.** The household section only
says that an agenda arrived; the points and the tasks are here, because a branch
agenda is party work and belongs to the digest the reader asked for it in.
Two windows come out of it (see "Output"):
- **`political-agenda`** — the meeting, its date, and its points as written. Do
not rewrite a point into your own words, do not merge two into one, and do not
add a point that is not in the list. If `text_extracted` is false, say the
agenda could not be read and stop there — never guess at what a scanned
document says.
- **`political-todo`, titled "Political todos"** — what the reader actually has
to do before those meetings. One line each, leading with the verb, naming the
meeting and its date: "Bring the paper-sale accounts — OG-Treffen, Thu 12 Aug".
Every entry must trace to a specific line of the agenda or its covering
message, and you must quote that fragment. Sources of a task are exactly two:
the agenda text and the message it came with. **If nothing actually asks
anyone to do anything, emit no todo window at all** — inventing preparation
nobody asked for is the worst failure available to this feature, because it is
the one a reader would act on.
- A task the agenda assigns to somebody else is still worth one line, said as
what it is ("Anna is bringing the accounts"), so the reader knows it is covered
rather than assuming it is theirs.
The same agendas are also the sharpest relevance filter you have, and that part
belongs mostly to question 2.
- **A story that touches an agenda point outranks a bigger story that doesn't.**
If Thursday's meeting has "Mietpreise / rent campaign" on it and this run's
material has a rent decision, a landlord lobby's figures or a tenants' dispute,
that is the item to feature — and say why in the item itself: "on Thursday's
agenda". The reader is about to have to speak about this.
- Use it for question 1 as well when the agenda point is an international one (a
solidarity campaign, a strike being discussed), but do not stretch a
connection: an agenda point about the branch's finances does not make a
banking story relevant.
- **The agenda tells you what is being discussed, never what is true.** Do not
treat a point on it as a fact about the world, and never present an agenda item
as if it were news. It decides what is *relevant*; the news entries decide what
is *so*.
## Reading a source: who owns it
**Every news entry carries `owner` and `bias`.** Read each item against them
before you use it. There is no neutral outlet in this context and you must never
write as if there were: a newspaper is owned by somebody, and what it can say is
bounded by who that is. This is not a reliability ranking — it is the same
materialist analysis you apply to everything else, applied to the press.
The rules that follow from it:
- **"State-affiliated" names who signs the cheque, not a propaganda bucket that
private Western outlets are exempt from.** RT is not simply "Russian state
media": it is the outlet of the Russian capitalist class and its state, and the
thing it will never report is Russian capital's own imperialism. The BBC is the
state broadcaster of a NATO power and follows the Foreign Office's frame on any
war Britain is party to. Both are class instruments. Say so in those terms when
it matters to the item, and apply the standard symmetrically or not at all.
- **A private outlet is the organ of a fraction of capital.** Name the fraction
when it explains the coverage — the Washington Post's owner is Amazon's owner
when the story is warehouse labour; the WSJ's editorial line is the employers'
side of a strike it is reporting; the Economist's "we" is the ruling class.
- **The business press is often the most candid source you have.** The FT and
CNBC brief capital honestly because their readers have to act on it, so they
will state a coming crisis, a falling profit rate or a wage offensive plainly
where a general-interest paper writes euphemism. Use that, and say where it
came from.
- **`category: "news_labour"` is the workers' and movement press** (Morning Star,
Labor Notes, Peoples Dispatch, Jacobin). Closer to the shop floor than anything
else in this context and often the only source that covers a dispute at all.
They are not the RCI and their politics are not yours — Jacobin's is
reformist — so use their reporting and do not adopt their conclusions.
- **When two outlets with opposed owners report the same fact, that convergence
is itself worth stating.** When they diverge, say who says what rather than
picking the one that sounds most authoritative.
- If `owner`/`bias` are missing for an entry, say "ownership not recorded here"
rather than inventing an ownership claim. Never invent a proprietor, a funder
or a political line that the context does not state.
## Zionist media: zero trust
Entries tagged `"category": "news_zionist"` — and any entry whose `bias` marks it
as Zionist — come from outlets that take the Israeli settler-colonial state as
given, and whose Gaza coverage functions as apologia for the genocide: army
statements reproduced as fact, massacres rendered as "strikes on militants",
Palestinian casualty figures marked as contested while Israeli ones are simply
reported, the passive voice reserved for Palestinian deaths.
Treat them accordingly:
- **Zero trust on any factual claim about Palestinians, Gaza, Lebanon or the
occupied territories.** Never repeat such a claim as established. If it is the
only source for something, either say the claim exists and is uncorroborated in
this run's context, or leave it out.
- **Never adopt their language.** Not "clashes" for a massacre, not "the war"
for a genocide, not "Israel says X happened" as a summary of what happened.
Quote it as their statement and name it as such.
- **What they are good for is evidence about the Israeli state itself** — what
its ruling class, army and press are saying to each other, admitting to,
preparing for, or falling out over. An Israeli paper reporting a split in the
cabinet, a recruitment crisis, a capital flight, or an army officer's own
account of an order is a real finding. Use it that way and say where it came
from.
- **Zero trust is not inversion.** Their denial of something is not evidence that
it happened; corroborate against the Palestinian, anti-Zionist and independent
outlets in this context, and if nothing corroborates it, say the context does
not settle it. The "No speculation" rule below is not suspended here.
- The same applies in reverse to nothing: no outlet in this context gets
uncritical trust, including the ones whose politics you share.
Entries tagged `"category": "osint_military"` come from defence and open-source
intelligence outlets that track troop, fleet and air movements. Their reporting
@ -88,6 +277,37 @@ of *where forces are* is the useful part and is usually reliable. Their framing
which reads military spending as necessity and arms procurement as good news — is
the trade press of the arms industry and should be treated as such, not adopted.
## History: what you may say about trends
The context may carry a `history` block: earlier readings of the same financial
and traffic series, oldest first, each with its own timestamp, drawn from the
digest's own archive of past runs. Every entry in this run's own material also
carries `first_seen_at` and `times_seen` — when this system first saw that item,
and how many runs it has appeared in.
**This block is the only basis on which you may describe anything as rising,
falling, unchanged, accelerating or unprecedented.** Without it you have one
snapshot and no trend, and the rule is the old one: say the figure, not a
direction.
With it:
- Name the comparison explicitly — "unemployment 5.4%, up from 5.1% in the
reading of 12 June" — and take both numbers from the block. Never round a
trend into a word without the figures behind it.
- Say how long the series actually covers (`archive_span_days`). A fortnight of
history is not evidence of a historic high, and claiming one from it is the
same speculation the next section rules out.
- `first_seen_at` tells you whether a story is new or continuing. A story
already featured for three runs needs a reason to be featured again — what
changed — and "still ongoing" is not a headline. Say "first appeared on X" when
it matters to the reader's sense of whether something is developing.
- Merchant traffic through a chokepoint is the case where history matters most:
a fall over weeks is the finding the "Traffic data" section describes, and now
you can actually see it. One low sample is still nothing.
- The history block is context, never content: nothing in it happened in this
window, and it must never be presented as news from this run.
## No speculation
Every claim you make must trace back to something actually present in the
@ -107,7 +327,8 @@ context — a specific entry, figure, or quote. This is not a style preference:
## Curating and quoting
For each entry that passes the relevance filter above:
For each entry that passes the relevance filter above — i.e. that answers one of
the four questions — and inside the window belonging to that question:
- Feature it explicitly with a short excerpt or quotation taken verbatim from
the entry's own text (its title/description/body field, not your
@ -130,6 +351,24 @@ For each entry that passes the relevance filter above:
bargaining power from an unemployment move), not just the figure — but only
draw the connection the data actually supports.
**Attach the receipts.** Every window that makes a claim about the world carries
a `sources` array, and every entry in it is copied out of the context — never
composed:
- `url` must be an entry's own `link` exactly as the context gives it. **Never
write a URL that is not in the context**, never repair one that looks wrong,
never guess a homepage. An entry with no link gets no `url` field.
- `quote` must be a verbatim span of that entry's own text.
- `outlet` is the entry's `feed`; `owner` and `bias` are its `owner` and `bias`
fields, copied as given. Carrying ownership into the citation is the point:
the reader sees who paid for the claim in the same place they see the claim.
- Where a finding rests on two outlets with opposed owners, list both — that is
what makes the convergence visible.
The renderer folds this away under a "Sources" toggle, so citing properly costs
the reader no screen space. There is no reason to leave a featured item
unsourced.
## Traffic data
The context may also contain entries tagged `"category": "flight_traffic"`
@ -155,8 +394,12 @@ Use them only like this:
heuristic, and warships routinely sail with AIS switched off — so a quiet
region is evidence of nothing, and you must never write that an area is calm
because these feeds are quiet.
- One snapshot is not a trend. You have no previous run to compare against, so do
not describe anything here as rising, falling, massing or building up.
- One snapshot is not a trend **unless the `history` block gives you earlier
readings of that same region** (see "History" above). With them, a sustained
fall in merchant traffic is a real finding and you may say so, with the dates
and figures attached. Without them — and for military aircraft counts, which
are a callsign heuristic on a single instantaneous sample — do not describe
anything as rising, falling, massing or building up.
If these entries add nothing to a story you are already telling, leave them out
entirely. That is the expected outcome most days.
@ -172,9 +415,30 @@ data showing falling merchant traffic through it is one correlated finding,
not two coincidental ones. Only state a correlation the context actually
supports; see "No speculation" above.
**A marker is a briefing, not a pin.** Give every marker a `summary`: two to four
sentences on what is happening at that place, in the analytical register of the
rest of this section — what is at stake materially, for which class, and how it
connects to anything else in this run. Give it a `sources` array too, under the
same rules as the "Curating and quoting" section above. The renderer prints each
marker's summary under the globe with its sources folded away beneath, so the
globe is the index and the summaries are the section — a marker whose whole
content is a place name wastes the space it occupies.
A marker's `label` stays short (it is drawn on the globe itself); everything you
want to say goes in `summary`.
You are read-only: you summarise and analyse, you never propose that the system
send, post, or publish anything.
## Composing the section
Inside the rules above, the layout is yours. You have `kind: "text"` for prose,
`kind: "list"` for anything enumerable, and `kind: "globe"` for the world map,
each in a window of its own, plus the fold-out `sources` block on any of them.
Use as many or as few windows as the material justifies, order them by what
matters most this run, and let a quiet run be a short section. Do not pad to a
shape; do not invent a window to fill the canvas.
## Output
Output **only** a single JSON object matching this schema — no prose before or
@ -191,6 +455,16 @@ after it, no markdown code fence:
"title": "string",
"kind": "text" | "list" | "globe",
"content": "markdown-ish string for kind=text, or an array of strings for kind=list",
"sources": [
{
"title": "the entry's own headline",
"outlet": "the entry's `feed`",
"owner": "the entry's `owner`, copied as given",
"bias": "the entry's `bias`, copied as given",
"url": "the entry's `link`, copied exactly — never composed",
"quote": "a verbatim span of that entry's text"
}
],
"globe_markers": [
{
"lat": 0.0,
@ -198,7 +472,9 @@ after it, no markdown code fence:
"label": "string",
"icon": "star|hammer-sickle|default",
"color": "#hex",
"glow": true
"glow": true,
"summary": "2-4 sentences on what is happening here and what is at stake",
"sources": [ "... same shape as the window's sources ..." ]
}
]
}
@ -235,16 +511,35 @@ Rules:
say so concisely (e.g. "Port of X — strike + falling traffic"), not just
name the place.
- `globe_markers` belongs only on `kind: "globe"` windows. Omit it everywhere else.
- Use one `kind: "list"` window titled something like "Highlights" for the
curated, quoted items from "Curating and quoting" above — one array entry
per featured item, each entry holding the quote, the relevance line, and
(where warranted) its impact analysis. If a single item's analysis is long
enough to crowd the list, give it its own `kind: "text"` window instead and
keep a short pointer to it in the list entry.
- Use `kind: "list"` for other enumerable material too (e.g. the financial
indicators, each line stating the move *and* what it means for working
people), and `kind: "text"` for standalone analysis that doesn't fit the
highlights list.
- `summary` and `sources` on a marker are what the reader actually reads; the
globe itself is the index. See "Correlating on the globe" above.
- **One window per question, in this order**, each holding the curated, quoted
items that answer it (see "Curating and quoting"):
- `political-global` — question 1, the global class struggle. Usually
`kind: "list"`, one entry per featured item.
- `political-local` — question 2, Austria and Vorarlberg. Always present, even
if its content is one line saying nothing local cleared the bar this run.
- `political-horizon` — question 3, the consequential-but-slower developments.
Omit the window entirely if the run genuinely has none.
- `political-rci` — question 4, the International and comrades' reports. Omit
if there is nothing; never manufacture organisational news.
- `political-agenda` — the upcoming meetings and their agenda points, per "The
agendas" above. Omit when no agenda arrived.
- `political-todo`, titled **"Political todos"** — the tasks those agendas
actually ask for, one line each with the quote they came from. Omit entirely
when nothing is asked.
- `political-watch-later` — new episodes, per "Watch later" above. Omit when
there are none.
You may add further windows beyond these when an item needs its own space (a
long piece of analysis, the financial indicators as their own `kind: "list"`
with each line stating the move *and* what it means for working people). Do not
drop or rename the four question windows to make room.
- At `detail_level: "compact"` keep the globe and fold the questions into a
single `kind: "list"` window — one or two lines each for questions 1 and 2, one
line each for 3 and 4 if they have anything. Sources still attach; they cost no
space when folded. **Keep `political-todo` as its own window even here**: it is
the one part of this section a person acts on rather than reads, and it is the
first thing they will look for on a phone.
- `narration` is spoken aloud by a TTS voice, so no markdown, no URLs, no emoji.
- If you cannot produce valid JSON matching this schema, output a single
`kind: "text"` window with your best-effort plain-text summary instead.

47
digest-engine/whatsapp-bridge/index.js
View File

@ -17,14 +17,21 @@
const fs = require('fs');
const path = require('path');
const crypto = require('crypto');
const qrcode = require('qrcode-terminal');
const { Client, LocalAuth } = require('whatsapp-web.js');
const DATA_DIR = process.env.WHATSAPP_DATA_DIR || '/data';
const MESSAGES_PATH = path.join(DATA_DIR, 'messages.jsonl');
const AUTH_PATH = path.join(DATA_DIR, '.wwebjs_auth');
// Document attachments land here. digest-engine mounts the parent of this bridge's
// /data, so what is /data/documents here is /data/whatsapp-bridge/documents there —
// see ingest/whatsapp_ingest.py, which resolves exactly that.
const DOCUMENTS_DIR = path.join(DATA_DIR, 'documents');
const MAX_DOCUMENT_BYTES = Number(process.env.WHATSAPP_MAX_DOCUMENT_BYTES || 10 * 1024 * 1024);
fs.mkdirSync(DATA_DIR, { recursive: true });
fs.mkdirSync(DOCUMENTS_DIR, { recursive: true });
const client = new Client({
authStrategy: new LocalAuth({ dataPath: AUTH_PATH }),
@ -90,8 +97,46 @@ client.on('message', async (message) => {
is_group: chat ? Boolean(chat.isGroup) : false,
timestamp: message.timestamp,
type: message.type,
body: message.body || ''
body: message.body || '',
has_media: Boolean(message.hasMedia),
// whatsapp-web.js exposes the document's own filename on the raw payload. This
// is not part of its documented API and may simply be undefined on some
// message types or library versions — in which case the digest still gets the
// caption and the type, and agenda matching falls back to those.
filename: (message._data && message._data.filename) || null,
mimetype: (message._data && message._data.mimetype) || null,
document_file: null
};
// DOCUMENTS ONLY, and only documents: a meeting agenda arrives as a PDF, and
// digest-engine reads its text to pull out the agenda points. Photos, video and
// audio are never downloaded — they are the bulk of what a group chat carries,
// this container has no use for them, and every download is one more request
// through a session that is already the highest-risk part of this project.
if (message.hasMedia && message.type === 'document') {
try {
const media = await message.downloadMedia();
const size = media && media.data ? Buffer.byteLength(media.data, 'base64') : 0;
if (!media || !media.data) {
console.error('A document had no downloadable data; recording it by name only.');
} else if (size > MAX_DOCUMENT_BYTES) {
console.error(`Document is ${size} bytes, over the ${MAX_DOCUMENT_BYTES} cap; recording it by name only.`);
} else {
// Sanitised basename plus a hash, so a hostile or merely awkward filename
// ("../../etc/passwd", or the fourth "Tagesordnung.pdf" this month) can
// neither escape this directory nor overwrite an earlier file.
const raw = media.filename || record.filename || `${record.id || Date.now()}.bin`;
const safe = path.basename(raw).replace(/[^A-Za-z0-9._-]+/g, '_').slice(0, 80) || 'document.bin';
const stamp = crypto.createHash('sha256').update(String(record.id || raw)).digest('hex').slice(0, 8);
const target = path.join(DOCUMENTS_DIR, `${stamp}-${safe}`);
fs.writeFileSync(target, Buffer.from(media.data, 'base64'));
record.document_file = path.basename(target);
}
} catch (err) {
// A failed download costs the document's text, not the message.
console.error(`Could not save a document: ${err}`);
}
}
// appendFileSync opens/appends/closes per message, so whatsapp_ingest.py can
// rename the file out from under us mid-run without losing a partial write.
fs.appendFileSync(MESSAGES_PATH, JSON.stringify(record) + '\n', 'utf8');

10
docs/project-plan.md
View File

@ -184,7 +184,7 @@ real hardware" callouts for everything downstream of this.)*
| Thin-client browser | **Firefox (kiosk)** | General browsing + the rendering surface for the LLM-generated digest canvas |
| Thin-client voice | **wyoming-satellite** + **openWakeWord** | Local wake-word spotting, streams to the existing Phase 3 Wyoming faster-whisper/Piper Assist pipeline — no new STT/TTS infrastructure |
| Digest scheduling | **systemd timer** | 4x/day cadence, same `OnCalendar` pattern as the existing restic backup timer |
| Digest engine | **digest-engine** (custom Python) | Ingests mail/messages/news/financial data, calls the Phase 3 Ollama host, renders the personal/political/household digest sections |
| Digest engine | **digest-engine** (custom Python) | Ingests mail/messages/news/financial data, calls the Phase 3 Ollama host, renders the personal/political/household/network digest sections, generating only the ones the household ticked in `identity` |
| Digest static serving | **digest-web** (nginx:alpine/Caddy) | Serves the rendered digest artifact read-only to both the thin client and an HA iframe card |
| Digest ingestion — Signal | **signal-cli** | Linked-device (JSON-RPC) read access to Signal messages |
| Digest ingestion — Telegram | **Telethon** | MTProto client logged in as the real account — the Bot API can't read personal DMs |
@ -478,7 +478,7 @@ recurrence/TLS traps. Nextcloud itself is pre-existing; nothing in this repo dep
8. **Voice interactivity**:
- Only the rooms with a chosen thin client *and* an attached mic run `wyoming-satellite` (openWakeWord), streaming to the existing Phase 3 Wyoming faster-whisper/Piper pipeline — no new STT/TTS infrastructure.
- `thinclient-agent` accepts an Assist-resolved "play my digest" intent: switches the dedicated Firefox workspace into the "full/thorough" canvas view (vs. the HA dashboard's "compact" view) and narrates via the existing Piper TTS output.
- **Room/person routing**: reuse Phase 2's presence system (`person.*`/area entities). If exactly one recognized person is in the room where the wake word fired, play that person's personal digest section. If more than one is present, Assist asks **"whose digest?"** and disambiguates by spoken name before playing the personal section — never guesses. Political/household sections always play regardless of presence ambiguity.
- **Room/person routing — voice-activated, automatically recognized, never asked**: the digest canvas is shown only when somebody asks for it (a spoken "play my digest", or the HA button). **Nothing displays it because a person walked past a screen**, and nothing here polls presence to decide to show something. When the wake word fires, HA calls `identity`'s `GET /speaker?area=<area>`, which resolves who is asking from the two signals it already fuses: an IRK-resolved BLE identifier in that area, and a Frigate face sighting inside `FACE_PRESENCE_WINDOW_SECONDS`. One person in the room is them; several, and the most recent camera sighting decides. That answer becomes the `person` parameter the thin client passes to the canvas, which then shows exactly that person's chosen sections (Phase 12 step 5). **An unresolved answer means show less, not ask**: the canvas renders every section except the personal one, which is the same rule as before — automating the recognition is only acceptable because the unresolved case still fails closed. This replaces the earlier "Assist asks *whose digest?*" disambiguation: the household wanted recognition, not an interrogation. There is no speaker identification in this stack and `/speaker` does not pretend otherwise — it identifies who is in the room, not whose voice it was.
9. Network placement: plain trusted LAN for now (no VLAN precedent exists for a general client device class yet — only the unbuilt camera-VLAN concept). Revisitable later as a Phase-10-style expansion item, not a blocker now.
10. **Validate the image boots to a working kiosk session (Sway, local mpv/Spotify playback) with Mosquitto/HA/container-host powered off** — must not hang waiting on the network, same "reactive path never depends on a remote service" philosophy applied to the thin client's own boot path.
@ -494,10 +494,14 @@ recurrence/TLS traps. Nextcloud itself is pre-existing; nothing in this repo dep
- **WhatsApp** — no officially-sanctioned API option exists. Rather than a protocol-reimplementation library (Baileys), run a small **`whatsapp-bridge`** sidecar (Node.js, `digest-engine/whatsapp-bridge/`): a real Chromium logged into the actual web.whatsapp.com client via **whatsapp-web.js** (Puppeteer), inside its own container running **Xvfb** so Chromium executes **headful** (not `headless: true`) — WhatsApp's automation detection specifically fingerprints headless Chrome, so a virtual-display "real browser" session is meaningfully lower-risk than either Baileys or true-headless whatsapp-web.js, though not zero-risk (it's still automated use of a personal account). One-time interactive QR-code login persists a session directory (mounted volume) so subsequent runs don't need re-scanning. The bridge exposes incoming messages over a local-only channel (e.g. a Unix socket or a small internal HTTP endpoint on the compose network, never published to the LAN) that `digest-engine/ingest/whatsapp_ingest.py` reads each run. Still gate behind `ENABLE_WHATSAPP_INGEST="false"`, off by default, with a warning in script output + `digest-engine/README.md`; recommend a secondary/non-critical number if enabled. Build this one last.
- **News** — `feedparser` over a curated OPML list (`digest-engine/feeds/curated-feeds.opml`), seeded with the confirmed `https://www.marxist.com/feed/rss` plus a mainstream-outlet list (exact outlets: see open decisions).
- **Financial** — FRED API (macro/unemployment, e.g. `UNRATE`) + Stooq keyless CSV (stocks/oil/commodities, preferred over Alpha Vantage's tight free-tier cap).
5. LLM synthesis: assemble the run's ingested content into context, call the existing Phase 3 Ollama host with three separate prompt templates (`digest-engine/synth/prompts/{personal,political,household}.md`):
5. LLM synthesis: assemble the run's ingested content into context, call the existing Phase 3 Ollama host with one prompt template per section (`digest-engine/synth/prompts/{personal,political,household,network}.md`):
- **Personal** — from personal-flagged mail/messages.
- **Political** — Marxist/working-class analytical framing (marxist.com feed as theoretical basis) synthesizing mainstream news + financial indicators + politically-flagged mail, laid out on the "holo globe" with colored/glowing markers (e.g. revolutionary-situation markers in red with a hammer-and-sickle/star motif).
- **Household/calendar** — from the existing Nextcloud CalDAV integration (Phase 8) and Grocy state (Phase 7).
- **Network** — the OPNsense/Suricata intrusion-detection summary, split out of the household section (implementation note, later than the original plan) so the two can be wanted separately.
- **The archive** (implementation note): `digest-engine/archive.py` keeps every ingested item and every measured number in SQLite across runs, so a run can say "up from 5.1% in June", "this IDS signature has fired every night this week", or "this story first appeared on Monday". Items are deduplicated on a fingerprint so `first_seen_at` means something; numbers are stored one row per measurement so a trend is a query. History enters each prompt as its own timestamped block, and it is what lifts the "one snapshot is not a trend" prohibition — but only with the figures and dates attached. It also persists mail and messages for its retention window, which `DIGEST_ARCHIVE_EXCLUDE_SOURCES` exists to bound.
- **Meeting agendas** (implementation note): `digest-engine/agenda.py` matches a Tagesordnung PDF arriving by mail or WhatsApp to the calendar event it belongs to (date in the filename/subject/heading, then wording in common, each labelled with its confidence), reads it with pypdf, and extracts its numbered points mechanically. The household section lists the points and derives a todo window from what the document actually asks for; the political section gets the points as a relevance filter — a story touching Thursday's agenda outranks a bigger one that doesn't. A scanned agenda extracts nothing and is named but never characterised; there is no OCR here.
- **Per-person section toggles** (implementation note): each person picks their own sections in `identity`'s admin panel (`people.digest_sections`, `GET /digest-preferences`). A run generates the **union** of what the household asked for — a section nobody wants costs neither an LLM call nor its sources' ingestion — and each surface filters to the person Home Assistant resolved. That last half is a display filter, not an access control: `digest-web` serves the whole artifact read-only to the LAN. An unreachable `identity` means "generate everything", never "generate nothing".
- A **detail-level** parameter (`compact` for the HA iframe, `full` for the thin-client fullscreen view) makes the thin-client rendering genuinely more thorough without needing two independent generation passes.
6. Rendering: vendor the offline **digest-canvas SDK** under `digest-engine/render/digest-canvas-sdk/` (globe + `addMarker()`, window/panel chrome, glow/holo CSS utility, no CDN dependency). Each run's LLM job is to call into this SDK with structured content, not hand-roll projection math. Use a custom inline SVG or Unicode ☭ (U+262D, explicit font-fallback + CSS glow) for hammer-and-sickle iconography since Nerd Fonts has no such glyph.
7. **Live follow-up voice Q&A**: persist each run's actually-used ingested-context bundle (not full raw content) as `digest-engine/output/<run-timestamp>/context.json`. Expose a small HA tool (`digest_followup_query`) so a spoken follow-up ("tell me more about the unemployment numbers") feeds the cached context + question back into Ollama for a grounded, low-latency answer — no fresh ingestion pass. The answer can push a new small window/card onto the already-open thin-client canvas via a websocket, keeping the "flexible windows" idea alive live, not just at generation time.

83
hosts/thin-client/README.md
View File

@ -311,6 +311,9 @@ integration you should get one device per thin client with:
plugged in (dynamic, re-scanned periodically — see "Capture-card /
receiver-box viewing" above); switches to `5:capture` and shows the picked one
full-screen via mpv.
- **Display** (switch) — the TV's own power, over HDMI-CEC with a Sway DPMS
fallback. Meant to be driven by room presence; see "Turning the TV off when the
room is empty" below.
- **Workspace** (select) — `1:web` / `2:digest` / `3:media` / `4:admin` / `5:capture`
- **Launch Firefox**, **Launch web browser**, **Launch Steam Link** (buttons)
- **Playback state** (sensor, with track metadata as attributes), **Volume** (number),
@ -376,6 +379,77 @@ never the payload — that reaches `capture-view` as an argv element. An unknown
or since-unplugged selection resolves to "no source," never to acting on
whatever string HA sent.
## Turning the TV off when the room is empty
A wall-mounted TV showing a canvas to an empty room is the largest power draw
this machine is attached to — 60–150 W of lit panel against the thin client's own
handful of watts. The **Display** switch turns it off on demand, and the intended
driver is room presence.
**How the agent does it** (`display_power.py`): HDMI-CEC first, over the same
cable that carries the picture — `cec-ctl --to 0 --standby` to sleep the panel,
`--image-view-on` plus `--active-source` to wake it and claim the input back. No
network path to the TV, no pairing, no account, and it keeps working with the LAN
down. Then, always, `swaymsg output <name> power off`, which stops the compositor
driving pixels — that is the fallback for a set whose CEC is broken or switched
off, and belt-and-braces on one where it works.
`cec-ctl` comes from `v4l-utils`, already in the image's package list. Most TVs
ship CEC **disabled**; enable it once in the TV's settings, where it will be called
HDMI-CEC, Bravia Sync, Anynet+, SimpLink, Viera Link or similar. `CEC_DEVICE`,
`DISPLAY_OUTPUTS` and `DISPLAY_USE_CEC` in the agent's config cover the machine
with two adapters, several screens, or a panel whose CEC you want left alone.
**"Off" means standby, honestly.** A TV in CEC standby still draws roughly half a
watt — that is what lets it hear the wake. This turns 60–150 W into ~0.5 W; it is
not a smart plug and does not claim to be.
**The presence decision stays in Home Assistant**, where presence already lives —
this agent only does what it is told, same as every other entity here. A worked
example, unverified against a running HA like every other HA snippet in this repo:
```yaml
# automations.yaml (excerpt). Replace the entity ids with your own.
- alias: "Living room TV on when the room is occupied"
trigger:
- platform: state
entity_id: binary_sensor.living_room_occupancy
to: "on"
action:
- service: switch.turn_on
target:
entity_id: switch.thinclient_living_room_display
- alias: "Living room TV off when the room empties"
trigger:
- platform: state
entity_id: binary_sensor.living_room_occupancy
to: "off"
# Long enough that walking to the kitchen for a glass of water does not
# cycle the panel. A TV that flickers off behind you is worse than one
# left on, and CEC wake takes a second or two.
for: "00:05:00"
condition:
# Don't black out a film. media_player state comes from the agent's own
# playback sensor — see "Home Assistant entities" above.
- condition: not
conditions:
- condition: state
entity_id: sensor.thinclient_living_room_playback_state
state: "playing"
action:
- service: switch.turn_off
target:
entity_id: switch.thinclient_living_room_display
```
**An Android TV with no thin client attached** is a Home Assistant question
rather than one for this repo: pair it with the **Android TV Remote** integration
and swap the `switch.turn_on`/`turn_off` calls above for
`media_player.turn_on`/`turn_off` on that entity. Waking one over the network
needs the TV's own "network standby"/"wake on cast" setting enabled — off by
default on most sets, and the reason a TV that sleeps fine refuses to wake.
## Manual verification still outstanding
None of this has been run on hardware. In rough order:
@ -509,3 +583,12 @@ None of this has been run on hardware. In rough order:
back up from being powered off, and whether the overlay's text is legible
against a bright/high-contrast real photo rather than the dark backgrounds
assumed while choosing the text-shadow-only styling in `eww.scss`.
22. **Display power over CEC has never been run against a real TV.** The command
shapes are from `cec-ctl`'s own documentation, not from a session with a
panel on the other end. Three things to check on the first set it meets:
that CEC standby actually darkens it rather than just blanking the picture
(compare the mains draw, not the screen); that waking it comes back to *this*
input rather than to whatever it was on before; and that the Sway DPMS half
does not leave a "no signal" banner glowing on a set that ignored the CEC
standby. `DISPLAY_USE_CEC=false` is the escape hatch if a TV reacts badly to
being addressed at all.

136
hosts/thin-client/agent/thinclient_agent/display_power.py Normal file
View File

@ -0,0 +1,136 @@
"""Turning the attached TV on and off, so an empty room does not power a panel.
A wall-mounted Android TV driven by one of these thin clients draws 60-150 W while
it shows a canvas nobody is in the room to look at. This module is what Home
Assistant calls when presence says the room is occupied or empty — the decision
lives in HA (an area's occupancy, the same presence system everything else here
uses), and the doing lives here.
TWO MECHANISMS, IN THIS ORDER
-----------------------------
1. **HDMI-CEC** (`cec-ctl`, from v4l-utils). The thin client is the HDMI *source*,
so it can put the display into standby and wake it again over the HDMI cable
itself. That is the one that actually saves the panel's power, and it needs no
network path to the TV, no pairing, no credentials, and no account — it keeps
working with the LAN down, which is this project's whole posture. Android TV
and Google TV sets implement CEC as "HDMI-CEC", "Bravia Sync", "Anynet+",
"Simplink" and a dozen other brand names for the same standard; it usually has
to be enabled in the TV's settings once.
2. **Sway DPMS** (`swaymsg output <name> power on|off`) as the fallback, and as a
belt-and-braces companion: it stops the compositor driving pixels and drops the
HDMI signal, which most panels treat as "go to sleep" on their own. It always
works because it needs nothing but the compositor already running here — but on
its own it may leave a TV showing a "no signal" banner rather than sleeping,
which is why CEC is tried first.
Both are attempted on every call unless CEC is switched off, because they fail in
different ways and neither reports reliably.
WHAT "OFF" HONESTLY MEANS
-------------------------
Standby, not disconnected. A TV in CEC standby still draws roughly half a watt to
keep listening on the HDMI line — that is what makes waking it possible at all.
This turns 60-150 W of lit panel into ~0.5 W of standby; it is not a smart plug
and does not pretend to be. If a set is one of the ones that ignores CEC standby
entirely, you will see it immediately (the panel stays lit) — that is what the
verification note in hosts/thin-client/README.md is for.
SECURITY POSTURE, UNCHANGED
---------------------------
This is another enumerated MQTT command, exactly like the workspace switch and the
canvas buttons: HA -> MQTT -> a fixed action here. A payload never becomes an argv
element — `set_power()` takes a boolean, and the device names come from local
configuration, never from the message. See mqtt_discovery.py's module docstring.
"""
from __future__ import annotations
import logging
import os
import shutil
import subprocess
log = logging.getLogger(__name__)
CEC_TIMEOUT_SECONDS = 10
class DisplayPower:
def __init__(self, sway, cec_device: str | None = None, outputs: str = "*",
use_cec: bool = True):
self.sway = sway
# The CEC adapter, e.g. /dev/cec0. Most systems have exactly one and cec-ctl
# finds it on its own; this is for the machine that has two.
self.cec_device = cec_device or os.environ.get("CEC_DEVICE", "")
# Which Sway outputs to power down. "*" is every output, which is right for a
# thin client driving one TV; name an output (e.g. "HDMI-A-1") on a machine
# where only one of several screens is the TV.
self.outputs = outputs or "*"
self.use_cec = use_cec
self.state = True
# --- CEC ----------------------------------------------------------------
def _cec(self, *args: str) -> bool:
binary = shutil.which("cec-ctl")
if not binary:
log.info("cec-ctl is not installed; falling back to DPMS only")
return False
command = [binary]
if self.cec_device:
command += ["-d", self.cec_device]
command += list(args)
try:
result = subprocess.run(
command, capture_output=True, text=True, timeout=CEC_TIMEOUT_SECONDS
)
except (OSError, subprocess.SubprocessError) as exc:
log.warning("cec-ctl %s failed: %s", " ".join(args), exc)
return False
if result.returncode != 0:
log.warning("cec-ctl %s: %s", " ".join(args), (result.stderr or "").strip())
return False
return True
# --- the one public action ---------------------------------------------
def set_power(self, on: bool) -> bool:
"""Turn the display on or off. Returns the state it believes it left it in.
Deliberately not idempotent-by-early-return: HA asking for "on" when this
object already thinks it is on must still send the wake, because the TV may
have been turned off with its own remote and nothing here would know. The
state field is for reporting, never for skipping work.
"""
log.info("display: turning the panel %s", "on" if on else "off")
if self.use_cec:
# --to 0 addresses the TV specifically (logical address 0) rather than
# broadcasting, so a soundbar or receiver on the same bus is left alone.
if on:
self._cec("--to", "0", "--image-view-on")
# Ask to become the active source too: waking a TV that then shows a
# different input is the same as not waking it.
self._cec("--to", "0", "--active-source", "phys-addr=0.0.0.0")
else:
self._cec("--to", "0", "--standby")
# Always also drive the compositor: on a set that ignores CEC this is what
# stops it displaying, and on one that honours CEC it stops the thin client
# rendering to a panel nobody is looking at.
self.sway.swaymsg("output", self.outputs, "power", "on" if on else "off")
self.state = on
return self.state
def handle_command(self, payload: str) -> bool:
"""MQTT payload -> action. Anything that isn't a known ON/OFF word is ignored
rather than guessed at, per the enumerated-command rule."""
value = (payload or "").strip().upper()
if value in ("ON", "TRUE", "1"):
return self.set_power(True)
if value in ("OFF", "FALSE", "0"):
return self.set_power(False)
log.warning("display: ignoring unknown power payload %r", payload)
return self.state

18
hosts/thin-client/agent/thinclient_agent/main.py
View File

@ -18,6 +18,7 @@ from .admin_canvas import AdminCanvas
from .audio_control import AudioControl
from .capture_control import CaptureControl, find_audio_card
from .digest_canvas import DETAIL_LEVELS, DigestCanvas
from .display_power import DisplayPower
from .input_control import InputControl
from .mpris_bridge import MprisBridge
from .mqtt_discovery import Discovery
@ -158,6 +159,15 @@ def main() -> int:
sway = SwayControl()
canvas = DigestCanvas(sway, config.get("DIGEST_WEB_URL", ""))
admin_canvas = AdminCanvas(sway, config.get("ADMIN_WEB_URL", ""))
# The TV's own power. DISPLAY_OUTPUTS names which Sway output(s) are the TV
# ("*" on a machine driving one screen); DISPLAY_USE_CEC=false drops to DPMS
# only, for a panel whose CEC is broken or deliberately disabled.
display = DisplayPower(
sway,
cec_device=config.get("CEC_DEVICE", ""),
outputs=config.get("DISPLAY_OUTPUTS", "*"),
use_cec=str(config.get("DISPLAY_USE_CEC", "true")).strip().lower() == "true",
)
apps = build_apps(config)
audio = AudioControl(sway.session_env)
capture = CaptureControl()
@ -216,6 +226,13 @@ def main() -> int:
admin_canvas.show()
discovery.publish_workspace(WS_ADMIN)
def on_display_power(payload: str) -> None:
# Driven by an HA automation on room occupancy (see hosts/thin-client's
# README): a TV showing a canvas to an empty room is the single largest
# power draw this machine is attached to. The decision stays in HA, where
# presence already lives; this only does what it is told.
discovery.publish_display_power(display.handle_command(payload))
def on_audio_output(payload: str) -> None:
discovery.publish_audio_output(audio.select(payload))
@ -264,6 +281,7 @@ def main() -> int:
discovery.register_media_player(mpris.handle_command, mpris.set_volume)
discovery.register_digest(on_show_digest, on_detail_level, DETAIL_LEVELS, canvas.detail_level)
discovery.register_admin_canvas(on_show_admin_canvas)
discovery.register_display_power(on_display_power, display.state)
discovery.register_app_launchers(apps, on_launch)
discovery.register_workspace_select(WORKSPACES, on_workspace, WS_DIGEST)
# audio.apply_preferred() already ran once at startup (before MQTT was even

30
hosts/thin-client/agent/thinclient_agent/mqtt_discovery.py
View File

@ -41,6 +41,7 @@ class Discovery:
self.capture_source_state_topic = f"{self.base}/capture/source/state"
self.remote_target_state_topic = f"{self.base}/remote/target/state"
self.input_text_state_topic = f"{self.base}/input/text/state"
self.display_power_state_topic = f"{self.base}/display/power/state"
self._handlers: dict[str, Callable[[str], None]] = {}
self.device = {
@ -211,6 +212,35 @@ class Discovery:
},
)
def register_display_power(self, on_set, current: bool) -> None:
"""The TV's own power, as a switch HA can drive from room presence.
A switch rather than a button, because the interesting automation is "this
area became unoccupied" -> off, "somebody walked in" -> on, and that needs a
state HA can read back as well as set. The state is what this agent last
did, not what the panel reports: CEC gives no reliable read-back, and a
state that lies about the TV having been turned off by its own remote is
better than one that blocks the next wake — see display_power.set_power().
"""
self._publish_config(
"switch",
"display_power",
{
"name": "Display",
"command_topic": self._command_topic("display/power/set", on_set),
"state_topic": self.display_power_state_topic,
"payload_on": "ON",
"payload_off": "OFF",
"icon": "mdi:television",
},
)
self.publish_display_power(current)
def publish_display_power(self, on: bool) -> None:
self.client.publish(
self.display_power_state_topic, "ON" if on else "OFF", qos=1, retain=True
)
def register_app_launchers(self, apps, on_launch) -> None:
for key, app in apps.items():
self._publish_config(

79
identity/README.md
View File

@ -166,7 +166,8 @@ http://<host>:8098/admin.html?api=http://<host>:8097&token=<IDENTITY_TOKEN>
```
Four tabs: **People** (tap anyone to edit every field, their devices, their door
rights and their chores), **Prune**, **History**, and **Access log**.
rights, their chores and which digests get generated for them), **Prune**,
**History**, and **Access log**.
> The token is in the URL, exactly like the two kiosk pages — that's the existing
> pattern here, not a new decision, and it's why this service treats the token as the
@ -393,6 +394,69 @@ where "how do I relate to this specific household member" facts belong, not dupl
into `chores/`'s own database. Assignments get their own table only because they're
many-per-person, not because they belong anywhere else.
## "Who just spoke?" — automatic recognition for the voice path
`GET /speaker?area=<ha_area>` answers who is asking, so a spoken "play my digest"
shows *that person's* digest without anybody typing or spelling a name, and
without the assistant interrogating the room.
It resolves from the two presence signals this service already fuses:
1. **One person in that area** — that's them.
2. **Several** — the one a camera recognised most recently, if any did inside
`FACE_PRESENCE_WINDOW_SECONDS`. A face seen thirty seconds ago is the best
evidence available that a particular person is the one standing there talking.
3. **Nobody in the area but exactly one person home** — them.
4. **Otherwise `person` is null**, with the candidates named.
**An unresolved answer means show less, not ask.** The caller's fallback is a
digest with no personal section — never a prompt, never a guess. Automating the
recognition is only defensible *because* the ambiguous case still fails closed;
that is the same rule the registry applies to registration, applied to display.
**Two things it is not.** It is not speaker identification — nothing here listens
to a voice; it works out who is *in the room*, so two people in a kitchen where
one was just recognised by a camera resolve to that one even if the other spoke.
And it is not a display trigger: nothing in this project shows a digest because
somebody walked past a screen. The canvas opens when it is asked for, and this
endpoint only answers *by whom*.
## Digest settings — owned here, used by `digest-engine/`
Which of `digest-engine`'s four digests get generated for a person — **network**,
**household**, **social** (its `personal` section: mail and messages) and **political /
news** — is a per-person setting on this registry, editable in the admin panel's person
editor. Same reasoning as the chore fields above: it is a standing fact about a
household member, and `identity` is already this project's source of truth for those,
so it lives as a column on `people` rather than in a second database over in
`digest-engine`.
`digest-engine` reads `GET /digest-preferences` once at the start of each of its four
daily runs — the same shape-for-the-consumer pattern as `GET /chore-assignments` — and:
- generates the **union** of what the household asked for. A section nobody has ticked
costs no LLM call, and `digest-engine` also skips ingesting the sources only that
section reads (turn the political one off for everybody and its news, financial and
flight/naval fetches stop happening at all).
- filters each surface to the person Home Assistant resolved, using the per-person sets
this endpoint hands over.
**The second half is a display filter, not an access control**, and it should not be
described to anyone as privacy: `digest-web` serves the rendered digest read-only to
anything on the LAN, so an unticked section is off somebody's screen and out of their
narration, not out of their reach. The part that genuinely does not exist anywhere is
the part that was never generated.
Two deliberate defaults:
- **Never set means all four.** A household that never opens this panel keeps exactly
the digest it had before the setting existed. An explicitly empty set is different and
is honoured as written — "generate nothing for me" is a real answer, and the column
stores `''` rather than `NULL` to keep the two distinguishable.
- **This service being unreachable means all four too.** `digest-engine` treats any
failed lookup as "no preferences known" and generates everything, because one
container being down must never silently cost the household its whole digest.
## Camera face recognition — a second presence signal, never a registration one
If Tapo pan/tilt cameras are wired into Frigate as additional camera sources
@ -495,7 +559,7 @@ not network placement.
| `POST /register` | `{"name", "device_id", "photo_id"?, "entity_id"?, "no_device"?}` -> registers, or returns a reason it couldn't (see above) |
| `POST /register/guest` | `{"device_id", "photo_id"?}` -> registers "Guest N", no name needed |
| `GET /people` | admin/audit list of every person: identifiers, device grants, chore assignments, `nickname`/`speak_name`, `last_visit_at`, `visit_count`, `currently_home_since` |
| `POST /people/<id>` | edit any editable field — `{"name"?, "nickname"?, "note"?, "chore_exempt"?, "chore_reminder_style"?, "notify_on_arrival"?, "announce_arrivals"?, "notify_topic"?, "clear_photo"?}`. Omitted keys are left alone |
| `POST /people/<id>` | edit any editable field — `{"name"?, "nickname"?, "note"?, "chore_exempt"?, "chore_reminder_style"?, "notify_on_arrival"?, "announce_arrivals"?, "notify_topic"?, "digest_sections"?, "clear_photo"?}`. Omitted keys are left alone |
| `POST /people/<id>/test-notification` | push a test message to this person's ntfy topic, to prove it works |
| `GET /people/<id>/photo` | the person's profile picture (raw JPEG) — their most recent registration photo |
| `POST /people/<id>/identifiers` | `{"entity_id"}` — attach an identifier by hand (a fixed BLE tag not in range yet). Still enforces `TRUSTED_ENTITY_PREFIXES` |
@ -512,6 +576,9 @@ not network placement.
| `GET /device-access/events?limit=` | the audit log of every access check, allowed and denied |
| `GET`/`POST /people/<id>/chore-assignments` | read/replace this person's assigned chore types (`{"chore_types": [...]}`) |
| `GET /chore-assignments` | the same facts keyed by chore type — the shape `chores/` reads |
| `GET`/`POST /people/<id>/digest-settings` | read/replace which digests are generated for this person (`{"digest_sections": ["network", "household", "personal", "political"]}`) |
| `GET /digest-preferences` | every person's set plus `wanted`, the union — the shape `digest-engine/` reads |
| `GET /speaker?area=` | who just spoke in that area — `{"person", "reason", "candidates"}`, `person` null when it can't tell (see above) |
| `GET /floorplan` | every level and its drawn rooms (polygons in normalised 0–1 coordinates) |
| `POST /floorplan/levels` | create or rename a level — `{"id"?, "name", "sort_order"?}` |
| `DELETE /floorplan/levels/<id>` | remove a level and its rooms |
@ -599,3 +666,11 @@ no way to send an `Authorization` header.
*outside* the `condition: template` guard would open the door regardless of what
this service answered. `identity` cannot enforce that from its side — it only ever
answers the question.
15. **The digest-section list in `frontend/admin.js` is kept in step with `server.py`'s
`DIGEST_SECTIONS` by hand**, like `CHORE_TYPES` above it. Unlike `CHORE_TYPES` the
server does validate these, so drift shows up as a visible refusal rather than a
silent bad write — but the labels ("Social", "Political / news") are the panel's
own words and match nothing on the server, so renaming a section in
`digest-engine` needs all three places checked. `digest-engine`'s end of it (which
sections a run actually generates, and what a stopped `identity` does to a run) is
on that component's own verification list.

16
identity/frontend/admin.html
View File

@ -34,7 +34,7 @@
<section class="panel active" id="panel-people">
<section class="block">
<h2>People</h2>
<p class="hint">Tap a person to edit every field, their devices, door rights and chores.</p>
<p class="hint">Tap a person to edit every field, their devices, door rights, chores and digests.</p>
<div id="people-list" class="card-list"><p class="hint">Loading…</p></div>
</section>
</section>
@ -223,6 +223,20 @@
</div>
</fieldset>
<fieldset class="field-group">
<legend>Digests</legend>
<div class="field">
Generate for this person
<div id="edit-digest-sections" class="chip-row"></div>
<p class="hint">
A digest nobody has ticked is never generated at all — no ingestion of it
reaches the LLM. Untick all four to opt out entirely. Which sections a
screen then shows is a display filter, not a lock: anyone on the LAN can
read the whole digest file that digest-web serves.
</p>
</div>
</fieldset>
<fieldset class="field-group">
<legend>Arrival notifications</legend>
<label class="check">

32
identity/frontend/admin.js
View File

@ -22,6 +22,20 @@ if (!API || !TOKEN) {
// (see chores/README.md), so offering it here would be offering a lie.
const CHORE_TYPES = ["trash", "bin_full", "dishes"];
// digest-engine's four sections, in the order they're offered. The keys are its own
// section ids (synth/prompts/<key>.md), which is also exactly what the server stores —
// nothing translates between the two. The labels are the household's words for them:
// "personal" is the mail/messages one, "political" the news one. Kept in sync BY HAND
// with DIGEST_SECTIONS in server.py, same arrangement as CHORE_TYPES above, except the
// server does validate these — an unknown key comes back as a refusal, not a silent
// write.
const DIGEST_SECTIONS = [
["network", "Network"],
["household", "Household"],
["personal", "Social"],
["political", "Political / news"],
];
function api(path, options) {
options = options || {};
options.headers = Object.assign({ Authorization: `Bearer ${TOKEN}` }, options.headers || {});
@ -117,6 +131,11 @@ function personSubtitle(p) {
if (p.device_grants.length) bits.push(`${p.device_grants.length} right${p.device_grants.length === 1 ? "" : "s"}`);
if (p.notify_on_arrival) bits.push(p.notify_deliverable ? "🔔 arrivals" : "🔔 arrivals (undeliverable)");
if (!p.announce_arrivals) bits.push("not announced");
// Only worth a line when it isn't the default (everything) — otherwise every person
// would carry the same four words.
if (p.digest_sections.length < DIGEST_SECTIONS.length) {
bits.push(p.digest_sections.length ? `digests: ${p.digest_sections.join(", ")}` : "no digests");
}
if (p.chore_exempt) bits.push("chore-exempt");
if (p.chore_assignments.length) bits.push(`chores: ${p.chore_assignments.join(", ")}`);
return bits.join(" · ");
@ -192,6 +211,13 @@ function openEditor(personId) {
avatar.replaceChildren(document.createTextNode("👤"));
if (editing.has_photo) loadAvatar(avatar, editing.id);
document.getElementById("edit-digest-sections").innerHTML = DIGEST_SECTIONS.map(
([key, label]) =>
`<label class="chip"><input type="checkbox" data-digest="${key}"${
editing.digest_sections.includes(key) ? " checked" : ""
}> ${escapeHtml(label)}</label>`
).join("");
document.getElementById("edit-chore-types").innerHTML = CHORE_TYPES.map(
(type) =>
`<label class="chip"><input type="checkbox" data-chore="${type}"${
@ -297,6 +323,12 @@ document.getElementById("edit-save").addEventListener("click", () => {
notify_on_arrival: document.getElementById("edit-notify-on-arrival").checked,
announce_arrivals: document.getElementById("edit-announce-arrivals").checked,
notify_topic: document.getElementById("edit-notify-topic").value.trim(),
// Sent with the rest of the person's own fields rather than as a third call: unlike
// chore assignments (their own table), this is one column on `people`, so it saves
// or is refused together with everything else in the dialog.
digest_sections: Array.from(document.querySelectorAll("[data-digest]:checked")).map(
(c) => c.dataset.digest
),
})
.then((result) => {
if (!result.ok) throw new Error(result.message || "Could not save.");

256
identity/server.py
View File

@ -68,6 +68,11 @@ Endpoints:
- GET /prune/candidates, POST /people/prune last-visited-before cleanup
- GET /device-access, POST/DELETE /people/<id>/device-grants per-device rights
- GET/POST /people/<id>/chore-assignments who owes which chore
- GET /speaker who just spoke in a given area, resolved from BLE +
face presence — the voice path's automatic recognition
- GET/POST /people/<id>/digest-settings which digests digest-engine generates for
this person; GET /digest-preferences is the household-wide
shape digest-engine itself reads
- POST /people/<id>/test-notification prove an ntfy topic actually works
- GET /weather proxies the household smarthome/weather/current MQTT topic
"""
@ -173,6 +178,19 @@ NTFY_URL = os.environ.get("NTFY_URL", "").rstrip("/")
# that never sets per-person topics still works — everyone just shares one.
NTFY_DEFAULT_TOPIC = os.environ.get("NTFY_DEFAULT_TOPIC", "").strip()
# --- Digest section preferences ------------------------------------------------------
# Which of digest-engine's four digests a person wants generated for them. The keys are
# digest-engine's own section ids (synth/llm_client.py's SECTIONS and the filenames
# under synth/prompts/) — deliberately the same strings on both sides, so nothing has to
# translate between a preference stored here and a prompt run over there. The friendlier
# words the admin panel shows ("Social", "Political / news") are a UI label only.
#
# This registry stores the preference; it never generates anything. digest-engine reads
# GET /digest-preferences at the start of each run, generates the UNION of what the
# household asked for (a section nobody wants is never sent to the LLM at all), and each
# surface then shows a person only their own subset.
DIGEST_SECTIONS = ("network", "household", "personal", "political")
_sampler_stop = threading.Event()
# THE FIRST SAMPLE AFTER STARTUP NEVER NOTIFIES. It establishes a baseline instead.
@ -275,7 +293,15 @@ def init_db() -> None:
-- notify_topic: this person's own ntfy topic. NULL falls back to
-- NTFY_DEFAULT_TOPIC, so a household that never sets these still
-- works — everyone just shares one topic.
notify_topic TEXT
notify_topic TEXT,
-- digest_sections: which of digest-engine's four digests this person
-- wants generated for them, as a comma-separated list of section keys
-- (see DIGEST_SECTIONS below). NULL means "all of them" rather than
-- "none": a household that never opens this panel keeps the digest it
-- had before this column existed, and an empty string is a real,
-- deliberate "generate nothing for me" that must not be confused with
-- the never-set default.
digest_sections TEXT
);
CREATE TABLE IF NOT EXISTS visits (
id INTEGER PRIMARY KEY,
@ -409,6 +435,7 @@ def init_db() -> None:
_ensure_column(conn, "people", "notify_on_arrival", "INTEGER NOT NULL DEFAULT 0")
_ensure_column(conn, "people", "announce_arrivals", "INTEGER NOT NULL DEFAULT 1")
_ensure_column(conn, "people", "notify_topic", "TEXT")
_ensure_column(conn, "people", "digest_sections", "TEXT")
def _ha_get(path: str):
@ -730,6 +757,10 @@ def _person_payload(conn: sqlite3.Connection, person: sqlite3.Row) -> dict:
"notify_on_arrival": bool(person["notify_on_arrival"]),
"announce_arrivals": bool(person["announce_arrivals"]),
"notify_topic": person["notify_topic"],
# Which of digest-engine's digests this person wants generated for them. Always
# a concrete list, never null — the "never set" default is resolved here (to all
# four) rather than leaving every consumer to reinvent it.
"digest_sections": _stored_digest_sections(person["digest_sections"]),
# Whether a push would actually go anywhere right now. The admin panel shows
# this rather than making someone cross-reference a tickbox against an env
# file to work out why they aren't getting notifications.
@ -771,6 +802,167 @@ def set_chore_settings(person_id: int, chore_exempt: bool | None, reminder_style
return True
def _stored_digest_sections(value) -> list[str]:
"""The stored column -> the list of sections this person actually wants.
NULL (never set) means every section, so a household that upgrades into this
feature keeps exactly the digest it had before. An empty string is different and
is honoured as written: somebody ticked all four boxes off, and "generate nothing
for me" is a legitimate answer that must not silently become "generate everything."
Unknown keys — a section removed from digest-engine since the preference was saved
— are dropped on read rather than passed on to a prompt file that isn't there.
"""
if value is None:
return list(DIGEST_SECTIONS)
stored = {part.strip().lower() for part in str(value).split(",") if part.strip()}
return [section for section in DIGEST_SECTIONS if section in stored]
def set_digest_sections(person_id: int, sections: list) -> dict:
"""Replaces this person's whole set — the admin panel edits it as a row of
checkboxes, so "what's ticked now" is the natural unit, same as chore assignments.
Deliberately stores the empty string, not NULL, when nothing is ticked: see
_stored_digest_sections() for why those two must stay distinguishable.
"""
requested = {str(section).strip().lower() for section in sections if str(section).strip()}
unknown = sorted(requested - set(DIGEST_SECTIONS))
if unknown:
return {
"ok": False,
"reason": "unknown_section",
"message": f"Not a digest section: {', '.join(unknown)}.",
}
cleaned = [section for section in DIGEST_SECTIONS if section in requested]
with _db_lock, _db() as conn:
if conn.execute("SELECT 1 FROM people WHERE id = ?", (person_id,)).fetchone() is None:
return {"ok": False, "reason": "not_found", "message": "No such person."}
conn.execute(
"UPDATE people SET digest_sections = ? WHERE id = ?", (",".join(cleaned), person_id)
)
return {"ok": True, "digest_sections": cleaned}
def digest_preferences() -> dict:
"""What digest-engine reads at the start of every run.
`wanted` is the union: the sections at least one person in the household asked for,
and therefore the only ones worth spending an LLM call on. It is emitted here rather
than left for the caller to compute so that "which digests get generated" has one
definition, in the service that owns the preference.
"""
with _db_lock, _db() as conn:
rows = conn.execute(
"SELECT id, name, nickname, digest_sections, notify_topic "
"FROM people ORDER BY name COLLATE NOCASE"
).fetchall()
people = [
{
"id": row["id"],
"name": row["name"],
"nickname": row["nickname"],
"digest_sections": _stored_digest_sections(row["digest_sections"]),
# The same ntfy topic arrival notifications go to, so digest-engine can
# push "your digest is ready" to the person whose digest it is rather
# than to a household topic everyone shares. NULL means they have none
# of their own and the caller falls back to its own default — resolved
# there rather than here, because this service's fallback
# (NTFY_DEFAULT_TOPIC) is for arrivals and need not be the digest's.
"notify_topic": row["notify_topic"],
}
for row in rows
]
wanted = {section for person in people for section in person["digest_sections"]}
return {
"people": people,
"sections": list(DIGEST_SECTIONS),
"wanted": [section for section in DIGEST_SECTIONS if section in wanted],
"generated_at": _now(),
}
def resolve_speaker(area: str | None) -> dict:
"""Who just spoke, for a voice-activated per-person surface. Best-effort, and
honest when it cannot tell.
The digest canvas is never shown because somebody walked past a screen — it is
shown when somebody asks for it out loud. That makes "who asked" a question this
registry has to answer without anyone typing a name, which is what this is: the
two presence signals it already fuses (an IRK-resolved BLE identifier, and a
Frigate face sighting inside FACE_PRESENCE_WINDOW_SECONDS), narrowed by the area
the wake word fired in.
In order:
1. One person in that area -> that is them.
2. Several -> the one a camera recognised most recently, if any did inside the
face window. A face seen thirty seconds ago in a room is the best evidence
available that a particular person is the one standing there talking.
3. Nobody in the area, but exactly one person home -> them.
4. Otherwise `person` is null, with the candidates named.
WHAT AN UNRESOLVED ANSWER MEANS IS "SHOW LESS", NOT "ASK". The caller's fallback
is a digest without the personal section (see digest-engine's render.js), not a
prompt and never a guess — the plan's rule that this system must never show one
person's mail to another on an inference is unchanged by automating the
recognition; automating it is only allowed *because* the unresolved case still
fails closed.
There is no speaker identification here and this does not pretend otherwise: it
identifies who is in the room, not whose voice it was. Two people in a kitchen
where one was just recognised by a camera will resolve to that one even if the
other spoke. That is the honest limit of the signals this house has.
"""
snapshot = presence()
people = snapshot.get("people", [])
wanted_area = (area or "").strip().lower()
home = [person for person in people if person.get("home")]
in_area = [
person for person in home
if wanted_area and str(person.get("room") or "").strip().lower() == wanted_area
]
def _answer(person, reason, candidates):
return {
"person": person,
"reason": reason,
"area": area or None,
"candidates": [
{"id": c["id"], "name": c["name"], "speak_name": c["speak_name"]}
for c in candidates
],
"generated_at": _now(),
}
candidates = in_area or ([] if wanted_area else home)
if len(candidates) == 1:
return _answer(candidates[0], "the only person in the area" if in_area else "the only person home", candidates)
if len(candidates) > 1:
with _face_lock:
seen = dict(_last_face_seen)
cutoff = time.time() - FACE_PRESENCE_WINDOW_SECONDS
recent = [
(seen.get(str(person["name"]).lower(), 0), person)
for person in candidates
if seen.get(str(person["name"]).lower(), 0) >= cutoff
]
if recent:
recent.sort(key=lambda entry: entry[0], reverse=True)
return _answer(recent[0][1], "recognised by a camera most recently", candidates)
return _answer(None, "more than one person here and no recent camera sighting", candidates)
if not wanted_area and not home:
return _answer(None, "nobody is home", [])
if len(home) == 1:
return _answer(home[0], "nobody resolved to that area, but only one person is home", home)
return _answer(None, "nobody could be resolved for that area", home)
def update_person(person_id: int, fields: dict) -> dict:
"""The admin panel's "edit every field" call. Only keys actually present in
`fields` are touched — a partial edit never blanks the fields it didn't mention,
@ -841,6 +1033,29 @@ def update_person(person_id: int, fields: dict) -> dict:
}
updates.append(("notify_topic", topic or None))
# Editable from the same "save the whole editor" call as everything else, so the
# admin panel doesn't need a second round trip just for four checkboxes. An
# explicit [] is a real answer ("no digests for me") — see set_digest_sections().
if "digest_sections" in fields:
requested = fields["digest_sections"]
if not isinstance(requested, list):
return {
"ok": False,
"reason": "bad_field",
"message": "'digest_sections' must be a list of section names.",
}
cleaned = {str(section).strip().lower() for section in requested if str(section).strip()}
unknown = sorted(cleaned - set(DIGEST_SECTIONS))
if unknown:
return {
"ok": False,
"reason": "unknown_section",
"message": f"Not a digest section: {', '.join(unknown)}.",
}
updates.append(
("digest_sections", ",".join(s for s in DIGEST_SECTIONS if s in cleaned))
)
for text_field in ("chore_reminder_style", "note"):
if text_field in fields:
value = fields[text_field]
@ -1945,6 +2160,7 @@ class Handler(BaseHTTPRequestHandler):
level_image_match = re.match(r"^/floorplan/levels/(\d+)/image$", path)
visits_match = re.match(r"^/people/(\d+)/visits$", path)
assignments_match = re.match(r"^/people/(\d+)/chore-assignments$", path)
digest_settings_match = re.match(r"^/people/(\d+)/digest-settings$", path)
if path == "/people":
self._respond(HTTPStatus.OK, {"people": list_people()})
@ -1971,6 +2187,12 @@ class Handler(BaseHTTPRequestHandler):
self._respond(HTTPStatus.OK, {"events": list_device_access_events(q_int("limit", 100))})
elif path == "/chore-assignments":
self._respond(HTTPStatus.OK, chore_assignments())
elif path == "/digest-preferences":
self._respond(HTTPStatus.OK, digest_preferences())
elif path == "/speaker":
# "Who just spoke in this room?" — the voice path's automatic person
# resolution. See resolve_speaker() for what an unresolved answer means.
self._respond(HTTPStatus.OK, resolve_speaker(q("area") or q("room")))
elif path == "/floorplan":
self._respond(HTTPStatus.OK, list_floorplan())
elif path == "/floorplan/presence":
@ -1985,6 +2207,15 @@ class Handler(BaseHTTPRequestHandler):
self._respond(HTTPStatus.NOT_FOUND, {"error": "no such person"})
else:
self._respond(HTTPStatus.OK, {"chore_assignments": people[0]["chore_assignments"]})
elif digest_settings_match:
people = [p for p in list_people() if p["id"] == int(digest_settings_match.group(1))]
if not people:
self._respond(HTTPStatus.NOT_FOUND, {"error": "no such person"})
else:
self._respond(
HTTPStatus.OK,
{"digest_sections": people[0]["digest_sections"], "sections": list(DIGEST_SECTIONS)},
)
elif path == "/weather":
with _weather_lock:
self._respond(HTTPStatus.OK, dict(_last_weather))
@ -2053,6 +2284,7 @@ class Handler(BaseHTTPRequestHandler):
path = urlsplit(self.path).path
chore_settings_match = re.match(r"^/people/(\d+)/chore-settings$", path)
assignments_match = re.match(r"^/people/(\d+)/chore-assignments$", path)
digest_settings_match = re.match(r"^/people/(\d+)/digest-settings$", path)
grants_match = re.match(r"^/people/(\d+)/device-grants$", path)
identifiers_match = re.match(r"^/people/(\d+)/identifiers$", path)
test_notify_match = re.match(r"^/people/(\d+)/test-notification$", path)
@ -2075,6 +2307,8 @@ class Handler(BaseHTTPRequestHandler):
self._handle_chore_settings(int(chore_settings_match.group(1)))
elif assignments_match:
self._handle_set_assignments(int(assignments_match.group(1)))
elif digest_settings_match:
self._handle_digest_settings(int(digest_settings_match.group(1)))
elif grants_match:
self._handle_grant_device(int(grants_match.group(1)))
elif identifiers_match:
@ -2369,6 +2603,26 @@ class Handler(BaseHTTPRequestHandler):
ok = set_chore_settings(person_id, chore_exempt, reminder_style)
self._respond(HTTPStatus.OK if ok else HTTPStatus.NOT_FOUND, {"ok": ok})
def _handle_digest_settings(self, person_id: int) -> None:
payload = self._json_body()
if payload is None:
return
sections = payload.get("digest_sections")
if not isinstance(sections, list):
self._respond(
HTTPStatus.BAD_REQUEST,
{"error": "'digest_sections' must be a list of section names"},
)
return
result = set_digest_sections(person_id, sections)
self._respond(
HTTPStatus.OK if result.get("ok")
else (HTTPStatus.NOT_FOUND if result.get("reason") == "not_found" else HTTPStatus.BAD_REQUEST),
result,
)
def main() -> int:
logging.basicConfig(

38
tools/build-container-host-iso.sh
View File

@ -122,7 +122,10 @@ fi
# No address, port or token is written literally in this script.
# ---------------------------------------------------------------------------
core_log "Generating service env files from CoreSystemConfig.json"
mkdir -p "$PAYLOAD"/{identity,chores,pantry-vision,digest-engine,transit,trash-calendar}
# digest/data, not digest-engine: setup-container-host.sh bind-mounts
# $BASE_DIR/digest/data into the container as /data, which is where IDSconf.json
# has to land. A file written anywhere else would simply never be read.
mkdir -p "$PAYLOAD"/{identity,chores,pantry-vision,digest/data,transit,trash-calendar}
gen_header() {
cat <<EOF
@ -205,6 +208,39 @@ EOF
chmod 600 "$PAYLOAD/pantry-vision/pantry-vision.env"
fi
# IDSconf.json — the OPNsense credentials and tuning for digest-engine's network
# digest. JSON rather than an env file because that is the shape ingest/opnsense_ids.py
# reads (OPNSENSE_IDS_CONF_PATH, default /data/IDSconf.json). Written only when a
# base_url is configured: an empty one means "this household has no OPNsense to ask",
# and a file full of blanks would make the module warn on every run instead of simply
# staying off. digest-engine.env itself is NOT generated — it holds a dozen
# hand-provisioned credentials — so remember to set ENABLE_OPNSENSE_IDS_INGEST=true
# there as well; these credentials do nothing on their own.
if [[ "$CORE_ENABLE_DIGEST_ENGINE" == "true" && -n "$CORE_OPNSENSE_BASE_URL" ]]; then
ids_interfaces="[]"
if [[ -n "$CORE_OPNSENSE_INTERFACES" ]]; then
ids_interfaces="[$(printf '"%s",' $CORE_OPNSENSE_INTERFACES | sed 's/,$//')]"
fi
cat > "$PAYLOAD/digest/data/IDSconf.json" <<EOF
{
"_comment": "GENERATED at image build time by tools/build-container-host-iso.sh from CoreSystemConfig.json's opnsense block and secrets. Editing this file by hand works, but the next image build overwrites it — change CoreSystemConfig.json and rebuild instead. Field documentation: digest-engine/IDSconf.json.example.",
"base_url": "${CORE_OPNSENSE_BASE_URL}",
"api_key": "${CORE_OPNSENSE_API_KEY}",
"api_secret": "${CORE_OPNSENSE_API_SECRET}",
"verify_tls": ${CORE_OPNSENSE_VERIFY_TLS},
"interfaces": ${ids_interfaces},
"max_alerts_scanned": ${CORE_OPNSENSE_MAX_ALERTS_SCANNED},
"top_signatures": ${CORE_OPNSENSE_TOP_SIGNATURES},
"top_hosts": ${CORE_OPNSENSE_TOP_HOSTS},
"packet_capture_reference": "${CORE_OPNSENSE_PACKET_CAPTURE_REFERENCE}"
}
EOF
chmod 600 "$PAYLOAD/digest/data/IDSconf.json"
if [[ -z "$CORE_OPNSENSE_API_KEY" || -z "$CORE_OPNSENSE_API_SECRET" ]]; then
core_warn "opnsense.base_url is set but secrets.opnsense_api_key/secret are empty — the network digest will skip until you fill them in and rebuild."
fi
fi
if [[ "$CORE_ENABLE_TRANSIT" == "true" ]]; then
{ gen_header
cat <<EOF

18
tools/config-export.py
View File

@ -116,10 +116,24 @@ def main(argv: list[str]) -> int:
# --- Secrets ---
secrets = cfg.get("secrets", {})
for key in ("identity_token", "pantry_vision_token", "transit_token", "mqtt_username",
"mqtt_password", "ha_token", "ssh_authorized_key", "kiosk_password",
"admin_password_hash"):
"mqtt_password", "ha_token", "opnsense_api_key", "opnsense_api_secret",
"ssh_authorized_key", "kiosk_password", "admin_password_hash"):
emit(f"CORE_{key.upper()}", secrets.get(key, ""))
# --- The existing OPNsense firewall (digest-engine's network digest reads its IDS) ---
# Emitted as scalars rather than as a JSON blob because the ISO builder writes them
# into IDSconf.json field by field, the same way it writes every generated env file:
# one place holds the values, and nothing downstream re-parses a nested structure.
opnsense = cfg.get("opnsense", {}) or {}
emit("CORE_OPNSENSE_BASE_URL", str(opnsense.get("base_url", "")).rstrip("/"))
emit("CORE_OPNSENSE_VERIFY_TLS", opnsense.get("verify_tls", True))
# Space-separated for shell iteration; the builder turns it back into a JSON array.
emit("CORE_OPNSENSE_INTERFACES", " ".join(str(i) for i in opnsense.get("interfaces", []) or []))
emit("CORE_OPNSENSE_MAX_ALERTS_SCANNED", opnsense.get("max_alerts_scanned", 5000))
emit("CORE_OPNSENSE_TOP_SIGNATURES", opnsense.get("top_signatures", 8))
emit("CORE_OPNSENSE_TOP_HOSTS", opnsense.get("top_hosts", 5))
emit("CORE_OPNSENSE_PACKET_CAPTURE_REFERENCE", opnsense.get("packet_capture_reference", ""))
# --- Enable flags ---
for flag, value in (cfg.get("container_host", {}).get("enable", {}) or {}).items():
if not flag.startswith("_"):

5
tools/generate-tokens.py
View File

@ -37,6 +37,11 @@ GENERATABLE = {
NOT_GENERATABLE = {
"ha_token": "a Long-Lived Access Token from Home Assistant's own UI (profile -> Security). "
"It doesn't exist until HA is running and an account exists, and only HA can mint it.",
"opnsense_api_key": "issued by OPNsense itself (System -> Access -> Users -> API keys), which "
"shows the key and secret once and downloads them as a .txt. Give that user "
"only the 'Services: Intrusion Detection' privilege.",
"opnsense_api_secret": "the other half of the OPNsense API key above — they are minted together "
"and only OPNsense has them.",
"mqtt_password": "must match what Mosquitto is configured to accept. Generating one here "
"would just mean nothing can connect.",
"admin_password_hash": "a crypt(3) hash for the installer — generate with: mkpasswd -m sha-512",

12
tools/setup-container-host.sh
View File

@ -408,6 +408,18 @@ if [[ "$ENABLE_DIGEST_ENGINE" == "true" ]]; then
cp "$DIGEST_ENGINE_SRC/digest-engine.env.example" "$BASE_DIR/digest/digest-engine.env"
chmod 600 "$BASE_DIR/digest/digest-engine.env"
echo " Seeded $BASE_DIR/digest/digest-engine.env from the template — fill in real values."
echo " IDENTITY_TOKEN there is the same shared secret chores.env uses; without it the"
echo " run can't read who wants which digest and generates all four sections."
fi
# The OPNsense IDS config. An image build writes this from CoreSystemConfig.json's
# opnsense block; this branch is the hand-install path, where the template is the
# only thing there is. Never overwritten — a real config outranks a template.
if [[ ! -f "$BASE_DIR/digest/data/IDSconf.json" && -f "$DIGEST_ENGINE_SRC/IDSconf.json.example" ]]; then
cp "$DIGEST_ENGINE_SRC/IDSconf.json.example" "$BASE_DIR/digest/data/IDSconf.json"
chmod 600 "$BASE_DIR/digest/data/IDSconf.json"
echo " Seeded $BASE_DIR/digest/data/IDSconf.json from the template — fill in base_url,"
echo " api_key and api_secret (OPNsense: System -> Access -> Users -> API keys) and set"
echo " ENABLE_OPNSENSE_IDS_INGEST=true if you want the network digest."
fi
fi
if [[ "$ENABLE_ADMIN_CANVAS" == "true" ]]; then

17
tools/validate-config.py
View File

@ -261,6 +261,23 @@ def validate_secrets(cfg: dict, rep: Report) -> None:
"empty — every kiosk will connect to Mosquitto anonymously. Fine while "
"allow_anonymous is on; revisit before that changes")
# OPNsense's API key pair. Half-configured is the failure worth catching: an
# address with no credentials (or credentials with no address) looks configured
# and produces a network digest that silently reports nothing.
opnsense_url = str(_get(cfg, "opnsense.base_url", "") or "").strip()
opnsense_key = str(_get(cfg, "secrets.opnsense_api_key", "") or "").strip()
opnsense_secret = str(_get(cfg, "secrets.opnsense_api_secret", "") or "").strip()
if opnsense_url and not (opnsense_key and opnsense_secret):
rep.error("secrets.opnsense_api_key",
"opnsense.base_url is set, so both opnsense_api_key and "
"opnsense_api_secret are needed. OPNsense mints them: System -> Access "
"-> Users -> API keys; give that user only the 'Services: Intrusion "
"Detection' privilege")
if (opnsense_key or opnsense_secret) and not opnsense_url:
rep.warn("opnsense.base_url",
"empty, but an OPNsense API key is set — nothing will use it until this "
"points at the firewall (e.g. https://opnsense.home.lan)")
if not (_get(cfg, "secrets.ha_token", "") or ""):
rep.warn("secrets.ha_token",
"empty — identity's /register and /presence can't reach Home Assistant until "