NewsAtlas/README.md

# NewsAtlas

A self-hosted, nginx-fronted "news globe": pulls RSS from a deliberately
ideologically-mixed set of outlets, localizes each story onto a 3D globe,
merges same-place/same-topic stories into a single expandable point, tracks
major stock indices and oil prices over time, and overlays weather and
(where available) conflict-event data. Any place name or article text can be
looked up on Wikipedia, and country panels show that country's parliament
composition diagram when Wikipedia has one.

## Layout

- **Left rail** — latest news, newest first, auto-refreshing. A 📍 button on
  any geocoded story flies the globe to it.
- **Right rail** — top: the 10 highest-24h-volume Polymarket prediction
  markets (title, leading outcome, volume), via their public Gamma REST API
  (`gamma-api.polymarket.com`, no key/auth — the same endpoint their own
  site's frontend calls; their site itself renders market data client-side,
  so there's nothing to scrape from the HTML). Below that: the index/oil
  ticker with a 7-day sparkline per instrument, plus a ❄ FROZEN badge on
  any instrument with a known-stale feed (MOEX) or a ⚠ ERROR badge if its
  last poll genuinely failed. Click an instrument to expand its
  recent-spikes-with-candidate-articles preview (see "Economic Incident
  History" below for the full log).
- **Center** — the globe. News clusters render as newspaper icons (tinted
  violet→pink by corroborating-source count); conflict events render as
  magenta helmet icons plus a pulsing ring. Clicking a news icon opens a
  modal with its stories, a Wikipedia summary of the place, and (if
  available) its country's parliament composition diagram. Text selected
  anywhere can be searched on Wikipedia via the popup that appears.
- **Top-left "i" button** — opens the **Economic Incident History** view: every
  detected market spike, full clock-hour window, with the complete list of
  articles published in it (not just the top few) — a comprehensive log,
  distinct from the quick preview in the right rail. Spikes across
  instruments within an hour of each other are merged into one incident
  (e.g. WTI and Brent crude spiking together shows as one entry, not two —
  a timestamp heuristic, not evidence the moves actually share a cause).
  Each incident also shows its **likely factors**: the words that recur
  most across every headline published in that window, stopwords filtered
  out — plain word-frequency counting, no AI/LLM involved (`textutil.py`).
  Read it as "worth checking these articles," not an explanation.
  A **"Show comparison charts"** button per incident (lazy-loaded) plots two
  overlaid line charts, each instrument normalized to % change from the
  window's start so wildly different price scales/currencies are
  comparable: one with every tracked instrument, one with just the
  incident's own — useful for checking whether a move was isolated to a
  couple of correlated instruments or part of a broader swing, and for
  spotting counter-reactions (something moving the opposite way at the same
  time). Hover for a hairline crosshair with a per-instrument reading at
  that moment.
- **Bottom drawer** — collapsible conflict/military-event log (ACLED-backed;
  see below).
- **Flights toggle** — live global air traffic (OpenSky Network) as airplane
  icons: magenta for flagged-military aircraft, the same dark violet as the
  globe surface for everything else. A "Military only" checkbox hides
  civilian traffic. Hover an aircraft for callsign/country/altitude/speed.

Globe points use **zoom-adaptive clustering**: the backend groups articles
onto a fixed ~11km grid, then the frontend progressively folds nearby grid
cells into a single point as you zoom out (`regroupForAltitude` in
`frontend/js/app.js`), so a continent-level view doesn't leave dozens of
separate dots. The fold radius grows with camera altitude, capped at
subcontinent scale — tune it via `mergeRadiusDeg()` if you want tighter or
looser regional grouping.

Color theme is the user's own "CyberQueer" palette (near-black base, hot
pink `#E40046` + electric violet `#5018DD` accents), including a tinted
globe material — see `frontend/css/style.css` `:root` variables to swap it.

National borders render as a bright violet stroke over a transparent fill
(Natural Earth admin-0 polygons, loaded from `three-globe`'s own npm
package via unpkg — no key or backend endpoint needed).

Each market instrument in the right rail shows a 7-day sparkline
(line + area, hover for a crosshair/tooltip) built from
`/api/markets/history`. Article listings (news feed, cluster modal, spike
candidate articles) show a small favicon next to each headline, resolved
from the article's own URL — decorative only, not a claim of any
publisher's official branding.

## Stack

- **backend/** — FastAPI + SQLite + APScheduler. Polls RSS feeds, geocodes
  articles against a curated gazetteer, clusters them, polls Yahoo Finance
  for markets/oil, optionally polls ACLED for conflict events, and proxies
  Wikipedia + OpenWeatherMap so API keys never reach the browser.
- **frontend/** — static HTML/CSS/JS using [globe.gl](https://globe.gl)
  (three.js) for the 3D globe. No build step.
- **nginx/** — reverse proxy (`/api/*` → backend) + static file server, with
  a short-lived cache in front of the API so many browser tabs don't hammer
  SQLite.

## Running it

```bash
cp .env.example .env   # fill in optional API keys, see below
docker compose up --build
```

Then open http://localhost:8080 (or whatever `HTTP_PORT` you set).

On first boot the globe will be empty for a few seconds until the first RSS
poll completes — hit "Refresh now" if you don't want to wait for the
10-minute interval.

The Economic Incident History, on the other hand, doesn't start empty: on
first startup the backend backfills a week of hourly price history per
instrument (`backfill_market_history()` in `markets.py`) and runs spike
detection retroactively across it, so there's already a populated week of
incidents rather than a week-long wait for live polling to accumulate
enough data. It's a one-time thing — skipped on later restarts once an
instrument has more than a day of real history. Backfilled incidents will
generally have few or no candidate articles, since RSS feeds only carry
recent items and there's no way to backfill a week of news to match a week
of backfilled prices — the price/incident data itself is real, the article
correlation just can't reach that far back.

## Optional API keys (`.env`)

Everything works with `.env` left blank except that the weather and conflict
overlays disable themselves. Nothing else requires a key.

| Var | What it unlocks | Get one |
|---|---|---|
| `OWM_API_KEY` | Weather overlay (clouds/precipitation/temp/wind/pressure) | free, [openweathermap.org](https://home.openweathermap.org/users/sign_up) |
| `ACLED_API_KEY` + `ACLED_EMAIL` | Conflict/"military movement" event overlay | free for registered (incl. non-commercial/research) use, [acleddata.com](https://acleddata.com/register/) |
| `OPENSKY_USERNAME` + `OPENSKY_PASSWORD` | Higher-frequency flights overlay polling | optional, free — anonymous access already works, see below |

**Why ACLED for "military movements":** there is no single free, public,
real-time feed of military movements. ACLED is the closest widely-used open
dataset of sourced, georeferenced conflict/political-violence events and is
what the conflict overlay is built on. Treat it as "reported conflict
events," not live troop tracking.

**Flights overlay** ("Flights" toggle) plots live airborne traffic from the
[OpenSky Network](https://opensky-network.org) REST API — works anonymously
out of the box. Anonymous access is capped at 400 credits/day, so the
default poll interval (`FLIGHTS_POLL_SECONDS=300`) is conservative; register
for free and set `OPENSKY_USERNAME`/`OPENSKY_PASSWORD` for a 4000/day cap and
you can safely poll more often. Aircraft are colored gray (civilian) or red
(flagged military) — see the heuristic caveat below before reading too much
into the red dots.

## What's a heuristic, not ground truth

- **Geocoding** (`backend/app/geocode.py`, `data/gazetteer.csv`) is keyword
  matching against a curated ~180-place gazetteer, not full NLP/NER. It will
  miss unlisted places and can mismatch generic names. Add rows to the CSV
  to extend coverage.
- **Clustering** groups articles whose matched location falls in the same
  ~11km grid cell within the last `ARTICLE_WINDOW_HOURS` (default 72h). The
  "topic" label shown per cluster is just the most frequent significant
  words across its headlines, not topic modeling.
- **Weather overlay** stitches a handful of OpenWeatherMap Web-Mercator
  tiles into one texture and maps it onto the globe's equirectangular UVs.
  It's visually indicative (clouds/precip patterns are recognizable) but not
  pixel-accurate, especially near the poles.
- **Parliament diagrams** (`backend/app/wikipedia.py`,
  `data/parliaments.yaml`) reuse the infobox image of each country's
  legislature article, which is usually — not always — the semicircle/
  hemicycle composition chart. Countries without a curated mapping are
  resolved by a live Wikipedia search at request time, so accuracy varies;
  fix a bad match by adding an override to `parliaments.yaml`.
- **MOEX (Russia) index** data from Yahoo Finance appears frozen at mid-2022
  values — Western data providers largely cut off live Russian market feeds
  after sanctions. It's shown for continuity, not as a live price.
- **RSS feed URLs** (`backend/app/sources.yaml`) belong to third parties and
  can change without notice — if a source goes quiet, check its site for a
  current feed URL and update the file (no rebuild needed, it's a read-only
  mount).
- **Military flight classification** (`backend/app/flights.py`,
  `data/military_ranges.yaml`) is a heuristic, not an authoritative flag —
  there's no public "is this military" field on live ADS-B data. It combines
  an ICAO24 hex-address prefix (only the US block is backed by a citable
  source: OpenSky Network's own 2017 research paper) and a couple of
  well-known military callsign prefixes (`RCH`, `NATO`). This reliably
  catches US/NATO transport and tanker traffic — the bulk of what's actually
  visible on public ADS-B — and under-flags everyone else. Extend
  `military_ranges.yaml` if you can source ranges/callsigns for other air
  forces with similar confidence; don't guess, a wrong range misclassifies
  real civilian flights.

## API surface (consumed by the frontend, but usable standalone)

- `GET /api/clusters` — merged globe points
- `GET /api/clusters/{cluster_key}/articles` — articles behind one point
- `GET /api/articles?q=&limit=` — raw article search
- `POST /api/refresh` — force an immediate RSS poll
- `GET /api/markets/latest`, `GET /api/markets/history?symbol=^GSPC` (accepts
  either `hours=` or an explicit `start=&end=` ISO-datetime window — the
  latter is what the incident overlay charts use)
- `GET /api/markets/spikes?symbol=&hours=` — sudden index/oil moves, each
  paired with every article published in a full-clock-hour window around it
  (`MARKET_SPIKE_LOOKBACK_HOURS` sets the minimum lookback; the window then
  snaps outward to whole hours). A move must clear an absolute floor
  (`MARKET_SPIKE_THRESHOLD_PCT`) *and*, once there's enough price history,
  be at least `MARKET_SPIKE_VOLATILITY_MULTIPLIER`× that instrument's own
  recent typical move (`MARKET_SPIKE_HISTORY_DAYS` lookback,
  `MARKET_SPIKE_MIN_SAMPLES` minimum observations before it applies) — so a
  routinely volatile instrument needs a bigger move to register than a
  normally-calm one. Candidate articles are ranked by relevance
  (keyword/country matching) but not filtered by it — timing alone
  qualifies a story. Click a ticker item for a quick preview, or the "i"
  button (top-left) for the full log. Heuristic correlation, not a verified
  causal link.
- `GET /api/markets/incidents?symbol=&hours=` — the same spikes merged
  across instruments within `INCIDENT_MERGE_HOURS` (1h, `markets.py`) of
  each other, each with a `top_keywords` word-frequency list. Backs the
  Economic Incident History view.
- `GET /api/conflict-events?hours=168`
- `GET /api/flights` — latest global OpenSky snapshot, with `is_military`
  per aircraft (see the heuristic caveat above)
- `GET /api/polymarket` — top 10 trending prediction markets by 24h volume
  (Polymarket's public Gamma API, no key needed)
- `GET /api/weather/tiles/{layer}/{z}/{x}/{y}.png` — OWM tile proxy
- `GET /api/wikipedia/summary?title=`, `GET /api/wikipedia/search?q=`,
  `GET /api/wikipedia/parliament?country=`

## Extending

- Add/edit RSS sources: `backend/app/sources.yaml`.
- Add gazetteer locations: `backend/app/data/gazetteer.csv`.
- Add/fix parliament page mappings: `backend/app/data/parliaments.yaml`.
- Add/fix military ICAO24/callsign ranges: `backend/app/data/military_ranges.yaml`.
- Extend stopwords for keyword extraction (cluster topics, incident "likely
  factors"): `backend/app/textutil.py`.
- Change poll intervals / clustering window: `.env`.

New model fields need a matching entry in `_COLUMN_MIGRATIONS` in
`backend/app/db.py` — `create_all()` only creates missing tables, not
columns on tables that already exist in a deployed `data/newsatlas.db`.

## Deploying as a Proxmox LXC

See `deploy/lxc/README.md` for a `pct`-based build script that provisions a
Debian/Ubuntu/Alpine LXC container running the full stack, and packages it
as a reusable `.tar.zst` template.