NewsAtlas/README.md

# NewsAtlas

A self-hosted, nginx-fronted "news globe": pulls RSS from a deliberately
ideologically-mixed set of outlets, localizes each story onto a 3D globe,
merges same-place/same-topic stories into a single expandable point, tracks
major stock indices and oil prices over time, and overlays weather and
(where available) conflict-event data. Any place name or article text can be
looked up on Wikipedia, and country panels show that country's parliament
composition diagram when Wikipedia has one.

## Layout

- **Left rail** — latest news, newest first, auto-refreshing. A 📍 button on
  any geocoded story flies the globe to it.
- **Right rail** — index/oil ticker with a 7-day sparkline per instrument.
  Click one to expand its recent-spikes-with-candidate-articles preview
  (see "Economic Incident History" below for the full log).
- **Center** — the globe. News clusters render as newspaper icons (tinted
  violet→pink by corroborating-source count); conflict events render as
  magenta helmet icons plus a pulsing ring. Clicking a news icon opens a
  modal with its stories, a Wikipedia summary of the place, and (if
  available) its country's parliament composition diagram. Text selected
  anywhere can be searched on Wikipedia via the popup that appears.
- **Top-left "i" button** — opens the **Economic Incident History** view: every
  detected market spike, full clock-hour window, with the complete list of
  articles published in it (not just the top few) — a comprehensive log,
  distinct from the quick preview in the right rail.
- **Bottom drawer** — collapsible conflict/military-event log (ACLED-backed;
  see below).
- **Flights toggle** — live global air traffic (OpenSky Network) as airplane
  icons: magenta for flagged-military aircraft, the same dark violet as the
  globe surface for everything else. A "Military only" checkbox hides
  civilian traffic. Hover an aircraft for callsign/country/altitude/speed.

Globe points use **zoom-adaptive clustering**: the backend groups articles
onto a fixed ~11km grid, then the frontend progressively folds nearby grid
cells into a single point as you zoom out (`regroupForAltitude` in
`frontend/js/app.js`), so a continent-level view doesn't leave dozens of
separate dots. The fold radius grows with camera altitude, capped at
subcontinent scale — tune it via `mergeRadiusDeg()` if you want tighter or
looser regional grouping.

Color theme is the user's own "CyberQueer" palette (near-black base, hot
pink `#E40046` + electric violet `#5018DD` accents), including a tinted
globe material — see `frontend/css/style.css` `:root` variables to swap it.

National borders render as a bright violet stroke over a transparent fill
(Natural Earth admin-0 polygons, loaded from `three-globe`'s own npm
package via unpkg — no key or backend endpoint needed).

Each market instrument in the right rail shows a 7-day sparkline
(line + area, hover for a crosshair/tooltip) built from
`/api/markets/history`. Article listings (news feed, cluster modal, spike
candidate articles) show a small favicon next to each headline, resolved
from the article's own URL — decorative only, not a claim of any
publisher's official branding.

## Stack

- **backend/** — FastAPI + SQLite + APScheduler. Polls RSS feeds, geocodes
  articles against a curated gazetteer, clusters them, polls Yahoo Finance
  for markets/oil, optionally polls ACLED for conflict events, and proxies
  Wikipedia + OpenWeatherMap so API keys never reach the browser.
- **frontend/** — static HTML/CSS/JS using [globe.gl](https://globe.gl)
  (three.js) for the 3D globe. No build step.
- **nginx/** — reverse proxy (`/api/*` → backend) + static file server, with
  a short-lived cache in front of the API so many browser tabs don't hammer
  SQLite.

## Running it

```bash
cp .env.example .env   # fill in optional API keys, see below
docker compose up --build
```

Then open http://localhost:8080 (or whatever `HTTP_PORT` you set).

On first boot the globe will be empty for a few seconds until the first RSS
poll completes — hit "Refresh now" if you don't want to wait for the
10-minute interval.

## Optional API keys (`.env`)

Everything works with `.env` left blank except that the weather and conflict
overlays disable themselves. Nothing else requires a key.

| Var | What it unlocks | Get one |
|---|---|---|
| `OWM_API_KEY` | Weather overlay (clouds/precipitation/temp/wind/pressure) | free, [openweathermap.org](https://home.openweathermap.org/users/sign_up) |
| `ACLED_API_KEY` + `ACLED_EMAIL` | Conflict/"military movement" event overlay | free for registered (incl. non-commercial/research) use, [acleddata.com](https://acleddata.com/register/) |
| `OPENSKY_USERNAME` + `OPENSKY_PASSWORD` | Higher-frequency flights overlay polling | optional, free — anonymous access already works, see below |

**Why ACLED for "military movements":** there is no single free, public,
real-time feed of military movements. ACLED is the closest widely-used open
dataset of sourced, georeferenced conflict/political-violence events and is
what the conflict overlay is built on. Treat it as "reported conflict
events," not live troop tracking.

**Flights overlay** ("Flights" toggle) plots live airborne traffic from the
[OpenSky Network](https://opensky-network.org) REST API — works anonymously
out of the box. Anonymous access is capped at 400 credits/day, so the
default poll interval (`FLIGHTS_POLL_SECONDS=300`) is conservative; register
for free and set `OPENSKY_USERNAME`/`OPENSKY_PASSWORD` for a 4000/day cap and
you can safely poll more often. Aircraft are colored gray (civilian) or red
(flagged military) — see the heuristic caveat below before reading too much
into the red dots.

## What's a heuristic, not ground truth

- **Geocoding** (`backend/app/geocode.py`, `data/gazetteer.csv`) is keyword
  matching against a curated ~180-place gazetteer, not full NLP/NER. It will
  miss unlisted places and can mismatch generic names. Add rows to the CSV
  to extend coverage.
- **Clustering** groups articles whose matched location falls in the same
  ~11km grid cell within the last `ARTICLE_WINDOW_HOURS` (default 72h). The
  "topic" label shown per cluster is just the most frequent significant
  words across its headlines, not topic modeling.
- **Weather overlay** stitches a handful of OpenWeatherMap Web-Mercator
  tiles into one texture and maps it onto the globe's equirectangular UVs.
  It's visually indicative (clouds/precip patterns are recognizable) but not
  pixel-accurate, especially near the poles.
- **Parliament diagrams** (`backend/app/wikipedia.py`,
  `data/parliaments.yaml`) reuse the infobox image of each country's
  legislature article, which is usually — not always — the semicircle/
  hemicycle composition chart. Countries without a curated mapping are
  resolved by a live Wikipedia search at request time, so accuracy varies;
  fix a bad match by adding an override to `parliaments.yaml`.
- **MOEX (Russia) index** data from Yahoo Finance appears frozen at mid-2022
  values — Western data providers largely cut off live Russian market feeds
  after sanctions. It's shown for continuity, not as a live price.
- **RSS feed URLs** (`backend/app/sources.yaml`) belong to third parties and
  can change without notice — if a source goes quiet, check its site for a
  current feed URL and update the file (no rebuild needed, it's a read-only
  mount).
- **Military flight classification** (`backend/app/flights.py`,
  `data/military_ranges.yaml`) is a heuristic, not an authoritative flag —
  there's no public "is this military" field on live ADS-B data. It combines
  an ICAO24 hex-address prefix (only the US block is backed by a citable
  source: OpenSky Network's own 2017 research paper) and a couple of
  well-known military callsign prefixes (`RCH`, `NATO`). This reliably
  catches US/NATO transport and tanker traffic — the bulk of what's actually
  visible on public ADS-B — and under-flags everyone else. Extend
  `military_ranges.yaml` if you can source ranges/callsigns for other air
  forces with similar confidence; don't guess, a wrong range misclassifies
  real civilian flights.

## API surface (consumed by the frontend, but usable standalone)

- `GET /api/clusters` — merged globe points
- `GET /api/clusters/{cluster_key}/articles` — articles behind one point
- `GET /api/articles?q=&limit=` — raw article search
- `POST /api/refresh` — force an immediate RSS poll
- `GET /api/markets/latest`, `GET /api/markets/history?symbol=^GSPC`
- `GET /api/markets/spikes?symbol=&hours=` — sudden index/oil moves
  (`MARKET_SPIKE_THRESHOLD_PCT`), each paired with every article published in
  a full-clock-hour window around it (`MARKET_SPIKE_LOOKBACK_HOURS` sets the
  minimum lookback; the window then snaps outward to whole hours). Ranked
  by relevance (keyword/country matching) but not filtered by it — timing
  alone qualifies a story as a candidate. Click a ticker item for a quick
  preview, or the "i" button (top-left) for the full log. Heuristic
  correlation, not a verified causal link.
- `GET /api/conflict-events?hours=168`
- `GET /api/flights` — latest global OpenSky snapshot, with `is_military`
  per aircraft (see the heuristic caveat above)
- `GET /api/weather/tiles/{layer}/{z}/{x}/{y}.png` — OWM tile proxy
- `GET /api/wikipedia/summary?title=`, `GET /api/wikipedia/search?q=`,
  `GET /api/wikipedia/parliament?country=`

## Extending

- Add/edit RSS sources: `backend/app/sources.yaml`.
- Add gazetteer locations: `backend/app/data/gazetteer.csv`.
- Add/fix parliament page mappings: `backend/app/data/parliaments.yaml`.
- Add/fix military ICAO24/callsign ranges: `backend/app/data/military_ranges.yaml`.
- Change poll intervals / clustering window: `.env`.

## Deploying as a Proxmox LXC

See `deploy/lxc/README.md` for a `pct`-based build script that provisions a
Debian/Ubuntu/Alpine LXC container running the full stack, and packages it
as a reusable `.tar.zst` template.