NewsAtlas/README.md

# NewsAtlas

A self-hosted, nginx-fronted "news globe": pulls RSS from a deliberately
ideologically-mixed set of outlets, localizes each story onto a 3D globe,
merges same-place/same-topic stories into a single expandable point, tracks
major stock indices and oil prices over time, and overlays weather and
(where available) conflict-event data. Any place name or article text can be
looked up on Wikipedia, and country panels show that country's parliament
composition diagram when Wikipedia has one.

## Layout

- **Left rail** — latest news, newest first, auto-refreshing. A 📍 button on
  any geocoded story flies the globe to it.
- **Right rail** — index/oil ticker. Click an instrument to expand its
  recent-spikes-with-candidate-articles view (see below).
- **Center** — the globe. Clicking a point opens a modal with its stories,
  a Wikipedia summary of the place, and (if available) its country's
  parliament composition diagram. Text selected anywhere can be searched on
  Wikipedia via the popup that appears.
- **Bottom drawer** — collapsible conflict/military-event log (ACLED-backed;
  see below).

Globe points use **zoom-adaptive clustering**: the backend groups articles
onto a fixed ~11km grid, then the frontend progressively folds nearby grid
cells into a single point as you zoom out (`regroupForAltitude` in
`frontend/js/app.js`), so a continent-level view doesn't leave dozens of
separate dots. The fold radius grows with camera altitude, capped at
subcontinent scale — tune it via `mergeRadiusDeg()` if you want tighter or
looser regional grouping.

Color theme is the user's own "CyberQueer" palette (near-black base, hot
pink `#E40046` + electric violet `#5018DD` accents), including a tinted
globe material — see `frontend/css/style.css` `:root` variables to swap it.

National borders render as a bright violet stroke over a transparent fill
(Natural Earth admin-0 polygons, loaded from `three-globe`'s own npm
package via unpkg — no key or backend endpoint needed).

Each market instrument in the right rail shows a 7-day sparkline
(line + area, hover for a crosshair/tooltip) built from
`/api/markets/history`. Article listings (news feed, cluster modal, spike
candidate articles) show a small favicon next to each headline, resolved
from the article's own URL — decorative only, not a claim of any
publisher's official branding.

## Stack

- **backend/** — FastAPI + SQLite + APScheduler. Polls RSS feeds, geocodes
  articles against a curated gazetteer, clusters them, polls Yahoo Finance
  for markets/oil, optionally polls ACLED for conflict events, and proxies
  Wikipedia + OpenWeatherMap so API keys never reach the browser.
- **frontend/** — static HTML/CSS/JS using [globe.gl](https://globe.gl)
  (three.js) for the 3D globe. No build step.
- **nginx/** — reverse proxy (`/api/*` → backend) + static file server, with
  a short-lived cache in front of the API so many browser tabs don't hammer
  SQLite.

## Running it

```bash
cp .env.example .env   # fill in optional API keys, see below
docker compose up --build
```

Then open http://localhost:8080 (or whatever `HTTP_PORT` you set).

On first boot the globe will be empty for a few seconds until the first RSS
poll completes — hit "Refresh now" if you don't want to wait for the
10-minute interval.

## Optional API keys (`.env`)

Everything works with `.env` left blank except that the weather and conflict
overlays disable themselves. Nothing else requires a key.

| Var | What it unlocks | Get one |
|---|---|---|
| `OWM_API_KEY` | Weather overlay (clouds/precipitation/temp/wind/pressure) | free, [openweathermap.org](https://home.openweathermap.org/users/sign_up) |
| `ACLED_API_KEY` + `ACLED_EMAIL` | Conflict/"military movement" event overlay | free for registered (incl. non-commercial/research) use, [acleddata.com](https://acleddata.com/register/) |

**Why ACLED for "military movements":** there is no single free, public,
real-time feed of military movements. ACLED is the closest widely-used open
dataset of sourced, georeferenced conflict/political-violence events and is
what the conflict overlay is built on. Treat it as "reported conflict
events," not live troop tracking.

## What's a heuristic, not ground truth

- **Geocoding** (`backend/app/geocode.py`, `data/gazetteer.csv`) is keyword
  matching against a curated ~180-place gazetteer, not full NLP/NER. It will
  miss unlisted places and can mismatch generic names. Add rows to the CSV
  to extend coverage.
- **Clustering** groups articles whose matched location falls in the same
  ~11km grid cell within the last `ARTICLE_WINDOW_HOURS` (default 72h). The
  "topic" label shown per cluster is just the most frequent significant
  words across its headlines, not topic modeling.
- **Weather overlay** stitches a handful of OpenWeatherMap Web-Mercator
  tiles into one texture and maps it onto the globe's equirectangular UVs.
  It's visually indicative (clouds/precip patterns are recognizable) but not
  pixel-accurate, especially near the poles.
- **Parliament diagrams** (`backend/app/wikipedia.py`,
  `data/parliaments.yaml`) reuse the infobox image of each country's
  legislature article, which is usually — not always — the semicircle/
  hemicycle composition chart. Countries without a curated mapping are
  resolved by a live Wikipedia search at request time, so accuracy varies;
  fix a bad match by adding an override to `parliaments.yaml`.
- **MOEX (Russia) index** data from Yahoo Finance appears frozen at mid-2022
  values — Western data providers largely cut off live Russian market feeds
  after sanctions. It's shown for continuity, not as a live price.
- **RSS feed URLs** (`backend/app/sources.yaml`) belong to third parties and
  can change without notice — if a source goes quiet, check its site for a
  current feed URL and update the file (no rebuild needed, it's a read-only
  mount).

## API surface (consumed by the frontend, but usable standalone)

- `GET /api/clusters` — merged globe points
- `GET /api/clusters/{cluster_key}/articles` — articles behind one point
- `GET /api/articles?q=&limit=` — raw article search
- `POST /api/refresh` — force an immediate RSS poll
- `GET /api/markets/latest`, `GET /api/markets/history?symbol=^GSPC`
- `GET /api/markets/spikes?symbol=&hours=` — sudden index/oil moves
  (`MARKET_SPIKE_THRESHOLD_PCT`) paired with relevance-ranked articles
  published in the surrounding window (`MARKET_SPIKE_LOOKBACK_HOURS`) — click
  a ticker item in the UI for this. It's a heuristic correlation (keyword +
  country/location matching), not a verified causal link.
- `GET /api/conflict-events?hours=168`
- `GET /api/weather/tiles/{layer}/{z}/{x}/{y}.png` — OWM tile proxy
- `GET /api/wikipedia/summary?title=`, `GET /api/wikipedia/search?q=`,
  `GET /api/wikipedia/parliament?country=`

## Extending

- Add/edit RSS sources: `backend/app/sources.yaml`.
- Add gazetteer locations: `backend/app/data/gazetteer.csv`.
- Add/fix parliament page mappings: `backend/app/data/parliaments.yaml`.
- Change poll intervals / clustering window: `.env`.