The_miro
/
NewsAtlas
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
NewsAtlas/README.md

11 KiB
Raw Blame History

NewsAtlas

A self-hosted, nginx-fronted "news globe": pulls RSS from a deliberately ideologically-mixed set of outlets, localizes each story onto a 3D globe, merges same-place/same-topic stories into a single expandable point, tracks major stock indices and oil prices over time, and overlays weather and (where available) conflict-event data. Any place name or article text can be looked up on Wikipedia, and country panels show that country's parliament composition diagram when Wikipedia has one.

Layout

  • Left rail — latest news, newest first, auto-refreshing. A 📍 button on any geocoded story flies the globe to it.
  • Right rail — index/oil ticker with a 7-day sparkline per instrument. Click one to expand its recent-spikes-with-candidate-articles preview (see "Economic Incident History" below for the full log).
  • Center — the globe. News clusters render as newspaper icons (tinted violet→pink by corroborating-source count); conflict events render as magenta helmet icons plus a pulsing ring. Clicking a news icon opens a modal with its stories, a Wikipedia summary of the place, and (if available) its country's parliament composition diagram. Text selected anywhere can be searched on Wikipedia via the popup that appears.
  • Top-left "i" button — opens the Economic Incident History view: every detected market spike, full clock-hour window, with the complete list of articles published in it (not just the top few) — a comprehensive log, distinct from the quick preview in the right rail. Spikes across instruments within an hour of each other are merged into one incident (e.g. WTI and Brent crude spiking together shows as one entry, not two — a timestamp heuristic, not evidence the moves actually share a cause). Each incident also shows its likely factors: the words that recur most across every headline published in that window, stopwords filtered out — plain word-frequency counting, no AI/LLM involved (textutil.py). Read it as "worth checking these articles," not an explanation.
  • Bottom drawer — collapsible conflict/military-event log (ACLED-backed; see below).
  • Flights toggle — live global air traffic (OpenSky Network) as airplane icons: magenta for flagged-military aircraft, the same dark violet as the globe surface for everything else. A "Military only" checkbox hides civilian traffic. Hover an aircraft for callsign/country/altitude/speed.

Globe points use zoom-adaptive clustering: the backend groups articles onto a fixed ~11km grid, then the frontend progressively folds nearby grid cells into a single point as you zoom out (regroupForAltitude in frontend/js/app.js), so a continent-level view doesn't leave dozens of separate dots. The fold radius grows with camera altitude, capped at subcontinent scale — tune it via mergeRadiusDeg() if you want tighter or looser regional grouping.

Color theme is the user's own "CyberQueer" palette (near-black base, hot pink #E40046 + electric violet #5018DD accents), including a tinted globe material — see frontend/css/style.css :root variables to swap it.

National borders render as a bright violet stroke over a transparent fill (Natural Earth admin-0 polygons, loaded from three-globe's own npm package via unpkg — no key or backend endpoint needed).

Each market instrument in the right rail shows a 7-day sparkline (line + area, hover for a crosshair/tooltip) built from /api/markets/history. Article listings (news feed, cluster modal, spike candidate articles) show a small favicon next to each headline, resolved from the article's own URL — decorative only, not a claim of any publisher's official branding.

Stack

  • backend/ — FastAPI + SQLite + APScheduler. Polls RSS feeds, geocodes articles against a curated gazetteer, clusters them, polls Yahoo Finance for markets/oil, optionally polls ACLED for conflict events, and proxies Wikipedia + OpenWeatherMap so API keys never reach the browser.
  • frontend/ — static HTML/CSS/JS using globe.gl (three.js) for the 3D globe. No build step.
  • nginx/ — reverse proxy (/api/* → backend) + static file server, with a short-lived cache in front of the API so many browser tabs don't hammer SQLite.

Running it

cp .env.example .env   # fill in optional API keys, see below
docker compose up --build

Then open http://localhost:8080 (or whatever HTTP_PORT you set).

On first boot the globe will be empty for a few seconds until the first RSS poll completes — hit "Refresh now" if you don't want to wait for the 10-minute interval.

Optional API keys (.env)

Everything works with .env left blank except that the weather and conflict overlays disable themselves. Nothing else requires a key.

Var What it unlocks Get one
OWM_API_KEY Weather overlay (clouds/precipitation/temp/wind/pressure) free, openweathermap.org
ACLED_API_KEY + ACLED_EMAIL Conflict/"military movement" event overlay free for registered (incl. non-commercial/research) use, acleddata.com
OPENSKY_USERNAME + OPENSKY_PASSWORD Higher-frequency flights overlay polling optional, free — anonymous access already works, see below

Why ACLED for "military movements": there is no single free, public, real-time feed of military movements. ACLED is the closest widely-used open dataset of sourced, georeferenced conflict/political-violence events and is what the conflict overlay is built on. Treat it as "reported conflict events," not live troop tracking.

Flights overlay ("Flights" toggle) plots live airborne traffic from the OpenSky Network REST API — works anonymously out of the box. Anonymous access is capped at 400 credits/day, so the default poll interval (FLIGHTS_POLL_SECONDS=300) is conservative; register for free and set OPENSKY_USERNAME/OPENSKY_PASSWORD for a 4000/day cap and you can safely poll more often. Aircraft are colored gray (civilian) or red (flagged military) — see the heuristic caveat below before reading too much into the red dots.

What's a heuristic, not ground truth

  • Geocoding (backend/app/geocode.py, data/gazetteer.csv) is keyword matching against a curated ~180-place gazetteer, not full NLP/NER. It will miss unlisted places and can mismatch generic names. Add rows to the CSV to extend coverage.
  • Clustering groups articles whose matched location falls in the same ~11km grid cell within the last ARTICLE_WINDOW_HOURS (default 72h). The "topic" label shown per cluster is just the most frequent significant words across its headlines, not topic modeling.
  • Weather overlay stitches a handful of OpenWeatherMap Web-Mercator tiles into one texture and maps it onto the globe's equirectangular UVs. It's visually indicative (clouds/precip patterns are recognizable) but not pixel-accurate, especially near the poles.
  • Parliament diagrams (backend/app/wikipedia.py, data/parliaments.yaml) reuse the infobox image of each country's legislature article, which is usually — not always — the semicircle/ hemicycle composition chart. Countries without a curated mapping are resolved by a live Wikipedia search at request time, so accuracy varies; fix a bad match by adding an override to parliaments.yaml.
  • MOEX (Russia) index data from Yahoo Finance appears frozen at mid-2022 values — Western data providers largely cut off live Russian market feeds after sanctions. It's shown for continuity, not as a live price.
  • RSS feed URLs (backend/app/sources.yaml) belong to third parties and can change without notice — if a source goes quiet, check its site for a current feed URL and update the file (no rebuild needed, it's a read-only mount).
  • Military flight classification (backend/app/flights.py, data/military_ranges.yaml) is a heuristic, not an authoritative flag — there's no public "is this military" field on live ADS-B data. It combines an ICAO24 hex-address prefix (only the US block is backed by a citable source: OpenSky Network's own 2017 research paper) and a couple of well-known military callsign prefixes (RCH, NATO). This reliably catches US/NATO transport and tanker traffic — the bulk of what's actually visible on public ADS-B — and under-flags everyone else. Extend military_ranges.yaml if you can source ranges/callsigns for other air forces with similar confidence; don't guess, a wrong range misclassifies real civilian flights.

API surface (consumed by the frontend, but usable standalone)

  • GET /api/clusters — merged globe points
  • GET /api/clusters/{cluster_key}/articles — articles behind one point
  • GET /api/articles?q=&limit= — raw article search
  • POST /api/refresh — force an immediate RSS poll
  • GET /api/markets/latest, GET /api/markets/history?symbol=^GSPC
  • GET /api/markets/spikes?symbol=&hours= — sudden index/oil moves (MARKET_SPIKE_THRESHOLD_PCT), each paired with every article published in a full-clock-hour window around it (MARKET_SPIKE_LOOKBACK_HOURS sets the minimum lookback; the window then snaps outward to whole hours). Ranked by relevance (keyword/country matching) but not filtered by it — timing alone qualifies a story as a candidate. Click a ticker item for a quick preview, or the "i" button (top-left) for the full log. Heuristic correlation, not a verified causal link.
  • GET /api/markets/incidents?symbol=&hours= — the same spikes merged across instruments within INCIDENT_MERGE_HOURS (1h, markets.py) of each other, each with a top_keywords word-frequency list. Backs the Economic Incident History view.
  • GET /api/conflict-events?hours=168
  • GET /api/flights — latest global OpenSky snapshot, with is_military per aircraft (see the heuristic caveat above)
  • GET /api/weather/tiles/{layer}/{z}/{x}/{y}.png — OWM tile proxy
  • GET /api/wikipedia/summary?title=, GET /api/wikipedia/search?q=, GET /api/wikipedia/parliament?country=

Extending

  • Add/edit RSS sources: backend/app/sources.yaml.
  • Add gazetteer locations: backend/app/data/gazetteer.csv.
  • Add/fix parliament page mappings: backend/app/data/parliaments.yaml.
  • Add/fix military ICAO24/callsign ranges: backend/app/data/military_ranges.yaml.
  • Extend stopwords for keyword extraction (cluster topics, incident "likely factors"): backend/app/textutil.py.
  • Change poll intervals / clustering window: .env.

New model fields need a matching entry in _COLUMN_MIGRATIONS in backend/app/db.py — create_all() only creates missing tables, not columns on tables that already exist in a deployed data/newsatlas.db.

Deploying as a Proxmox LXC

See deploy/lxc/README.md for a pct-based build script that provisions a Debian/Ubuntu/Alpine LXC container running the full stack, and packages it as a reusable .tar.zst template.