Add flights overlay, globe icon sprites, and economic incident history

Flights: live OpenSky global air traffic as airplane-icon sprites, magenta
for heuristically-flagged military aircraft vs. globe-tint violet for
civilian, with a military-only filter and hover tooltips.

Globe rendering: news clusters and conflict events switch from three-globe's
built-in point/ring primitives to a custom Sprite-based objects layer so
they render as tinted newspaper/helmet icons respectively, while keeping
existing click/hover behavior and the conflict pulsing-ring effect.

Economic Incident History: a dedicated view (new top-left "i" button) listing
every detected market spike with a full-clock-hour correlation window and
the complete list of articles published in it, not just a relevance-capped
preview.

Also: LXC/Proxmox deployment scripts (deploy/lxc/) driven by a gitignored
prebuild-config.json, and three additional RSS sources (CNN, Jacobin, Der
Standard).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Um48tTvZDrEgDeweFyhPYC
main
Amir Alexander Abdelbaki 2026-07-20 14:31:21 +02:00
parent d52cb8b12e
commit 8e2c5efdbe
17 changed files with 1153 additions and 41 deletions

View File

@ -17,6 +17,14 @@ OWM_API_KEY=
ACLED_API_KEY=
ACLED_EMAIL=
# --- Flights overlay (OpenSky Network) ---
# Works anonymously out of the box (400 credits/day, hence the conservative
# default poll interval below). Free registration raises that to 4000/day:
# https://opensky-network.org/index.php?option=com_users&view=registration
OPENSKY_USERNAME=
OPENSKY_PASSWORD=
FLIGHTS_POLL_SECONDS=300
# --- Polling intervals (minutes) ---
RSS_POLL_MINUTES=10
MARKET_POLL_MINUTES=15

1
.gitignore vendored
View File

@ -2,3 +2,4 @@
/data/
__pycache__/
*.pyc
/deploy/lxc/prebuild-config.json

View File

@ -12,14 +12,25 @@ composition diagram when Wikipedia has one.
- **Left rail** — latest news, newest first, auto-refreshing. A 📍 button on
any geocoded story flies the globe to it.
- **Right rail** — index/oil ticker. Click an instrument to expand its
recent-spikes-with-candidate-articles view (see below).
- **Center** — the globe. Clicking a point opens a modal with its stories,
a Wikipedia summary of the place, and (if available) its country's
parliament composition diagram. Text selected anywhere can be searched on
Wikipedia via the popup that appears.
- **Right rail** — index/oil ticker with a 7-day sparkline per instrument.
Click one to expand its recent-spikes-with-candidate-articles preview
(see "Economic Incident History" below for the full log).
- **Center** — the globe. News clusters render as newspaper icons (tinted
violet→pink by corroborating-source count); conflict events render as
magenta helmet icons plus a pulsing ring. Clicking a news icon opens a
modal with its stories, a Wikipedia summary of the place, and (if
available) its country's parliament composition diagram. Text selected
anywhere can be searched on Wikipedia via the popup that appears.
- **Top-left "i" button** — opens the **Economic Incident History** view: every
detected market spike, full clock-hour window, with the complete list of
articles published in it (not just the top few) — a comprehensive log,
distinct from the quick preview in the right rail.
- **Bottom drawer** — collapsible conflict/military-event log (ACLED-backed;
see below).
- **Flights toggle** — live global air traffic (OpenSky Network) as airplane
icons: magenta for flagged-military aircraft, the same dark violet as the
globe surface for everything else. A "Military only" checkbox hides
civilian traffic. Hover an aircraft for callsign/country/altitude/speed.
Globe points use **zoom-adaptive clustering**: the backend groups articles
onto a fixed ~11km grid, then the frontend progressively folds nearby grid
@ -78,6 +89,7 @@ overlays disable themselves. Nothing else requires a key.
|---|---|---|
| `OWM_API_KEY` | Weather overlay (clouds/precipitation/temp/wind/pressure) | free, [openweathermap.org](https://home.openweathermap.org/users/sign_up) |
| `ACLED_API_KEY` + `ACLED_EMAIL` | Conflict/"military movement" event overlay | free for registered (incl. non-commercial/research) use, [acleddata.com](https://acleddata.com/register/) |
| `OPENSKY_USERNAME` + `OPENSKY_PASSWORD` | Higher-frequency flights overlay polling | optional, free — anonymous access already works, see below |
**Why ACLED for "military movements":** there is no single free, public,
real-time feed of military movements. ACLED is the closest widely-used open
@ -85,6 +97,15 @@ dataset of sourced, georeferenced conflict/political-violence events and is
what the conflict overlay is built on. Treat it as "reported conflict
events," not live troop tracking.
**Flights overlay** ("Flights" toggle) plots live airborne traffic from the
[OpenSky Network](https://opensky-network.org) REST API — works anonymously
out of the box. Anonymous access is capped at 400 credits/day, so the
default poll interval (`FLIGHTS_POLL_SECONDS=300`) is conservative; register
for free and set `OPENSKY_USERNAME`/`OPENSKY_PASSWORD` for a 4000/day cap and
you can safely poll more often. Aircraft are colored gray (civilian) or red
(flagged military) — see the heuristic caveat below before reading too much
into the red dots.
## What's a heuristic, not ground truth
- **Geocoding** (`backend/app/geocode.py`, `data/gazetteer.csv`) is keyword
@ -112,6 +133,17 @@ events," not live troop tracking.
can change without notice — if a source goes quiet, check its site for a
current feed URL and update the file (no rebuild needed, it's a read-only
mount).
- **Military flight classification** (`backend/app/flights.py`,
`data/military_ranges.yaml`) is a heuristic, not an authoritative flag —
there's no public "is this military" field on live ADS-B data. It combines
an ICAO24 hex-address prefix (only the US block is backed by a citable
source: OpenSky Network's own 2017 research paper) and a couple of
well-known military callsign prefixes (`RCH`, `NATO`). This reliably
catches US/NATO transport and tanker traffic — the bulk of what's actually
visible on public ADS-B — and under-flags everyone else. Extend
`military_ranges.yaml` if you can source ranges/callsigns for other air
forces with similar confidence; don't guess, a wrong range misclassifies
real civilian flights.
## API surface (consumed by the frontend, but usable standalone)
@ -121,11 +153,16 @@ events," not live troop tracking.
- `POST /api/refresh` — force an immediate RSS poll
- `GET /api/markets/latest`, `GET /api/markets/history?symbol=^GSPC`
- `GET /api/markets/spikes?symbol=&hours=` — sudden index/oil moves
(`MARKET_SPIKE_THRESHOLD_PCT`) paired with relevance-ranked articles
published in the surrounding window (`MARKET_SPIKE_LOOKBACK_HOURS`) — click
a ticker item in the UI for this. It's a heuristic correlation (keyword +
country/location matching), not a verified causal link.
(`MARKET_SPIKE_THRESHOLD_PCT`), each paired with every article published in
a full-clock-hour window around it (`MARKET_SPIKE_LOOKBACK_HOURS` sets the
minimum lookback; the window then snaps outward to whole hours). Ranked
by relevance (keyword/country matching) but not filtered by it — timing
alone qualifies a story as a candidate. Click a ticker item for a quick
preview, or the "i" button (top-left) for the full log. Heuristic
correlation, not a verified causal link.
- `GET /api/conflict-events?hours=168`
- `GET /api/flights` — latest global OpenSky snapshot, with `is_military`
per aircraft (see the heuristic caveat above)
- `GET /api/weather/tiles/{layer}/{z}/{x}/{y}.png` — OWM tile proxy
- `GET /api/wikipedia/summary?title=`, `GET /api/wikipedia/search?q=`,
`GET /api/wikipedia/parliament?country=`
@ -135,4 +172,11 @@ events," not live troop tracking.
- Add/edit RSS sources: `backend/app/sources.yaml`.
- Add gazetteer locations: `backend/app/data/gazetteer.csv`.
- Add/fix parliament page mappings: `backend/app/data/parliaments.yaml`.
- Add/fix military ICAO24/callsign ranges: `backend/app/data/military_ranges.yaml`.
- Change poll intervals / clustering window: `.env`.
## Deploying as a Proxmox LXC
See `deploy/lxc/README.md` for a `pct`-based build script that provisions a
Debian/Ubuntu/Alpine LXC container running the full stack, and packages it
as a reusable `.tar.zst` template.

View File

@ -28,6 +28,17 @@ class Settings(BaseSettings):
sources_file: str = str(APP_DIR / "sources.yaml")
gazetteer_file: str = str(APP_DIR / "data" / "gazetteer.csv")
# --- Flights overlay (OpenSky Network) ---
# Anonymous access works out of the box but is capped at 400 credits/day;
# a full-globe states/all poll costs several credits, so the default
# interval below is deliberately conservative. Free registration
# (opensky-network.org) raises the cap to 4000/day — set these and drop
# FLIGHTS_POLL_SECONDS if you want closer-to-live updates.
opensky_username: str = ""
opensky_password: str = ""
flights_poll_seconds: int = 300
military_ranges_file: str = str(APP_DIR / "data" / "military_ranges.yaml")
class Config:
env_file = ".env"

View File

@ -0,0 +1,32 @@
# Best-effort military-aircraft classification for the flights overlay.
#
# There is no single public, authoritative "is this plane military" flag on
# live ADS-B data. This combines two heuristics:
#
# 1. icao24_prefixes — hex-address blocks that air forces reserve within
# their country's ICAO24 allocation. Only the US block below is backed
# by a citable source (OpenSky Network's own published research,
# "Mode S and ADS-B Usage of Military and other State Aircraft", 2017:
# "identifiers used by the US Air Force tend to begin with 'AE'"). Add
# more countries' ranges here if you can source them with similar
# confidence — do not guess, a wrong range misclassifies real civilian
# traffic.
# 2. callsign_prefixes — well-known, publicly documented military radio
# callsigns. "RCH" (US Air Mobility Command, "Reach") is the big one
# for ADS-B-visible traffic; "NATO" covers NATO AWACS/transport flights.
#
# Net effect: this overlay reliably flags US military and NATO transport/
# tanker traffic (the bulk of what's actually visible on public ADS-B — most
# other countries' military aircraft either don't broadcast ADS-B or use
# civilian-style codes) and will under-flag everyone else. Extend the lists
# below as you find sourced ranges/callsigns for other air forces.
icao24_prefixes:
- prefix: "ae"
label: "US military (DoD)"
callsign_prefixes:
- prefix: "RCH"
label: "US Air Mobility Command (Reach)"
- prefix: "NATO"
label: "NATO"

92
backend/app/flights.py Normal file
View File

@ -0,0 +1,92 @@
"""Live flight overlay, backed by the OpenSky Network REST API.
Positions are high-churn and ephemeral unlike articles/markets/conflict
events we don't persist history to SQLite, just keep the latest global
snapshot in memory and let the scheduler refresh it.
Military classification is a best-effort heuristic (see
data/military_ranges.yaml for sourcing and caveats) there is no public
"is this military" flag on live ADS-B data.
"""
import logging
from functools import lru_cache
import httpx
import yaml
from .config import settings
log = logging.getLogger("newsatlas.flights")
STATES_URL = "https://opensky-network.org/api/states/all"
_latest_flights: list[dict] = []
@lru_cache(maxsize=1)
def _military_rules() -> tuple[list[str], list[str]]:
with open(settings.military_ranges_file, encoding="utf-8") as f:
data = yaml.safe_load(f) or {}
icao_prefixes = [r["prefix"].lower() for r in data.get("icao24_prefixes", [])]
callsign_prefixes = [r["prefix"].upper() for r in data.get("callsign_prefixes", [])]
return icao_prefixes, callsign_prefixes
def _is_military(icao24: str, callsign: str) -> bool:
icao_prefixes, callsign_prefixes = _military_rules()
icao24 = (icao24 or "").lower()
callsign = (callsign or "").strip().upper()
if any(icao24.startswith(p) for p in icao_prefixes):
return True
if any(callsign.startswith(p) for p in callsign_prefixes):
return True
return False
def poll_flights() -> int:
global _latest_flights
auth = None
if settings.opensky_username and settings.opensky_password:
auth = (settings.opensky_username, settings.opensky_password)
try:
resp = httpx.get(STATES_URL, auth=auth, timeout=30)
resp.raise_for_status()
payload = resp.json()
except Exception:
log.exception("Failed to fetch OpenSky states")
return 0
flights = []
for s in payload.get("states") or []:
icao24, callsign, origin_country = s[0], s[1], s[2]
lon, lat, baro_altitude, on_ground = s[5], s[6], s[7], s[8]
velocity, heading, vertical_rate = s[9], s[10], s[11]
if lat is None or lon is None or on_ground:
continue
callsign = (callsign or "").strip()
flights.append(
{
"icao24": icao24,
"callsign": callsign,
"origin_country": origin_country,
"lat": lat,
"lon": lon,
"altitude_m": baro_altitude,
"velocity_ms": velocity,
"heading": heading,
"vertical_rate_ms": vertical_rate,
"is_military": _is_military(icao24, callsign),
}
)
_latest_flights = flights
log.info("Flight poll complete: %d airborne aircraft (%d flagged military)", len(flights), sum(f["is_military"] for f in flights))
return len(flights)
def get_latest_flights() -> list[dict]:
return _latest_flights

View File

@ -13,6 +13,7 @@ from . import wikipedia
from .clustering import build_clusters
from .config import settings
from .db import get_session, init_db, SessionLocal
from .flights import get_latest_flights
from .ingest import fetch_all
from .markets import poll_markets, INSTRUMENTS
from .conflict import enabled as conflict_enabled, poll_conflict_events
@ -56,9 +57,17 @@ def public_config():
"weather_enabled": bool(settings.owm_api_key),
"conflict_enabled": conflict_enabled(),
"article_window_hours": settings.article_window_hours,
"flights_poll_seconds": settings.flights_poll_seconds,
}
# ---- Flights ----------------------------------------------------------
@app.get("/api/flights")
def api_flights():
return get_latest_flights()
# ---- Articles / clusters -----------------------------------------------
@app.get("/api/clusters")
@ -174,7 +183,7 @@ def api_markets_history(symbol: str, hours: int = Query(168, le=24 * 30)):
@app.get("/api/markets/spikes")
def api_markets_spikes(symbol: str | None = None, hours: int = Query(168, le=24 * 30)):
def api_markets_spikes(symbol: str | None = None, hours: int = Query(168, le=24 * 365)):
since = dt.datetime.utcnow() - dt.timedelta(hours=hours)
session = next(get_session())
try:

View File

@ -100,7 +100,7 @@ def _score_article(article: Article, keywords: list[str], country: str | None) -
return score
def _find_candidate_articles(session: Session, symbol: str, window_start: dt.datetime, window_end: dt.datetime, limit: int = 8) -> list[int]:
def _find_candidate_articles(session: Session, symbol: str, window_start: dt.datetime, window_end: dt.datetime, limit: int = 60) -> list[int]:
hints = _SPIKE_HINTS.get(symbol, {"keywords": [], "country": None})
rows = session.execute(
select(Article)
@ -109,12 +109,24 @@ def _find_candidate_articles(session: Session, symbol: str, window_start: dt.dat
.limit(300) # cap the scan; this is a heuristic ranker, not a full-text search index
).scalars().all()
# Every article published in the window is a candidate — timing alone is
# "possibly related" for the purposes of this log, not just keyword
# matches. Score still decides ordering (keyword/country hits surface
# first), it just no longer excludes zero-score articles.
scored = [(_score_article(a, hints["keywords"], hints["country"]), a) for a in rows]
scored = [(s, a) for s, a in scored if s > 0]
scored.sort(key=lambda pair: (pair[0], pair[1].published_at), reverse=True)
return [a.id for _, a in scored[:limit]]
def _floor_hour(t: dt.datetime) -> dt.datetime:
return t.replace(minute=0, second=0, microsecond=0)
def _ceil_hour(t: dt.datetime) -> dt.datetime:
floored = _floor_hour(t)
return floored if floored == t else floored + dt.timedelta(hours=1)
def _detect_and_record_spike(
session: Session, symbol: str, label: str, prev: MarketPrice | None, price: float, recorded_at: dt.datetime
) -> None:
@ -124,8 +136,11 @@ def _detect_and_record_spike(
if abs(pct_change) < settings.market_spike_threshold_pct:
return
window_start = prev.recorded_at - dt.timedelta(hours=settings.market_spike_lookback_hours)
article_ids = _find_candidate_articles(session, symbol, window_start, recorded_at)
# Snapped to whole clock hours so the log reads as clean ranges (e.g.
# "14:00-16:00"), covering at least the full hour the spike landed in.
window_start = _floor_hour(prev.recorded_at - dt.timedelta(hours=settings.market_spike_lookback_hours))
window_end = _ceil_hour(recorded_at)
article_ids = _find_candidate_articles(session, symbol, window_start, window_end)
session.add(
MarketSpike(
@ -135,7 +150,7 @@ def _detect_and_record_spike(
to_price=price,
pct_change=round(pct_change, 3),
window_start=window_start,
window_end=recorded_at,
window_end=window_end,
detected_at=recorded_at,
article_ids_json=json.dumps(article_ids),
)

View File

@ -5,6 +5,7 @@ from apscheduler.schedulers.background import BackgroundScheduler
from .conflict import poll_conflict_events
from .config import settings
from .db import SessionLocal
from .flights import poll_flights
from .ingest import fetch_all
from .markets import poll_markets
@ -38,11 +39,17 @@ def _run_conflict_job() -> None:
session.close()
def _run_flights_job() -> None:
count = poll_flights()
log.info("Flight poll complete: %d aircraft", count)
def start_scheduler() -> BackgroundScheduler:
scheduler = BackgroundScheduler(timezone="UTC")
scheduler.add_job(_run_rss_job, "interval", minutes=settings.rss_poll_minutes, next_run_time=None)
scheduler.add_job(_run_market_job, "interval", minutes=settings.market_poll_minutes, next_run_time=None)
scheduler.add_job(_run_conflict_job, "interval", minutes=settings.conflict_poll_minutes, next_run_time=None)
scheduler.add_job(_run_flights_job, "interval", seconds=settings.flights_poll_seconds, next_run_time=None)
scheduler.start()
# Kick off an immediate first run of each job in the background so the

92
deploy/lxc/README.md Normal file
View File

@ -0,0 +1,92 @@
# NewsAtlas — Proxmox LXC deployment
Builds a Proxmox LXC container running the full NewsAtlas stack (Docker +
docker-compose inside the CT), and packages it as a reusable `.tar.zst`
template. Supports Debian, Ubuntu, and Alpine base images.
> **Not executed against a real Proxmox host while writing this** — there
> wasn't one available in that environment. The docker-compose stack itself
> (`../../docker-compose.yml`) *has* been built and run end-to-end; this LXC
> layer just automates `pct`/`pveam`/`vzdump` calls around it. Run
> `./build.sh <target> --dry-run` first and read the printed commands before
> running for real.
## Security model — read before using
`build.sh` writes real secrets (whatever you put in `prebuild-config.json`)
into `/opt/newsatlas/.env` **inside the container's filesystem**, and that
filesystem gets packaged into the `.tar.zst` template. This was a deliberate
choice — the safer default (inject secrets only into the running container,
never into a distributable template) was offered and explicitly declined in
favor of "clone-and-go" templates that boot with everything already
configured.
The trade-off that comes with that: **every copy of the resulting
`.tar.zst` is a credentials-bearing artifact**, for every distro variant you
build. Concretely:
- Keep template files (`local:vztmpl/...` / your `vzdump` dump directory)
on storage only trusted operators can read — never a public/shared LXC
template store. The whole point of a reusable template is that it gets
cloned, and each clone carries the same baked-in credentials.
- If a copy of the tarball is ever backed up somewhere less trusted, copied
off-host, or shared, **rotate every key in `prebuild-config.json`**
(OpenWeatherMap, ACLED, OpenSky) — there's no way to invalidate just the
copy that leaked.
- `prebuild-config.json` itself is gitignored; don't commit it, don't paste
it into issues/chat.
If you'd rather not carry that risk, don't use this path — run
`docker compose up --build` directly on any host (see the root README) and
manage `.env` there normally; nothing gets baked into a shareable artifact.
## Setup
```bash
cp prebuild-config.json.template prebuild-config.json
$EDITOR prebuild-config.json # fill in secrets, LXC sizing, network, etc.
```
Key fields:
- `app.repo_url` / `app.branch` — where `provision.sh` clones the app from
inside the container. Defaults to this repo's own remote.
- `secrets.*` — same keys as the root `.env.example`, baked in as described
above.
- `lxc.*` — sizing (`cores`, `memory_mb`, `disk_gb`), Proxmox storage IDs
(`storage` for the rootfs, `template_storage` for both the base template
and the packaged output), networking (`bridge`, `vlan`, `ip`, `gateway`),
and either `root_password` or `ssh_public_key_file` for access.
- `distros.*` — the `pveam available` template-name prefix used to resolve
the newest matching template for each distro. Check `pveam available`
on your host if these drift.
## Building
Run as root on the Proxmox VE host:
```bash
./build.sh debian # one distro
./build.sh all # debian + ubuntu + alpine
./build.sh ubuntu --dry-run # print the pct/pveam/vzdump commands, run nothing
./build.sh alpine --no-keep # destroy the scratch CT after packaging (template file only)
./build.sh debian --no-template # deploy and leave the CT running; skip packaging
```
Each run: allocates the next free CTID, creates the CT with `nesting=1`
(required to run Docker inside an LXC), installs Docker via `provision.sh`,
clones the app, pushes the generated `.env`, runs `docker compose up
--build`, waits for `/api/health`, then (unless `--no-template`) stops the
CT and runs `vzdump --mode stop --compress zstd` to produce the template,
restarting the CT afterward (unless `--no-keep`).
## Redeploying from a packaged template
```bash
pct restore <newid> <template_storage>:backup/vzdump-lxc-<ctid>-*.tar.zst --storage <storage>
pct start <newid>
```
The container comes up with the app already running and `.env` already in
place — no further provisioning needed (that's the whole point, and also
exactly why the security note above matters).

248
deploy/lxc/build.sh Executable file
View File

@ -0,0 +1,248 @@
#!/bin/bash
# Builds a NewsAtlas LXC container on a Proxmox VE host: creates the CT,
# installs Docker inside it, deploys the app, bakes secrets/config from
# prebuild-config.json into it, and (by default) packages the result as a
# reusable .tar.zst template via vzdump.
#
# Run this ON the Proxmox VE host, as root. It has NOT been executed against
# a real Proxmox host while developing this (no such host is available in
# that environment) — read it before running, and use --dry-run first.
#
# SECURITY: the packaged template tarball contains the real values from
# prebuild-config.json (API keys, etc.) baked into /opt/newsatlas/.env
# inside its filesystem. That was an explicit choice (see deploy/lxc/README.md)
# over the safer default of injecting secrets at deploy time — treat every
# copy of the resulting .tar.zst as a credentials-bearing artifact: keep it
# on trusted storage only, never upload it to a public template store, and
# rotate the keys if a copy ever leaves your control.
#
# Usage:
# ./build.sh <debian|ubuntu|alpine|all> [--dry-run] [--no-keep] [--no-template]
#
# --dry-run print every pct/pveam/vzdump command instead of running it
# --no-keep destroy the scratch CT after packaging (template only)
# --no-template deploy and leave the CT running; skip vzdump packaging
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
CONFIG_FILE="$SCRIPT_DIR/prebuild-config.json"
PROVISION_SCRIPT="$SCRIPT_DIR/provision.sh"
APP_DIR_IN_CT="/opt/newsatlas"
DRY_RUN=0
KEEP_CT=1
MAKE_TEMPLATE=1
TARGET="${1:-}"
shift || true
for arg in "$@"; do
case "$arg" in
--dry-run) DRY_RUN=1 ;;
--no-keep) KEEP_CT=0 ;;
--no-template) MAKE_TEMPLATE=0 ;;
*) echo "Unknown flag: $arg" >&2; exit 1 ;;
esac
done
run() {
if [ "$DRY_RUN" = "1" ]; then
printf '[dry-run]'
printf ' %q' "$@"
printf '\n'
else
"$@"
fi
}
# ---- preflight --------------------------------------------------------
if [ "$(id -u)" != "0" ]; then
echo "build.sh must run as root on the Proxmox VE host." >&2
exit 1
fi
for bin in pct pveam pvesh jq; do
if ! command -v "$bin" >/dev/null 2>&1; then
echo "Missing required command: $bin (this must run on a Proxmox VE host with jq installed)." >&2
exit 1
fi
done
if [ "$MAKE_TEMPLATE" = "1" ] && ! command -v vzdump >/dev/null 2>&1; then
echo "Missing required command: vzdump (needed to package a template; pass --no-template to skip)." >&2
exit 1
fi
if [ ! -f "$CONFIG_FILE" ]; then
cat >&2 <<EOF
Missing $CONFIG_FILE.
Copy prebuild-config.json.template to prebuild-config.json and fill it in:
cp "$SCRIPT_DIR/prebuild-config.json.template" "$CONFIG_FILE"
EOF
exit 1
fi
case "$TARGET" in
debian|ubuntu|alpine|all) ;;
*)
echo "Usage: $0 <debian|ubuntu|alpine|all> [--dry-run] [--no-keep] [--no-template]" >&2
exit 1
;;
esac
jqc() { jq -r "$1" "$CONFIG_FILE"; }
REPO_URL=$(jqc '.app.repo_url')
BRANCH=$(jqc '.app.branch')
HTTP_PORT=$(jqc '.app.http_port')
HOSTNAME_BASE=$(jqc '.lxc.hostname')
CORES=$(jqc '.lxc.cores')
MEMORY_MB=$(jqc '.lxc.memory_mb')
SWAP_MB=$(jqc '.lxc.swap_mb')
DISK_GB=$(jqc '.lxc.disk_gb')
STORAGE=$(jqc '.lxc.storage')
TEMPLATE_STORAGE=$(jqc '.lxc.template_storage')
BRIDGE=$(jqc '.lxc.bridge')
VLAN=$(jqc '.lxc.vlan')
IP_CFG=$(jqc '.lxc.ip')
GATEWAY=$(jqc '.lxc.gateway')
UNPRIVILEGED=$(jqc '.lxc.unprivileged')
START_ON_BOOT=$(jqc '.lxc.start_on_boot')
ROOT_PASSWORD=$(jqc '.lxc.root_password')
SSH_KEY_FILE=$(jqc '.lxc.ssh_public_key_file')
if [ "$TARGET" = "all" ]; then
DISTROS="debian ubuntu alpine"
else
DISTROS="$TARGET"
fi
# ---- per-distro build ---------------------------------------------------
build_one() {
local distro="$1"
local template_prefix ctid template hostname net0 rootfs features
local unpriv_flag=""
[ "$UNPRIVILEGED" = "true" ] && unpriv_flag="1" || unpriv_flag="0"
template_prefix=$(jqc ".distros.${distro}")
if [ -z "$template_prefix" ] || [ "$template_prefix" = "null" ]; then
echo "No distros.${distro} entry in prebuild-config.json, skipping." >&2
return 1
fi
echo "=== Building NewsAtlas LXC: $distro ==="
echo "-- Resolving template for prefix '$template_prefix'..."
pveam update >/dev/null 2>&1 || true
template=$(pveam available --section system 2>/dev/null | awk '{print $2}' | grep "^${template_prefix}" | sort -V | tail -1)
if [ -z "$template" ]; then
echo "Could not find a template matching '${template_prefix}' via 'pveam available'. Check prebuild-config.json's distros.${distro} value against 'pveam available'." >&2
return 1
fi
echo "-- Using template: $template"
if ! pveam list "$TEMPLATE_STORAGE" 2>/dev/null | grep -q "$template"; then
run pveam download "$TEMPLATE_STORAGE" "$template"
fi
ctid=$(pvesh get /cluster/nextid)
hostname="${HOSTNAME_BASE}-${distro}"
rootfs="${STORAGE}:${DISK_GB}"
features="nesting=1,keyctl=1" # required for Docker-in-LXC
net0="name=eth0,bridge=${BRIDGE},ip=${IP_CFG}"
[ -n "$VLAN" ] && net0="${net0},tag=${VLAN}"
[ "$IP_CFG" != "dhcp" ] && [ -n "$GATEWAY" ] && net0="${net0},gw=${GATEWAY}"
echo "-- Creating CT $ctid ($hostname)..."
create_args=(pct create "$ctid" "${TEMPLATE_STORAGE}:vztmpl/${template}"
--hostname "$hostname"
--cores "$CORES"
--memory "$MEMORY_MB"
--swap "$SWAP_MB"
--rootfs "$rootfs"
--net0 "$net0"
--unprivileged "$unpriv_flag"
--features "$features"
--onboot "$([ "$START_ON_BOOT" = "true" ] && echo 1 || echo 0)"
)
if [ -n "$ROOT_PASSWORD" ]; then
create_args+=(--password "$ROOT_PASSWORD")
fi
if [ -n "$SSH_KEY_FILE" ] && [ -f "$SSH_KEY_FILE" ]; then
create_args+=(--ssh-public-keys "$SSH_KEY_FILE")
fi
run "${create_args[@]}"
run pct start "$ctid"
if [ "$DRY_RUN" != "1" ]; then
echo "-- Waiting for CT $ctid to come up..."
for i in $(seq 1 30); do
pct exec "$ctid" -- true 2>/dev/null && break
sleep 2
[ "$i" = "30" ] && { echo "CT $ctid did not become ready in time." >&2; return 1; }
done
fi
echo "-- Provisioning (installing Docker, cloning repo)..."
run pct push "$ctid" "$PROVISION_SCRIPT" /root/provision.sh --perms 0755
run pct exec "$ctid" -- /root/provision.sh "$REPO_URL" "$BRANCH" "$APP_DIR_IN_CT"
echo "-- Writing .env (baking config from prebuild-config.json)..."
local envfile
envfile=$(mktemp)
{
echo "HTTP_PORT=${HTTP_PORT}"
echo "OWM_API_KEY=$(jqc '.secrets.owm_api_key')"
echo "ACLED_API_KEY=$(jqc '.secrets.acled_api_key')"
echo "ACLED_EMAIL=$(jqc '.secrets.acled_email')"
echo "OPENSKY_USERNAME=$(jqc '.secrets.opensky_username')"
echo "OPENSKY_PASSWORD=$(jqc '.secrets.opensky_password')"
echo "RSS_POLL_MINUTES=$(jqc '.polling.rss_poll_minutes')"
echo "MARKET_POLL_MINUTES=$(jqc '.polling.market_poll_minutes')"
echo "MARKET_SPIKE_THRESHOLD_PCT=$(jqc '.polling.market_spike_threshold_pct')"
echo "MARKET_SPIKE_LOOKBACK_HOURS=$(jqc '.polling.market_spike_lookback_hours')"
echo "CONFLICT_POLL_MINUTES=$(jqc '.polling.conflict_poll_minutes')"
echo "FLIGHTS_POLL_SECONDS=$(jqc '.polling.flights_poll_seconds')"
echo "ARTICLE_WINDOW_HOURS=$(jqc '.polling.article_window_hours')"
echo "DATABASE_PATH=/data/newsatlas.db"
} > "$envfile"
run pct push "$ctid" "$envfile" "${APP_DIR_IN_CT}/.env" --perms 0600
rm -f "$envfile"
echo "-- Starting the stack..."
run pct exec "$ctid" -- bash -c "cd ${APP_DIR_IN_CT} && docker compose up -d --build"
if [ "$DRY_RUN" != "1" ]; then
echo "-- Waiting for the app to answer /api/health..."
for i in $(seq 1 30); do
pct exec "$ctid" -- curl -sf "http://localhost:${HTTP_PORT}/api/health" >/dev/null 2>&1 && break
sleep 3
[ "$i" = "30" ] && echo "Warning: /api/health didn't respond in time; check 'pct exec $ctid -- docker compose -f ${APP_DIR_IN_CT}/docker-compose.yml logs'." >&2
done
fi
if [ "$MAKE_TEMPLATE" = "1" ]; then
echo "-- Stopping CT $ctid to package as a template..."
run pct shutdown "$ctid" --timeout 60
echo "-- Packaging via vzdump..."
run vzdump "$ctid" --mode stop --compress zstd --storage "$TEMPLATE_STORAGE"
if [ "$KEEP_CT" = "1" ]; then
run pct start "$ctid"
echo "-- CT $ctid restarted (kept running in addition to the packaged template)."
else
run pct destroy "$ctid"
echo "-- CT $ctid destroyed after packaging (--no-keep)."
fi
fi
echo "=== Done: $distro (CTID $ctid, hostname $hostname) ==="
echo " Reminder: any packaged .tar.zst for this CT has live secrets baked in — see deploy/lxc/README.md."
}
for d in $DISTROS; do
build_one "$d"
done

View File

@ -0,0 +1,53 @@
{
"_comment": "Copy this file to prebuild-config.json (gitignored) and fill it in. build.sh reads secrets/config from prebuild-config.json only — never from this template.",
"app": {
"repo_url": "https://git.abdelbaki.eu/The_miro/NewsAtlas.git",
"branch": "main",
"http_port": 8080
},
"secrets": {
"_comment": "Written into /opt/newsatlas/.env inside the container (mode 600, root-only) and baked into the packaged template tarball. Anyone with a copy of that tarball has these values — see deploy/lxc/README.md before sharing/backing one up anywhere untrusted.",
"owm_api_key": "",
"acled_api_key": "",
"acled_email": "",
"opensky_username": "",
"opensky_password": ""
},
"polling": {
"rss_poll_minutes": 10,
"market_poll_minutes": 15,
"market_spike_threshold_pct": 1.5,
"market_spike_lookback_hours": 6,
"conflict_poll_minutes": 60,
"flights_poll_seconds": 300,
"article_window_hours": 72
},
"lxc": {
"hostname": "newsatlas",
"cores": 2,
"memory_mb": 2048,
"swap_mb": 512,
"disk_gb": 8,
"storage": "local-lvm",
"template_storage": "local",
"bridge": "vmbr0",
"vlan": "",
"ip": "dhcp",
"gateway": "",
"unprivileged": true,
"start_on_boot": true,
"root_password": "",
"ssh_public_key_file": ""
},
"distros": {
"_comment": "Which base-OS variants build.sh should build when run with --all. Each maps to a `pveam available` template name prefix; build.sh picks the newest matching version.",
"debian": "debian-12-standard",
"ubuntu": "ubuntu-24.04-standard",
"alpine": "alpine-3.19-default"
}
}

48
deploy/lxc/provision.sh Executable file
View File

@ -0,0 +1,48 @@
#!/bin/bash
# Runs INSIDE the LXC container as root (pushed + executed by build.sh).
# Installs Docker and clones NewsAtlas. Deliberately does NOT touch .env or
# start the stack — build.sh pushes the generated .env and starts the stack
# as separate steps afterward, so a plain `git clone` here never collides
# with an already-placed .env file.
set -euo pipefail
REPO_URL="${1:?usage: provision.sh <repo_url> <branch> <app_dir>}"
BRANCH="${2:?usage: provision.sh <repo_url> <branch> <app_dir>}"
APP_DIR="${3:?usage: provision.sh <repo_url> <branch> <app_dir>}"
. /etc/os-release
install_docker_apt() {
export DEBIAN_FRONTEND=noninteractive
apt-get update
apt-get install -y ca-certificates curl gnupg git jq
curl -fsSL https://get.docker.com | sh
systemctl enable docker
systemctl start docker
}
install_docker_apk() {
apk update
apk add --no-cache docker docker-cli-compose git jq openrc
rc-update add docker default
service docker start
}
case "$ID" in
debian|ubuntu) install_docker_apt ;;
alpine) install_docker_apk ;;
*)
echo "provision.sh: unsupported distro '$ID' (expected debian, ubuntu, or alpine)" >&2
exit 1
;;
esac
if [ -d "$APP_DIR/.git" ]; then
git -C "$APP_DIR" fetch origin "$BRANCH"
git -C "$APP_DIR" checkout "$BRANCH"
git -C "$APP_DIR" reset --hard "origin/$BRANCH"
else
git clone --branch "$BRANCH" --depth 1 "$REPO_URL" "$APP_DIR"
fi
echo "provision.sh: done. Docker installed, repo at $APP_DIR."

View File

@ -10,6 +10,7 @@ services:
- ./backend/app/sources.yaml:/app/app/sources.yaml:ro
- ./backend/app/data/gazetteer.csv:/app/app/data/gazetteer.csv:ro
- ./backend/app/data/parliaments.yaml:/app/app/data/parliaments.yaml:ro
- ./backend/app/data/military_ranges.yaml:/app/app/data/military_ranges.yaml:ro
expose:
- "8000"

View File

@ -21,6 +21,7 @@
--c-dark-soft: #9d7cff; /* lightened violet for small text/links on dark bg */
--c-red: #f50505; /* danger / down */
--c-green: #34d399; /* gain — kept distinct from theme for finance semantics */
--c-magenta: #ff33ff; /* flagged-military flights — must match MILITARY_FLIGHT_HEX in app.js */
--c-panel: rgba(24, 15, 33, 0.97);
--c-panel-border: #3a2159;
@ -60,6 +61,8 @@ html, body {
z-index: 30;
}
#topbarLeft { display: flex; align-items: center; }
#topbar h1 {
font-size: 18px;
letter-spacing: 0.08em;
@ -218,7 +221,7 @@ html, body {
.modal { position: absolute; inset: 0; z-index: 40; display: flex; align-items: center; justify-content: center; }
.modal.hidden { display: none; }
#modalBackdrop { position: absolute; inset: 0; background: rgba(10, 0, 20, 0.65); }
.modal-backdrop { position: absolute; inset: 0; background: rgba(10, 0, 20, 0.65); }
.modal-card {
position: relative;
width: 420px;
@ -231,7 +234,8 @@ html, body {
padding: 20px 18px;
box-shadow: 0 12px 40px rgba(0,0,0,0.6), 0 0 60px rgba(80, 24, 221, 0.15);
}
#closeModal {
.modal-card.wide { width: 760px; max-height: 88vh; }
.modal-close {
position: absolute;
top: 8px;
right: 10px;
@ -241,7 +245,37 @@ html, body {
font-size: 22px;
cursor: pointer;
}
#closeModal:hover { color: var(--c-highlight-soft); }
.modal-close:hover { color: var(--c-highlight-soft); }
#infoBtn {
width: 26px;
height: 26px;
border-radius: 50%;
background: var(--c-tag-bg);
border: 1px solid var(--c-panel-border);
color: var(--c-dark-soft);
font-size: 13px;
font-weight: 700;
font-style: italic;
font-family: Georgia, "Times New Roman", serif;
cursor: pointer;
margin-left: 4px;
}
#infoBtn:hover { background: #3d2260; color: #fff; }
.econ-select {
background: #1c1030;
color: var(--c-text);
border: 1px solid var(--c-panel-border);
border-radius: 4px;
font-size: 12px;
padding: 4px 6px;
}
.econ-incident { padding: 14px 0; border-bottom: 1px solid #241533; }
.econ-incident h3 { margin: 0 0 4px; font-size: 14px; color: #f2dede; }
.econ-incident .window { font-size: 11px; color: var(--c-text-muted); margin-bottom: 8px; }
.econ-incident .articles { margin-top: 8px; }
.panel-section { margin-bottom: 20px; }
.panel-section h2 {
@ -271,6 +305,23 @@ html, body {
/* ---- wikipedia text-selection popup ---- */
#flightTooltip {
position: absolute;
z-index: 45;
background: #1c1030;
border: 1px solid var(--c-panel-border);
border-radius: 6px;
padding: 6px 9px;
box-shadow: 0 4px 16px rgba(0,0,0,0.5);
font-size: 11px;
line-height: 1.5;
pointer-events: none;
max-width: 220px;
}
#flightTooltip.hidden { display: none; }
#flightTooltip b { color: var(--c-text); }
#flightTooltip .mil { color: var(--c-magenta); font-weight: 600; }
#wikiSelectPopup {
position: absolute;
z-index: 50;

View File

@ -10,7 +10,10 @@
<div id="globeViz"></div>
<header id="topbar">
<h1>NewsAtlas</h1>
<div id="topbarLeft">
<h1>NewsAtlas</h1>
<button id="infoBtn" title="Economic Incident History">i</button>
</div>
<div id="layerToggles">
<label><input type="checkbox" id="toggleWeather" /> Weather</label>
<select id="weatherLayer">
@ -21,6 +24,9 @@
<option value="pressure">Pressure</option>
</select>
<label><input type="checkbox" id="toggleConflict" /> Conflict overlay</label>
<label><input type="checkbox" id="toggleFlights" /> Flights</label>
<label><input type="checkbox" id="toggleMilitaryOnly" /> Military only</label>
<span id="flightCount" class="subtle"></span>
<span id="lastUpdated"></span>
<button id="refreshBtn" title="Force an immediate RSS poll">Refresh now</button>
</div>
@ -37,14 +43,34 @@
</aside>
<div id="clusterModal" class="modal hidden">
<div id="modalBackdrop"></div>
<div class="modal-backdrop" data-close="clusterModal"></div>
<div class="modal-card">
<button id="closeModal" title="Close">&times;</button>
<button class="modal-close" data-close="clusterModal" title="Close">&times;</button>
<div id="modalContent"></div>
</div>
</div>
<div id="econHistoryView" class="modal hidden">
<div class="modal-backdrop" data-close="econHistoryView"></div>
<div class="modal-card wide">
<button class="modal-close" data-close="econHistoryView" title="Close">&times;</button>
<div class="headline" style="margin-top:0;">Economic Incident History</div>
<p class="subtle">
Sudden index/oil price moves and every story published in that window, newest first.
Ordering favors keyword/country-matched stories, but timing alone qualifies a story as a
candidate here — this is a correlation log, not a verified causal record.
</p>
<div style="margin: 10px 0 16px;">
<select id="econHistorySymbolFilter" class="econ-select">
<option value="">All instruments</option>
</select>
</div>
<div id="econHistoryList"></div>
</div>
</div>
<div id="wikiSelectPopup" class="hidden"></div>
<div id="flightTooltip" class="hidden"></div>
<footer id="militaryDrawer" class="collapsed">
<button id="drawerHandle">

View File

@ -1,5 +1,11 @@
const API = "/api";
// The globe surface's violet tint (see onGlobeReady below) — reused as the
// civilian-flight color so those icons read as "part of the globe" rather
// than a separate alert layer, per the flights-overlay color spec.
const GLOBE_TINT_HEX = 0x6a3fd9;
const MILITARY_FLIGHT_HEX = 0xff33ff;
let config = { weather_enabled: false, conflict_enabled: false };
let clusters = []; // raw, server-side ~11km-grid clusters
let displayedClusters = []; // zoom-adaptive regrouping of `clusters`, currently on screen
@ -15,20 +21,12 @@ const globe = Globe()(document.getElementById("globeViz"))
.globeImageUrl("https://unpkg.com/three-globe/example/img/earth-blue-marble.jpg")
.bumpImageUrl("https://unpkg.com/three-globe/example/img/earth-topology.png")
.backgroundColor("#0f0d16")
.pointLat("lat")
.pointLng("lon")
.pointAltitude(0.012)
.pointRadius(pointRadiusFor)
.pointColor(colorForCluster)
.pointLabel(
(d) =>
`<div style="background:#170b28;border:1px solid #3a2159;padding:6px 8px;border-radius:6px;max-width:220px;font-size:12px;">
<b>${escapeHtml(d.headline)}</b><br/>
${d.article_count} stor${d.article_count === 1 ? "y" : "ies"} · ${d.source_count} source(s)
${d.location_name ? `<br/>${escapeHtml(d.location_name)}` : ""}
</div>`
)
.onPointClick(onClusterClick)
.objectLat("lat")
.objectLng("lon")
.objectAltitude(0.014)
.objectThreeObject(buildGlobeObject)
.objectLabel(objectLabelFor)
.onObjectClick(onGlobeObjectClick)
.onZoom(({ altitude }) => {
currentAltitude = altitude;
scheduleRegroup();
@ -42,6 +40,12 @@ const globe = Globe()(document.getElementById("globeViz"))
.ringPropagationSpeed(3)
.ringRepeatPeriod((d) => d.repeatPeriod);
// Degrees-of-arc -> scene units, so icon sprite sizes can be specified in
// the same "degrees" unit pointRadiusFor/declutter already use, and stay
// visually consistent with them.
const sceneUnitsPerDeg = (2 * Math.PI * globe.getGlobeRadius()) / 360;
const HELMET_ICON_DEG = 1.6;
globe.pointOfView({ lat: 20, lng: 20, altitude: 2.2 }, 0);
// Tint the globe surface (blue-marble texture) into the theme's violet
@ -53,7 +57,7 @@ globe.pointOfView({ lat: 20, lng: 20, altitude: 2.2 }, 0);
// tint applied immediately after construction.
globe.onGlobeReady(() => {
const globeMaterial = globe.globeMaterial();
globeMaterial.color = new THREE.Color(0x6a3fd9);
globeMaterial.color = new THREE.Color(GLOBE_TINT_HEX);
globeMaterial.emissive = new THREE.Color(0x1a0e33);
globeMaterial.emissiveIntensity = 0.25;
});
@ -88,6 +92,106 @@ function colorForCluster(d) {
return palette[n - 1] || palette[0];
}
// ---- globe icon sprites (news clusters + conflict events) --------------
//
// Both are white-on-transparent alpha-mask canvases so a SpriteMaterial's
// `color` can tint them per-datum (news: the violet->pink gradient above;
// conflict: magenta) without needing separate colored art assets — same
// technique as the flight-plane icon further down.
let _newspaperIconTexture = null;
function newspaperIconTexture() {
if (_newspaperIconTexture) return _newspaperIconTexture;
const size = 64;
const canvas = document.createElement("canvas");
canvas.width = size;
canvas.height = size;
const ctx = canvas.getContext("2d");
ctx.fillStyle = "#ffffff";
ctx.beginPath();
ctx.moveTo(8, 8);
ctx.lineTo(46, 8);
ctx.lineTo(58, 20);
ctx.lineTo(58, 58);
ctx.lineTo(8, 58);
ctx.closePath();
ctx.fill();
ctx.globalCompositeOperation = "destination-out"; // cut "text line" gaps into the page
ctx.fillRect(14, 16, 30, 4);
ctx.fillRect(14, 26, 38, 3);
ctx.fillRect(14, 33, 38, 3);
ctx.fillRect(14, 40, 34, 3);
ctx.fillRect(14, 47, 28, 3);
ctx.globalCompositeOperation = "source-over";
_newspaperIconTexture = new THREE.CanvasTexture(canvas);
return _newspaperIconTexture;
}
let _helmetIconTexture = null;
function helmetIconTexture() {
if (_helmetIconTexture) return _helmetIconTexture;
const size = 64;
const canvas = document.createElement("canvas");
canvas.width = size;
canvas.height = size;
const ctx = canvas.getContext("2d");
ctx.fillStyle = "#ffffff";
ctx.beginPath();
ctx.arc(32, 30, 21, Math.PI, 0, false); // dome
ctx.lineTo(53, 36);
ctx.quadraticCurveTo(53, 43, 45, 43); // brim, right
ctx.lineTo(19, 43);
ctx.quadraticCurveTo(11, 43, 11, 36); // brim, left
ctx.closePath();
ctx.fill();
_helmetIconTexture = new THREE.CanvasTexture(canvas);
return _helmetIconTexture;
}
function makeIconSprite(texture, colorHex, scale) {
const material = new THREE.SpriteMaterial({
map: texture,
color: colorHex,
transparent: true,
depthWrite: false,
});
const sprite = new THREE.Sprite(material);
sprite.scale.set(scale, scale, 1);
return sprite;
}
function buildGlobeObject(d) {
if (d.kind === "conflict") {
return makeIconSprite(helmetIconTexture(), MILITARY_FLIGHT_HEX, HELMET_ICON_DEG * sceneUnitsPerDeg);
}
return makeIconSprite(newspaperIconTexture(), colorForCluster(d), pointRadiusFor(d) * sceneUnitsPerDeg * 2.4);
}
function objectLabelFor(d) {
if (d.kind === "conflict") {
return `<div style="background:#170b28;border:1px solid #3a2159;padding:6px 8px;border-radius:6px;max-width:220px;font-size:12px;">
<b>${escapeHtml(d.event_type)}</b><br/>
${escapeHtml(d.actor1)}${d.actor2 ? " vs " + escapeHtml(d.actor2) : ""}<br/>
${escapeHtml(d.location_name)}, ${escapeHtml(d.country)}
${d.fatalities ? `<br/><span style="color:#f50505;">${d.fatalities} fatalities</span>` : ""}
</div>`;
}
return `<div style="background:#170b28;border:1px solid #3a2159;padding:6px 8px;border-radius:6px;max-width:220px;font-size:12px;">
<b>${escapeHtml(d.headline)}</b><br/>
${d.article_count} stor${d.article_count === 1 ? "y" : "ies"} · ${d.source_count} source(s)
${d.location_name ? `<br/>${escapeHtml(d.location_name)}` : ""}
</div>`;
}
function onGlobeObjectClick(d) {
if (d.kind === "conflict") {
flyTo(d.lat, d.lon);
militaryDrawer.classList.remove("collapsed");
return;
}
onClusterClick(d);
}
// pointRadius is in degrees of arc on the globe surface (three-globe's own
// unit for this accessor — not pixels, not a fraction of globe radius), so
// it's directly comparable to lat/lon distances for the decluttering pass
@ -118,10 +222,21 @@ function scheduleRegroup() {
requestAnimationFrame(() => {
regroupScheduled = false;
displayedClusters = regroupForAltitude(clusters, currentAltitude);
globe.pointsData(displayedClusters);
syncGlobeObjects();
});
}
// News clusters and conflict events share one three-globe "objects" layer
// (see buildGlobeObject) so both can be custom icon sprites; this merges
// the two data sources into it, respecting the conflict-overlay toggle.
function syncGlobeObjects() {
const newsObjs = displayedClusters.map((c) => ({ ...c, kind: "news" }));
const conflictObjs = document.getElementById("toggleConflict").checked
? conflictEvents.map((e) => ({ ...e, kind: "conflict" }))
: [];
globe.objectsData([...newsObjs, ...conflictObjs]);
}
function mergeRadiusDeg(altitude) {
// altitude ~0.4 (close, city-level) -> ~0deg (no extra merging beyond the server grid)
// altitude ~2.2 (default view) -> ~13deg (nearby-country scale, e.g. Levant stays
@ -292,8 +407,9 @@ function onClusterClick(cluster) {
const clusterModal = document.getElementById("clusterModal");
const modalContent = document.getElementById("modalContent");
document.getElementById("closeModal").onclick = () => clusterModal.classList.add("hidden");
document.getElementById("modalBackdrop").onclick = () => clusterModal.classList.add("hidden");
document.querySelectorAll("[data-close]").forEach((el) => {
el.onclick = () => document.getElementById(el.dataset.close).classList.add("hidden");
});
async function openClusterModal(cluster) {
clusterModal.classList.remove("hidden");
@ -573,6 +689,7 @@ async function loadConflictEvents() {
.join("")
: `<p class="subtle">No events recorded in the current window.</p>`;
syncRings();
syncGlobeObjects();
} catch (e) {
console.error("Failed to load conflict events", e);
}
@ -763,6 +880,254 @@ async function toggleMarketDetail(symbol, idx) {
}
}
// ----------------------------------------------- economic incident history
const econHistoryView = document.getElementById("econHistoryView");
let econHistorySymbolsLoaded = false;
document.getElementById("infoBtn").onclick = () => {
econHistoryView.classList.remove("hidden");
loadEconHistory();
};
document.getElementById("econHistorySymbolFilter").onchange = () => loadEconHistory();
async function loadEconHistory() {
const listEl = document.getElementById("econHistoryList");
const filterEl = document.getElementById("econHistorySymbolFilter");
const symbol = filterEl.value;
if (!econHistorySymbolsLoaded) {
try {
const rows = await (await fetch(`${API}/markets/latest`)).json();
filterEl.innerHTML =
`<option value="">All instruments</option>` +
rows.map((r) => `<option value="${r.symbol}">${escapeHtml(r.label)}</option>`).join("");
econHistorySymbolsLoaded = true;
} catch (e) {
// non-fatal — the filter just stays at "All instruments"
}
}
listEl.innerHTML = `<p class="subtle">Loading…</p>`;
try {
const url = symbol
? `${API}/markets/spikes?symbol=${encodeURIComponent(symbol)}&hours=8760`
: `${API}/markets/spikes?hours=8760`;
const spikes = await (await fetch(url)).json();
if (!spikes.length) {
listEl.innerHTML = `<p class="subtle">No moves ≥ the spike threshold recorded yet.</p>`;
return;
}
listEl.innerHTML = spikes.map(econIncidentHtml).join("");
} catch (e) {
listEl.innerHTML = `<p class="subtle">Could not load incident history.</p>`;
}
}
function econIncidentHtml(s) {
const dir = s.pct_change > 0 ? "up" : "down";
const arrow = s.pct_change > 0 ? "▲" : "▼";
const fmtHour = (iso) =>
new Date(iso).toLocaleString(undefined, {
month: "short",
day: "numeric",
hour: "2-digit",
minute: "2-digit",
});
const articlesHtml = s.candidate_articles.length
? s.candidate_articles.map(articleItemHtml).join("")
: `<p class="subtle">No stories found published in that window.</p>`;
return `
<div class="econ-incident">
<h3>${escapeHtml(s.label)} <span class="${dir}">${arrow} ${s.pct_change.toFixed(2)}%</span></h3>
<div class="window">
${fmtHour(s.window_start)} ${fmtHour(s.window_end)}
· ${s.from_price.toLocaleString()} ${s.to_price.toLocaleString()}
· ${s.candidate_articles.length} stor${s.candidate_articles.length === 1 ? "y" : "ies"} in window
</div>
<div class="articles">${articlesHtml}</div>
</div>
`;
}
// -------------------------------------------------------------- flights --
//
// Rendered as a single THREE.Points cloud (not three-globe's per-datum
// pointsData/objectsData layers, already used by news clusters) since a
// live global snapshot can be several thousand aircraft — one draw call
// keeps that cheap. The tradeoff: point sprites all face the camera and
// don't support per-instance rotation, so icons aren't heading-aligned.
// Military classification is a best-effort heuristic from the backend (see
// backend/app/data/military_ranges.yaml); color is the only encoding
// (magenta = flagged military, globe-tint purple = everything else) and
// isn't itself a claim of certainty — the hover tooltip states as much.
let flightsPoints = null;
let flightsData = [];
let flightsPollTimer = null;
let flightsPollSeconds = 300;
const flightRaycaster = new THREE.Raycaster();
flightRaycaster.params.Points.threshold = 2.5;
function altitudeForFlight(f) {
const m = f.altitude_m || 0;
return 0.02 + Math.min(1, m / 12000) * 0.03;
}
// A small top-down airplane silhouette, drawn once as a white-on-transparent
// alpha mask so PointsMaterial's vertexColors can tint it per-aircraft
// (military vs civilian) without needing separate red/gray sprite sheets.
let _planeIconTexture = null;
function planeIconTexture() {
if (_planeIconTexture) return _planeIconTexture;
const size = 64;
const canvas = document.createElement("canvas");
canvas.width = size;
canvas.height = size;
const ctx = canvas.getContext("2d");
ctx.fillStyle = "#ffffff";
ctx.beginPath();
ctx.moveTo(32, 3); // nose
ctx.lineTo(36, 24);
ctx.lineTo(61, 39); // right wingtip
ctx.lineTo(61, 45);
ctx.lineTo(37, 38);
ctx.lineTo(35, 50);
ctx.lineTo(47, 59); // right tailtip
ctx.lineTo(47, 63);
ctx.lineTo(32, 57); // tail center
ctx.lineTo(17, 63); // left tailtip
ctx.lineTo(17, 59);
ctx.lineTo(29, 50);
ctx.lineTo(27, 38);
ctx.lineTo(3, 45); // left wingtip
ctx.lineTo(3, 39);
ctx.lineTo(28, 24);
ctx.closePath();
ctx.fill();
_planeIconTexture = new THREE.CanvasTexture(canvas);
return _planeIconTexture;
}
function buildFlightsPoints(flights) {
const geometry = new THREE.BufferGeometry();
const positions = new Float32Array(flights.length * 3);
const colors = new Float32Array(flights.length * 3);
const milColor = new THREE.Color(MILITARY_FLIGHT_HEX);
const civColor = new THREE.Color(GLOBE_TINT_HEX);
flights.forEach((f, i) => {
const { x, y, z } = globe.getCoords(f.lat, f.lon, altitudeForFlight(f));
positions[i * 3] = x;
positions[i * 3 + 1] = y;
positions[i * 3 + 2] = z;
const c = f.is_military ? milColor : civColor;
colors[i * 3] = c.r;
colors[i * 3 + 1] = c.g;
colors[i * 3 + 2] = c.b;
});
geometry.setAttribute("position", new THREE.BufferAttribute(positions, 3));
geometry.setAttribute("color", new THREE.BufferAttribute(colors, 3));
const material = new THREE.PointsMaterial({
size: 4,
map: planeIconTexture(),
alphaTest: 0.3,
vertexColors: true,
sizeAttenuation: true,
transparent: true,
depthWrite: false,
});
return new THREE.Points(geometry, material);
}
async function loadFlights() {
try {
const res = await fetch(`${API}/flights`);
flightsData = await res.json();
renderFlights();
} catch (e) {
console.error("Failed to load flights", e);
}
}
// Rebuilds the points cloud from the last-fetched snapshot, applying the
// "Military only" filter — kept separate from loadFlights so toggling the
// filter is instant and doesn't wait on a network round-trip.
let flightsRendered = []; // the array actually backing the current geometry (post-filter), for hover lookups
function renderFlights() {
if (flightsPoints) {
globe.scene().remove(flightsPoints);
flightsPoints.geometry.dispose();
flightsPoints.material.dispose();
flightsPoints = null;
}
if (!document.getElementById("toggleFlights").checked) return;
const militaryOnly = document.getElementById("toggleMilitaryOnly").checked;
flightsRendered = militaryOnly ? flightsData.filter((f) => f.is_military) : flightsData;
flightsPoints = buildFlightsPoints(flightsRendered);
globe.scene().add(flightsPoints);
const milCount = flightsData.filter((f) => f.is_military).length;
document.getElementById("flightCount").textContent = flightsData.length
? militaryOnly
? `${milCount} military-flagged (of ${flightsData.length})`
: `${flightsData.length} aircraft · ${milCount} military-flagged`
: "no data yet";
}
function setFlightsVisible(visible) {
if (visible) {
loadFlights();
flightsPollTimer = setInterval(loadFlights, Math.max(15, flightsPollSeconds) * 1000);
} else {
if (flightsPollTimer) clearInterval(flightsPollTimer);
flightsPollTimer = null;
if (flightsPoints) {
globe.scene().remove(flightsPoints);
flightsPoints.geometry.dispose();
flightsPoints.material.dispose();
flightsPoints = null;
}
document.getElementById("flightCount").textContent = "";
document.getElementById("flightTooltip").classList.add("hidden");
}
}
const flightTooltip = document.getElementById("flightTooltip");
document.getElementById("globeViz").addEventListener("mousemove", (ev) => {
if (!flightsPoints) return;
const el = document.getElementById("globeViz");
const rect = el.getBoundingClientRect();
const mouse = new THREE.Vector2(
((ev.clientX - rect.left) / rect.width) * 2 - 1,
-((ev.clientY - rect.top) / rect.height) * 2 + 1
);
flightRaycaster.setFromCamera(mouse, globe.camera());
const hits = flightRaycaster.intersectObject(flightsPoints);
const f = hits.length ? flightsRendered[hits[0].index] : null;
if (!f) {
flightTooltip.classList.add("hidden");
return;
}
flightTooltip.innerHTML = `
<b>${escapeHtml(f.callsign || f.icao24)}</b>${f.is_military ? ' <span class="mil">MIL (heuristic)</span>' : ""}<br/>
${escapeHtml(f.origin_country || "")}<br/>
${f.altitude_m != null ? Math.round(f.altitude_m) + " m" : ""}${
f.velocity_ms != null ? " · " + Math.round(f.velocity_ms * 3.6) + " km/h" : ""
}
`;
flightTooltip.style.left = ev.clientX + 12 + "px";
flightTooltip.style.top = ev.clientY + 12 + "px";
flightTooltip.classList.remove("hidden");
});
// --------------------------------------------------------------- wiring --
document.getElementById("refreshBtn").onclick = async () => {
@ -781,7 +1146,15 @@ document.getElementById("weatherLayer").onchange = (ev) => {
document.getElementById("toggleConflict").onchange = () => {
if (!conflictEvents.length) loadConflictEvents();
else syncRings();
else {
syncRings();
syncGlobeObjects();
}
};
document.getElementById("toggleFlights").onchange = (ev) => setFlightsVisible(ev.target.checked);
document.getElementById("toggleMilitaryOnly").onchange = () => {
if (document.getElementById("toggleFlights").checked) renderFlights();
};
async function init() {
@ -792,6 +1165,7 @@ async function init() {
}
document.getElementById("toggleWeather").disabled = !config.weather_enabled;
document.getElementById("toggleConflict").disabled = !config.conflict_enabled;
flightsPollSeconds = config.flights_poll_seconds || 300;
await loadClusters();
await loadNewsFeed();