diff --git a/.env.example b/.env.example index 19af685..8ff02ce 100644 --- a/.env.example +++ b/.env.example @@ -17,6 +17,14 @@ OWM_API_KEY= ACLED_API_KEY= ACLED_EMAIL= +# --- Flights overlay (OpenSky Network) --- +# Works anonymously out of the box (400 credits/day, hence the conservative +# default poll interval below). Free registration raises that to 4000/day: +# https://opensky-network.org/index.php?option=com_users&view=registration +OPENSKY_USERNAME= +OPENSKY_PASSWORD= +FLIGHTS_POLL_SECONDS=300 + # --- Polling intervals (minutes) --- RSS_POLL_MINUTES=10 MARKET_POLL_MINUTES=15 diff --git a/.gitignore b/.gitignore index 848de11..888901a 100644 --- a/.gitignore +++ b/.gitignore @@ -2,3 +2,4 @@ /data/ __pycache__/ *.pyc +/deploy/lxc/prebuild-config.json diff --git a/README.md b/README.md index ee9e2ec..17349de 100644 --- a/README.md +++ b/README.md @@ -12,14 +12,25 @@ composition diagram when Wikipedia has one. - **Left rail** β€” latest news, newest first, auto-refreshing. A πŸ“ button on any geocoded story flies the globe to it. -- **Right rail** β€” index/oil ticker. Click an instrument to expand its - recent-spikes-with-candidate-articles view (see below). -- **Center** β€” the globe. Clicking a point opens a modal with its stories, - a Wikipedia summary of the place, and (if available) its country's - parliament composition diagram. Text selected anywhere can be searched on - Wikipedia via the popup that appears. +- **Right rail** β€” index/oil ticker with a 7-day sparkline per instrument. + Click one to expand its recent-spikes-with-candidate-articles preview + (see "Economic Incident History" below for the full log). +- **Center** β€” the globe. News clusters render as newspaper icons (tinted + violetβ†’pink by corroborating-source count); conflict events render as + magenta helmet icons plus a pulsing ring. Clicking a news icon opens a + modal with its stories, a Wikipedia summary of the place, and (if + available) its country's parliament composition diagram. Text selected + anywhere can be searched on Wikipedia via the popup that appears. +- **Top-left "i" button** β€” opens the **Economic Incident History** view: every + detected market spike, full clock-hour window, with the complete list of + articles published in it (not just the top few) β€” a comprehensive log, + distinct from the quick preview in the right rail. - **Bottom drawer** β€” collapsible conflict/military-event log (ACLED-backed; see below). +- **Flights toggle** β€” live global air traffic (OpenSky Network) as airplane + icons: magenta for flagged-military aircraft, the same dark violet as the + globe surface for everything else. A "Military only" checkbox hides + civilian traffic. Hover an aircraft for callsign/country/altitude/speed. Globe points use **zoom-adaptive clustering**: the backend groups articles onto a fixed ~11km grid, then the frontend progressively folds nearby grid @@ -78,6 +89,7 @@ overlays disable themselves. Nothing else requires a key. |---|---|---| | `OWM_API_KEY` | Weather overlay (clouds/precipitation/temp/wind/pressure) | free, [openweathermap.org](https://home.openweathermap.org/users/sign_up) | | `ACLED_API_KEY` + `ACLED_EMAIL` | Conflict/"military movement" event overlay | free for registered (incl. non-commercial/research) use, [acleddata.com](https://acleddata.com/register/) | +| `OPENSKY_USERNAME` + `OPENSKY_PASSWORD` | Higher-frequency flights overlay polling | optional, free β€” anonymous access already works, see below | **Why ACLED for "military movements":** there is no single free, public, real-time feed of military movements. ACLED is the closest widely-used open @@ -85,6 +97,15 @@ dataset of sourced, georeferenced conflict/political-violence events and is what the conflict overlay is built on. Treat it as "reported conflict events," not live troop tracking. +**Flights overlay** ("Flights" toggle) plots live airborne traffic from the +[OpenSky Network](https://opensky-network.org) REST API β€” works anonymously +out of the box. Anonymous access is capped at 400 credits/day, so the +default poll interval (`FLIGHTS_POLL_SECONDS=300`) is conservative; register +for free and set `OPENSKY_USERNAME`/`OPENSKY_PASSWORD` for a 4000/day cap and +you can safely poll more often. Aircraft are colored gray (civilian) or red +(flagged military) β€” see the heuristic caveat below before reading too much +into the red dots. + ## What's a heuristic, not ground truth - **Geocoding** (`backend/app/geocode.py`, `data/gazetteer.csv`) is keyword @@ -112,6 +133,17 @@ events," not live troop tracking. can change without notice β€” if a source goes quiet, check its site for a current feed URL and update the file (no rebuild needed, it's a read-only mount). +- **Military flight classification** (`backend/app/flights.py`, + `data/military_ranges.yaml`) is a heuristic, not an authoritative flag β€” + there's no public "is this military" field on live ADS-B data. It combines + an ICAO24 hex-address prefix (only the US block is backed by a citable + source: OpenSky Network's own 2017 research paper) and a couple of + well-known military callsign prefixes (`RCH`, `NATO`). This reliably + catches US/NATO transport and tanker traffic β€” the bulk of what's actually + visible on public ADS-B β€” and under-flags everyone else. Extend + `military_ranges.yaml` if you can source ranges/callsigns for other air + forces with similar confidence; don't guess, a wrong range misclassifies + real civilian flights. ## API surface (consumed by the frontend, but usable standalone) @@ -121,11 +153,16 @@ events," not live troop tracking. - `POST /api/refresh` β€” force an immediate RSS poll - `GET /api/markets/latest`, `GET /api/markets/history?symbol=^GSPC` - `GET /api/markets/spikes?symbol=&hours=` β€” sudden index/oil moves - (`MARKET_SPIKE_THRESHOLD_PCT`) paired with relevance-ranked articles - published in the surrounding window (`MARKET_SPIKE_LOOKBACK_HOURS`) β€” click - a ticker item in the UI for this. It's a heuristic correlation (keyword + - country/location matching), not a verified causal link. + (`MARKET_SPIKE_THRESHOLD_PCT`), each paired with every article published in + a full-clock-hour window around it (`MARKET_SPIKE_LOOKBACK_HOURS` sets the + minimum lookback; the window then snaps outward to whole hours). Ranked + by relevance (keyword/country matching) but not filtered by it β€” timing + alone qualifies a story as a candidate. Click a ticker item for a quick + preview, or the "i" button (top-left) for the full log. Heuristic + correlation, not a verified causal link. - `GET /api/conflict-events?hours=168` +- `GET /api/flights` β€” latest global OpenSky snapshot, with `is_military` + per aircraft (see the heuristic caveat above) - `GET /api/weather/tiles/{layer}/{z}/{x}/{y}.png` β€” OWM tile proxy - `GET /api/wikipedia/summary?title=`, `GET /api/wikipedia/search?q=`, `GET /api/wikipedia/parliament?country=` @@ -135,4 +172,11 @@ events," not live troop tracking. - Add/edit RSS sources: `backend/app/sources.yaml`. - Add gazetteer locations: `backend/app/data/gazetteer.csv`. - Add/fix parliament page mappings: `backend/app/data/parliaments.yaml`. +- Add/fix military ICAO24/callsign ranges: `backend/app/data/military_ranges.yaml`. - Change poll intervals / clustering window: `.env`. + +## Deploying as a Proxmox LXC + +See `deploy/lxc/README.md` for a `pct`-based build script that provisions a +Debian/Ubuntu/Alpine LXC container running the full stack, and packages it +as a reusable `.tar.zst` template. diff --git a/backend/app/config.py b/backend/app/config.py index e66da13..0791191 100644 --- a/backend/app/config.py +++ b/backend/app/config.py @@ -28,6 +28,17 @@ class Settings(BaseSettings): sources_file: str = str(APP_DIR / "sources.yaml") gazetteer_file: str = str(APP_DIR / "data" / "gazetteer.csv") + # --- Flights overlay (OpenSky Network) --- + # Anonymous access works out of the box but is capped at 400 credits/day; + # a full-globe states/all poll costs several credits, so the default + # interval below is deliberately conservative. Free registration + # (opensky-network.org) raises the cap to 4000/day β€” set these and drop + # FLIGHTS_POLL_SECONDS if you want closer-to-live updates. + opensky_username: str = "" + opensky_password: str = "" + flights_poll_seconds: int = 300 + military_ranges_file: str = str(APP_DIR / "data" / "military_ranges.yaml") + class Config: env_file = ".env" diff --git a/backend/app/data/military_ranges.yaml b/backend/app/data/military_ranges.yaml new file mode 100644 index 0000000..d0a0c68 --- /dev/null +++ b/backend/app/data/military_ranges.yaml @@ -0,0 +1,32 @@ +# Best-effort military-aircraft classification for the flights overlay. +# +# There is no single public, authoritative "is this plane military" flag on +# live ADS-B data. This combines two heuristics: +# +# 1. icao24_prefixes β€” hex-address blocks that air forces reserve within +# their country's ICAO24 allocation. Only the US block below is backed +# by a citable source (OpenSky Network's own published research, +# "Mode S and ADS-B Usage of Military and other State Aircraft", 2017: +# "identifiers used by the US Air Force tend to begin with 'AE'"). Add +# more countries' ranges here if you can source them with similar +# confidence β€” do not guess, a wrong range misclassifies real civilian +# traffic. +# 2. callsign_prefixes β€” well-known, publicly documented military radio +# callsigns. "RCH" (US Air Mobility Command, "Reach") is the big one +# for ADS-B-visible traffic; "NATO" covers NATO AWACS/transport flights. +# +# Net effect: this overlay reliably flags US military and NATO transport/ +# tanker traffic (the bulk of what's actually visible on public ADS-B β€” most +# other countries' military aircraft either don't broadcast ADS-B or use +# civilian-style codes) and will under-flag everyone else. Extend the lists +# below as you find sourced ranges/callsigns for other air forces. + +icao24_prefixes: + - prefix: "ae" + label: "US military (DoD)" + +callsign_prefixes: + - prefix: "RCH" + label: "US Air Mobility Command (Reach)" + - prefix: "NATO" + label: "NATO" diff --git a/backend/app/flights.py b/backend/app/flights.py new file mode 100644 index 0000000..6acdce6 --- /dev/null +++ b/backend/app/flights.py @@ -0,0 +1,92 @@ +"""Live flight overlay, backed by the OpenSky Network REST API. + +Positions are high-churn and ephemeral β€” unlike articles/markets/conflict +events we don't persist history to SQLite, just keep the latest global +snapshot in memory and let the scheduler refresh it. + +Military classification is a best-effort heuristic (see +data/military_ranges.yaml for sourcing and caveats) β€” there is no public +"is this military" flag on live ADS-B data. +""" + +import logging +from functools import lru_cache + +import httpx +import yaml + +from .config import settings + +log = logging.getLogger("newsatlas.flights") + +STATES_URL = "https://opensky-network.org/api/states/all" + +_latest_flights: list[dict] = [] + + +@lru_cache(maxsize=1) +def _military_rules() -> tuple[list[str], list[str]]: + with open(settings.military_ranges_file, encoding="utf-8") as f: + data = yaml.safe_load(f) or {} + icao_prefixes = [r["prefix"].lower() for r in data.get("icao24_prefixes", [])] + callsign_prefixes = [r["prefix"].upper() for r in data.get("callsign_prefixes", [])] + return icao_prefixes, callsign_prefixes + + +def _is_military(icao24: str, callsign: str) -> bool: + icao_prefixes, callsign_prefixes = _military_rules() + icao24 = (icao24 or "").lower() + callsign = (callsign or "").strip().upper() + if any(icao24.startswith(p) for p in icao_prefixes): + return True + if any(callsign.startswith(p) for p in callsign_prefixes): + return True + return False + + +def poll_flights() -> int: + global _latest_flights + + auth = None + if settings.opensky_username and settings.opensky_password: + auth = (settings.opensky_username, settings.opensky_password) + + try: + resp = httpx.get(STATES_URL, auth=auth, timeout=30) + resp.raise_for_status() + payload = resp.json() + except Exception: + log.exception("Failed to fetch OpenSky states") + return 0 + + flights = [] + for s in payload.get("states") or []: + icao24, callsign, origin_country = s[0], s[1], s[2] + lon, lat, baro_altitude, on_ground = s[5], s[6], s[7], s[8] + velocity, heading, vertical_rate = s[9], s[10], s[11] + if lat is None or lon is None or on_ground: + continue + + callsign = (callsign or "").strip() + flights.append( + { + "icao24": icao24, + "callsign": callsign, + "origin_country": origin_country, + "lat": lat, + "lon": lon, + "altitude_m": baro_altitude, + "velocity_ms": velocity, + "heading": heading, + "vertical_rate_ms": vertical_rate, + "is_military": _is_military(icao24, callsign), + } + ) + + _latest_flights = flights + log.info("Flight poll complete: %d airborne aircraft (%d flagged military)", len(flights), sum(f["is_military"] for f in flights)) + return len(flights) + + +def get_latest_flights() -> list[dict]: + return _latest_flights diff --git a/backend/app/main.py b/backend/app/main.py index 802c279..7cc471f 100644 --- a/backend/app/main.py +++ b/backend/app/main.py @@ -13,6 +13,7 @@ from . import wikipedia from .clustering import build_clusters from .config import settings from .db import get_session, init_db, SessionLocal +from .flights import get_latest_flights from .ingest import fetch_all from .markets import poll_markets, INSTRUMENTS from .conflict import enabled as conflict_enabled, poll_conflict_events @@ -56,9 +57,17 @@ def public_config(): "weather_enabled": bool(settings.owm_api_key), "conflict_enabled": conflict_enabled(), "article_window_hours": settings.article_window_hours, + "flights_poll_seconds": settings.flights_poll_seconds, } +# ---- Flights ---------------------------------------------------------- + +@app.get("/api/flights") +def api_flights(): + return get_latest_flights() + + # ---- Articles / clusters ----------------------------------------------- @app.get("/api/clusters") @@ -174,7 +183,7 @@ def api_markets_history(symbol: str, hours: int = Query(168, le=24 * 30)): @app.get("/api/markets/spikes") -def api_markets_spikes(symbol: str | None = None, hours: int = Query(168, le=24 * 30)): +def api_markets_spikes(symbol: str | None = None, hours: int = Query(168, le=24 * 365)): since = dt.datetime.utcnow() - dt.timedelta(hours=hours) session = next(get_session()) try: diff --git a/backend/app/markets.py b/backend/app/markets.py index ffc5efe..2c40cc1 100644 --- a/backend/app/markets.py +++ b/backend/app/markets.py @@ -100,7 +100,7 @@ def _score_article(article: Article, keywords: list[str], country: str | None) - return score -def _find_candidate_articles(session: Session, symbol: str, window_start: dt.datetime, window_end: dt.datetime, limit: int = 8) -> list[int]: +def _find_candidate_articles(session: Session, symbol: str, window_start: dt.datetime, window_end: dt.datetime, limit: int = 60) -> list[int]: hints = _SPIKE_HINTS.get(symbol, {"keywords": [], "country": None}) rows = session.execute( select(Article) @@ -109,12 +109,24 @@ def _find_candidate_articles(session: Session, symbol: str, window_start: dt.dat .limit(300) # cap the scan; this is a heuristic ranker, not a full-text search index ).scalars().all() + # Every article published in the window is a candidate β€” timing alone is + # "possibly related" for the purposes of this log, not just keyword + # matches. Score still decides ordering (keyword/country hits surface + # first), it just no longer excludes zero-score articles. scored = [(_score_article(a, hints["keywords"], hints["country"]), a) for a in rows] - scored = [(s, a) for s, a in scored if s > 0] scored.sort(key=lambda pair: (pair[0], pair[1].published_at), reverse=True) return [a.id for _, a in scored[:limit]] +def _floor_hour(t: dt.datetime) -> dt.datetime: + return t.replace(minute=0, second=0, microsecond=0) + + +def _ceil_hour(t: dt.datetime) -> dt.datetime: + floored = _floor_hour(t) + return floored if floored == t else floored + dt.timedelta(hours=1) + + def _detect_and_record_spike( session: Session, symbol: str, label: str, prev: MarketPrice | None, price: float, recorded_at: dt.datetime ) -> None: @@ -124,8 +136,11 @@ def _detect_and_record_spike( if abs(pct_change) < settings.market_spike_threshold_pct: return - window_start = prev.recorded_at - dt.timedelta(hours=settings.market_spike_lookback_hours) - article_ids = _find_candidate_articles(session, symbol, window_start, recorded_at) + # Snapped to whole clock hours so the log reads as clean ranges (e.g. + # "14:00-16:00"), covering at least the full hour the spike landed in. + window_start = _floor_hour(prev.recorded_at - dt.timedelta(hours=settings.market_spike_lookback_hours)) + window_end = _ceil_hour(recorded_at) + article_ids = _find_candidate_articles(session, symbol, window_start, window_end) session.add( MarketSpike( @@ -135,7 +150,7 @@ def _detect_and_record_spike( to_price=price, pct_change=round(pct_change, 3), window_start=window_start, - window_end=recorded_at, + window_end=window_end, detected_at=recorded_at, article_ids_json=json.dumps(article_ids), ) diff --git a/backend/app/scheduler.py b/backend/app/scheduler.py index dcf853a..fafde33 100644 --- a/backend/app/scheduler.py +++ b/backend/app/scheduler.py @@ -5,6 +5,7 @@ from apscheduler.schedulers.background import BackgroundScheduler from .conflict import poll_conflict_events from .config import settings from .db import SessionLocal +from .flights import poll_flights from .ingest import fetch_all from .markets import poll_markets @@ -38,11 +39,17 @@ def _run_conflict_job() -> None: session.close() +def _run_flights_job() -> None: + count = poll_flights() + log.info("Flight poll complete: %d aircraft", count) + + def start_scheduler() -> BackgroundScheduler: scheduler = BackgroundScheduler(timezone="UTC") scheduler.add_job(_run_rss_job, "interval", minutes=settings.rss_poll_minutes, next_run_time=None) scheduler.add_job(_run_market_job, "interval", minutes=settings.market_poll_minutes, next_run_time=None) scheduler.add_job(_run_conflict_job, "interval", minutes=settings.conflict_poll_minutes, next_run_time=None) + scheduler.add_job(_run_flights_job, "interval", seconds=settings.flights_poll_seconds, next_run_time=None) scheduler.start() # Kick off an immediate first run of each job in the background so the diff --git a/deploy/lxc/README.md b/deploy/lxc/README.md new file mode 100644 index 0000000..0f87488 --- /dev/null +++ b/deploy/lxc/README.md @@ -0,0 +1,92 @@ +# NewsAtlas β€” Proxmox LXC deployment + +Builds a Proxmox LXC container running the full NewsAtlas stack (Docker + +docker-compose inside the CT), and packages it as a reusable `.tar.zst` +template. Supports Debian, Ubuntu, and Alpine base images. + +> **Not executed against a real Proxmox host while writing this** β€” there +> wasn't one available in that environment. The docker-compose stack itself +> (`../../docker-compose.yml`) *has* been built and run end-to-end; this LXC +> layer just automates `pct`/`pveam`/`vzdump` calls around it. Run +> `./build.sh --dry-run` first and read the printed commands before +> running for real. + +## Security model β€” read before using + +`build.sh` writes real secrets (whatever you put in `prebuild-config.json`) +into `/opt/newsatlas/.env` **inside the container's filesystem**, and that +filesystem gets packaged into the `.tar.zst` template. This was a deliberate +choice β€” the safer default (inject secrets only into the running container, +never into a distributable template) was offered and explicitly declined in +favor of "clone-and-go" templates that boot with everything already +configured. + +The trade-off that comes with that: **every copy of the resulting +`.tar.zst` is a credentials-bearing artifact**, for every distro variant you +build. Concretely: + +- Keep template files (`local:vztmpl/...` / your `vzdump` dump directory) + on storage only trusted operators can read β€” never a public/shared LXC + template store. The whole point of a reusable template is that it gets + cloned, and each clone carries the same baked-in credentials. +- If a copy of the tarball is ever backed up somewhere less trusted, copied + off-host, or shared, **rotate every key in `prebuild-config.json`** + (OpenWeatherMap, ACLED, OpenSky) β€” there's no way to invalidate just the + copy that leaked. +- `prebuild-config.json` itself is gitignored; don't commit it, don't paste + it into issues/chat. + +If you'd rather not carry that risk, don't use this path β€” run +`docker compose up --build` directly on any host (see the root README) and +manage `.env` there normally; nothing gets baked into a shareable artifact. + +## Setup + +```bash +cp prebuild-config.json.template prebuild-config.json +$EDITOR prebuild-config.json # fill in secrets, LXC sizing, network, etc. +``` + +Key fields: + +- `app.repo_url` / `app.branch` β€” where `provision.sh` clones the app from + inside the container. Defaults to this repo's own remote. +- `secrets.*` β€” same keys as the root `.env.example`, baked in as described + above. +- `lxc.*` β€” sizing (`cores`, `memory_mb`, `disk_gb`), Proxmox storage IDs + (`storage` for the rootfs, `template_storage` for both the base template + and the packaged output), networking (`bridge`, `vlan`, `ip`, `gateway`), + and either `root_password` or `ssh_public_key_file` for access. +- `distros.*` β€” the `pveam available` template-name prefix used to resolve + the newest matching template for each distro. Check `pveam available` + on your host if these drift. + +## Building + +Run as root on the Proxmox VE host: + +```bash +./build.sh debian # one distro +./build.sh all # debian + ubuntu + alpine +./build.sh ubuntu --dry-run # print the pct/pveam/vzdump commands, run nothing +./build.sh alpine --no-keep # destroy the scratch CT after packaging (template file only) +./build.sh debian --no-template # deploy and leave the CT running; skip packaging +``` + +Each run: allocates the next free CTID, creates the CT with `nesting=1` +(required to run Docker inside an LXC), installs Docker via `provision.sh`, +clones the app, pushes the generated `.env`, runs `docker compose up +--build`, waits for `/api/health`, then (unless `--no-template`) stops the +CT and runs `vzdump --mode stop --compress zstd` to produce the template, +restarting the CT afterward (unless `--no-keep`). + +## Redeploying from a packaged template + +```bash +pct restore :backup/vzdump-lxc--*.tar.zst --storage +pct start +``` + +The container comes up with the app already running and `.env` already in +place β€” no further provisioning needed (that's the whole point, and also +exactly why the security note above matters). diff --git a/deploy/lxc/build.sh b/deploy/lxc/build.sh new file mode 100755 index 0000000..7ee5746 --- /dev/null +++ b/deploy/lxc/build.sh @@ -0,0 +1,248 @@ +#!/bin/bash +# Builds a NewsAtlas LXC container on a Proxmox VE host: creates the CT, +# installs Docker inside it, deploys the app, bakes secrets/config from +# prebuild-config.json into it, and (by default) packages the result as a +# reusable .tar.zst template via vzdump. +# +# Run this ON the Proxmox VE host, as root. It has NOT been executed against +# a real Proxmox host while developing this (no such host is available in +# that environment) β€” read it before running, and use --dry-run first. +# +# SECURITY: the packaged template tarball contains the real values from +# prebuild-config.json (API keys, etc.) baked into /opt/newsatlas/.env +# inside its filesystem. That was an explicit choice (see deploy/lxc/README.md) +# over the safer default of injecting secrets at deploy time β€” treat every +# copy of the resulting .tar.zst as a credentials-bearing artifact: keep it +# on trusted storage only, never upload it to a public template store, and +# rotate the keys if a copy ever leaves your control. +# +# Usage: +# ./build.sh [--dry-run] [--no-keep] [--no-template] +# +# --dry-run print every pct/pveam/vzdump command instead of running it +# --no-keep destroy the scratch CT after packaging (template only) +# --no-template deploy and leave the CT running; skip vzdump packaging + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +CONFIG_FILE="$SCRIPT_DIR/prebuild-config.json" +PROVISION_SCRIPT="$SCRIPT_DIR/provision.sh" +APP_DIR_IN_CT="/opt/newsatlas" + +DRY_RUN=0 +KEEP_CT=1 +MAKE_TEMPLATE=1 +TARGET="${1:-}" +shift || true +for arg in "$@"; do + case "$arg" in + --dry-run) DRY_RUN=1 ;; + --no-keep) KEEP_CT=0 ;; + --no-template) MAKE_TEMPLATE=0 ;; + *) echo "Unknown flag: $arg" >&2; exit 1 ;; + esac +done + +run() { + if [ "$DRY_RUN" = "1" ]; then + printf '[dry-run]' + printf ' %q' "$@" + printf '\n' + else + "$@" + fi +} + +# ---- preflight -------------------------------------------------------- + +if [ "$(id -u)" != "0" ]; then + echo "build.sh must run as root on the Proxmox VE host." >&2 + exit 1 +fi + +for bin in pct pveam pvesh jq; do + if ! command -v "$bin" >/dev/null 2>&1; then + echo "Missing required command: $bin (this must run on a Proxmox VE host with jq installed)." >&2 + exit 1 + fi +done + +if [ "$MAKE_TEMPLATE" = "1" ] && ! command -v vzdump >/dev/null 2>&1; then + echo "Missing required command: vzdump (needed to package a template; pass --no-template to skip)." >&2 + exit 1 +fi + +if [ ! -f "$CONFIG_FILE" ]; then + cat >&2 < [--dry-run] [--no-keep] [--no-template]" >&2 + exit 1 + ;; +esac + +jqc() { jq -r "$1" "$CONFIG_FILE"; } + +REPO_URL=$(jqc '.app.repo_url') +BRANCH=$(jqc '.app.branch') +HTTP_PORT=$(jqc '.app.http_port') + +HOSTNAME_BASE=$(jqc '.lxc.hostname') +CORES=$(jqc '.lxc.cores') +MEMORY_MB=$(jqc '.lxc.memory_mb') +SWAP_MB=$(jqc '.lxc.swap_mb') +DISK_GB=$(jqc '.lxc.disk_gb') +STORAGE=$(jqc '.lxc.storage') +TEMPLATE_STORAGE=$(jqc '.lxc.template_storage') +BRIDGE=$(jqc '.lxc.bridge') +VLAN=$(jqc '.lxc.vlan') +IP_CFG=$(jqc '.lxc.ip') +GATEWAY=$(jqc '.lxc.gateway') +UNPRIVILEGED=$(jqc '.lxc.unprivileged') +START_ON_BOOT=$(jqc '.lxc.start_on_boot') +ROOT_PASSWORD=$(jqc '.lxc.root_password') +SSH_KEY_FILE=$(jqc '.lxc.ssh_public_key_file') + +if [ "$TARGET" = "all" ]; then + DISTROS="debian ubuntu alpine" +else + DISTROS="$TARGET" +fi + +# ---- per-distro build --------------------------------------------------- + +build_one() { + local distro="$1" + local template_prefix ctid template hostname net0 rootfs features + local unpriv_flag="" + [ "$UNPRIVILEGED" = "true" ] && unpriv_flag="1" || unpriv_flag="0" + + template_prefix=$(jqc ".distros.${distro}") + if [ -z "$template_prefix" ] || [ "$template_prefix" = "null" ]; then + echo "No distros.${distro} entry in prebuild-config.json, skipping." >&2 + return 1 + fi + + echo "=== Building NewsAtlas LXC: $distro ===" + + echo "-- Resolving template for prefix '$template_prefix'..." + pveam update >/dev/null 2>&1 || true + template=$(pveam available --section system 2>/dev/null | awk '{print $2}' | grep "^${template_prefix}" | sort -V | tail -1) + if [ -z "$template" ]; then + echo "Could not find a template matching '${template_prefix}' via 'pveam available'. Check prebuild-config.json's distros.${distro} value against 'pveam available'." >&2 + return 1 + fi + echo "-- Using template: $template" + + if ! pveam list "$TEMPLATE_STORAGE" 2>/dev/null | grep -q "$template"; then + run pveam download "$TEMPLATE_STORAGE" "$template" + fi + + ctid=$(pvesh get /cluster/nextid) + hostname="${HOSTNAME_BASE}-${distro}" + rootfs="${STORAGE}:${DISK_GB}" + features="nesting=1,keyctl=1" # required for Docker-in-LXC + + net0="name=eth0,bridge=${BRIDGE},ip=${IP_CFG}" + [ -n "$VLAN" ] && net0="${net0},tag=${VLAN}" + [ "$IP_CFG" != "dhcp" ] && [ -n "$GATEWAY" ] && net0="${net0},gw=${GATEWAY}" + + echo "-- Creating CT $ctid ($hostname)..." + create_args=(pct create "$ctid" "${TEMPLATE_STORAGE}:vztmpl/${template}" + --hostname "$hostname" + --cores "$CORES" + --memory "$MEMORY_MB" + --swap "$SWAP_MB" + --rootfs "$rootfs" + --net0 "$net0" + --unprivileged "$unpriv_flag" + --features "$features" + --onboot "$([ "$START_ON_BOOT" = "true" ] && echo 1 || echo 0)" + ) + if [ -n "$ROOT_PASSWORD" ]; then + create_args+=(--password "$ROOT_PASSWORD") + fi + if [ -n "$SSH_KEY_FILE" ] && [ -f "$SSH_KEY_FILE" ]; then + create_args+=(--ssh-public-keys "$SSH_KEY_FILE") + fi + run "${create_args[@]}" + + run pct start "$ctid" + + if [ "$DRY_RUN" != "1" ]; then + echo "-- Waiting for CT $ctid to come up..." + for i in $(seq 1 30); do + pct exec "$ctid" -- true 2>/dev/null && break + sleep 2 + [ "$i" = "30" ] && { echo "CT $ctid did not become ready in time." >&2; return 1; } + done + fi + + echo "-- Provisioning (installing Docker, cloning repo)..." + run pct push "$ctid" "$PROVISION_SCRIPT" /root/provision.sh --perms 0755 + run pct exec "$ctid" -- /root/provision.sh "$REPO_URL" "$BRANCH" "$APP_DIR_IN_CT" + + echo "-- Writing .env (baking config from prebuild-config.json)..." + local envfile + envfile=$(mktemp) + { + echo "HTTP_PORT=${HTTP_PORT}" + echo "OWM_API_KEY=$(jqc '.secrets.owm_api_key')" + echo "ACLED_API_KEY=$(jqc '.secrets.acled_api_key')" + echo "ACLED_EMAIL=$(jqc '.secrets.acled_email')" + echo "OPENSKY_USERNAME=$(jqc '.secrets.opensky_username')" + echo "OPENSKY_PASSWORD=$(jqc '.secrets.opensky_password')" + echo "RSS_POLL_MINUTES=$(jqc '.polling.rss_poll_minutes')" + echo "MARKET_POLL_MINUTES=$(jqc '.polling.market_poll_minutes')" + echo "MARKET_SPIKE_THRESHOLD_PCT=$(jqc '.polling.market_spike_threshold_pct')" + echo "MARKET_SPIKE_LOOKBACK_HOURS=$(jqc '.polling.market_spike_lookback_hours')" + echo "CONFLICT_POLL_MINUTES=$(jqc '.polling.conflict_poll_minutes')" + echo "FLIGHTS_POLL_SECONDS=$(jqc '.polling.flights_poll_seconds')" + echo "ARTICLE_WINDOW_HOURS=$(jqc '.polling.article_window_hours')" + echo "DATABASE_PATH=/data/newsatlas.db" + } > "$envfile" + run pct push "$ctid" "$envfile" "${APP_DIR_IN_CT}/.env" --perms 0600 + rm -f "$envfile" + + echo "-- Starting the stack..." + run pct exec "$ctid" -- bash -c "cd ${APP_DIR_IN_CT} && docker compose up -d --build" + + if [ "$DRY_RUN" != "1" ]; then + echo "-- Waiting for the app to answer /api/health..." + for i in $(seq 1 30); do + pct exec "$ctid" -- curl -sf "http://localhost:${HTTP_PORT}/api/health" >/dev/null 2>&1 && break + sleep 3 + [ "$i" = "30" ] && echo "Warning: /api/health didn't respond in time; check 'pct exec $ctid -- docker compose -f ${APP_DIR_IN_CT}/docker-compose.yml logs'." >&2 + done + fi + + if [ "$MAKE_TEMPLATE" = "1" ]; then + echo "-- Stopping CT $ctid to package as a template..." + run pct shutdown "$ctid" --timeout 60 + echo "-- Packaging via vzdump..." + run vzdump "$ctid" --mode stop --compress zstd --storage "$TEMPLATE_STORAGE" + if [ "$KEEP_CT" = "1" ]; then + run pct start "$ctid" + echo "-- CT $ctid restarted (kept running in addition to the packaged template)." + else + run pct destroy "$ctid" + echo "-- CT $ctid destroyed after packaging (--no-keep)." + fi + fi + + echo "=== Done: $distro (CTID $ctid, hostname $hostname) ===" + echo " Reminder: any packaged .tar.zst for this CT has live secrets baked in β€” see deploy/lxc/README.md." +} + +for d in $DISTROS; do + build_one "$d" +done diff --git a/deploy/lxc/prebuild-config.json.template b/deploy/lxc/prebuild-config.json.template new file mode 100644 index 0000000..eac88a5 --- /dev/null +++ b/deploy/lxc/prebuild-config.json.template @@ -0,0 +1,53 @@ +{ + "_comment": "Copy this file to prebuild-config.json (gitignored) and fill it in. build.sh reads secrets/config from prebuild-config.json only β€” never from this template.", + + "app": { + "repo_url": "https://git.abdelbaki.eu/The_miro/NewsAtlas.git", + "branch": "main", + "http_port": 8080 + }, + + "secrets": { + "_comment": "Written into /opt/newsatlas/.env inside the container (mode 600, root-only) and baked into the packaged template tarball. Anyone with a copy of that tarball has these values β€” see deploy/lxc/README.md before sharing/backing one up anywhere untrusted.", + "owm_api_key": "", + "acled_api_key": "", + "acled_email": "", + "opensky_username": "", + "opensky_password": "" + }, + + "polling": { + "rss_poll_minutes": 10, + "market_poll_minutes": 15, + "market_spike_threshold_pct": 1.5, + "market_spike_lookback_hours": 6, + "conflict_poll_minutes": 60, + "flights_poll_seconds": 300, + "article_window_hours": 72 + }, + + "lxc": { + "hostname": "newsatlas", + "cores": 2, + "memory_mb": 2048, + "swap_mb": 512, + "disk_gb": 8, + "storage": "local-lvm", + "template_storage": "local", + "bridge": "vmbr0", + "vlan": "", + "ip": "dhcp", + "gateway": "", + "unprivileged": true, + "start_on_boot": true, + "root_password": "", + "ssh_public_key_file": "" + }, + + "distros": { + "_comment": "Which base-OS variants build.sh should build when run with --all. Each maps to a `pveam available` template name prefix; build.sh picks the newest matching version.", + "debian": "debian-12-standard", + "ubuntu": "ubuntu-24.04-standard", + "alpine": "alpine-3.19-default" + } +} diff --git a/deploy/lxc/provision.sh b/deploy/lxc/provision.sh new file mode 100755 index 0000000..ffb7a68 --- /dev/null +++ b/deploy/lxc/provision.sh @@ -0,0 +1,48 @@ +#!/bin/bash +# Runs INSIDE the LXC container as root (pushed + executed by build.sh). +# Installs Docker and clones NewsAtlas. Deliberately does NOT touch .env or +# start the stack β€” build.sh pushes the generated .env and starts the stack +# as separate steps afterward, so a plain `git clone` here never collides +# with an already-placed .env file. +set -euo pipefail + +REPO_URL="${1:?usage: provision.sh }" +BRANCH="${2:?usage: provision.sh }" +APP_DIR="${3:?usage: provision.sh }" + +. /etc/os-release + +install_docker_apt() { + export DEBIAN_FRONTEND=noninteractive + apt-get update + apt-get install -y ca-certificates curl gnupg git jq + curl -fsSL https://get.docker.com | sh + systemctl enable docker + systemctl start docker +} + +install_docker_apk() { + apk update + apk add --no-cache docker docker-cli-compose git jq openrc + rc-update add docker default + service docker start +} + +case "$ID" in + debian|ubuntu) install_docker_apt ;; + alpine) install_docker_apk ;; + *) + echo "provision.sh: unsupported distro '$ID' (expected debian, ubuntu, or alpine)" >&2 + exit 1 + ;; +esac + +if [ -d "$APP_DIR/.git" ]; then + git -C "$APP_DIR" fetch origin "$BRANCH" + git -C "$APP_DIR" checkout "$BRANCH" + git -C "$APP_DIR" reset --hard "origin/$BRANCH" +else + git clone --branch "$BRANCH" --depth 1 "$REPO_URL" "$APP_DIR" +fi + +echo "provision.sh: done. Docker installed, repo at $APP_DIR." diff --git a/docker-compose.yml b/docker-compose.yml index 431900a..11bab1b 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -10,6 +10,7 @@ services: - ./backend/app/sources.yaml:/app/app/sources.yaml:ro - ./backend/app/data/gazetteer.csv:/app/app/data/gazetteer.csv:ro - ./backend/app/data/parliaments.yaml:/app/app/data/parliaments.yaml:ro + - ./backend/app/data/military_ranges.yaml:/app/app/data/military_ranges.yaml:ro expose: - "8000" diff --git a/frontend/css/style.css b/frontend/css/style.css index c4617e6..08cd17d 100644 --- a/frontend/css/style.css +++ b/frontend/css/style.css @@ -21,6 +21,7 @@ --c-dark-soft: #9d7cff; /* lightened violet for small text/links on dark bg */ --c-red: #f50505; /* danger / down */ --c-green: #34d399; /* gain β€” kept distinct from theme for finance semantics */ + --c-magenta: #ff33ff; /* flagged-military flights β€” must match MILITARY_FLIGHT_HEX in app.js */ --c-panel: rgba(24, 15, 33, 0.97); --c-panel-border: #3a2159; @@ -60,6 +61,8 @@ html, body { z-index: 30; } +#topbarLeft { display: flex; align-items: center; } + #topbar h1 { font-size: 18px; letter-spacing: 0.08em; @@ -218,7 +221,7 @@ html, body { .modal { position: absolute; inset: 0; z-index: 40; display: flex; align-items: center; justify-content: center; } .modal.hidden { display: none; } -#modalBackdrop { position: absolute; inset: 0; background: rgba(10, 0, 20, 0.65); } +.modal-backdrop { position: absolute; inset: 0; background: rgba(10, 0, 20, 0.65); } .modal-card { position: relative; width: 420px; @@ -231,7 +234,8 @@ html, body { padding: 20px 18px; box-shadow: 0 12px 40px rgba(0,0,0,0.6), 0 0 60px rgba(80, 24, 221, 0.15); } -#closeModal { +.modal-card.wide { width: 760px; max-height: 88vh; } +.modal-close { position: absolute; top: 8px; right: 10px; @@ -241,7 +245,37 @@ html, body { font-size: 22px; cursor: pointer; } -#closeModal:hover { color: var(--c-highlight-soft); } +.modal-close:hover { color: var(--c-highlight-soft); } + +#infoBtn { + width: 26px; + height: 26px; + border-radius: 50%; + background: var(--c-tag-bg); + border: 1px solid var(--c-panel-border); + color: var(--c-dark-soft); + font-size: 13px; + font-weight: 700; + font-style: italic; + font-family: Georgia, "Times New Roman", serif; + cursor: pointer; + margin-left: 4px; +} +#infoBtn:hover { background: #3d2260; color: #fff; } + +.econ-select { + background: #1c1030; + color: var(--c-text); + border: 1px solid var(--c-panel-border); + border-radius: 4px; + font-size: 12px; + padding: 4px 6px; +} + +.econ-incident { padding: 14px 0; border-bottom: 1px solid #241533; } +.econ-incident h3 { margin: 0 0 4px; font-size: 14px; color: #f2dede; } +.econ-incident .window { font-size: 11px; color: var(--c-text-muted); margin-bottom: 8px; } +.econ-incident .articles { margin-top: 8px; } .panel-section { margin-bottom: 20px; } .panel-section h2 { @@ -271,6 +305,23 @@ html, body { /* ---- wikipedia text-selection popup ---- */ +#flightTooltip { + position: absolute; + z-index: 45; + background: #1c1030; + border: 1px solid var(--c-panel-border); + border-radius: 6px; + padding: 6px 9px; + box-shadow: 0 4px 16px rgba(0,0,0,0.5); + font-size: 11px; + line-height: 1.5; + pointer-events: none; + max-width: 220px; +} +#flightTooltip.hidden { display: none; } +#flightTooltip b { color: var(--c-text); } +#flightTooltip .mil { color: var(--c-magenta); font-weight: 600; } + #wikiSelectPopup { position: absolute; z-index: 50; diff --git a/frontend/index.html b/frontend/index.html index ddf8df7..40605c8 100644 --- a/frontend/index.html +++ b/frontend/index.html @@ -10,7 +10,10 @@
-

NewsAtlas

+
+

NewsAtlas

+ +
+ + +
@@ -37,14 +43,34 @@ + + +