Add flights overlay, globe icon sprites, and economic incident history
Flights: live OpenSky global air traffic as airplane-icon sprites, magenta for heuristically-flagged military aircraft vs. globe-tint violet for civilian, with a military-only filter and hover tooltips. Globe rendering: news clusters and conflict events switch from three-globe's built-in point/ring primitives to a custom Sprite-based objects layer so they render as tinted newspaper/helmet icons respectively, while keeping existing click/hover behavior and the conflict pulsing-ring effect. Economic Incident History: a dedicated view (new top-left "i" button) listing every detected market spike with a full-clock-hour correlation window and the complete list of articles published in it, not just a relevance-capped preview. Also: LXC/Proxmox deployment scripts (deploy/lxc/) driven by a gitignored prebuild-config.json, and three additional RSS sources (CNN, Jacobin, Der Standard). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Um48tTvZDrEgDeweFyhPYCmain
parent
d52cb8b12e
commit
8e2c5efdbe
|
|
@ -17,6 +17,14 @@ OWM_API_KEY=
|
||||||
ACLED_API_KEY=
|
ACLED_API_KEY=
|
||||||
ACLED_EMAIL=
|
ACLED_EMAIL=
|
||||||
|
|
||||||
|
# --- Flights overlay (OpenSky Network) ---
|
||||||
|
# Works anonymously out of the box (400 credits/day, hence the conservative
|
||||||
|
# default poll interval below). Free registration raises that to 4000/day:
|
||||||
|
# https://opensky-network.org/index.php?option=com_users&view=registration
|
||||||
|
OPENSKY_USERNAME=
|
||||||
|
OPENSKY_PASSWORD=
|
||||||
|
FLIGHTS_POLL_SECONDS=300
|
||||||
|
|
||||||
# --- Polling intervals (minutes) ---
|
# --- Polling intervals (minutes) ---
|
||||||
RSS_POLL_MINUTES=10
|
RSS_POLL_MINUTES=10
|
||||||
MARKET_POLL_MINUTES=15
|
MARKET_POLL_MINUTES=15
|
||||||
|
|
|
||||||
|
|
@ -2,3 +2,4 @@
|
||||||
/data/
|
/data/
|
||||||
__pycache__/
|
__pycache__/
|
||||||
*.pyc
|
*.pyc
|
||||||
|
/deploy/lxc/prebuild-config.json
|
||||||
|
|
|
||||||
64
README.md
64
README.md
|
|
@ -12,14 +12,25 @@ composition diagram when Wikipedia has one.
|
||||||
|
|
||||||
- **Left rail** — latest news, newest first, auto-refreshing. A 📍 button on
|
- **Left rail** — latest news, newest first, auto-refreshing. A 📍 button on
|
||||||
any geocoded story flies the globe to it.
|
any geocoded story flies the globe to it.
|
||||||
- **Right rail** — index/oil ticker. Click an instrument to expand its
|
- **Right rail** — index/oil ticker with a 7-day sparkline per instrument.
|
||||||
recent-spikes-with-candidate-articles view (see below).
|
Click one to expand its recent-spikes-with-candidate-articles preview
|
||||||
- **Center** — the globe. Clicking a point opens a modal with its stories,
|
(see "Economic Incident History" below for the full log).
|
||||||
a Wikipedia summary of the place, and (if available) its country's
|
- **Center** — the globe. News clusters render as newspaper icons (tinted
|
||||||
parliament composition diagram. Text selected anywhere can be searched on
|
violet→pink by corroborating-source count); conflict events render as
|
||||||
Wikipedia via the popup that appears.
|
magenta helmet icons plus a pulsing ring. Clicking a news icon opens a
|
||||||
|
modal with its stories, a Wikipedia summary of the place, and (if
|
||||||
|
available) its country's parliament composition diagram. Text selected
|
||||||
|
anywhere can be searched on Wikipedia via the popup that appears.
|
||||||
|
- **Top-left "i" button** — opens the **Economic Incident History** view: every
|
||||||
|
detected market spike, full clock-hour window, with the complete list of
|
||||||
|
articles published in it (not just the top few) — a comprehensive log,
|
||||||
|
distinct from the quick preview in the right rail.
|
||||||
- **Bottom drawer** — collapsible conflict/military-event log (ACLED-backed;
|
- **Bottom drawer** — collapsible conflict/military-event log (ACLED-backed;
|
||||||
see below).
|
see below).
|
||||||
|
- **Flights toggle** — live global air traffic (OpenSky Network) as airplane
|
||||||
|
icons: magenta for flagged-military aircraft, the same dark violet as the
|
||||||
|
globe surface for everything else. A "Military only" checkbox hides
|
||||||
|
civilian traffic. Hover an aircraft for callsign/country/altitude/speed.
|
||||||
|
|
||||||
Globe points use **zoom-adaptive clustering**: the backend groups articles
|
Globe points use **zoom-adaptive clustering**: the backend groups articles
|
||||||
onto a fixed ~11km grid, then the frontend progressively folds nearby grid
|
onto a fixed ~11km grid, then the frontend progressively folds nearby grid
|
||||||
|
|
@ -78,6 +89,7 @@ overlays disable themselves. Nothing else requires a key.
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| `OWM_API_KEY` | Weather overlay (clouds/precipitation/temp/wind/pressure) | free, [openweathermap.org](https://home.openweathermap.org/users/sign_up) |
|
| `OWM_API_KEY` | Weather overlay (clouds/precipitation/temp/wind/pressure) | free, [openweathermap.org](https://home.openweathermap.org/users/sign_up) |
|
||||||
| `ACLED_API_KEY` + `ACLED_EMAIL` | Conflict/"military movement" event overlay | free for registered (incl. non-commercial/research) use, [acleddata.com](https://acleddata.com/register/) |
|
| `ACLED_API_KEY` + `ACLED_EMAIL` | Conflict/"military movement" event overlay | free for registered (incl. non-commercial/research) use, [acleddata.com](https://acleddata.com/register/) |
|
||||||
|
| `OPENSKY_USERNAME` + `OPENSKY_PASSWORD` | Higher-frequency flights overlay polling | optional, free — anonymous access already works, see below |
|
||||||
|
|
||||||
**Why ACLED for "military movements":** there is no single free, public,
|
**Why ACLED for "military movements":** there is no single free, public,
|
||||||
real-time feed of military movements. ACLED is the closest widely-used open
|
real-time feed of military movements. ACLED is the closest widely-used open
|
||||||
|
|
@ -85,6 +97,15 @@ dataset of sourced, georeferenced conflict/political-violence events and is
|
||||||
what the conflict overlay is built on. Treat it as "reported conflict
|
what the conflict overlay is built on. Treat it as "reported conflict
|
||||||
events," not live troop tracking.
|
events," not live troop tracking.
|
||||||
|
|
||||||
|
**Flights overlay** ("Flights" toggle) plots live airborne traffic from the
|
||||||
|
[OpenSky Network](https://opensky-network.org) REST API — works anonymously
|
||||||
|
out of the box. Anonymous access is capped at 400 credits/day, so the
|
||||||
|
default poll interval (`FLIGHTS_POLL_SECONDS=300`) is conservative; register
|
||||||
|
for free and set `OPENSKY_USERNAME`/`OPENSKY_PASSWORD` for a 4000/day cap and
|
||||||
|
you can safely poll more often. Aircraft are colored gray (civilian) or red
|
||||||
|
(flagged military) — see the heuristic caveat below before reading too much
|
||||||
|
into the red dots.
|
||||||
|
|
||||||
## What's a heuristic, not ground truth
|
## What's a heuristic, not ground truth
|
||||||
|
|
||||||
- **Geocoding** (`backend/app/geocode.py`, `data/gazetteer.csv`) is keyword
|
- **Geocoding** (`backend/app/geocode.py`, `data/gazetteer.csv`) is keyword
|
||||||
|
|
@ -112,6 +133,17 @@ events," not live troop tracking.
|
||||||
can change without notice — if a source goes quiet, check its site for a
|
can change without notice — if a source goes quiet, check its site for a
|
||||||
current feed URL and update the file (no rebuild needed, it's a read-only
|
current feed URL and update the file (no rebuild needed, it's a read-only
|
||||||
mount).
|
mount).
|
||||||
|
- **Military flight classification** (`backend/app/flights.py`,
|
||||||
|
`data/military_ranges.yaml`) is a heuristic, not an authoritative flag —
|
||||||
|
there's no public "is this military" field on live ADS-B data. It combines
|
||||||
|
an ICAO24 hex-address prefix (only the US block is backed by a citable
|
||||||
|
source: OpenSky Network's own 2017 research paper) and a couple of
|
||||||
|
well-known military callsign prefixes (`RCH`, `NATO`). This reliably
|
||||||
|
catches US/NATO transport and tanker traffic — the bulk of what's actually
|
||||||
|
visible on public ADS-B — and under-flags everyone else. Extend
|
||||||
|
`military_ranges.yaml` if you can source ranges/callsigns for other air
|
||||||
|
forces with similar confidence; don't guess, a wrong range misclassifies
|
||||||
|
real civilian flights.
|
||||||
|
|
||||||
## API surface (consumed by the frontend, but usable standalone)
|
## API surface (consumed by the frontend, but usable standalone)
|
||||||
|
|
||||||
|
|
@ -121,11 +153,16 @@ events," not live troop tracking.
|
||||||
- `POST /api/refresh` — force an immediate RSS poll
|
- `POST /api/refresh` — force an immediate RSS poll
|
||||||
- `GET /api/markets/latest`, `GET /api/markets/history?symbol=^GSPC`
|
- `GET /api/markets/latest`, `GET /api/markets/history?symbol=^GSPC`
|
||||||
- `GET /api/markets/spikes?symbol=&hours=` — sudden index/oil moves
|
- `GET /api/markets/spikes?symbol=&hours=` — sudden index/oil moves
|
||||||
(`MARKET_SPIKE_THRESHOLD_PCT`) paired with relevance-ranked articles
|
(`MARKET_SPIKE_THRESHOLD_PCT`), each paired with every article published in
|
||||||
published in the surrounding window (`MARKET_SPIKE_LOOKBACK_HOURS`) — click
|
a full-clock-hour window around it (`MARKET_SPIKE_LOOKBACK_HOURS` sets the
|
||||||
a ticker item in the UI for this. It's a heuristic correlation (keyword +
|
minimum lookback; the window then snaps outward to whole hours). Ranked
|
||||||
country/location matching), not a verified causal link.
|
by relevance (keyword/country matching) but not filtered by it — timing
|
||||||
|
alone qualifies a story as a candidate. Click a ticker item for a quick
|
||||||
|
preview, or the "i" button (top-left) for the full log. Heuristic
|
||||||
|
correlation, not a verified causal link.
|
||||||
- `GET /api/conflict-events?hours=168`
|
- `GET /api/conflict-events?hours=168`
|
||||||
|
- `GET /api/flights` — latest global OpenSky snapshot, with `is_military`
|
||||||
|
per aircraft (see the heuristic caveat above)
|
||||||
- `GET /api/weather/tiles/{layer}/{z}/{x}/{y}.png` — OWM tile proxy
|
- `GET /api/weather/tiles/{layer}/{z}/{x}/{y}.png` — OWM tile proxy
|
||||||
- `GET /api/wikipedia/summary?title=`, `GET /api/wikipedia/search?q=`,
|
- `GET /api/wikipedia/summary?title=`, `GET /api/wikipedia/search?q=`,
|
||||||
`GET /api/wikipedia/parliament?country=`
|
`GET /api/wikipedia/parliament?country=`
|
||||||
|
|
@ -135,4 +172,11 @@ events," not live troop tracking.
|
||||||
- Add/edit RSS sources: `backend/app/sources.yaml`.
|
- Add/edit RSS sources: `backend/app/sources.yaml`.
|
||||||
- Add gazetteer locations: `backend/app/data/gazetteer.csv`.
|
- Add gazetteer locations: `backend/app/data/gazetteer.csv`.
|
||||||
- Add/fix parliament page mappings: `backend/app/data/parliaments.yaml`.
|
- Add/fix parliament page mappings: `backend/app/data/parliaments.yaml`.
|
||||||
|
- Add/fix military ICAO24/callsign ranges: `backend/app/data/military_ranges.yaml`.
|
||||||
- Change poll intervals / clustering window: `.env`.
|
- Change poll intervals / clustering window: `.env`.
|
||||||
|
|
||||||
|
## Deploying as a Proxmox LXC
|
||||||
|
|
||||||
|
See `deploy/lxc/README.md` for a `pct`-based build script that provisions a
|
||||||
|
Debian/Ubuntu/Alpine LXC container running the full stack, and packages it
|
||||||
|
as a reusable `.tar.zst` template.
|
||||||
|
|
|
||||||
|
|
@ -28,6 +28,17 @@ class Settings(BaseSettings):
|
||||||
sources_file: str = str(APP_DIR / "sources.yaml")
|
sources_file: str = str(APP_DIR / "sources.yaml")
|
||||||
gazetteer_file: str = str(APP_DIR / "data" / "gazetteer.csv")
|
gazetteer_file: str = str(APP_DIR / "data" / "gazetteer.csv")
|
||||||
|
|
||||||
|
# --- Flights overlay (OpenSky Network) ---
|
||||||
|
# Anonymous access works out of the box but is capped at 400 credits/day;
|
||||||
|
# a full-globe states/all poll costs several credits, so the default
|
||||||
|
# interval below is deliberately conservative. Free registration
|
||||||
|
# (opensky-network.org) raises the cap to 4000/day — set these and drop
|
||||||
|
# FLIGHTS_POLL_SECONDS if you want closer-to-live updates.
|
||||||
|
opensky_username: str = ""
|
||||||
|
opensky_password: str = ""
|
||||||
|
flights_poll_seconds: int = 300
|
||||||
|
military_ranges_file: str = str(APP_DIR / "data" / "military_ranges.yaml")
|
||||||
|
|
||||||
class Config:
|
class Config:
|
||||||
env_file = ".env"
|
env_file = ".env"
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,32 @@
|
||||||
|
# Best-effort military-aircraft classification for the flights overlay.
|
||||||
|
#
|
||||||
|
# There is no single public, authoritative "is this plane military" flag on
|
||||||
|
# live ADS-B data. This combines two heuristics:
|
||||||
|
#
|
||||||
|
# 1. icao24_prefixes — hex-address blocks that air forces reserve within
|
||||||
|
# their country's ICAO24 allocation. Only the US block below is backed
|
||||||
|
# by a citable source (OpenSky Network's own published research,
|
||||||
|
# "Mode S and ADS-B Usage of Military and other State Aircraft", 2017:
|
||||||
|
# "identifiers used by the US Air Force tend to begin with 'AE'"). Add
|
||||||
|
# more countries' ranges here if you can source them with similar
|
||||||
|
# confidence — do not guess, a wrong range misclassifies real civilian
|
||||||
|
# traffic.
|
||||||
|
# 2. callsign_prefixes — well-known, publicly documented military radio
|
||||||
|
# callsigns. "RCH" (US Air Mobility Command, "Reach") is the big one
|
||||||
|
# for ADS-B-visible traffic; "NATO" covers NATO AWACS/transport flights.
|
||||||
|
#
|
||||||
|
# Net effect: this overlay reliably flags US military and NATO transport/
|
||||||
|
# tanker traffic (the bulk of what's actually visible on public ADS-B — most
|
||||||
|
# other countries' military aircraft either don't broadcast ADS-B or use
|
||||||
|
# civilian-style codes) and will under-flag everyone else. Extend the lists
|
||||||
|
# below as you find sourced ranges/callsigns for other air forces.
|
||||||
|
|
||||||
|
icao24_prefixes:
|
||||||
|
- prefix: "ae"
|
||||||
|
label: "US military (DoD)"
|
||||||
|
|
||||||
|
callsign_prefixes:
|
||||||
|
- prefix: "RCH"
|
||||||
|
label: "US Air Mobility Command (Reach)"
|
||||||
|
- prefix: "NATO"
|
||||||
|
label: "NATO"
|
||||||
|
|
@ -0,0 +1,92 @@
|
||||||
|
"""Live flight overlay, backed by the OpenSky Network REST API.
|
||||||
|
|
||||||
|
Positions are high-churn and ephemeral — unlike articles/markets/conflict
|
||||||
|
events we don't persist history to SQLite, just keep the latest global
|
||||||
|
snapshot in memory and let the scheduler refresh it.
|
||||||
|
|
||||||
|
Military classification is a best-effort heuristic (see
|
||||||
|
data/military_ranges.yaml for sourcing and caveats) — there is no public
|
||||||
|
"is this military" flag on live ADS-B data.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import logging
|
||||||
|
from functools import lru_cache
|
||||||
|
|
||||||
|
import httpx
|
||||||
|
import yaml
|
||||||
|
|
||||||
|
from .config import settings
|
||||||
|
|
||||||
|
log = logging.getLogger("newsatlas.flights")
|
||||||
|
|
||||||
|
STATES_URL = "https://opensky-network.org/api/states/all"
|
||||||
|
|
||||||
|
_latest_flights: list[dict] = []
|
||||||
|
|
||||||
|
|
||||||
|
@lru_cache(maxsize=1)
|
||||||
|
def _military_rules() -> tuple[list[str], list[str]]:
|
||||||
|
with open(settings.military_ranges_file, encoding="utf-8") as f:
|
||||||
|
data = yaml.safe_load(f) or {}
|
||||||
|
icao_prefixes = [r["prefix"].lower() for r in data.get("icao24_prefixes", [])]
|
||||||
|
callsign_prefixes = [r["prefix"].upper() for r in data.get("callsign_prefixes", [])]
|
||||||
|
return icao_prefixes, callsign_prefixes
|
||||||
|
|
||||||
|
|
||||||
|
def _is_military(icao24: str, callsign: str) -> bool:
|
||||||
|
icao_prefixes, callsign_prefixes = _military_rules()
|
||||||
|
icao24 = (icao24 or "").lower()
|
||||||
|
callsign = (callsign or "").strip().upper()
|
||||||
|
if any(icao24.startswith(p) for p in icao_prefixes):
|
||||||
|
return True
|
||||||
|
if any(callsign.startswith(p) for p in callsign_prefixes):
|
||||||
|
return True
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def poll_flights() -> int:
|
||||||
|
global _latest_flights
|
||||||
|
|
||||||
|
auth = None
|
||||||
|
if settings.opensky_username and settings.opensky_password:
|
||||||
|
auth = (settings.opensky_username, settings.opensky_password)
|
||||||
|
|
||||||
|
try:
|
||||||
|
resp = httpx.get(STATES_URL, auth=auth, timeout=30)
|
||||||
|
resp.raise_for_status()
|
||||||
|
payload = resp.json()
|
||||||
|
except Exception:
|
||||||
|
log.exception("Failed to fetch OpenSky states")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
flights = []
|
||||||
|
for s in payload.get("states") or []:
|
||||||
|
icao24, callsign, origin_country = s[0], s[1], s[2]
|
||||||
|
lon, lat, baro_altitude, on_ground = s[5], s[6], s[7], s[8]
|
||||||
|
velocity, heading, vertical_rate = s[9], s[10], s[11]
|
||||||
|
if lat is None or lon is None or on_ground:
|
||||||
|
continue
|
||||||
|
|
||||||
|
callsign = (callsign or "").strip()
|
||||||
|
flights.append(
|
||||||
|
{
|
||||||
|
"icao24": icao24,
|
||||||
|
"callsign": callsign,
|
||||||
|
"origin_country": origin_country,
|
||||||
|
"lat": lat,
|
||||||
|
"lon": lon,
|
||||||
|
"altitude_m": baro_altitude,
|
||||||
|
"velocity_ms": velocity,
|
||||||
|
"heading": heading,
|
||||||
|
"vertical_rate_ms": vertical_rate,
|
||||||
|
"is_military": _is_military(icao24, callsign),
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
_latest_flights = flights
|
||||||
|
log.info("Flight poll complete: %d airborne aircraft (%d flagged military)", len(flights), sum(f["is_military"] for f in flights))
|
||||||
|
return len(flights)
|
||||||
|
|
||||||
|
|
||||||
|
def get_latest_flights() -> list[dict]:
|
||||||
|
return _latest_flights
|
||||||
|
|
@ -13,6 +13,7 @@ from . import wikipedia
|
||||||
from .clustering import build_clusters
|
from .clustering import build_clusters
|
||||||
from .config import settings
|
from .config import settings
|
||||||
from .db import get_session, init_db, SessionLocal
|
from .db import get_session, init_db, SessionLocal
|
||||||
|
from .flights import get_latest_flights
|
||||||
from .ingest import fetch_all
|
from .ingest import fetch_all
|
||||||
from .markets import poll_markets, INSTRUMENTS
|
from .markets import poll_markets, INSTRUMENTS
|
||||||
from .conflict import enabled as conflict_enabled, poll_conflict_events
|
from .conflict import enabled as conflict_enabled, poll_conflict_events
|
||||||
|
|
@ -56,9 +57,17 @@ def public_config():
|
||||||
"weather_enabled": bool(settings.owm_api_key),
|
"weather_enabled": bool(settings.owm_api_key),
|
||||||
"conflict_enabled": conflict_enabled(),
|
"conflict_enabled": conflict_enabled(),
|
||||||
"article_window_hours": settings.article_window_hours,
|
"article_window_hours": settings.article_window_hours,
|
||||||
|
"flights_poll_seconds": settings.flights_poll_seconds,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
# ---- Flights ----------------------------------------------------------
|
||||||
|
|
||||||
|
@app.get("/api/flights")
|
||||||
|
def api_flights():
|
||||||
|
return get_latest_flights()
|
||||||
|
|
||||||
|
|
||||||
# ---- Articles / clusters -----------------------------------------------
|
# ---- Articles / clusters -----------------------------------------------
|
||||||
|
|
||||||
@app.get("/api/clusters")
|
@app.get("/api/clusters")
|
||||||
|
|
@ -174,7 +183,7 @@ def api_markets_history(symbol: str, hours: int = Query(168, le=24 * 30)):
|
||||||
|
|
||||||
|
|
||||||
@app.get("/api/markets/spikes")
|
@app.get("/api/markets/spikes")
|
||||||
def api_markets_spikes(symbol: str | None = None, hours: int = Query(168, le=24 * 30)):
|
def api_markets_spikes(symbol: str | None = None, hours: int = Query(168, le=24 * 365)):
|
||||||
since = dt.datetime.utcnow() - dt.timedelta(hours=hours)
|
since = dt.datetime.utcnow() - dt.timedelta(hours=hours)
|
||||||
session = next(get_session())
|
session = next(get_session())
|
||||||
try:
|
try:
|
||||||
|
|
|
||||||
|
|
@ -100,7 +100,7 @@ def _score_article(article: Article, keywords: list[str], country: str | None) -
|
||||||
return score
|
return score
|
||||||
|
|
||||||
|
|
||||||
def _find_candidate_articles(session: Session, symbol: str, window_start: dt.datetime, window_end: dt.datetime, limit: int = 8) -> list[int]:
|
def _find_candidate_articles(session: Session, symbol: str, window_start: dt.datetime, window_end: dt.datetime, limit: int = 60) -> list[int]:
|
||||||
hints = _SPIKE_HINTS.get(symbol, {"keywords": [], "country": None})
|
hints = _SPIKE_HINTS.get(symbol, {"keywords": [], "country": None})
|
||||||
rows = session.execute(
|
rows = session.execute(
|
||||||
select(Article)
|
select(Article)
|
||||||
|
|
@ -109,12 +109,24 @@ def _find_candidate_articles(session: Session, symbol: str, window_start: dt.dat
|
||||||
.limit(300) # cap the scan; this is a heuristic ranker, not a full-text search index
|
.limit(300) # cap the scan; this is a heuristic ranker, not a full-text search index
|
||||||
).scalars().all()
|
).scalars().all()
|
||||||
|
|
||||||
|
# Every article published in the window is a candidate — timing alone is
|
||||||
|
# "possibly related" for the purposes of this log, not just keyword
|
||||||
|
# matches. Score still decides ordering (keyword/country hits surface
|
||||||
|
# first), it just no longer excludes zero-score articles.
|
||||||
scored = [(_score_article(a, hints["keywords"], hints["country"]), a) for a in rows]
|
scored = [(_score_article(a, hints["keywords"], hints["country"]), a) for a in rows]
|
||||||
scored = [(s, a) for s, a in scored if s > 0]
|
|
||||||
scored.sort(key=lambda pair: (pair[0], pair[1].published_at), reverse=True)
|
scored.sort(key=lambda pair: (pair[0], pair[1].published_at), reverse=True)
|
||||||
return [a.id for _, a in scored[:limit]]
|
return [a.id for _, a in scored[:limit]]
|
||||||
|
|
||||||
|
|
||||||
|
def _floor_hour(t: dt.datetime) -> dt.datetime:
|
||||||
|
return t.replace(minute=0, second=0, microsecond=0)
|
||||||
|
|
||||||
|
|
||||||
|
def _ceil_hour(t: dt.datetime) -> dt.datetime:
|
||||||
|
floored = _floor_hour(t)
|
||||||
|
return floored if floored == t else floored + dt.timedelta(hours=1)
|
||||||
|
|
||||||
|
|
||||||
def _detect_and_record_spike(
|
def _detect_and_record_spike(
|
||||||
session: Session, symbol: str, label: str, prev: MarketPrice | None, price: float, recorded_at: dt.datetime
|
session: Session, symbol: str, label: str, prev: MarketPrice | None, price: float, recorded_at: dt.datetime
|
||||||
) -> None:
|
) -> None:
|
||||||
|
|
@ -124,8 +136,11 @@ def _detect_and_record_spike(
|
||||||
if abs(pct_change) < settings.market_spike_threshold_pct:
|
if abs(pct_change) < settings.market_spike_threshold_pct:
|
||||||
return
|
return
|
||||||
|
|
||||||
window_start = prev.recorded_at - dt.timedelta(hours=settings.market_spike_lookback_hours)
|
# Snapped to whole clock hours so the log reads as clean ranges (e.g.
|
||||||
article_ids = _find_candidate_articles(session, symbol, window_start, recorded_at)
|
# "14:00-16:00"), covering at least the full hour the spike landed in.
|
||||||
|
window_start = _floor_hour(prev.recorded_at - dt.timedelta(hours=settings.market_spike_lookback_hours))
|
||||||
|
window_end = _ceil_hour(recorded_at)
|
||||||
|
article_ids = _find_candidate_articles(session, symbol, window_start, window_end)
|
||||||
|
|
||||||
session.add(
|
session.add(
|
||||||
MarketSpike(
|
MarketSpike(
|
||||||
|
|
@ -135,7 +150,7 @@ def _detect_and_record_spike(
|
||||||
to_price=price,
|
to_price=price,
|
||||||
pct_change=round(pct_change, 3),
|
pct_change=round(pct_change, 3),
|
||||||
window_start=window_start,
|
window_start=window_start,
|
||||||
window_end=recorded_at,
|
window_end=window_end,
|
||||||
detected_at=recorded_at,
|
detected_at=recorded_at,
|
||||||
article_ids_json=json.dumps(article_ids),
|
article_ids_json=json.dumps(article_ids),
|
||||||
)
|
)
|
||||||
|
|
|
||||||
|
|
@ -5,6 +5,7 @@ from apscheduler.schedulers.background import BackgroundScheduler
|
||||||
from .conflict import poll_conflict_events
|
from .conflict import poll_conflict_events
|
||||||
from .config import settings
|
from .config import settings
|
||||||
from .db import SessionLocal
|
from .db import SessionLocal
|
||||||
|
from .flights import poll_flights
|
||||||
from .ingest import fetch_all
|
from .ingest import fetch_all
|
||||||
from .markets import poll_markets
|
from .markets import poll_markets
|
||||||
|
|
||||||
|
|
@ -38,11 +39,17 @@ def _run_conflict_job() -> None:
|
||||||
session.close()
|
session.close()
|
||||||
|
|
||||||
|
|
||||||
|
def _run_flights_job() -> None:
|
||||||
|
count = poll_flights()
|
||||||
|
log.info("Flight poll complete: %d aircraft", count)
|
||||||
|
|
||||||
|
|
||||||
def start_scheduler() -> BackgroundScheduler:
|
def start_scheduler() -> BackgroundScheduler:
|
||||||
scheduler = BackgroundScheduler(timezone="UTC")
|
scheduler = BackgroundScheduler(timezone="UTC")
|
||||||
scheduler.add_job(_run_rss_job, "interval", minutes=settings.rss_poll_minutes, next_run_time=None)
|
scheduler.add_job(_run_rss_job, "interval", minutes=settings.rss_poll_minutes, next_run_time=None)
|
||||||
scheduler.add_job(_run_market_job, "interval", minutes=settings.market_poll_minutes, next_run_time=None)
|
scheduler.add_job(_run_market_job, "interval", minutes=settings.market_poll_minutes, next_run_time=None)
|
||||||
scheduler.add_job(_run_conflict_job, "interval", minutes=settings.conflict_poll_minutes, next_run_time=None)
|
scheduler.add_job(_run_conflict_job, "interval", minutes=settings.conflict_poll_minutes, next_run_time=None)
|
||||||
|
scheduler.add_job(_run_flights_job, "interval", seconds=settings.flights_poll_seconds, next_run_time=None)
|
||||||
scheduler.start()
|
scheduler.start()
|
||||||
|
|
||||||
# Kick off an immediate first run of each job in the background so the
|
# Kick off an immediate first run of each job in the background so the
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,92 @@
|
||||||
|
# NewsAtlas — Proxmox LXC deployment
|
||||||
|
|
||||||
|
Builds a Proxmox LXC container running the full NewsAtlas stack (Docker +
|
||||||
|
docker-compose inside the CT), and packages it as a reusable `.tar.zst`
|
||||||
|
template. Supports Debian, Ubuntu, and Alpine base images.
|
||||||
|
|
||||||
|
> **Not executed against a real Proxmox host while writing this** — there
|
||||||
|
> wasn't one available in that environment. The docker-compose stack itself
|
||||||
|
> (`../../docker-compose.yml`) *has* been built and run end-to-end; this LXC
|
||||||
|
> layer just automates `pct`/`pveam`/`vzdump` calls around it. Run
|
||||||
|
> `./build.sh <target> --dry-run` first and read the printed commands before
|
||||||
|
> running for real.
|
||||||
|
|
||||||
|
## Security model — read before using
|
||||||
|
|
||||||
|
`build.sh` writes real secrets (whatever you put in `prebuild-config.json`)
|
||||||
|
into `/opt/newsatlas/.env` **inside the container's filesystem**, and that
|
||||||
|
filesystem gets packaged into the `.tar.zst` template. This was a deliberate
|
||||||
|
choice — the safer default (inject secrets only into the running container,
|
||||||
|
never into a distributable template) was offered and explicitly declined in
|
||||||
|
favor of "clone-and-go" templates that boot with everything already
|
||||||
|
configured.
|
||||||
|
|
||||||
|
The trade-off that comes with that: **every copy of the resulting
|
||||||
|
`.tar.zst` is a credentials-bearing artifact**, for every distro variant you
|
||||||
|
build. Concretely:
|
||||||
|
|
||||||
|
- Keep template files (`local:vztmpl/...` / your `vzdump` dump directory)
|
||||||
|
on storage only trusted operators can read — never a public/shared LXC
|
||||||
|
template store. The whole point of a reusable template is that it gets
|
||||||
|
cloned, and each clone carries the same baked-in credentials.
|
||||||
|
- If a copy of the tarball is ever backed up somewhere less trusted, copied
|
||||||
|
off-host, or shared, **rotate every key in `prebuild-config.json`**
|
||||||
|
(OpenWeatherMap, ACLED, OpenSky) — there's no way to invalidate just the
|
||||||
|
copy that leaked.
|
||||||
|
- `prebuild-config.json` itself is gitignored; don't commit it, don't paste
|
||||||
|
it into issues/chat.
|
||||||
|
|
||||||
|
If you'd rather not carry that risk, don't use this path — run
|
||||||
|
`docker compose up --build` directly on any host (see the root README) and
|
||||||
|
manage `.env` there normally; nothing gets baked into a shareable artifact.
|
||||||
|
|
||||||
|
## Setup
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cp prebuild-config.json.template prebuild-config.json
|
||||||
|
$EDITOR prebuild-config.json # fill in secrets, LXC sizing, network, etc.
|
||||||
|
```
|
||||||
|
|
||||||
|
Key fields:
|
||||||
|
|
||||||
|
- `app.repo_url` / `app.branch` — where `provision.sh` clones the app from
|
||||||
|
inside the container. Defaults to this repo's own remote.
|
||||||
|
- `secrets.*` — same keys as the root `.env.example`, baked in as described
|
||||||
|
above.
|
||||||
|
- `lxc.*` — sizing (`cores`, `memory_mb`, `disk_gb`), Proxmox storage IDs
|
||||||
|
(`storage` for the rootfs, `template_storage` for both the base template
|
||||||
|
and the packaged output), networking (`bridge`, `vlan`, `ip`, `gateway`),
|
||||||
|
and either `root_password` or `ssh_public_key_file` for access.
|
||||||
|
- `distros.*` — the `pveam available` template-name prefix used to resolve
|
||||||
|
the newest matching template for each distro. Check `pveam available`
|
||||||
|
on your host if these drift.
|
||||||
|
|
||||||
|
## Building
|
||||||
|
|
||||||
|
Run as root on the Proxmox VE host:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./build.sh debian # one distro
|
||||||
|
./build.sh all # debian + ubuntu + alpine
|
||||||
|
./build.sh ubuntu --dry-run # print the pct/pveam/vzdump commands, run nothing
|
||||||
|
./build.sh alpine --no-keep # destroy the scratch CT after packaging (template file only)
|
||||||
|
./build.sh debian --no-template # deploy and leave the CT running; skip packaging
|
||||||
|
```
|
||||||
|
|
||||||
|
Each run: allocates the next free CTID, creates the CT with `nesting=1`
|
||||||
|
(required to run Docker inside an LXC), installs Docker via `provision.sh`,
|
||||||
|
clones the app, pushes the generated `.env`, runs `docker compose up
|
||||||
|
--build`, waits for `/api/health`, then (unless `--no-template`) stops the
|
||||||
|
CT and runs `vzdump --mode stop --compress zstd` to produce the template,
|
||||||
|
restarting the CT afterward (unless `--no-keep`).
|
||||||
|
|
||||||
|
## Redeploying from a packaged template
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pct restore <newid> <template_storage>:backup/vzdump-lxc-<ctid>-*.tar.zst --storage <storage>
|
||||||
|
pct start <newid>
|
||||||
|
```
|
||||||
|
|
||||||
|
The container comes up with the app already running and `.env` already in
|
||||||
|
place — no further provisioning needed (that's the whole point, and also
|
||||||
|
exactly why the security note above matters).
|
||||||
|
|
@ -0,0 +1,248 @@
|
||||||
|
#!/bin/bash
|
||||||
|
# Builds a NewsAtlas LXC container on a Proxmox VE host: creates the CT,
|
||||||
|
# installs Docker inside it, deploys the app, bakes secrets/config from
|
||||||
|
# prebuild-config.json into it, and (by default) packages the result as a
|
||||||
|
# reusable .tar.zst template via vzdump.
|
||||||
|
#
|
||||||
|
# Run this ON the Proxmox VE host, as root. It has NOT been executed against
|
||||||
|
# a real Proxmox host while developing this (no such host is available in
|
||||||
|
# that environment) — read it before running, and use --dry-run first.
|
||||||
|
#
|
||||||
|
# SECURITY: the packaged template tarball contains the real values from
|
||||||
|
# prebuild-config.json (API keys, etc.) baked into /opt/newsatlas/.env
|
||||||
|
# inside its filesystem. That was an explicit choice (see deploy/lxc/README.md)
|
||||||
|
# over the safer default of injecting secrets at deploy time — treat every
|
||||||
|
# copy of the resulting .tar.zst as a credentials-bearing artifact: keep it
|
||||||
|
# on trusted storage only, never upload it to a public template store, and
|
||||||
|
# rotate the keys if a copy ever leaves your control.
|
||||||
|
#
|
||||||
|
# Usage:
|
||||||
|
# ./build.sh <debian|ubuntu|alpine|all> [--dry-run] [--no-keep] [--no-template]
|
||||||
|
#
|
||||||
|
# --dry-run print every pct/pveam/vzdump command instead of running it
|
||||||
|
# --no-keep destroy the scratch CT after packaging (template only)
|
||||||
|
# --no-template deploy and leave the CT running; skip vzdump packaging
|
||||||
|
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
CONFIG_FILE="$SCRIPT_DIR/prebuild-config.json"
|
||||||
|
PROVISION_SCRIPT="$SCRIPT_DIR/provision.sh"
|
||||||
|
APP_DIR_IN_CT="/opt/newsatlas"
|
||||||
|
|
||||||
|
DRY_RUN=0
|
||||||
|
KEEP_CT=1
|
||||||
|
MAKE_TEMPLATE=1
|
||||||
|
TARGET="${1:-}"
|
||||||
|
shift || true
|
||||||
|
for arg in "$@"; do
|
||||||
|
case "$arg" in
|
||||||
|
--dry-run) DRY_RUN=1 ;;
|
||||||
|
--no-keep) KEEP_CT=0 ;;
|
||||||
|
--no-template) MAKE_TEMPLATE=0 ;;
|
||||||
|
*) echo "Unknown flag: $arg" >&2; exit 1 ;;
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
|
||||||
|
run() {
|
||||||
|
if [ "$DRY_RUN" = "1" ]; then
|
||||||
|
printf '[dry-run]'
|
||||||
|
printf ' %q' "$@"
|
||||||
|
printf '\n'
|
||||||
|
else
|
||||||
|
"$@"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---- preflight --------------------------------------------------------
|
||||||
|
|
||||||
|
if [ "$(id -u)" != "0" ]; then
|
||||||
|
echo "build.sh must run as root on the Proxmox VE host." >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
for bin in pct pveam pvesh jq; do
|
||||||
|
if ! command -v "$bin" >/dev/null 2>&1; then
|
||||||
|
echo "Missing required command: $bin (this must run on a Proxmox VE host with jq installed)." >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
if [ "$MAKE_TEMPLATE" = "1" ] && ! command -v vzdump >/dev/null 2>&1; then
|
||||||
|
echo "Missing required command: vzdump (needed to package a template; pass --no-template to skip)." >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [ ! -f "$CONFIG_FILE" ]; then
|
||||||
|
cat >&2 <<EOF
|
||||||
|
Missing $CONFIG_FILE.
|
||||||
|
Copy prebuild-config.json.template to prebuild-config.json and fill it in:
|
||||||
|
cp "$SCRIPT_DIR/prebuild-config.json.template" "$CONFIG_FILE"
|
||||||
|
EOF
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
case "$TARGET" in
|
||||||
|
debian|ubuntu|alpine|all) ;;
|
||||||
|
*)
|
||||||
|
echo "Usage: $0 <debian|ubuntu|alpine|all> [--dry-run] [--no-keep] [--no-template]" >&2
|
||||||
|
exit 1
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
|
||||||
|
jqc() { jq -r "$1" "$CONFIG_FILE"; }
|
||||||
|
|
||||||
|
REPO_URL=$(jqc '.app.repo_url')
|
||||||
|
BRANCH=$(jqc '.app.branch')
|
||||||
|
HTTP_PORT=$(jqc '.app.http_port')
|
||||||
|
|
||||||
|
HOSTNAME_BASE=$(jqc '.lxc.hostname')
|
||||||
|
CORES=$(jqc '.lxc.cores')
|
||||||
|
MEMORY_MB=$(jqc '.lxc.memory_mb')
|
||||||
|
SWAP_MB=$(jqc '.lxc.swap_mb')
|
||||||
|
DISK_GB=$(jqc '.lxc.disk_gb')
|
||||||
|
STORAGE=$(jqc '.lxc.storage')
|
||||||
|
TEMPLATE_STORAGE=$(jqc '.lxc.template_storage')
|
||||||
|
BRIDGE=$(jqc '.lxc.bridge')
|
||||||
|
VLAN=$(jqc '.lxc.vlan')
|
||||||
|
IP_CFG=$(jqc '.lxc.ip')
|
||||||
|
GATEWAY=$(jqc '.lxc.gateway')
|
||||||
|
UNPRIVILEGED=$(jqc '.lxc.unprivileged')
|
||||||
|
START_ON_BOOT=$(jqc '.lxc.start_on_boot')
|
||||||
|
ROOT_PASSWORD=$(jqc '.lxc.root_password')
|
||||||
|
SSH_KEY_FILE=$(jqc '.lxc.ssh_public_key_file')
|
||||||
|
|
||||||
|
if [ "$TARGET" = "all" ]; then
|
||||||
|
DISTROS="debian ubuntu alpine"
|
||||||
|
else
|
||||||
|
DISTROS="$TARGET"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# ---- per-distro build ---------------------------------------------------
|
||||||
|
|
||||||
|
build_one() {
|
||||||
|
local distro="$1"
|
||||||
|
local template_prefix ctid template hostname net0 rootfs features
|
||||||
|
local unpriv_flag=""
|
||||||
|
[ "$UNPRIVILEGED" = "true" ] && unpriv_flag="1" || unpriv_flag="0"
|
||||||
|
|
||||||
|
template_prefix=$(jqc ".distros.${distro}")
|
||||||
|
if [ -z "$template_prefix" ] || [ "$template_prefix" = "null" ]; then
|
||||||
|
echo "No distros.${distro} entry in prebuild-config.json, skipping." >&2
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "=== Building NewsAtlas LXC: $distro ==="
|
||||||
|
|
||||||
|
echo "-- Resolving template for prefix '$template_prefix'..."
|
||||||
|
pveam update >/dev/null 2>&1 || true
|
||||||
|
template=$(pveam available --section system 2>/dev/null | awk '{print $2}' | grep "^${template_prefix}" | sort -V | tail -1)
|
||||||
|
if [ -z "$template" ]; then
|
||||||
|
echo "Could not find a template matching '${template_prefix}' via 'pveam available'. Check prebuild-config.json's distros.${distro} value against 'pveam available'." >&2
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
echo "-- Using template: $template"
|
||||||
|
|
||||||
|
if ! pveam list "$TEMPLATE_STORAGE" 2>/dev/null | grep -q "$template"; then
|
||||||
|
run pveam download "$TEMPLATE_STORAGE" "$template"
|
||||||
|
fi
|
||||||
|
|
||||||
|
ctid=$(pvesh get /cluster/nextid)
|
||||||
|
hostname="${HOSTNAME_BASE}-${distro}"
|
||||||
|
rootfs="${STORAGE}:${DISK_GB}"
|
||||||
|
features="nesting=1,keyctl=1" # required for Docker-in-LXC
|
||||||
|
|
||||||
|
net0="name=eth0,bridge=${BRIDGE},ip=${IP_CFG}"
|
||||||
|
[ -n "$VLAN" ] && net0="${net0},tag=${VLAN}"
|
||||||
|
[ "$IP_CFG" != "dhcp" ] && [ -n "$GATEWAY" ] && net0="${net0},gw=${GATEWAY}"
|
||||||
|
|
||||||
|
echo "-- Creating CT $ctid ($hostname)..."
|
||||||
|
create_args=(pct create "$ctid" "${TEMPLATE_STORAGE}:vztmpl/${template}"
|
||||||
|
--hostname "$hostname"
|
||||||
|
--cores "$CORES"
|
||||||
|
--memory "$MEMORY_MB"
|
||||||
|
--swap "$SWAP_MB"
|
||||||
|
--rootfs "$rootfs"
|
||||||
|
--net0 "$net0"
|
||||||
|
--unprivileged "$unpriv_flag"
|
||||||
|
--features "$features"
|
||||||
|
--onboot "$([ "$START_ON_BOOT" = "true" ] && echo 1 || echo 0)"
|
||||||
|
)
|
||||||
|
if [ -n "$ROOT_PASSWORD" ]; then
|
||||||
|
create_args+=(--password "$ROOT_PASSWORD")
|
||||||
|
fi
|
||||||
|
if [ -n "$SSH_KEY_FILE" ] && [ -f "$SSH_KEY_FILE" ]; then
|
||||||
|
create_args+=(--ssh-public-keys "$SSH_KEY_FILE")
|
||||||
|
fi
|
||||||
|
run "${create_args[@]}"
|
||||||
|
|
||||||
|
run pct start "$ctid"
|
||||||
|
|
||||||
|
if [ "$DRY_RUN" != "1" ]; then
|
||||||
|
echo "-- Waiting for CT $ctid to come up..."
|
||||||
|
for i in $(seq 1 30); do
|
||||||
|
pct exec "$ctid" -- true 2>/dev/null && break
|
||||||
|
sleep 2
|
||||||
|
[ "$i" = "30" ] && { echo "CT $ctid did not become ready in time." >&2; return 1; }
|
||||||
|
done
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "-- Provisioning (installing Docker, cloning repo)..."
|
||||||
|
run pct push "$ctid" "$PROVISION_SCRIPT" /root/provision.sh --perms 0755
|
||||||
|
run pct exec "$ctid" -- /root/provision.sh "$REPO_URL" "$BRANCH" "$APP_DIR_IN_CT"
|
||||||
|
|
||||||
|
echo "-- Writing .env (baking config from prebuild-config.json)..."
|
||||||
|
local envfile
|
||||||
|
envfile=$(mktemp)
|
||||||
|
{
|
||||||
|
echo "HTTP_PORT=${HTTP_PORT}"
|
||||||
|
echo "OWM_API_KEY=$(jqc '.secrets.owm_api_key')"
|
||||||
|
echo "ACLED_API_KEY=$(jqc '.secrets.acled_api_key')"
|
||||||
|
echo "ACLED_EMAIL=$(jqc '.secrets.acled_email')"
|
||||||
|
echo "OPENSKY_USERNAME=$(jqc '.secrets.opensky_username')"
|
||||||
|
echo "OPENSKY_PASSWORD=$(jqc '.secrets.opensky_password')"
|
||||||
|
echo "RSS_POLL_MINUTES=$(jqc '.polling.rss_poll_minutes')"
|
||||||
|
echo "MARKET_POLL_MINUTES=$(jqc '.polling.market_poll_minutes')"
|
||||||
|
echo "MARKET_SPIKE_THRESHOLD_PCT=$(jqc '.polling.market_spike_threshold_pct')"
|
||||||
|
echo "MARKET_SPIKE_LOOKBACK_HOURS=$(jqc '.polling.market_spike_lookback_hours')"
|
||||||
|
echo "CONFLICT_POLL_MINUTES=$(jqc '.polling.conflict_poll_minutes')"
|
||||||
|
echo "FLIGHTS_POLL_SECONDS=$(jqc '.polling.flights_poll_seconds')"
|
||||||
|
echo "ARTICLE_WINDOW_HOURS=$(jqc '.polling.article_window_hours')"
|
||||||
|
echo "DATABASE_PATH=/data/newsatlas.db"
|
||||||
|
} > "$envfile"
|
||||||
|
run pct push "$ctid" "$envfile" "${APP_DIR_IN_CT}/.env" --perms 0600
|
||||||
|
rm -f "$envfile"
|
||||||
|
|
||||||
|
echo "-- Starting the stack..."
|
||||||
|
run pct exec "$ctid" -- bash -c "cd ${APP_DIR_IN_CT} && docker compose up -d --build"
|
||||||
|
|
||||||
|
if [ "$DRY_RUN" != "1" ]; then
|
||||||
|
echo "-- Waiting for the app to answer /api/health..."
|
||||||
|
for i in $(seq 1 30); do
|
||||||
|
pct exec "$ctid" -- curl -sf "http://localhost:${HTTP_PORT}/api/health" >/dev/null 2>&1 && break
|
||||||
|
sleep 3
|
||||||
|
[ "$i" = "30" ] && echo "Warning: /api/health didn't respond in time; check 'pct exec $ctid -- docker compose -f ${APP_DIR_IN_CT}/docker-compose.yml logs'." >&2
|
||||||
|
done
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [ "$MAKE_TEMPLATE" = "1" ]; then
|
||||||
|
echo "-- Stopping CT $ctid to package as a template..."
|
||||||
|
run pct shutdown "$ctid" --timeout 60
|
||||||
|
echo "-- Packaging via vzdump..."
|
||||||
|
run vzdump "$ctid" --mode stop --compress zstd --storage "$TEMPLATE_STORAGE"
|
||||||
|
if [ "$KEEP_CT" = "1" ]; then
|
||||||
|
run pct start "$ctid"
|
||||||
|
echo "-- CT $ctid restarted (kept running in addition to the packaged template)."
|
||||||
|
else
|
||||||
|
run pct destroy "$ctid"
|
||||||
|
echo "-- CT $ctid destroyed after packaging (--no-keep)."
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "=== Done: $distro (CTID $ctid, hostname $hostname) ==="
|
||||||
|
echo " Reminder: any packaged .tar.zst for this CT has live secrets baked in — see deploy/lxc/README.md."
|
||||||
|
}
|
||||||
|
|
||||||
|
for d in $DISTROS; do
|
||||||
|
build_one "$d"
|
||||||
|
done
|
||||||
|
|
@ -0,0 +1,53 @@
|
||||||
|
{
|
||||||
|
"_comment": "Copy this file to prebuild-config.json (gitignored) and fill it in. build.sh reads secrets/config from prebuild-config.json only — never from this template.",
|
||||||
|
|
||||||
|
"app": {
|
||||||
|
"repo_url": "https://git.abdelbaki.eu/The_miro/NewsAtlas.git",
|
||||||
|
"branch": "main",
|
||||||
|
"http_port": 8080
|
||||||
|
},
|
||||||
|
|
||||||
|
"secrets": {
|
||||||
|
"_comment": "Written into /opt/newsatlas/.env inside the container (mode 600, root-only) and baked into the packaged template tarball. Anyone with a copy of that tarball has these values — see deploy/lxc/README.md before sharing/backing one up anywhere untrusted.",
|
||||||
|
"owm_api_key": "",
|
||||||
|
"acled_api_key": "",
|
||||||
|
"acled_email": "",
|
||||||
|
"opensky_username": "",
|
||||||
|
"opensky_password": ""
|
||||||
|
},
|
||||||
|
|
||||||
|
"polling": {
|
||||||
|
"rss_poll_minutes": 10,
|
||||||
|
"market_poll_minutes": 15,
|
||||||
|
"market_spike_threshold_pct": 1.5,
|
||||||
|
"market_spike_lookback_hours": 6,
|
||||||
|
"conflict_poll_minutes": 60,
|
||||||
|
"flights_poll_seconds": 300,
|
||||||
|
"article_window_hours": 72
|
||||||
|
},
|
||||||
|
|
||||||
|
"lxc": {
|
||||||
|
"hostname": "newsatlas",
|
||||||
|
"cores": 2,
|
||||||
|
"memory_mb": 2048,
|
||||||
|
"swap_mb": 512,
|
||||||
|
"disk_gb": 8,
|
||||||
|
"storage": "local-lvm",
|
||||||
|
"template_storage": "local",
|
||||||
|
"bridge": "vmbr0",
|
||||||
|
"vlan": "",
|
||||||
|
"ip": "dhcp",
|
||||||
|
"gateway": "",
|
||||||
|
"unprivileged": true,
|
||||||
|
"start_on_boot": true,
|
||||||
|
"root_password": "",
|
||||||
|
"ssh_public_key_file": ""
|
||||||
|
},
|
||||||
|
|
||||||
|
"distros": {
|
||||||
|
"_comment": "Which base-OS variants build.sh should build when run with --all. Each maps to a `pveam available` template name prefix; build.sh picks the newest matching version.",
|
||||||
|
"debian": "debian-12-standard",
|
||||||
|
"ubuntu": "ubuntu-24.04-standard",
|
||||||
|
"alpine": "alpine-3.19-default"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,48 @@
|
||||||
|
#!/bin/bash
|
||||||
|
# Runs INSIDE the LXC container as root (pushed + executed by build.sh).
|
||||||
|
# Installs Docker and clones NewsAtlas. Deliberately does NOT touch .env or
|
||||||
|
# start the stack — build.sh pushes the generated .env and starts the stack
|
||||||
|
# as separate steps afterward, so a plain `git clone` here never collides
|
||||||
|
# with an already-placed .env file.
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
REPO_URL="${1:?usage: provision.sh <repo_url> <branch> <app_dir>}"
|
||||||
|
BRANCH="${2:?usage: provision.sh <repo_url> <branch> <app_dir>}"
|
||||||
|
APP_DIR="${3:?usage: provision.sh <repo_url> <branch> <app_dir>}"
|
||||||
|
|
||||||
|
. /etc/os-release
|
||||||
|
|
||||||
|
install_docker_apt() {
|
||||||
|
export DEBIAN_FRONTEND=noninteractive
|
||||||
|
apt-get update
|
||||||
|
apt-get install -y ca-certificates curl gnupg git jq
|
||||||
|
curl -fsSL https://get.docker.com | sh
|
||||||
|
systemctl enable docker
|
||||||
|
systemctl start docker
|
||||||
|
}
|
||||||
|
|
||||||
|
install_docker_apk() {
|
||||||
|
apk update
|
||||||
|
apk add --no-cache docker docker-cli-compose git jq openrc
|
||||||
|
rc-update add docker default
|
||||||
|
service docker start
|
||||||
|
}
|
||||||
|
|
||||||
|
case "$ID" in
|
||||||
|
debian|ubuntu) install_docker_apt ;;
|
||||||
|
alpine) install_docker_apk ;;
|
||||||
|
*)
|
||||||
|
echo "provision.sh: unsupported distro '$ID' (expected debian, ubuntu, or alpine)" >&2
|
||||||
|
exit 1
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
|
||||||
|
if [ -d "$APP_DIR/.git" ]; then
|
||||||
|
git -C "$APP_DIR" fetch origin "$BRANCH"
|
||||||
|
git -C "$APP_DIR" checkout "$BRANCH"
|
||||||
|
git -C "$APP_DIR" reset --hard "origin/$BRANCH"
|
||||||
|
else
|
||||||
|
git clone --branch "$BRANCH" --depth 1 "$REPO_URL" "$APP_DIR"
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "provision.sh: done. Docker installed, repo at $APP_DIR."
|
||||||
|
|
@ -10,6 +10,7 @@ services:
|
||||||
- ./backend/app/sources.yaml:/app/app/sources.yaml:ro
|
- ./backend/app/sources.yaml:/app/app/sources.yaml:ro
|
||||||
- ./backend/app/data/gazetteer.csv:/app/app/data/gazetteer.csv:ro
|
- ./backend/app/data/gazetteer.csv:/app/app/data/gazetteer.csv:ro
|
||||||
- ./backend/app/data/parliaments.yaml:/app/app/data/parliaments.yaml:ro
|
- ./backend/app/data/parliaments.yaml:/app/app/data/parliaments.yaml:ro
|
||||||
|
- ./backend/app/data/military_ranges.yaml:/app/app/data/military_ranges.yaml:ro
|
||||||
expose:
|
expose:
|
||||||
- "8000"
|
- "8000"
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -21,6 +21,7 @@
|
||||||
--c-dark-soft: #9d7cff; /* lightened violet for small text/links on dark bg */
|
--c-dark-soft: #9d7cff; /* lightened violet for small text/links on dark bg */
|
||||||
--c-red: #f50505; /* danger / down */
|
--c-red: #f50505; /* danger / down */
|
||||||
--c-green: #34d399; /* gain — kept distinct from theme for finance semantics */
|
--c-green: #34d399; /* gain — kept distinct from theme for finance semantics */
|
||||||
|
--c-magenta: #ff33ff; /* flagged-military flights — must match MILITARY_FLIGHT_HEX in app.js */
|
||||||
|
|
||||||
--c-panel: rgba(24, 15, 33, 0.97);
|
--c-panel: rgba(24, 15, 33, 0.97);
|
||||||
--c-panel-border: #3a2159;
|
--c-panel-border: #3a2159;
|
||||||
|
|
@ -60,6 +61,8 @@ html, body {
|
||||||
z-index: 30;
|
z-index: 30;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#topbarLeft { display: flex; align-items: center; }
|
||||||
|
|
||||||
#topbar h1 {
|
#topbar h1 {
|
||||||
font-size: 18px;
|
font-size: 18px;
|
||||||
letter-spacing: 0.08em;
|
letter-spacing: 0.08em;
|
||||||
|
|
@ -218,7 +221,7 @@ html, body {
|
||||||
|
|
||||||
.modal { position: absolute; inset: 0; z-index: 40; display: flex; align-items: center; justify-content: center; }
|
.modal { position: absolute; inset: 0; z-index: 40; display: flex; align-items: center; justify-content: center; }
|
||||||
.modal.hidden { display: none; }
|
.modal.hidden { display: none; }
|
||||||
#modalBackdrop { position: absolute; inset: 0; background: rgba(10, 0, 20, 0.65); }
|
.modal-backdrop { position: absolute; inset: 0; background: rgba(10, 0, 20, 0.65); }
|
||||||
.modal-card {
|
.modal-card {
|
||||||
position: relative;
|
position: relative;
|
||||||
width: 420px;
|
width: 420px;
|
||||||
|
|
@ -231,7 +234,8 @@ html, body {
|
||||||
padding: 20px 18px;
|
padding: 20px 18px;
|
||||||
box-shadow: 0 12px 40px rgba(0,0,0,0.6), 0 0 60px rgba(80, 24, 221, 0.15);
|
box-shadow: 0 12px 40px rgba(0,0,0,0.6), 0 0 60px rgba(80, 24, 221, 0.15);
|
||||||
}
|
}
|
||||||
#closeModal {
|
.modal-card.wide { width: 760px; max-height: 88vh; }
|
||||||
|
.modal-close {
|
||||||
position: absolute;
|
position: absolute;
|
||||||
top: 8px;
|
top: 8px;
|
||||||
right: 10px;
|
right: 10px;
|
||||||
|
|
@ -241,7 +245,37 @@ html, body {
|
||||||
font-size: 22px;
|
font-size: 22px;
|
||||||
cursor: pointer;
|
cursor: pointer;
|
||||||
}
|
}
|
||||||
#closeModal:hover { color: var(--c-highlight-soft); }
|
.modal-close:hover { color: var(--c-highlight-soft); }
|
||||||
|
|
||||||
|
#infoBtn {
|
||||||
|
width: 26px;
|
||||||
|
height: 26px;
|
||||||
|
border-radius: 50%;
|
||||||
|
background: var(--c-tag-bg);
|
||||||
|
border: 1px solid var(--c-panel-border);
|
||||||
|
color: var(--c-dark-soft);
|
||||||
|
font-size: 13px;
|
||||||
|
font-weight: 700;
|
||||||
|
font-style: italic;
|
||||||
|
font-family: Georgia, "Times New Roman", serif;
|
||||||
|
cursor: pointer;
|
||||||
|
margin-left: 4px;
|
||||||
|
}
|
||||||
|
#infoBtn:hover { background: #3d2260; color: #fff; }
|
||||||
|
|
||||||
|
.econ-select {
|
||||||
|
background: #1c1030;
|
||||||
|
color: var(--c-text);
|
||||||
|
border: 1px solid var(--c-panel-border);
|
||||||
|
border-radius: 4px;
|
||||||
|
font-size: 12px;
|
||||||
|
padding: 4px 6px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.econ-incident { padding: 14px 0; border-bottom: 1px solid #241533; }
|
||||||
|
.econ-incident h3 { margin: 0 0 4px; font-size: 14px; color: #f2dede; }
|
||||||
|
.econ-incident .window { font-size: 11px; color: var(--c-text-muted); margin-bottom: 8px; }
|
||||||
|
.econ-incident .articles { margin-top: 8px; }
|
||||||
|
|
||||||
.panel-section { margin-bottom: 20px; }
|
.panel-section { margin-bottom: 20px; }
|
||||||
.panel-section h2 {
|
.panel-section h2 {
|
||||||
|
|
@ -271,6 +305,23 @@ html, body {
|
||||||
|
|
||||||
/* ---- wikipedia text-selection popup ---- */
|
/* ---- wikipedia text-selection popup ---- */
|
||||||
|
|
||||||
|
#flightTooltip {
|
||||||
|
position: absolute;
|
||||||
|
z-index: 45;
|
||||||
|
background: #1c1030;
|
||||||
|
border: 1px solid var(--c-panel-border);
|
||||||
|
border-radius: 6px;
|
||||||
|
padding: 6px 9px;
|
||||||
|
box-shadow: 0 4px 16px rgba(0,0,0,0.5);
|
||||||
|
font-size: 11px;
|
||||||
|
line-height: 1.5;
|
||||||
|
pointer-events: none;
|
||||||
|
max-width: 220px;
|
||||||
|
}
|
||||||
|
#flightTooltip.hidden { display: none; }
|
||||||
|
#flightTooltip b { color: var(--c-text); }
|
||||||
|
#flightTooltip .mil { color: var(--c-magenta); font-weight: 600; }
|
||||||
|
|
||||||
#wikiSelectPopup {
|
#wikiSelectPopup {
|
||||||
position: absolute;
|
position: absolute;
|
||||||
z-index: 50;
|
z-index: 50;
|
||||||
|
|
|
||||||
|
|
@ -10,7 +10,10 @@
|
||||||
<div id="globeViz"></div>
|
<div id="globeViz"></div>
|
||||||
|
|
||||||
<header id="topbar">
|
<header id="topbar">
|
||||||
|
<div id="topbarLeft">
|
||||||
<h1>NewsAtlas</h1>
|
<h1>NewsAtlas</h1>
|
||||||
|
<button id="infoBtn" title="Economic Incident History">i</button>
|
||||||
|
</div>
|
||||||
<div id="layerToggles">
|
<div id="layerToggles">
|
||||||
<label><input type="checkbox" id="toggleWeather" /> Weather</label>
|
<label><input type="checkbox" id="toggleWeather" /> Weather</label>
|
||||||
<select id="weatherLayer">
|
<select id="weatherLayer">
|
||||||
|
|
@ -21,6 +24,9 @@
|
||||||
<option value="pressure">Pressure</option>
|
<option value="pressure">Pressure</option>
|
||||||
</select>
|
</select>
|
||||||
<label><input type="checkbox" id="toggleConflict" /> Conflict overlay</label>
|
<label><input type="checkbox" id="toggleConflict" /> Conflict overlay</label>
|
||||||
|
<label><input type="checkbox" id="toggleFlights" /> Flights</label>
|
||||||
|
<label><input type="checkbox" id="toggleMilitaryOnly" /> Military only</label>
|
||||||
|
<span id="flightCount" class="subtle"></span>
|
||||||
<span id="lastUpdated"></span>
|
<span id="lastUpdated"></span>
|
||||||
<button id="refreshBtn" title="Force an immediate RSS poll">Refresh now</button>
|
<button id="refreshBtn" title="Force an immediate RSS poll">Refresh now</button>
|
||||||
</div>
|
</div>
|
||||||
|
|
@ -37,14 +43,34 @@
|
||||||
</aside>
|
</aside>
|
||||||
|
|
||||||
<div id="clusterModal" class="modal hidden">
|
<div id="clusterModal" class="modal hidden">
|
||||||
<div id="modalBackdrop"></div>
|
<div class="modal-backdrop" data-close="clusterModal"></div>
|
||||||
<div class="modal-card">
|
<div class="modal-card">
|
||||||
<button id="closeModal" title="Close">×</button>
|
<button class="modal-close" data-close="clusterModal" title="Close">×</button>
|
||||||
<div id="modalContent"></div>
|
<div id="modalContent"></div>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
|
<div id="econHistoryView" class="modal hidden">
|
||||||
|
<div class="modal-backdrop" data-close="econHistoryView"></div>
|
||||||
|
<div class="modal-card wide">
|
||||||
|
<button class="modal-close" data-close="econHistoryView" title="Close">×</button>
|
||||||
|
<div class="headline" style="margin-top:0;">Economic Incident History</div>
|
||||||
|
<p class="subtle">
|
||||||
|
Sudden index/oil price moves and every story published in that window, newest first.
|
||||||
|
Ordering favors keyword/country-matched stories, but timing alone qualifies a story as a
|
||||||
|
candidate here — this is a correlation log, not a verified causal record.
|
||||||
|
</p>
|
||||||
|
<div style="margin: 10px 0 16px;">
|
||||||
|
<select id="econHistorySymbolFilter" class="econ-select">
|
||||||
|
<option value="">All instruments</option>
|
||||||
|
</select>
|
||||||
|
</div>
|
||||||
|
<div id="econHistoryList"></div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
<div id="wikiSelectPopup" class="hidden"></div>
|
<div id="wikiSelectPopup" class="hidden"></div>
|
||||||
|
<div id="flightTooltip" class="hidden"></div>
|
||||||
|
|
||||||
<footer id="militaryDrawer" class="collapsed">
|
<footer id="militaryDrawer" class="collapsed">
|
||||||
<button id="drawerHandle">
|
<button id="drawerHandle">
|
||||||
|
|
|
||||||
|
|
@ -1,5 +1,11 @@
|
||||||
const API = "/api";
|
const API = "/api";
|
||||||
|
|
||||||
|
// The globe surface's violet tint (see onGlobeReady below) — reused as the
|
||||||
|
// civilian-flight color so those icons read as "part of the globe" rather
|
||||||
|
// than a separate alert layer, per the flights-overlay color spec.
|
||||||
|
const GLOBE_TINT_HEX = 0x6a3fd9;
|
||||||
|
const MILITARY_FLIGHT_HEX = 0xff33ff;
|
||||||
|
|
||||||
let config = { weather_enabled: false, conflict_enabled: false };
|
let config = { weather_enabled: false, conflict_enabled: false };
|
||||||
let clusters = []; // raw, server-side ~11km-grid clusters
|
let clusters = []; // raw, server-side ~11km-grid clusters
|
||||||
let displayedClusters = []; // zoom-adaptive regrouping of `clusters`, currently on screen
|
let displayedClusters = []; // zoom-adaptive regrouping of `clusters`, currently on screen
|
||||||
|
|
@ -15,20 +21,12 @@ const globe = Globe()(document.getElementById("globeViz"))
|
||||||
.globeImageUrl("https://unpkg.com/three-globe/example/img/earth-blue-marble.jpg")
|
.globeImageUrl("https://unpkg.com/three-globe/example/img/earth-blue-marble.jpg")
|
||||||
.bumpImageUrl("https://unpkg.com/three-globe/example/img/earth-topology.png")
|
.bumpImageUrl("https://unpkg.com/three-globe/example/img/earth-topology.png")
|
||||||
.backgroundColor("#0f0d16")
|
.backgroundColor("#0f0d16")
|
||||||
.pointLat("lat")
|
.objectLat("lat")
|
||||||
.pointLng("lon")
|
.objectLng("lon")
|
||||||
.pointAltitude(0.012)
|
.objectAltitude(0.014)
|
||||||
.pointRadius(pointRadiusFor)
|
.objectThreeObject(buildGlobeObject)
|
||||||
.pointColor(colorForCluster)
|
.objectLabel(objectLabelFor)
|
||||||
.pointLabel(
|
.onObjectClick(onGlobeObjectClick)
|
||||||
(d) =>
|
|
||||||
`<div style="background:#170b28;border:1px solid #3a2159;padding:6px 8px;border-radius:6px;max-width:220px;font-size:12px;">
|
|
||||||
<b>${escapeHtml(d.headline)}</b><br/>
|
|
||||||
${d.article_count} stor${d.article_count === 1 ? "y" : "ies"} · ${d.source_count} source(s)
|
|
||||||
${d.location_name ? `<br/>${escapeHtml(d.location_name)}` : ""}
|
|
||||||
</div>`
|
|
||||||
)
|
|
||||||
.onPointClick(onClusterClick)
|
|
||||||
.onZoom(({ altitude }) => {
|
.onZoom(({ altitude }) => {
|
||||||
currentAltitude = altitude;
|
currentAltitude = altitude;
|
||||||
scheduleRegroup();
|
scheduleRegroup();
|
||||||
|
|
@ -42,6 +40,12 @@ const globe = Globe()(document.getElementById("globeViz"))
|
||||||
.ringPropagationSpeed(3)
|
.ringPropagationSpeed(3)
|
||||||
.ringRepeatPeriod((d) => d.repeatPeriod);
|
.ringRepeatPeriod((d) => d.repeatPeriod);
|
||||||
|
|
||||||
|
// Degrees-of-arc -> scene units, so icon sprite sizes can be specified in
|
||||||
|
// the same "degrees" unit pointRadiusFor/declutter already use, and stay
|
||||||
|
// visually consistent with them.
|
||||||
|
const sceneUnitsPerDeg = (2 * Math.PI * globe.getGlobeRadius()) / 360;
|
||||||
|
const HELMET_ICON_DEG = 1.6;
|
||||||
|
|
||||||
globe.pointOfView({ lat: 20, lng: 20, altitude: 2.2 }, 0);
|
globe.pointOfView({ lat: 20, lng: 20, altitude: 2.2 }, 0);
|
||||||
|
|
||||||
// Tint the globe surface (blue-marble texture) into the theme's violet
|
// Tint the globe surface (blue-marble texture) into the theme's violet
|
||||||
|
|
@ -53,7 +57,7 @@ globe.pointOfView({ lat: 20, lng: 20, altitude: 2.2 }, 0);
|
||||||
// tint applied immediately after construction.
|
// tint applied immediately after construction.
|
||||||
globe.onGlobeReady(() => {
|
globe.onGlobeReady(() => {
|
||||||
const globeMaterial = globe.globeMaterial();
|
const globeMaterial = globe.globeMaterial();
|
||||||
globeMaterial.color = new THREE.Color(0x6a3fd9);
|
globeMaterial.color = new THREE.Color(GLOBE_TINT_HEX);
|
||||||
globeMaterial.emissive = new THREE.Color(0x1a0e33);
|
globeMaterial.emissive = new THREE.Color(0x1a0e33);
|
||||||
globeMaterial.emissiveIntensity = 0.25;
|
globeMaterial.emissiveIntensity = 0.25;
|
||||||
});
|
});
|
||||||
|
|
@ -88,6 +92,106 @@ function colorForCluster(d) {
|
||||||
return palette[n - 1] || palette[0];
|
return palette[n - 1] || palette[0];
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ---- globe icon sprites (news clusters + conflict events) --------------
|
||||||
|
//
|
||||||
|
// Both are white-on-transparent alpha-mask canvases so a SpriteMaterial's
|
||||||
|
// `color` can tint them per-datum (news: the violet->pink gradient above;
|
||||||
|
// conflict: magenta) without needing separate colored art assets — same
|
||||||
|
// technique as the flight-plane icon further down.
|
||||||
|
|
||||||
|
let _newspaperIconTexture = null;
|
||||||
|
function newspaperIconTexture() {
|
||||||
|
if (_newspaperIconTexture) return _newspaperIconTexture;
|
||||||
|
const size = 64;
|
||||||
|
const canvas = document.createElement("canvas");
|
||||||
|
canvas.width = size;
|
||||||
|
canvas.height = size;
|
||||||
|
const ctx = canvas.getContext("2d");
|
||||||
|
ctx.fillStyle = "#ffffff";
|
||||||
|
ctx.beginPath();
|
||||||
|
ctx.moveTo(8, 8);
|
||||||
|
ctx.lineTo(46, 8);
|
||||||
|
ctx.lineTo(58, 20);
|
||||||
|
ctx.lineTo(58, 58);
|
||||||
|
ctx.lineTo(8, 58);
|
||||||
|
ctx.closePath();
|
||||||
|
ctx.fill();
|
||||||
|
ctx.globalCompositeOperation = "destination-out"; // cut "text line" gaps into the page
|
||||||
|
ctx.fillRect(14, 16, 30, 4);
|
||||||
|
ctx.fillRect(14, 26, 38, 3);
|
||||||
|
ctx.fillRect(14, 33, 38, 3);
|
||||||
|
ctx.fillRect(14, 40, 34, 3);
|
||||||
|
ctx.fillRect(14, 47, 28, 3);
|
||||||
|
ctx.globalCompositeOperation = "source-over";
|
||||||
|
_newspaperIconTexture = new THREE.CanvasTexture(canvas);
|
||||||
|
return _newspaperIconTexture;
|
||||||
|
}
|
||||||
|
|
||||||
|
let _helmetIconTexture = null;
|
||||||
|
function helmetIconTexture() {
|
||||||
|
if (_helmetIconTexture) return _helmetIconTexture;
|
||||||
|
const size = 64;
|
||||||
|
const canvas = document.createElement("canvas");
|
||||||
|
canvas.width = size;
|
||||||
|
canvas.height = size;
|
||||||
|
const ctx = canvas.getContext("2d");
|
||||||
|
ctx.fillStyle = "#ffffff";
|
||||||
|
ctx.beginPath();
|
||||||
|
ctx.arc(32, 30, 21, Math.PI, 0, false); // dome
|
||||||
|
ctx.lineTo(53, 36);
|
||||||
|
ctx.quadraticCurveTo(53, 43, 45, 43); // brim, right
|
||||||
|
ctx.lineTo(19, 43);
|
||||||
|
ctx.quadraticCurveTo(11, 43, 11, 36); // brim, left
|
||||||
|
ctx.closePath();
|
||||||
|
ctx.fill();
|
||||||
|
_helmetIconTexture = new THREE.CanvasTexture(canvas);
|
||||||
|
return _helmetIconTexture;
|
||||||
|
}
|
||||||
|
|
||||||
|
function makeIconSprite(texture, colorHex, scale) {
|
||||||
|
const material = new THREE.SpriteMaterial({
|
||||||
|
map: texture,
|
||||||
|
color: colorHex,
|
||||||
|
transparent: true,
|
||||||
|
depthWrite: false,
|
||||||
|
});
|
||||||
|
const sprite = new THREE.Sprite(material);
|
||||||
|
sprite.scale.set(scale, scale, 1);
|
||||||
|
return sprite;
|
||||||
|
}
|
||||||
|
|
||||||
|
function buildGlobeObject(d) {
|
||||||
|
if (d.kind === "conflict") {
|
||||||
|
return makeIconSprite(helmetIconTexture(), MILITARY_FLIGHT_HEX, HELMET_ICON_DEG * sceneUnitsPerDeg);
|
||||||
|
}
|
||||||
|
return makeIconSprite(newspaperIconTexture(), colorForCluster(d), pointRadiusFor(d) * sceneUnitsPerDeg * 2.4);
|
||||||
|
}
|
||||||
|
|
||||||
|
function objectLabelFor(d) {
|
||||||
|
if (d.kind === "conflict") {
|
||||||
|
return `<div style="background:#170b28;border:1px solid #3a2159;padding:6px 8px;border-radius:6px;max-width:220px;font-size:12px;">
|
||||||
|
<b>${escapeHtml(d.event_type)}</b><br/>
|
||||||
|
${escapeHtml(d.actor1)}${d.actor2 ? " vs " + escapeHtml(d.actor2) : ""}<br/>
|
||||||
|
${escapeHtml(d.location_name)}, ${escapeHtml(d.country)}
|
||||||
|
${d.fatalities ? `<br/><span style="color:#f50505;">${d.fatalities} fatalities</span>` : ""}
|
||||||
|
</div>`;
|
||||||
|
}
|
||||||
|
return `<div style="background:#170b28;border:1px solid #3a2159;padding:6px 8px;border-radius:6px;max-width:220px;font-size:12px;">
|
||||||
|
<b>${escapeHtml(d.headline)}</b><br/>
|
||||||
|
${d.article_count} stor${d.article_count === 1 ? "y" : "ies"} · ${d.source_count} source(s)
|
||||||
|
${d.location_name ? `<br/>${escapeHtml(d.location_name)}` : ""}
|
||||||
|
</div>`;
|
||||||
|
}
|
||||||
|
|
||||||
|
function onGlobeObjectClick(d) {
|
||||||
|
if (d.kind === "conflict") {
|
||||||
|
flyTo(d.lat, d.lon);
|
||||||
|
militaryDrawer.classList.remove("collapsed");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
onClusterClick(d);
|
||||||
|
}
|
||||||
|
|
||||||
// pointRadius is in degrees of arc on the globe surface (three-globe's own
|
// pointRadius is in degrees of arc on the globe surface (three-globe's own
|
||||||
// unit for this accessor — not pixels, not a fraction of globe radius), so
|
// unit for this accessor — not pixels, not a fraction of globe radius), so
|
||||||
// it's directly comparable to lat/lon distances for the decluttering pass
|
// it's directly comparable to lat/lon distances for the decluttering pass
|
||||||
|
|
@ -118,10 +222,21 @@ function scheduleRegroup() {
|
||||||
requestAnimationFrame(() => {
|
requestAnimationFrame(() => {
|
||||||
regroupScheduled = false;
|
regroupScheduled = false;
|
||||||
displayedClusters = regroupForAltitude(clusters, currentAltitude);
|
displayedClusters = regroupForAltitude(clusters, currentAltitude);
|
||||||
globe.pointsData(displayedClusters);
|
syncGlobeObjects();
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// News clusters and conflict events share one three-globe "objects" layer
|
||||||
|
// (see buildGlobeObject) so both can be custom icon sprites; this merges
|
||||||
|
// the two data sources into it, respecting the conflict-overlay toggle.
|
||||||
|
function syncGlobeObjects() {
|
||||||
|
const newsObjs = displayedClusters.map((c) => ({ ...c, kind: "news" }));
|
||||||
|
const conflictObjs = document.getElementById("toggleConflict").checked
|
||||||
|
? conflictEvents.map((e) => ({ ...e, kind: "conflict" }))
|
||||||
|
: [];
|
||||||
|
globe.objectsData([...newsObjs, ...conflictObjs]);
|
||||||
|
}
|
||||||
|
|
||||||
function mergeRadiusDeg(altitude) {
|
function mergeRadiusDeg(altitude) {
|
||||||
// altitude ~0.4 (close, city-level) -> ~0deg (no extra merging beyond the server grid)
|
// altitude ~0.4 (close, city-level) -> ~0deg (no extra merging beyond the server grid)
|
||||||
// altitude ~2.2 (default view) -> ~13deg (nearby-country scale, e.g. Levant stays
|
// altitude ~2.2 (default view) -> ~13deg (nearby-country scale, e.g. Levant stays
|
||||||
|
|
@ -292,8 +407,9 @@ function onClusterClick(cluster) {
|
||||||
const clusterModal = document.getElementById("clusterModal");
|
const clusterModal = document.getElementById("clusterModal");
|
||||||
const modalContent = document.getElementById("modalContent");
|
const modalContent = document.getElementById("modalContent");
|
||||||
|
|
||||||
document.getElementById("closeModal").onclick = () => clusterModal.classList.add("hidden");
|
document.querySelectorAll("[data-close]").forEach((el) => {
|
||||||
document.getElementById("modalBackdrop").onclick = () => clusterModal.classList.add("hidden");
|
el.onclick = () => document.getElementById(el.dataset.close).classList.add("hidden");
|
||||||
|
});
|
||||||
|
|
||||||
async function openClusterModal(cluster) {
|
async function openClusterModal(cluster) {
|
||||||
clusterModal.classList.remove("hidden");
|
clusterModal.classList.remove("hidden");
|
||||||
|
|
@ -573,6 +689,7 @@ async function loadConflictEvents() {
|
||||||
.join("")
|
.join("")
|
||||||
: `<p class="subtle">No events recorded in the current window.</p>`;
|
: `<p class="subtle">No events recorded in the current window.</p>`;
|
||||||
syncRings();
|
syncRings();
|
||||||
|
syncGlobeObjects();
|
||||||
} catch (e) {
|
} catch (e) {
|
||||||
console.error("Failed to load conflict events", e);
|
console.error("Failed to load conflict events", e);
|
||||||
}
|
}
|
||||||
|
|
@ -763,6 +880,254 @@ async function toggleMarketDetail(symbol, idx) {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ----------------------------------------------- economic incident history
|
||||||
|
|
||||||
|
const econHistoryView = document.getElementById("econHistoryView");
|
||||||
|
let econHistorySymbolsLoaded = false;
|
||||||
|
|
||||||
|
document.getElementById("infoBtn").onclick = () => {
|
||||||
|
econHistoryView.classList.remove("hidden");
|
||||||
|
loadEconHistory();
|
||||||
|
};
|
||||||
|
|
||||||
|
document.getElementById("econHistorySymbolFilter").onchange = () => loadEconHistory();
|
||||||
|
|
||||||
|
async function loadEconHistory() {
|
||||||
|
const listEl = document.getElementById("econHistoryList");
|
||||||
|
const filterEl = document.getElementById("econHistorySymbolFilter");
|
||||||
|
const symbol = filterEl.value;
|
||||||
|
|
||||||
|
if (!econHistorySymbolsLoaded) {
|
||||||
|
try {
|
||||||
|
const rows = await (await fetch(`${API}/markets/latest`)).json();
|
||||||
|
filterEl.innerHTML =
|
||||||
|
`<option value="">All instruments</option>` +
|
||||||
|
rows.map((r) => `<option value="${r.symbol}">${escapeHtml(r.label)}</option>`).join("");
|
||||||
|
econHistorySymbolsLoaded = true;
|
||||||
|
} catch (e) {
|
||||||
|
// non-fatal — the filter just stays at "All instruments"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
listEl.innerHTML = `<p class="subtle">Loading…</p>`;
|
||||||
|
try {
|
||||||
|
const url = symbol
|
||||||
|
? `${API}/markets/spikes?symbol=${encodeURIComponent(symbol)}&hours=8760`
|
||||||
|
: `${API}/markets/spikes?hours=8760`;
|
||||||
|
const spikes = await (await fetch(url)).json();
|
||||||
|
if (!spikes.length) {
|
||||||
|
listEl.innerHTML = `<p class="subtle">No moves ≥ the spike threshold recorded yet.</p>`;
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
listEl.innerHTML = spikes.map(econIncidentHtml).join("");
|
||||||
|
} catch (e) {
|
||||||
|
listEl.innerHTML = `<p class="subtle">Could not load incident history.</p>`;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function econIncidentHtml(s) {
|
||||||
|
const dir = s.pct_change > 0 ? "up" : "down";
|
||||||
|
const arrow = s.pct_change > 0 ? "▲" : "▼";
|
||||||
|
const fmtHour = (iso) =>
|
||||||
|
new Date(iso).toLocaleString(undefined, {
|
||||||
|
month: "short",
|
||||||
|
day: "numeric",
|
||||||
|
hour: "2-digit",
|
||||||
|
minute: "2-digit",
|
||||||
|
});
|
||||||
|
const articlesHtml = s.candidate_articles.length
|
||||||
|
? s.candidate_articles.map(articleItemHtml).join("")
|
||||||
|
: `<p class="subtle">No stories found published in that window.</p>`;
|
||||||
|
return `
|
||||||
|
<div class="econ-incident">
|
||||||
|
<h3>${escapeHtml(s.label)} <span class="${dir}">${arrow} ${s.pct_change.toFixed(2)}%</span></h3>
|
||||||
|
<div class="window">
|
||||||
|
${fmtHour(s.window_start)} – ${fmtHour(s.window_end)}
|
||||||
|
· ${s.from_price.toLocaleString()} → ${s.to_price.toLocaleString()}
|
||||||
|
· ${s.candidate_articles.length} stor${s.candidate_articles.length === 1 ? "y" : "ies"} in window
|
||||||
|
</div>
|
||||||
|
<div class="articles">${articlesHtml}</div>
|
||||||
|
</div>
|
||||||
|
`;
|
||||||
|
}
|
||||||
|
|
||||||
|
// -------------------------------------------------------------- flights --
|
||||||
|
//
|
||||||
|
// Rendered as a single THREE.Points cloud (not three-globe's per-datum
|
||||||
|
// pointsData/objectsData layers, already used by news clusters) since a
|
||||||
|
// live global snapshot can be several thousand aircraft — one draw call
|
||||||
|
// keeps that cheap. The tradeoff: point sprites all face the camera and
|
||||||
|
// don't support per-instance rotation, so icons aren't heading-aligned.
|
||||||
|
// Military classification is a best-effort heuristic from the backend (see
|
||||||
|
// backend/app/data/military_ranges.yaml); color is the only encoding
|
||||||
|
// (magenta = flagged military, globe-tint purple = everything else) and
|
||||||
|
// isn't itself a claim of certainty — the hover tooltip states as much.
|
||||||
|
|
||||||
|
let flightsPoints = null;
|
||||||
|
let flightsData = [];
|
||||||
|
let flightsPollTimer = null;
|
||||||
|
let flightsPollSeconds = 300;
|
||||||
|
const flightRaycaster = new THREE.Raycaster();
|
||||||
|
flightRaycaster.params.Points.threshold = 2.5;
|
||||||
|
|
||||||
|
function altitudeForFlight(f) {
|
||||||
|
const m = f.altitude_m || 0;
|
||||||
|
return 0.02 + Math.min(1, m / 12000) * 0.03;
|
||||||
|
}
|
||||||
|
|
||||||
|
// A small top-down airplane silhouette, drawn once as a white-on-transparent
|
||||||
|
// alpha mask so PointsMaterial's vertexColors can tint it per-aircraft
|
||||||
|
// (military vs civilian) without needing separate red/gray sprite sheets.
|
||||||
|
let _planeIconTexture = null;
|
||||||
|
function planeIconTexture() {
|
||||||
|
if (_planeIconTexture) return _planeIconTexture;
|
||||||
|
|
||||||
|
const size = 64;
|
||||||
|
const canvas = document.createElement("canvas");
|
||||||
|
canvas.width = size;
|
||||||
|
canvas.height = size;
|
||||||
|
const ctx = canvas.getContext("2d");
|
||||||
|
ctx.fillStyle = "#ffffff";
|
||||||
|
ctx.beginPath();
|
||||||
|
ctx.moveTo(32, 3); // nose
|
||||||
|
ctx.lineTo(36, 24);
|
||||||
|
ctx.lineTo(61, 39); // right wingtip
|
||||||
|
ctx.lineTo(61, 45);
|
||||||
|
ctx.lineTo(37, 38);
|
||||||
|
ctx.lineTo(35, 50);
|
||||||
|
ctx.lineTo(47, 59); // right tailtip
|
||||||
|
ctx.lineTo(47, 63);
|
||||||
|
ctx.lineTo(32, 57); // tail center
|
||||||
|
ctx.lineTo(17, 63); // left tailtip
|
||||||
|
ctx.lineTo(17, 59);
|
||||||
|
ctx.lineTo(29, 50);
|
||||||
|
ctx.lineTo(27, 38);
|
||||||
|
ctx.lineTo(3, 45); // left wingtip
|
||||||
|
ctx.lineTo(3, 39);
|
||||||
|
ctx.lineTo(28, 24);
|
||||||
|
ctx.closePath();
|
||||||
|
ctx.fill();
|
||||||
|
|
||||||
|
_planeIconTexture = new THREE.CanvasTexture(canvas);
|
||||||
|
return _planeIconTexture;
|
||||||
|
}
|
||||||
|
|
||||||
|
function buildFlightsPoints(flights) {
|
||||||
|
const geometry = new THREE.BufferGeometry();
|
||||||
|
const positions = new Float32Array(flights.length * 3);
|
||||||
|
const colors = new Float32Array(flights.length * 3);
|
||||||
|
const milColor = new THREE.Color(MILITARY_FLIGHT_HEX);
|
||||||
|
const civColor = new THREE.Color(GLOBE_TINT_HEX);
|
||||||
|
|
||||||
|
flights.forEach((f, i) => {
|
||||||
|
const { x, y, z } = globe.getCoords(f.lat, f.lon, altitudeForFlight(f));
|
||||||
|
positions[i * 3] = x;
|
||||||
|
positions[i * 3 + 1] = y;
|
||||||
|
positions[i * 3 + 2] = z;
|
||||||
|
const c = f.is_military ? milColor : civColor;
|
||||||
|
colors[i * 3] = c.r;
|
||||||
|
colors[i * 3 + 1] = c.g;
|
||||||
|
colors[i * 3 + 2] = c.b;
|
||||||
|
});
|
||||||
|
|
||||||
|
geometry.setAttribute("position", new THREE.BufferAttribute(positions, 3));
|
||||||
|
geometry.setAttribute("color", new THREE.BufferAttribute(colors, 3));
|
||||||
|
|
||||||
|
const material = new THREE.PointsMaterial({
|
||||||
|
size: 4,
|
||||||
|
map: planeIconTexture(),
|
||||||
|
alphaTest: 0.3,
|
||||||
|
vertexColors: true,
|
||||||
|
sizeAttenuation: true,
|
||||||
|
transparent: true,
|
||||||
|
depthWrite: false,
|
||||||
|
});
|
||||||
|
|
||||||
|
return new THREE.Points(geometry, material);
|
||||||
|
}
|
||||||
|
|
||||||
|
async function loadFlights() {
|
||||||
|
try {
|
||||||
|
const res = await fetch(`${API}/flights`);
|
||||||
|
flightsData = await res.json();
|
||||||
|
renderFlights();
|
||||||
|
} catch (e) {
|
||||||
|
console.error("Failed to load flights", e);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Rebuilds the points cloud from the last-fetched snapshot, applying the
|
||||||
|
// "Military only" filter — kept separate from loadFlights so toggling the
|
||||||
|
// filter is instant and doesn't wait on a network round-trip.
|
||||||
|
let flightsRendered = []; // the array actually backing the current geometry (post-filter), for hover lookups
|
||||||
|
function renderFlights() {
|
||||||
|
if (flightsPoints) {
|
||||||
|
globe.scene().remove(flightsPoints);
|
||||||
|
flightsPoints.geometry.dispose();
|
||||||
|
flightsPoints.material.dispose();
|
||||||
|
flightsPoints = null;
|
||||||
|
}
|
||||||
|
if (!document.getElementById("toggleFlights").checked) return;
|
||||||
|
|
||||||
|
const militaryOnly = document.getElementById("toggleMilitaryOnly").checked;
|
||||||
|
flightsRendered = militaryOnly ? flightsData.filter((f) => f.is_military) : flightsData;
|
||||||
|
flightsPoints = buildFlightsPoints(flightsRendered);
|
||||||
|
globe.scene().add(flightsPoints);
|
||||||
|
|
||||||
|
const milCount = flightsData.filter((f) => f.is_military).length;
|
||||||
|
document.getElementById("flightCount").textContent = flightsData.length
|
||||||
|
? militaryOnly
|
||||||
|
? `${milCount} military-flagged (of ${flightsData.length})`
|
||||||
|
: `${flightsData.length} aircraft · ${milCount} military-flagged`
|
||||||
|
: "no data yet";
|
||||||
|
}
|
||||||
|
|
||||||
|
function setFlightsVisible(visible) {
|
||||||
|
if (visible) {
|
||||||
|
loadFlights();
|
||||||
|
flightsPollTimer = setInterval(loadFlights, Math.max(15, flightsPollSeconds) * 1000);
|
||||||
|
} else {
|
||||||
|
if (flightsPollTimer) clearInterval(flightsPollTimer);
|
||||||
|
flightsPollTimer = null;
|
||||||
|
if (flightsPoints) {
|
||||||
|
globe.scene().remove(flightsPoints);
|
||||||
|
flightsPoints.geometry.dispose();
|
||||||
|
flightsPoints.material.dispose();
|
||||||
|
flightsPoints = null;
|
||||||
|
}
|
||||||
|
document.getElementById("flightCount").textContent = "";
|
||||||
|
document.getElementById("flightTooltip").classList.add("hidden");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
const flightTooltip = document.getElementById("flightTooltip");
|
||||||
|
document.getElementById("globeViz").addEventListener("mousemove", (ev) => {
|
||||||
|
if (!flightsPoints) return;
|
||||||
|
const el = document.getElementById("globeViz");
|
||||||
|
const rect = el.getBoundingClientRect();
|
||||||
|
const mouse = new THREE.Vector2(
|
||||||
|
((ev.clientX - rect.left) / rect.width) * 2 - 1,
|
||||||
|
-((ev.clientY - rect.top) / rect.height) * 2 + 1
|
||||||
|
);
|
||||||
|
flightRaycaster.setFromCamera(mouse, globe.camera());
|
||||||
|
const hits = flightRaycaster.intersectObject(flightsPoints);
|
||||||
|
const f = hits.length ? flightsRendered[hits[0].index] : null;
|
||||||
|
if (!f) {
|
||||||
|
flightTooltip.classList.add("hidden");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
flightTooltip.innerHTML = `
|
||||||
|
<b>${escapeHtml(f.callsign || f.icao24)}</b>${f.is_military ? ' <span class="mil">MIL (heuristic)</span>' : ""}<br/>
|
||||||
|
${escapeHtml(f.origin_country || "")}<br/>
|
||||||
|
${f.altitude_m != null ? Math.round(f.altitude_m) + " m" : ""}${
|
||||||
|
f.velocity_ms != null ? " · " + Math.round(f.velocity_ms * 3.6) + " km/h" : ""
|
||||||
|
}
|
||||||
|
`;
|
||||||
|
flightTooltip.style.left = ev.clientX + 12 + "px";
|
||||||
|
flightTooltip.style.top = ev.clientY + 12 + "px";
|
||||||
|
flightTooltip.classList.remove("hidden");
|
||||||
|
});
|
||||||
|
|
||||||
// --------------------------------------------------------------- wiring --
|
// --------------------------------------------------------------- wiring --
|
||||||
|
|
||||||
document.getElementById("refreshBtn").onclick = async () => {
|
document.getElementById("refreshBtn").onclick = async () => {
|
||||||
|
|
@ -781,7 +1146,15 @@ document.getElementById("weatherLayer").onchange = (ev) => {
|
||||||
|
|
||||||
document.getElementById("toggleConflict").onchange = () => {
|
document.getElementById("toggleConflict").onchange = () => {
|
||||||
if (!conflictEvents.length) loadConflictEvents();
|
if (!conflictEvents.length) loadConflictEvents();
|
||||||
else syncRings();
|
else {
|
||||||
|
syncRings();
|
||||||
|
syncGlobeObjects();
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
document.getElementById("toggleFlights").onchange = (ev) => setFlightsVisible(ev.target.checked);
|
||||||
|
document.getElementById("toggleMilitaryOnly").onchange = () => {
|
||||||
|
if (document.getElementById("toggleFlights").checked) renderFlights();
|
||||||
};
|
};
|
||||||
|
|
||||||
async function init() {
|
async function init() {
|
||||||
|
|
@ -792,6 +1165,7 @@ async function init() {
|
||||||
}
|
}
|
||||||
document.getElementById("toggleWeather").disabled = !config.weather_enabled;
|
document.getElementById("toggleWeather").disabled = !config.weather_enabled;
|
||||||
document.getElementById("toggleConflict").disabled = !config.conflict_enabled;
|
document.getElementById("toggleConflict").disabled = !config.conflict_enabled;
|
||||||
|
flightsPollSeconds = config.flights_poll_seconds || 300;
|
||||||
|
|
||||||
await loadClusters();
|
await loadClusters();
|
||||||
await loadNewsFeed();
|
await loadNewsFeed();
|
||||||
|
|
|
||||||
Loading…
Reference in New Issue