An automated observational platform for long-term public indicators.
Explore the latest public observations published by World Observer:
World Observer records repeatable, long-running observations of public indicators and publishes them as small JSON archives and static-dashboard exports. It is built for durability: a stable observer pipeline, conservative public-data sources, historical state retention, and descriptive dashboards that can be maintained for years.
Observational only: World Observer records what public sources reported at a point in time. It does not predict, speculate, attribute causes, track people, or profile users.
| Capability | Description |
|---|---|
| 🕒 Automated daily observations | Runs observer modules on a fixed daily UTC cadence and stores canonical outputs. |
| 🧩 Modular observer framework | Each observer is a self-contained Python module that emits one JSON object. |
| 🗃️ Historical archives | Daily archives under data/daily/YYYY-MM-DD/ preserve time-indexed observations. |
| 📊 Interactive dashboard data | Compact dashboard exports in dashboard/ power a static website without coupling to raw schemas. |
| 📈 Reusable trend charts | Internet and media history exports provide ready-to-chart time series. |
| 💓 Heartbeat monitoring | Minimal hourly heartbeat files indicate automation liveness only. |
| 🔓 Public data only | Observers use public, non-invasive signals and avoid private or user-level data. |
| 🌐 Static website friendly | The publishing helper copies exported JSON into a GitHub Pages-style website checkout. |
World Observer is a descriptive archive, not an intelligence product. The platform is designed around these constraints:
- Public sources only — no private datasets, user tracking, fingerprinting, or account-level collection.
- No surveillance — observers describe aggregate public indicators, not individuals.
- No prediction — outputs describe historical and current observations rather than forecasting future events.
- No speculation — dashboards avoid causal claims and keep interpretation separate from measurement.
- Historical preservation — daily JSON outputs are retained so changes can be reviewed over time.
- Stable semantics — observer behavior should remain consistent so long-horizon comparisons stay meaningful.
For more detail, see ethics.md, methodology.md, and SIGNIFICANCE_MODEL.md.
The dashboard export summarizes observer health, category counts, latest dates, and website-safe status fields.
Internet observers cover reachability, DNS behavior, IPv6 adoption, mail exchange visibility, TLS changes, traceroute-style indicators, and cable-dependency metadata.
The Germany media-language observer exports website-safe descriptive fields and historical trend points for public media-language monitoring.
Society observers track public cost-of-living style indicators, currently including German fuel-price infrastructure and an East Frisian tea retail-price series.
| Category | Observers |
|---|---|
| 🌐 Internet | Area51 Reachability; HTTP Reachability Index; Cuba Internet Weather; DNS Time To Answer Index; DNS TTA Stress Index; Global Reachability Long Horizon; Global Reachability Score; Internet Shrinkage Index; IPv6 Adoption Locked States; IPv6 Global Compare; IPv6 Locked States; Iran DNS Behavior; MX Presence By Country; MX Presence Per Country; North Korea Connectivity; Silent Countries List; TLS Fingerprint Change; Traceroute To Nowhere; Undersea Cable Dependency; Undersea Cable Dependency Map |
| 📰 Media | Media Language Germany |
| 🏘️ Society | Germany Fuel Price Observer; East Frisian Tea Observer |
| 🌱 Environment | Geomagnetic Storm Observer |
| Category | Planned observers |
|---|---|
| 🌐 Internet | ASN Visibility By Country |
| 🏘️ Society | Fuel Prices Germany, Electricity Prices Germany, Food Prices Germany, Housing Costs Germany, Deutsche Bahn Punctuality, Deutsche Post Reliability |
| 🌱 Environment | Weather Germany, Climate Germany, Natural Disasters Germany |
The canonical category metadata lives in config/observer_metadata.json. The canonical daily runner list lives in scripts/run_daily.py.
Observer
↓
State / latest JSON
↓
Dashboard export
↓
Static website
| Layer | Responsibility | Main paths |
|---|---|---|
| Observer | Collect one public observation and emit one JSON object on stdout. | observers/*/observer.py |
| Daily runner | Execute canonical observers, normalize outputs, write daily archives, and run the meta observer. | scripts/run_daily.py, data/daily/, data/latest/ |
| State | Preserve observer-specific historical state needed for continuity. | state/ |
| Dashboard export | Convert latest and daily raw observer outputs into compact, website-safe JSON. | scripts/export_dashboard.py, dashboard/ |
| Static website publish | Copy exported dashboard JSON into a separate website checkout. | scripts/publish_dashboard_to_pages.py |
| Heartbeat | Publish liveness-only hourly heartbeat files. | scripts/heartbeat_push.py, state/heartbeat/ |
world-observer/
├── observers/ # Self-contained observer modules
├── data/
│ ├── daily/YYYY-MM-DD/ # Immutable daily JSON archives
│ └── latest/ # Rolling latest observer snapshots
├── dashboard/ # Static-website JSON export files
│ └── history/ # Compact trend-history exports
├── state/ # Local observer state and heartbeat files
├── scripts/ # Daily runner, exporter, publishing, health checks
├── visualizations/ # Significance visualization helpers
├── docs/ # Operational and dashboard documentation
├── tests/ # Pytest suite and fixtures
├── cron/ # Example cron schedule
├── reports/ # Operational reports and audits
├── ethics.md # Ethical scope and collection constraints
└── methodology.md # Measurement methodology
Raw local capture directories are intentionally ignored where appropriate so the repository can retain durable aggregate archives without bloating Git history.
- Python 3.12+
- Linux or another cron-capable Unix-like environment for long-running automation
pipand a virtual environment
git clone https://github.com/dennishilk/world-observer.git
cd world-observer
python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -r requirements.txt -r requirements-dev.txtpython scripts/run_daily.py --date 2026-07-01This writes per-observer outputs to data/daily/2026-07-01/, refreshes data/latest/, and writes the meta summary as summary.json and summary.md.
python scripts/export_dashboard.pyThe exporter writes compact static-website files to dashboard/, including category summaries and history exports.
python scripts/publish_dashboard_to_pages.py --pages-repo /path/to/dennishilk.github.ioThe publish helper validates the target checkout and replaces only world-observer/dashboard/ inside that website repository.
sudo ./setup_world_observer.shThe setup script installs idempotent cron jobs for:
- hourly heartbeat commits at minute
0, and - daily observer execution at 02:00 UTC.
Heartbeat commits are liveness indicators only. They are not observations, anomaly signals, daily summaries, or significance indicators.
cat state/heartbeat/2026-07-01T10Z.jsonOnly recent heartbeat files are retained, while observer archives remain under data/daily/ and latest snapshots remain under data/latest/.
python -m pytest
python scripts/verify_repository_health.py
python scripts/export_dashboard.pyThe test suite covers observer contracts, dashboard export behavior, metadata consistency, publishing safety, and selected observer parsers.
Operational notes: cron, deploy keys, and data contracts
For a run date YYYY-MM-DD, scripts/run_daily.py treats data/daily/YYYY-MM-DD/ as the canonical daily directory. Each observer listed in the runner emits one JSON object on stdout, the runner writes it to data/daily/YYYY-MM-DD/<observer>.json, and world-observer-meta reads that same directory through WORLD_OBSERVER_DAILY_DIR. The meta observer output is persisted as summary.json and rendered as summary.md; there is intentionally no world-observer-meta.json artifact.
world-observer-meta treats an observer as successful when its file exists, the JSON root is an object, and the top-level status is not "error". Missing files, invalid JSON, non-object roots, and explicit error statuses are counted as missing/degraded.
The automation layer installs an hourly heartbeat and a daily 02:00 UTC observer run. Inspect repository-level cron output with:
tail -f ~/world-observer/logs/cron.logFor scheduler-level problems, check:
sudo systemctl status cron
crontab -lLong-running automation usually pushes with a dedicated GitHub deploy key:
ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519_world_observer -C "world-observer-deploy-key"
ssh -o BatchMode=yes -T git@github.comAdd the public key to the repository deploy keys with write access, then configure the local checkout's SSH command as needed.
The Area51 observer aggregates bounded airspace activity into daily activity units and rolling baselines. Tracked outputs intentionally do not contain callsigns, tail numbers, routes, or per-aircraft identifiers.
World Observer is expanding from Internet-focused indicators into broader public-observation categories.
| Area | Planned work |
|---|---|
| ⚡ Electricity | Public electricity-price observer and chart-ready dashboard export. |
| 🍞 Food | Additional representative public food-price indicators. |
| 🏠 Housing | Descriptive public housing-cost indicators. |
| 🚆 Deutsche Bahn | Public punctuality/reliability observer. |
| 📮 Deutsche Post | Public postal-reliability observer. |
| 🌱 Environment | Weather, climate, and natural-disaster public-data observers. |
| 📊 Dashboard | Richer static website cards and reusable trend visualizations. |
Contributions should preserve the project's observational scope:
- Prefer public, stable, low-risk data sources.
- Keep observers small, deterministic, and readable.
- Emit one JSON object per observer run.
- Separate collection from dashboard export and presentation.
- Do not add user tracking, private data, invasive probing, or speculative claims.
World Observer is released under the MIT License.