Shadowbroker
A real-time, multi-domain OSINT dashboard that turns independent public feeds into an explainable operational picture.
Project identity
Shadowbroker is a self-hosted open-source intelligence workspace for observing public activity across aviation, maritime traffic, satellites, earthquakes, conflict zones, public cameras, GPS interference, and geopolitical events. Work recorded in the repository began March 4, 2026; the latest recorded release tag is v0.9.5.
The project is a dashboard, not an intelligence oracle. Its value comes from placing independent observations into a common map and timeline while keeping source, freshness, and uncertainty visible.
Intelligence domains
Movement and infrastructure
- Aircraft and maritime layers support geographic situational awareness.
- Satellite positions add orbital context without treating a calculated position as a verified event.
- Public camera sources can provide visual context when their availability and terms permit it.
Events and interference
- Earthquake data contributes time, magnitude, and location-based alerts.
- Conflict and geopolitical event layers bring separate public reporting into one interface.
- GPS-jamming observations help identify navigation-interference areas but must remain clearly attributed and time bounded.
Each domain has different schemas, refresh intervals, rate limits, geographic accuracy, and failure modes. Shadowbroker therefore treats every feed as a module rather than forcing all collectors into one large service.
Architecture and stack
This non-monolithic layout gives the map a stable API contract while letting Python handle feed-specific parsing and enrichment. It also makes a frontend rendering failure distinguishable from an upstream data failure.
Data flow, provenance, and uncertainty
- Observation time and retrieval time are separate; a newly fetched record can still describe an old event.
- Provider identifiers and attribution must survive normalization so an analyst can trace a marker back to its origin.
- Caching reduces provider pressure, but cached data requires explicit freshness indicators.
- Cross-domain proximity is a lead for investigation, not proof that two events are related.
Map and interface design
A dense operational map can become unusable before it becomes technically slow. The interface must manage layer visibility, marker clustering, selection state, time ranges, legends, and loading/error states without hiding uncertainty.
- MapLibre keeps the rendering layer independent from proprietary map SDK logic.
- Domain-specific icons and controls help prevent aircraft, vessels, satellites, and event reports from blending into one ambiguous marker type.
- Details should open on demand so the primary map remains readable.
- Responsive layouts need different information density rather than merely shrinking a desktop map.
- Accessibility includes keyboard-reachable controls, readable contrast, non-color status cues, and textual alternatives to spatial information.
Docker, Helm, and operational boundaries
The repository separates frontend, backend, and deployment configuration. Docker supports repeatable local operation, while the Helm chart models the services as distinct Kubernetes workloads. The chart is ingress-ready and uses restricted runtime user IDs such as 1001 rather than assuming root access.
Operational checks
- Test each service health endpoint independently.
- Verify the frontend uses the intended backend origin.
- Check provider failures without restarting healthy domains.
- Keep credentials in runtime configuration, never in client bundles or images.
Scaling rule
- Measure slow feeds, CPU, memory, response size, and request frequency first.
- Scale backend collectors separately only where load evidence supports it.
- Do not add Kubernetes complexity to solve a problem that a healthy two-container deployment does not have.
Security, privacy, and responsible OSINT
The project documentation states that the application does not need to collect or transmit user data and can keep its backend self-hosted. That reduces exposure, but OSINT still needs a threat model.
- Provider keys and collection credentials remain server-side and receive the minimum possible scope.
- Outbound requests need timeouts, size limits, controlled redirects, and safe URL allowlists to reduce server-side request-forgery risk.
- Untrusted feed text must be encoded before display; map popups are not exempt from cross-site-scripting controls.
- Rate limits, cache policy, attribution, and provider terms are part of application correctness.
- Retention should be limited to the operational need, especially for location history.
- The interface distinguishes public observation, automated inference, and analyst conclusion.
Testing and release verification
The frontend manifest records development, production build, lint, test, watch, and coverage workflows. Those checks cover code quality, but a reliable OSINT release also needs source-contract and degraded-mode tests.
Hurdles and what I learned
- Real-time systems are usually collections of different update speeds, not one truly live stream.
- A map marker without provenance and age can be more misleading than no marker.
- Normalizing every source into one contract is useful only if domain-specific meaning is preserved.
- Partial failure must be a first-class interface state; one unavailable provider should not present the whole platform as down.
- Frontend/backend separation protects credentials and makes debugging more precise.
- Helm security contexts and modular templates make production assumptions reviewable.
- Responsible OSINT requires minimization, attribution, and cautious language even when all sources are public.
Roadmap and future planning
The path from v0.9.5 should prioritize operational evidence over the number of visible layers: provider contract tests, per-source health and freshness, provenance panels, retention controls, documented licensing, accessible non-map views, signed images, dependency scanning, and a repeatable rollback drill. Higher-level correlation should be introduced only with explainable rules and explicit confidence—not as an unexplained AI conclusion.