Signal Horizon Telemetry Pipeline
From 10K signals/sec at the edge to hunt-ready rollups in ClickHouse. Authenticated ingest, tiered retention, and ten materialized views that keep dashboards snappy.
Ingest Path — Edge to Hunter
STAGE 01
Edge Sensors
Synapse Pingora + Apparatus agents emit signal events, HTTP transactions, and sensor logs.
EMIT
STAGE 02
Telemetry API
POST /telemetry · JWT-auth'd · replay-protected via nonce store · Zod-validated. Batches up to 5000 events.
INGEST
STAGE 03
Retry Buffer
In-memory retry queue catches transient ClickHouse outages without dropping samples.
RESILIENCE
STAGE 04
ClickHouse
Partitioned by month, ORDER BY tenant + time. ZSTD compression (~100B/signal). TTL-driven tiered retention.
STORE
STAGE 05
Dashboards & Hunts
Horizon UI & analyst queries read pre-aggregated materialized views for sub-second response.
QUERY
10K/s
Target Sustained
Per-tenant ingest target with bloom-filter indices for fast IP & fingerprint lookups.
~100B
Compressed/Signal
ZSTD-compressed column store. 100M signals ≈ 10 GB raw.
5000
Events/Batch Max
Enforced by TelemetryBatchSchema; larger batches rejected at the edge.
Tiered Retention — TTL-Driven
blocklist_history
BLOCK CHANGE LOG
365d
campaign_history
CAMPAIGN SNAPSHOTS
180d
signal_events
PRIMARY TIME-SERIES
90d
http_transactions
RAW REQUEST TELEMETRY
30d
sensor_logs
RAW SENSOR OUTPUT
30d
Retention enforced via ClickHouse TTL toDateTime(timestamp) + INTERVAL N DAY. Hot tables (short TTL) carry raw detail; warm tables (long TTL) carry summaries for historical hunts.
Pre-Aggregated Views — 10 Rollups For Instant Queries
signal_hourly_mv
TENANT · TYPE · HOUR
Hourly signal counts by tenant & signal type. Base for most dashboards.
ip_daily_mv
SOURCE_IP · DAY
Daily per-IP summary for dwell-time anomaly detection.
top_actors_hourly
SENSOR · HOUR · ACTOR
Ranked actors per sensor per hour — noisy-actor surfacing.
attack_trends_daily
CLASS · DAY
Classification counts over time for trend lines.
blocks_by_sensor_hourly
SENSOR · HOUR
Block volume per sensor — traffic vs. enforcement balance.
campaign_velocity_hourly
CAMPAIGN · HOUR
Rate of new members per campaign — escalation detection.
geo_distribution_daily
COUNTRY · DAY
Geo spread of signals; feeds world-map visualizations.
actor_sensor_matrix
ACTOR × SENSOR
Cross-sensor coverage per actor — confirms spread of an attacker.
fingerprint_spread_daily
FINGERPRINT · DAY
How many distinct IPs share a JA4 or TLS fingerprint.
daily_summary
TENANT · DAY
Top-level tenant totals — dashboards home page.