Component Specification: Telemetry & Data Fabric

Tier 3: Technical Specifications · Audience: Data Engineers, SecOps Infrastructure Leads · Normative Status: Reference Component
Prerequisites: Cyber Threat Intelligence · Next Step: Detection Engine

---

The Telemetry & Data Fabric provides the foundational data infrastructure for TIDIR. It guarantees reliable, high-throughput ingestion across the SOC Visibility Quad (Logs, Endpoint, Network, and Application / Cloud / Artificial Intelligence runtime observability), real-time normalisation into the Open Cybersecurity Schema Framework (OCSF), and tier-optimised storage across hot analytical indices and durable lakehouse repositories.

---

2. Core Functional Requirements

1. Scalable Ingestion & Buffering:

   • Resilient against downstream pipeline slowdowns using distributed partitioned commit logs.
   • Dynamic partition autoscaling based on incoming event rates (Events Per Second - EPS).
   • At-least-once message delivery semantics with consumer deduplication.

2. OCSF Schema Normalisation:

   • Decouple raw vendor telemetry from detection logic.
   • Mapping catalog across the SOC Visibility Quad:
     • Host Activity (Process Creation, Network Connections, File Operations) -> OCSF System Activity / Process Activity classes.
     • Cloud Management Plane -> OCSF Cloud / Account Activity classes.
     • Network Flows -> OCSF Network Activity classes.
     • Identity / Auth Events -> OCSF Authentication / Identity classes.
     • Application & AI Observability (API transactions, eBPF system call hooks, model inference audits, SaaS logs) -> OCSF Application Activity / API Activity classes.
   • Dead-Letter Queue (DLQ) for non-conforming or unparseable payloads with automated alerting.

3. Dual-Tier Storage Architecture:

   • Hot Tier (Search & Immediate Triage):
     • High-throughput inverted text and columnar indices.
     • Retains recent 15–30 days.
     • Optimised for needle-in-a-haystack lookups, timeline queries, and analyst interactive dashboards.
   • Lakehouse Tier (Historical, Deep Analytics & ML):
     • Open table metadata catalogue backed by highly durable object storage.
     • Columnar Parquet compression (Snappy / Zstd).
     • Partitioned by event timestamp (dt=YYYY-MM-DD/hh=HH) and OCSF class.
     • Queryable via distributed SQL execution engines.

4. Threat-Led Telemetry Validation & Coverage Closure:

   • Ingestion priorities are aligned with the empirical technique frequency curve established by Cyber Threat Intelligence (CTI).
   • Automated collection audits continuously verify that the mandatory OCSF attributes required by the top 20 high-prevalence adversary techniques (e.g. process execution ancestry, memory access permissions, token impersonation, network egress flows) are actively collected across critical asset classes.
   • Sensor instrumentation deficits that would blind detection of top-prevalence techniques trigger immediate gap notifications, ensuring collection depth where adversary frequency is highest before expending budget on peripheral log sources.

5. Distributed Ingestion & Federated Context Pushdown:

   • Ingests line-rate normalized OCSF Finding objects (Classes 2001 and 2004) from native security controls (ADR-0023), bypassing the need to stream bulk commodity events centrally for standard matches.
   • Preserves raw contextual telemetry in low-cost columnar object storage partitions or local forwarder ring buffers.
   • Exposes federated query pushdown connectors that allow investigative subagents to execute targeted SQL sweeps directly across edge storage nodes and lakehouse partitions on demand.

---

3. Architectural Capability Archetypes & Protocol Standards

Subsystem Component	Functional Capability Pattern	Data Model & Protocol Standards
Streaming Message Fabric	Distributed partitioned append-only log with horizontal partition rebalancing and consumer offset tracking.	Binary streaming protocol; SASL/SCRAM authentication; mTLS encryption.
Line-Rate Normalisation Engine	Stateless, horizontally scalable schema translation workers compiling proprietary event formats into standard records.	Open Cybersecurity Schema Framework (OCSF v1.3+); JSON/Avro serialization.
Hot Analytics Engine	Distributed columnar and inverted search index supporting sub-second aggregations and temporal range queries.	Open search query dialect; REST/HTTP API.
Columnar Lakehouse Engine	Serverless distributed query engine operating directly against immutable columnar file stores with ACID snapshot isolation.	Open table format metadata specifications; Apache Parquet format.