Cloud-Native SOC Architecture¶

A cloud-native Security Operations Center moves beyond legacy on-premises SIEM deployments to a scalable, API-driven, multi-cloud architecture. This document defines the reference architecture for building a SOC that is cloud-first — leveraging elastic compute, object storage, streaming pipelines, and infrastructure-as-code to handle modern detection and response at scale.

Cloud-Native SOC Reference Architecture¶

flowchart TB
    subgraph Sources["Log & Telemetry Sources"]
        CLOUD_LOGS[Cloud Provider Logs<br/>AWS CloudTrail / Azure Activity / GCP Audit]
        ENDPOINT[Endpoint Telemetry<br/>EDR Agents]
        NETWORK[Network Telemetry<br/>NDR / VPC Flow Logs]
        IDENTITY[Identity Logs<br/>IdP / SSO / MFA]
        SAAS[SaaS Application Logs<br/>M365 / Google Workspace / Salesforce]
        CUSTOM[Custom App Logs<br/>Webhooks / Syslog / API]
    end

    subgraph Collection["Collection Layer"]
        AGENTS[Log Shipping Agents]
        API_COLL[API Collectors]
        WEBHOOK[Webhook Receivers]
        STREAM[Streaming Ingestion<br/>Kafka / Kinesis / Event Hub]
    end

    subgraph Processing["Processing & Normalization"]
        NORM[Schema Normalization<br/>OCSF / ECS Mapping]
        ENRICH[Enrichment Pipeline<br/>GeoIP, TI, Asset Context]
        FILTER[Ingestion Filtering<br/>Drop / Downsample / Route]
        PARSE[Parsing Engine<br/>Field Extraction / Type Coercion]
    end

    subgraph Detection["Detection Engine"]
        REALTIME[Real-Time Streaming Detection<br/>Correlation Rules / Sigma]
        BATCH[Batch Analytics<br/>Scheduled Queries / Threat Hunting]
        ML[ML Anomaly Detection<br/>UEBA / Behavioral Baselines]
        TI_MATCH[Threat Intel Matching<br/>IOC Correlation]
    end

    subgraph Response["Response & Orchestration"]
        SOAR[SOAR Platform<br/>Playbook Execution]
        CASE[Case Management<br/>Investigation Workflow]
        NOTIFY[Notification Engine<br/>PagerDuty / Slack / Email]
        AUTO_RESP[Automated Response<br/>Contain / Isolate / Block]
    end

    subgraph Storage["Tiered Storage"]
        HOT[Hot Tier<br/>Real-time indexing — 30 days]
        WARM[Warm Tier<br/>Searchable archive — 90 days]
        COLD[Cold Tier<br/>Object storage — 1 year+]
    end

    CLOUD_LOGS --> AGENTS
    ENDPOINT --> AGENTS
    NETWORK --> API_COLL
    IDENTITY --> API_COLL
    SAAS --> WEBHOOK
    CUSTOM --> WEBHOOK
    AGENTS --> STREAM
    API_COLL --> STREAM
    WEBHOOK --> STREAM

    STREAM --> NORM
    NORM --> ENRICH
    ENRICH --> FILTER
    FILTER --> PARSE

    PARSE --> REALTIME
    PARSE --> HOT
    REALTIME --> SOAR
    REALTIME --> CASE
    REALTIME --> NOTIFY

    BATCH --> CASE
    ML --> SOAR
    TI_MATCH --> SOAR

    HOT --> BATCH
    HOT --> ML
    HOT --> TI_MATCH

    HOT -.->|age-off 30d| WARM
    WARM -.->|age-off 90d| COLD

    SOAR --> AUTO_RESP
    SOAR --> CASE

Data Pipeline Architecture¶

The data pipeline transforms raw telemetry into actionable security events through four stages:

Stage 1: Collection¶

Multiple collection methods ensure coverage across all source types:

Stage 2: Normalization (OCSF / ECS)¶

Raw logs are mapped to a unified schema for consistent detection and analysis:

Normalization pipeline example:

Raw log (vendor-specific)
    → Parser (extract fields)
    → Schema mapper (map to OCSF categories)
    → Type coercion (timestamps → ISO 8601, IPs → normalized)
    → Validation (reject malformed; route to dead letter queue)
    → Enriched event (ready for detection)

Stage 3: Enrichment¶

Events are enriched inline before they reach the detection engine:

Stage 4: Filtering and Routing¶

Not all events require real-time indexing. Selective ingestion reduces cost without sacrificing coverage:

Tiered Storage Architecture¶

Storage Tier Specifications¶

Cost Comparison (Synthetic Reference — Per TB/Month)¶

Data Lifecycle Policy¶

flowchart LR
    INGEST[Ingest] -->|real-time| HOT[Hot Tier<br/>Fully indexed<br/>30 days]
    HOT -->|age-off policy| WARM[Warm Tier<br/>Key fields indexed<br/>90 days]
    WARM -->|age-off policy| COLD[Cold Tier<br/>Compressed Parquet<br/>1 year+]
    COLD -->|retention expiry| DELETE[Secure Deletion<br/>Crypto-shredding]

    LEGAL[Legal Hold] -.->|override age-off| HOT
    LEGAL -.->|override age-off| WARM
    LEGAL -.->|preserve indefinitely| COLD

Detection Architecture¶

Three Detection Modes¶

The cloud-native SOC operates three complementary detection modes:

1. Real-Time Streaming Detection¶

2. Batch Analytics (Scheduled)¶

3. ML Anomaly Detection¶

Detection Rule Lifecycle¶

Author (Sigma YAML)
    → Peer review (PR-based)
    → Unit test (synthetic event validation)
    → Simulation (run against 30d historical data — measure FP rate)
    → Staged deployment (canary → 25% → 100%)
    → Production monitoring (alert volume, FP rate, MTTD)
    → Tuning (quarterly review; retire if no true positives in 180 days)

SOAR Integration Architecture¶

Playbook Architecture¶

flowchart TB
    ALERT[Detection Alert] --> TRIAGE[Auto-Triage Playbook]

    TRIAGE --> ENRICH_PB[Enrichment Playbook]
    ENRICH_PB --> |"query TI, CMDB, IdP"| DECISION{Severity + Confidence}

    DECISION -->|High severity + High confidence| AUTO[Automated Response Playbook]
    DECISION -->|High severity + Low confidence| HUMAN[Human-in-the-Loop Playbook]
    DECISION -->|Low severity| SUPPRESS[Suppress & Log]

    AUTO --> CONTAIN[Containment Actions]
    CONTAIN --> ISOLATE[Isolate Host — EDR API]
    CONTAIN --> DISABLE[Disable Account — IdP API]
    CONTAIN --> BLOCK[Block IP — Firewall API]

    HUMAN --> NOTIFY_ANALYST[Notify Analyst — Slack / PagerDuty]
    NOTIFY_ANALYST --> ANALYST_REVIEW[Analyst Review + Decision]
    ANALYST_REVIEW -->|Approve| CONTAIN
    ANALYST_REVIEW -->|False Positive| TUNE[Add Exception + Tune Rule]

    CONTAIN --> CASE_CREATE[Create / Update Case]
    SUPPRESS --> CASE_CREATE
    TUNE --> CASE_CREATE

API Mesh¶

The SOAR platform integrates with the security stack via a standardized API mesh:

Enrichment Pipeline¶

Every alert passes through automated enrichment before analyst review:

Alert received
    → IP reputation lookup (TI platform — 3 providers)
    → Domain reputation + WHOIS age check
    → File hash lookup (VirusTotal, internal sandbox)
    → Asset context (CMDB — owner, criticality, environment)
    → User context (IdP — role, department, risk score)
    → Related alerts (SIEM — same entity, last 7 days)
    → GeoIP + impossible travel check
    → Severity re-scoring based on enrichment
    → Route to appropriate playbook

Multi-Cloud Considerations¶

Unified Data Model Across Providers¶

Cross-Cloud Detection Challenges¶

Scaling Patterns¶

Horizontal Scaling Architecture¶

flowchart LR
    subgraph Ingestion["Ingestion Tier (auto-scale)"]
        IW1[Ingestion Worker 1]
        IW2[Ingestion Worker 2]
        IWN[Ingestion Worker N]
    end

    subgraph Queue["Message Queue (partitioned)"]
        Q[Kafka / Kinesis<br/>Partitioned by source type]
    end

    subgraph Detection_Scale["Detection Tier (auto-scale)"]
        DW1[Detection Worker 1<br/>Auth rules]
        DW2[Detection Worker 2<br/>Network rules]
        DWN[Detection Worker N<br/>Endpoint rules]
    end

    subgraph Index["Indexing Tier (auto-scale)"]
        IDX1[Index Writer 1]
        IDX2[Index Writer 2]
        IDXN[Index Writer N]
    end

    IW1 --> Q
    IW2 --> Q
    IWN --> Q
    Q --> DW1
    Q --> DW2
    Q --> DWN
    Q --> IDX1
    Q --> IDX2
    Q --> IDXN

Auto-Scaling Policies¶

Queue-Based Processing¶

All inter-tier communication uses message queues for resilience and backpressure management:

Cost Optimization¶

Ingestion Cost Reduction Strategies¶

Total Cost of Ownership Model (Synthetic — 10 TB/Day Ingestion)¶

Cloud-Native SOC Maturity Model¶

Reference Metrics and SLAs¶

Capacity Planning¶

Detection SLAs¶

Storage Cost SLAs¶

Common Cloud-Native SOC Mistakes¶

Nexus SecOps Control Mapping¶

See Zero Trust SOC | Zero Trust Network | Reference Architecture Related chapters: Chapter 7: SIEM Architecture | Chapter 8: SOAR | Chapter 11: Cloud Security Operations