SC-088: eBPF Rootkit — Operation KERNEL GHOST¶

Scenario Overview¶

Field	Detail
ID	SC-088
Category	Endpoint Security / Kernel Exploitation / Rootkit
Severity	Critical
ATT&CK Tactics	Persistence, Defense Evasion, Credential Access, Privilege Escalation
ATT&CK Techniques	T1014 (Rootkit), T1056 (Input Capture), T1547 (Boot or Logon Autostart Execution), T1068 (Exploitation for Privilege Escalation)
Target Environment	Linux production server fleet (Ubuntu 22.04, kernel 5.15+) running web applications, databases, and container workloads with eBPF-capable kernels and CAP_BPF-enabled monitoring tools
Difficulty	★★★★★
Duration	4–6 hours
Estimated Impact	Kernel-level rootkit installed on 4 production servers; all process, file, and network activity hidden from userspace tools; SSH and database credentials intercepted for 72 hours before detection; complete evasion of EDR, auditd, and file integrity monitoring; 96-hour containment and remediation requiring full OS reinstallation

Narrative¶

Ironclad Infrastructure, a fictional managed hosting provider, operates a fleet of 200+ bare-metal and virtual Linux servers hosting customer applications. Their infrastructure runs at 10.50.0.0/16 with a management network at 172.16.0.0/16. Servers run Ubuntu 22.04 LTS with kernel 5.15, which includes full eBPF (extended Berkeley Packet Filter) support for performance monitoring, networking, and observability.

Ironclad's security stack includes an EDR agent, auditd with comprehensive rules, file integrity monitoring (FIM), and centralized log aggregation at siem.ironclad.example.com (198.51.100.80). The operations team uses eBPF-based tools (bpftrace, BCC) for performance troubleshooting, and the network team uses eBPF-based load balancers and packet filtering. As a result, eBPF program loading is a normal and expected activity on these servers.

In April 2026, a threat actor group designated SILICON PHANTOM — a nation-state APT specializing in kernel-level exploitation and long-term persistence — targets Ironclad through an initial compromise of a web application server. After gaining root access through a known vulnerability chain, SILICON PHANTOM deploys a custom eBPF rootkit that operates entirely in kernel space, hiding processes, files, network connections, and credentials from all userspace security tools.

Attack Flow¶

graph TD
    A[Phase 1: Initial Compromise<br/>Web app exploit chain → root access] --> B[Phase 2: Kernel Capability Assessment<br/>Verify eBPF support, BTF, kernel version]
    B --> C[Phase 3: eBPF Rootkit Deployment<br/>Load malicious BPF programs into kernel]
    C --> D[Phase 4: Process & File Hiding<br/>Hook getdents64, openat syscalls]
    D --> E[Phase 5: Network Connection Hiding<br/>Hook TCP/UDP socket calls, manipulate /proc/net]
    E --> F[Phase 6: Credential Interception<br/>Hook read/write syscalls on SSH/DB processes]
    F --> G[Phase 7: Persistence & Lateral Movement<br/>Spread rootkit to additional servers]
    G --> H[Phase 8: Detection & Response<br/>Kernel-level anomaly detection + memory forensics]

Phase Details¶

Phase 1: Initial Compromise¶

ATT&CK Technique: T1068 (Exploitation for Privilege Escalation)

SILICON PHANTOM gains initial access through a vulnerability chain on a customer-facing web application server (web-prod-01.ironclad.example.com, 10.50.10.20). The attack exploits an unpatched deserialization vulnerability in the web framework for initial code execution as the application user, then escalates to root via a kernel privilege escalation vulnerability.

# Simulated initial compromise (educational only)
# Stage 1: Web application exploit (deserialization → code execution)
# Attacker exploits vulnerable endpoint on web-prod-01

$ curl -s -X POST https://web-prod-01.ironclad.example.com/api/import \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer REDACTED" \
    -d '{"data": "BASE64_SERIALIZED_PAYLOAD_REDACTED",
         "format": "binary"}'

# Result: Code execution as www-data user
# $ id
# uid=33(www-data) gid=33(www-data) groups=33(www-data)

# Stage 2: Local privilege escalation via kernel vulnerability
# Exploiting a race condition in filesystem subsystem (educational — CVE fictional)
# $ ./priv_esc_exploit
# [*] Triggering filesystem race condition...
# [*] Overwriting kernel credential structure...
# [*] Spawning root shell...
# $ id
# uid=0(root) gid=0(root) groups=0(root)

# Post-exploitation reconnaissance
$ uname -a
Linux web-prod-01 5.15.0-91-generic #101-Ubuntu SMP x86_64 GNU/Linux

$ hostname -I
10.50.10.20 172.16.10.20

$ cat /etc/os-release | head -3
NAME="Ubuntu"
VERSION="22.04.4 LTS (Jammy Jellyfish)"
VERSION_ID="22.04"

# Check for security tooling
$ ps aux | grep -E "(falcon|crowdstrike|edr|auditd|ossec|wazuh)" | head -5
root       1247  0.3  1.2  524288  98304 ?  Sl   Jan01 127:42 /opt/edr-agent/bin/edr-agentd
root       1305  0.0  0.0   12456   2048 ?  S    Jan01   0:45 /sbin/auditd -n
root       1892  0.1  0.4  128000  32768 ?  Sl   Jan01  42:18 /usr/bin/fim-agent

# EDR agent, auditd, and FIM are all running — traditional rootkit would be detected
# eBPF rootkit will operate below these tools' visibility

Phase 2: Kernel Capability Assessment¶

ATT&CK Technique: T1014 (Rootkit)

Before deploying the rootkit, SILICON PHANTOM assesses the kernel's eBPF capabilities to ensure the rootkit will function correctly. The attacker verifies BPF support, available program types, BTF (BPF Type Format) availability, and existing eBPF programs.

# Simulated kernel capability assessment (educational only)
# Check eBPF availability and features

# Verify BPF syscall support
$ cat /proc/sys/kernel/unprivileged_bpf_disabled
1
# Unprivileged BPF is disabled — but we have root, so this doesn't matter

# Check kernel BPF features
$ cat /proc/config.gz 2>/dev/null | zcat | grep -i bpf | head -10
CONFIG_BPF=y
CONFIG_BPF_SYSCALL=y
CONFIG_BPF_JIT=y
CONFIG_BPF_JIT_ALWAYS_ON=y
CONFIG_BPF_LSM=y
CONFIG_HAVE_EBPF_JIT=y
CONFIG_BPF_EVENTS=y

# Check BTF (BPF Type Format) support — required for CO-RE programs
$ ls -la /sys/kernel/btf/vmlinux
-r--r--r-- 1 root root 4587520 Apr  1 00:00 /sys/kernel/btf/vmlinux
# BTF available — enables portable eBPF programs across kernel versions

# Enumerate existing BPF programs (to understand what's normal)
$ bpftool prog list
1: cgroup_device  tag a04f5eef06a7f555  gpl
   loaded_at 2026-01-01T00:00:00+0000  uid 0
   xlated 648B  jited 409B  memlock 4096B
15: tracing  tag 8b5c4c12f7a5e3d1  gpl
   loaded_at 2026-01-15T10:00:00+0000  uid 0
   xlated 2048B  jited 1256B  memlock 8192B
   btf_id 42
23: cgroup_skb  tag c3d5a7b2e8f10946  gpl
   loaded_at 2026-01-01T00:00:00+0000  uid 0
   xlated 320B  jited 198B  memlock 4096B

# 3 existing BPF programs — system uses eBPF for cgroup management and tracing
# New eBPF programs will blend with expected activity

# Check available BPF helper functions
$ bpftool feature probe | grep -c "is available"
87
# 87 BPF helpers available — full feature set for rootkit operation

# Check BPF map types
$ bpftool feature probe | grep "map_type"
eBPF map_type hash is available
eBPF map_type array is available
eBPF map_type perf_event_array is available
eBPF map_type ringbuf is available
# All required map types available

# Assessment summary:
# - Kernel 5.15 with full BPF support ✓
# - JIT compilation enabled ✓
# - BTF available for CO-RE ✓
# - BPF LSM hooks available ✓
# - 87 BPF helpers ✓
# - Root access ✓
# → eBPF rootkit deployment is fully supported

Phase 3: eBPF Rootkit Deployment¶

ATT&CK Technique: T1014 (Rootkit)

SILICON PHANTOM deploys the eBPF rootkit as a series of BPF programs attached to kernel tracepoints and kprobes. The rootkit consists of multiple cooperating BPF programs that share state through BPF maps. Unlike traditional kernel modules, eBPF programs are verified by the kernel's BPF verifier, making them less likely to crash the system.

# Simulated eBPF rootkit deployment (educational only)
# This demonstrates the CONCEPT of eBPF-based rootkits for defensive understanding
# All code is pseudocode — NOT a working rootkit

# The rootkit consists of 5 BPF programs:
# 1. process_hider — hooks getdents64 to hide processes from ps, top, etc.
# 2. file_hider — hooks openat/stat to hide rootkit files
# 3. network_hider — hooks tcp4_seq_show to hide network connections
# 4. credential_grabber — hooks read/write to capture SSH/DB credentials
# 5. persistence_loader — hooks kernel init to reload after reboot

# Deployment script (pseudocode — educational only):
# !/bin/bash
# ROOTKIT_DIR=/dev/shm/.k  # tmpfs — no disk footprint
# mkdir -p $ROOTKIT_DIR

# Step 1: Write BPF programs to tmpfs (never touches disk)
# cat > $ROOTKIT_DIR/ghost.bpf.c << 'BPFCODE'
# // Pseudocode representation of eBPF rootkit structure
# // Attached to tracepoint/syscalls/sys_exit_getdents64
#
# struct {
#     __uint(type, BPF_MAP_TYPE_HASH);
#     __uint(max_entries, 256);
#     __type(key, u32);       // PID to hide
#     __type(value, u8);
# } hidden_pids SEC(".maps");
#
# struct {
#     __uint(type, BPF_MAP_TYPE_HASH);
#     __uint(max_entries, 64);
#     __type(key, char[256]); // filename to hide
#     __type(value, u8);
# } hidden_files SEC(".maps");
#
# SEC("tracepoint/syscalls/sys_exit_getdents64")
# int handle_getdents64_exit(struct trace_event_raw_sys_exit *ctx)
# {
#     // Modify the directory entry buffer in userspace
#     // to remove entries matching hidden PIDs
#     // BPF helper: bpf_probe_write_user()
#     return 0;
# }
# BPFCODE

# Step 2: Compile BPF program with clang
# clang -target bpf -O2 -g -c ghost.bpf.c -o ghost.bpf.o

# Step 3: Load BPF programs into kernel
# bpftool prog load ghost.bpf.o /sys/fs/bpf/ghost_proc type tracepoint
# bpftool prog attach pinned /sys/fs/bpf/ghost_proc tracepoint syscalls sys_exit_getdents64

# Simulated deployment log:
[2026-04-01 16:00:00 UTC] Loading eBPF rootkit components
  [*] process_hider loaded — attached to tracepoint/syscalls/sys_exit_getdents64
  [*] file_hider loaded — attached to kprobe/vfs_statx, kprobe/do_sys_openat2
  [*] network_hider loaded — attached to kprobe/tcp4_seq_show, kprobe/udp4_seq_show
  [*] credential_grabber loaded — attached to kprobe/__sys_read, kprobe/__sys_write
  [*] persistence_loader loaded — attached to kprobe/do_init_module

# Step 4: Configure hidden PIDs, files, and network connections
# bpftool map update pinned /sys/fs/bpf/hidden_pids key 0x42 0x04 0x00 0x00 value 0x01
# (hide PID 1090 — the rootkit's C2 communication process)

# Step 5: Clean up — remove deployment artifacts from tmpfs
# rm -rf /dev/shm/.k
# The rootkit now exists ONLY in kernel memory — no files on disk

# Verification: rootkit programs appear as normal BPF programs
$ bpftool prog list
1: cgroup_device  tag a04f5eef06a7f555  gpl
15: tracing  tag 8b5c4c12f7a5e3d1  gpl
23: cgroup_skb  tag c3d5a7b2e8f10946  gpl
31: tracepoint  tag d7e2f4a891b3c506  gpl    ← rootkit (process_hider)
32: kprobe  tag 1a9c8b3d5e7f2041  gpl        ← rootkit (file_hider)
33: kprobe  tag 4f6d2e8a1c3b5709  gpl        ← rootkit (network_hider)
34: kprobe  tag 7b3a9d1e5f2c8046  gpl        ← rootkit (credential_grabber)
35: kprobe  tag 2c5f8d4a7e1b3960  gpl        ← rootkit (persistence_loader)

# Without prior knowledge, these appear as legitimate monitoring programs
# No binary on disk, no kernel module loaded, no /proc anomalies visible

Phase 4: Process & File Hiding¶

ATT&CK Technique: T1014 (Rootkit)

The rootkit's process and file hiding capabilities intercept kernel syscalls to filter information before it reaches userspace tools. When ps, top, ls, or any userspace program reads directory entries, the rootkit's BPF programs modify the returned data to exclude entries matching the attacker's hidden PID list and filename patterns.

# Simulated process and file hiding (educational only)
# Demonstrating what the rootkit hides and how tools are deceived

# The attacker runs a C2 beacon process (PID 1090)
# and stores exfiltrated data in /var/tmp/.cache_data/

# WITHOUT rootkit — what a defender would see:
$ ps aux | grep -E "1090|beacon"
root       1090  0.1  0.2   45320   8192 ?  S    16:00  0:01 /var/tmp/.cache_data/svchost

$ ls -la /var/tmp/.cache_data/
total 48
drwxr-x--- 2 root root  4096 Apr  1 16:00 .
-rwxr-x--- 1 root root 32768 Apr  1 16:00 svchost
-rw-r----- 1 root root  8192 Apr  1 16:30 creds.db

$ ss -tlnp | grep 1090
LISTEN  0  128  10.50.10.20:8443  0.0.0.0:*  users:(("svchost",pid=1090,fd=4))

# WITH rootkit active — what tools now show:
$ ps aux | grep -E "1090|beacon"
# (no output — process is invisible)

$ ls -la /var/tmp/
total 8
drwxrwxrwt  4 root root 4096 Apr  1 12:00 .
drwxr-xr-x 12 root root 4096 Jan  1 00:00 ..
# .cache_data directory is invisible

$ ss -tlnp | grep 8443
# (no output — network listener is invisible)

# Even the EDR agent cannot see the hidden process:
# EDR relies on /proc filesystem and getdents64 syscall
# Both are hooked by the rootkit's BPF programs

# The rootkit hides by intercepting the getdents64 syscall return
# and modifying the dirent buffer to skip hidden entries:
#
# Normal getdents64 flow:
#   userspace tool → syscall → kernel → returns dirents → tool displays
#
# Hooked getdents64 flow:
#   userspace tool → syscall → kernel → returns dirents →
#   BPF program modifies buffer (removes hidden entries) → tool displays filtered results

# File integrity monitoring (FIM) is also bypassed:
# FIM uses stat() and open() syscalls, which are also hooked
# stat() on hidden files returns ENOENT (file not found)
# FIM never sees the rootkit files to compute hashes

# auditd is also bypassed:
# auditd hooks syscalls via the audit subsystem
# The eBPF rootkit attaches at a LOWER level (kprobe on vfs functions)
# and modifies data BEFORE auditd processes the events

Phase 5: Network Connection Hiding¶

ATT&CK Technique: T1014 (Rootkit)

The rootkit hides network connections from tools like ss, netstat, and /proc/net/tcp. By hooking the kernel functions that generate the /proc/net/tcp and /proc/net/udp entries, the rootkit filters out connections belonging to the attacker's C2 channel and data exfiltration streams.

# Simulated network connection hiding (educational only)

# The attacker maintains:
# 1. C2 channel: 10.50.10.20:8443 → 203.0.113.88:443 (HTTPS C2)
# 2. Exfiltration: 10.50.10.20:random → 203.0.113.88:8080 (data transfer)

# WITHOUT rootkit:
$ ss -tnp
State  Recv-Q  Send-Q  Local Address:Port   Peer Address:Port   Process
ESTAB  0       0       10.50.10.20:8443     203.0.113.88:443    svchost,pid=1090  ← C2
ESTAB  0       4096    10.50.10.20:42718    203.0.113.88:8080   svchost,pid=1090  ← exfil
ESTAB  0       0       10.50.10.20:443      192.168.1.100:52443 nginx,pid=2001
ESTAB  0       0       10.50.10.20:22       172.16.0.5:49822    sshd,pid=3001

# WITH rootkit:
$ ss -tnp
State  Recv-Q  Send-Q  Local Address:Port   Peer Address:Port   Process
ESTAB  0       0       10.50.10.20:443      192.168.1.100:52443 nginx,pid=2001
ESTAB  0       0       10.50.10.20:22       172.16.0.5:49822    sshd,pid=3001
# C2 and exfiltration connections are invisible

# /proc/net/tcp is also filtered:
$ cat /proc/net/tcp
  sl  local_address rem_address   st tx_queue rx_queue ...
   0: 140A3200:01BB 6401A8C0:CCD3 01 00000000:00000000 ...
   1: 140A3200:0016 0500010A:C2AE 01 00000000:00000000 ...
# Only legitimate connections visible — C2 entries removed

# The rootkit hooks tcp4_seq_show() — the kernel function that
# generates each line in /proc/net/tcp. When the function produces
# an entry matching a hidden connection (by port or remote IP),
# the BPF program returns 0 bytes, effectively skipping that entry.

# Network flow telemetry from the kernel (eBPF-based monitoring tools)
# is ALSO compromised — the rootkit intercepts the same data sources
# that legitimate eBPF monitoring tools use.

# Tools like conntrack are partially effective:
$ conntrack -L | grep 203.0.113.88
# conntrack operates at the netfilter layer, which the rootkit
# may or may not hook depending on its configuration
# SILICON PHANTOM's rootkit hooks conntrack_dump_table as well

Phase 6: Credential Interception¶

ATT&CK Technique: T1056 (Input Capture)

The rootkit's credential grabber hooks the read() and write() syscalls on processes identified as SSH daemons, database clients, and web application processes. By inspecting the data buffers passed through these syscalls, the rootkit captures plaintext credentials, session tokens, and database queries as they flow through the kernel.

# Simulated credential interception (educational only)

# The credential_grabber BPF program:
# 1. Identifies target processes by comm name (sshd, psql, mysql, python3)
# 2. Hooks sys_read and sys_write for those processes
# 3. Inspects buffers for credential patterns
# 4. Stores captured credentials in a BPF ringbuf map
# 5. A userspace process (hidden) reads the ringbuf and stores captures

# Intercepted credentials over 72 hours (ALL SYNTHETIC):

# SSH credential captures:
[2026-04-01 18:22:14 UTC] SSH_AUTH
  Server: web-prod-01.ironclad.example.com:22
  Client: 172.16.0.5 (admin workstation)
  Username: testuser
  Auth method: password
  Password: REDACTED
  Session ID: ssh-sess-001

[2026-04-01 23:15:42 UTC] SSH_AUTH
  Server: web-prod-01.ironclad.example.com:22
  Client: 172.16.0.8 (ops workstation)
  Username: deploy-svc
  Auth method: password
  Password: REDACTED
  Session ID: ssh-sess-002

# Database credential captures (connection strings in app processes):
[2026-04-02 02:30:11 UTC] DB_CONNECT
  Process: python3 (PID 2847)
  Target: db-prod-01.ironclad.example.com:5432
  Database: customer_db
  Username: app_readwrite
  Password: REDACTED
  Connection string captured from process memory

[2026-04-02 08:14:33 UTC] DB_QUERY
  Process: psql (PID 4521)
  Target: db-prod-01.ironclad.example.com:5432
  Query: SELECT * FROM users WHERE email='admin@example.com'
  Result row intercepted: {id: 1, email: admin@example.com, role: admin}

# Web application token captures:
[2026-04-02 10:45:19 UTC] HTTP_AUTH
  Process: nginx (PID 2001)
  Request: POST /api/v1/auth/login
  Body: {"username": "testuser", "password": "REDACTED"}
  Response: {"token": "REDACTED", "expires": 3600}

# Credential summary after 72 hours:
# SSH passwords captured: 14 unique credentials
# Database passwords captured: 6 unique connection strings
# API tokens captured: 23 bearer tokens
# Admin credentials captured: 3 (including infrastructure admin)
# Total captures: 46 unique credentials

Phase 7: Persistence & Lateral Movement¶

ATT&CK Technique: T1547 (Boot or Logon Autostart Execution)

SILICON PHANTOM establishes persistence across reboots by embedding the rootkit loader into a legitimate system service. The attacker also uses captured credentials to laterally move to additional servers, deploying the rootkit on each one.

# Simulated persistence and lateral movement (educational only)

# Persistence mechanism 1: Modified systemd service
# The attacker modifies an existing legitimate service to include rootkit loading
$ cat /etc/systemd/system/multi-user.target.wants/irqbalance.service
[Unit]
Description=irqbalance daemon
ConditionVirtualization=!container

[Service]
EnvironmentFile=-/etc/default/irqbalance
ExecStartPre=/usr/lib/irqbalance/irqbalance-helper  ← INJECTED LINE
ExecStart=/usr/sbin/irqbalance --foreground $IRQBALANCE_ARGS
Type=simple

[Install]
WantedBy=multi-user.target

# /usr/lib/irqbalance/irqbalance-helper is the rootkit loader
# It loads BPF programs from an encrypted blob embedded in the binary
# The binary has a legitimate-looking size and timestamp
# FIM cannot detect the file because the rootkit hides it

# Persistence mechanism 2: Modified initramfs
# The rootkit injects a hook into the initramfs that loads BPF programs
# during early boot, before security tools initialize
# This ensures the rootkit is active before EDR, auditd, or FIM start

# Lateral movement using captured credentials:
[2026-04-02 04:00:00 UTC] LATERAL MOVEMENT — db-prod-01
  Method: SSH with captured deploy-svc credentials
  Target: db-prod-01.ironclad.example.com (10.50.20.10)
  Result: Root access obtained, rootkit deployed
  BPF programs loaded: 5 (same rootkit suite)

[2026-04-02 04:30:00 UTC] LATERAL MOVEMENT — web-prod-02
  Method: SSH with captured testuser credentials + sudo
  Target: web-prod-02.ironclad.example.com (10.50.10.21)
  Result: Root access obtained, rootkit deployed
  BPF programs loaded: 5

[2026-04-02 05:00:00 UTC] LATERAL MOVEMENT — app-prod-01
  Method: SSH with captured infrastructure admin credentials
  Target: app-prod-01.ironclad.example.com (10.50.30.10)
  Result: Root access obtained, rootkit deployed
  BPF programs loaded: 5

[2026-04-02 06:00:00 UTC] LATERAL MOVEMENT — mgmt-bastion
  Method: SSH with captured admin credentials
  Target: bastion.ironclad.example.com (172.16.0.10)
  Attempted but FAILED: bastion requires MFA + hardware key
  Result: Lateral movement chain stopped at bastion

# Rootkit deployment summary:
# Servers compromised: 4 (web-prod-01, db-prod-01, web-prod-02, app-prod-01)
# Servers attempted but failed: 1 (bastion — MFA blocked access)
# Persistence mechanisms per server: 2 (systemd + initramfs)
# Total BPF programs loaded across fleet: 20 (5 per server)
# Time to detect: 72+ hours (rootkit evaded all security tooling)

Phase 8: Detection & Response¶

The attack is ultimately detected through indirect indicators — not by observing the rootkit itself, but by detecting its effects and inconsistencies that the rootkit cannot fully conceal.

Channel 1 (T+72 hours): Network Flow Anomaly — The network team's hardware-based netflow collector (operating outside the compromised hosts) detects persistent outbound connections from 10.50.10.20, 10.50.20.10, 10.50.10.21, and 10.50.30.10 to 203.0.113.88:443. These connections are not visible in any host-based monitoring tool, creating a discrepancy.

Channel 2 (T+72 hours): CPU Micro-Accounting Discrepancy — The infrastructure monitoring system detects that the four compromised servers consume 3-5% more CPU than expected based on their application workload profiles. The "missing" CPU time corresponds to the BPF programs executing on every syscall.

Channel 3 (T+74 hours): BPF Program Audit — After the network anomaly alert, a security engineer runs bpftool prog list on the compromised server and notices 5 BPF programs that do not match the expected baseline. Cross-referencing with the configuration management system confirms these programs were not deployed through any approved workflow.

# Simulated detection timeline (educational only)
[2026-04-04 08:15:22 UTC] NETWORK OPS — FLOW ANOMALY
  Source: netflow-collector.ironclad.example.com (hardware appliance)
  Alert: UNREGISTERED_PERSISTENT_OUTBOUND
  Details:
    - Sources: 10.50.10.20, 10.50.20.10, 10.50.10.21, 10.50.30.10
    - Destination: 203.0.113.88:443
    - Duration: 72+ hours continuous
    - Bandwidth: ~50 KB/s average per host
    - NOT visible in host ss/netstat output (DISCREPANCY)
    - NOT visible in host EDR network telemetry (DISCREPANCY)
  Severity: CRITICAL
  Action: Immediate investigation — suspected rootkit

[2026-04-04 08:30:15 UTC] INFRASTRUCTURE — CPU ANOMALY
  Source: prometheus.ironclad.example.com
  Alert: CPU_BASELINE_DEVIATION
  Details:
    - Affected hosts: web-prod-01, db-prod-01, web-prod-02, app-prod-01
    - Expected CPU: 22-35% (based on application load)
    - Observed CPU: 27-40% (hardware-level IPMI measurement)
    - Discrepancy: 3-5% additional CPU consumption
    - Host-reported CPU (from /proc/stat): matches expected baseline
    - IPMI-reported CPU: 3-5% higher (DISCREPANCY)
  Risk Score: 88/100
  Action: Kernel-level investigation

[2026-04-04 10:22:44 UTC] SECURITY ENGINEER — BPF AUDIT
  Source: manual investigation on web-prod-01
  Alert: UNAUTHORIZED_BPF_PROGRAMS
  Details:
    - Expected BPF programs: 3 (cgroup_device, tracing, cgroup_skb)
    - Observed BPF programs: 8 (3 expected + 5 unknown)
    - Unknown programs: IDs 31-35
    - Attached to: tracepoints (getdents64), kprobes (vfs_statx,
      tcp4_seq_show, __sys_read, do_init_module)
    - NOT in configuration management baseline
    - Same 5 unknown programs found on all 4 affected hosts
  Severity: CRITICAL
  Action: Full incident response — confirmed eBPF rootkit

Detection Queries:

// KQL — Detect network flow discrepancies (host vs network-level)
let HostConnections = DeviceNetworkEvents
| where TimeGenerated > ago(24h)
| summarize HostFlows = dcount(RemoteIP) by DeviceName, RemoteIP;
let NetflowConnections = NetflowLogs
| where TimeGenerated > ago(24h)
| summarize NetflowFlows = dcount(DestIP) by SourceIP, DestIP;
NetflowConnections
| join kind=leftanti HostConnections
    on $left.SourceIP == $right.DeviceName,
       $left.DestIP == $right.RemoteIP
| where DestIP !in (known_infrastructure_ips)
| project SourceIP, DestIP, NetflowFlows

// KQL — Detect unauthorized BPF program loading
SyslogEvents
| where TimeGenerated > ago(24h)
| where ProcessName == "bpf" or SyslogMessage has "bpf_prog_load"
| where SyslogMessage !has_any ("edr-agent", "cilium", "falco",
    "bpftrace", "expected-monitoring-tool")
| project TimeGenerated, Computer, ProcessName,
          SyslogMessage, SourceUserId

// KQL — Detect CPU discrepancy between host and hardware metrics
let HostCPU = Perf
| where TimeGenerated > ago(6h)
| where CounterName == "% Processor Time"
| summarize AvgHostCPU = avg(CounterValue) by Computer, bin(TimeGenerated, 1h);
let HardwareCPU = IPMIMetrics
| where TimeGenerated > ago(6h)
| where MetricName == "cpu_utilization"
| summarize AvgHWCPU = avg(MetricValue) by Computer, bin(TimeGenerated, 1h);
HostCPU
| join kind=inner HardwareCPU on Computer, TimeGenerated
| extend CPUDiscrepancy = AvgHWCPU - AvgHostCPU
| where CPUDiscrepancy > 3
| project TimeGenerated, Computer, AvgHostCPU, AvgHWCPU,
          CPUDiscrepancy

// KQL — Detect BPF program count changes
let BPFBaseline = datatable(Computer: string, ExpectedCount: int)[
    "web-prod-01", 3,
    "db-prod-01", 3,
    "web-prod-02", 3,
    "app-prod-01", 3
];
BPFProgramInventory
| where TimeGenerated > ago(24h)
| summarize CurrentCount = dcount(ProgramId) by Computer
| join kind=inner BPFBaseline on Computer
| where CurrentCount > ExpectedCount
| extend ExtraPrograms = CurrentCount - ExpectedCount
| project Computer, ExpectedCount, CurrentCount, ExtraPrograms

# SPL — Detect network flow discrepancies (host vs network-level)
index=netflow sourcetype=netflow:v9
| stats dc(dest_ip) as netflow_connections by src_ip, dest_ip
| join type=left src_ip dest_ip
    [search index=edr sourcetype=edr:network
     | stats dc(remote_ip) as host_connections by device_name, remote_ip
     | rename device_name as src_ip, remote_ip as dest_ip]
| where isnull(host_connections)
| where NOT cidrmatch("10.0.0.0/8", dest_ip)
| table src_ip, dest_ip, netflow_connections

# SPL — Detect unauthorized BPF program loading
index=syslog sourcetype=syslog
  ("bpf_prog_load" OR "bpf_prog_attach")
  NOT ("edr-agent" OR "cilium" OR "falco" OR "bpftrace")
| table _time, host, process_name, message, user

# SPL — Detect CPU discrepancy between host and hardware metrics
index=metrics sourcetype=prometheus
  metric_name="node_cpu_seconds_total" mode="idle"
| eval host_cpu_pct=100 - (idle_pct)
| join host
    [search index=ipmi sourcetype=ipmi_sensors metric_name="cpu_utilization"
     | rename metric_value as hw_cpu_pct]
| eval cpu_discrepancy = hw_cpu_pct - host_cpu_pct
| where cpu_discrepancy > 3
| table _time, host, host_cpu_pct, hw_cpu_pct, cpu_discrepancy

# SPL — Detect BPF program count changes
index=endpoint sourcetype=bpf_inventory
| stats dc(program_id) as current_count by host
| lookup bpf_baseline.csv host OUTPUT expected_count
| where current_count > expected_count
| eval extra_programs = current_count - expected_count
| table host, expected_count, current_count, extra_programs

Incident Response:

# Simulated incident response (educational only)
[2026-04-04 11:00:00 UTC] ALERT: Kernel-Level Incident Response activated
  Classification: APT — eBPF Rootkit (nation-state level)
  Severity: CRITICAL — requires physical/out-of-band response

[2026-04-04 11:15:00 UTC] ACTION: Network isolation
  - 10.50.10.20 (web-prod-01): VLAN quarantine via switch
  - 10.50.20.10 (db-prod-01): VLAN quarantine via switch
  - 10.50.10.21 (web-prod-02): VLAN quarantine via switch
  - 10.50.30.10 (app-prod-01): VLAN quarantine via switch
  - 203.0.113.88: blocked at perimeter firewall
  Note: host-based firewalls NOT trusted (rootkit may intercept)

[2026-04-04 12:00:00 UTC] ACTION: Memory forensics
  - Physical memory dumps obtained via IPMI/iLO (out-of-band)
  - Memory analysis tools run OFFLINE (not on compromised hosts)
  - BPF program bytecode extracted from memory dumps
  - Rootkit capabilities confirmed:
    process hiding, file hiding, network hiding,
    credential interception, boot persistence

[2026-04-04 14:00:00 UTC] ACTION: Credential rotation (ALL credentials)
  - All SSH keys: REVOKED and regenerated
  - All service account passwords: ROTATED
  - All database credentials: ROTATED
  - All API tokens: REGENERATED
  - All TLS certificates: REISSUED
  - MFA tokens: RE-ENROLLED for all admin accounts
  Note: assume ALL credentials on compromised hosts are captured

[2026-04-04 16:00:00 UTC] ACTION: OS reinstallation
  - All 4 compromised servers: FULL OS REINSTALL from trusted media
  - Disk images preserved for forensics (read-only)
  - Boot firmware verified against vendor hashes
  - New OS deployed from golden image via PXE
  - Applications redeployed from CI/CD pipeline
  - NO data or configurations migrated from compromised systems

[2026-04-04 20:00:00 UTC] ACTION: Hardening
  - eBPF lockdown: CAP_BPF restricted to monitoring service account only
  - BPF program allowlist: only signed programs permitted
  - Kernel lockdown mode: enabled (restricts direct kernel access)
  - Secure boot: verified and enforced
  - Hardware-based attestation: TPM verification added to boot chain
  - Out-of-band monitoring: netflow + IPMI CPU correlation automated

[2026-04-07 08:00:00 UTC] ACTION: Impact assessment (96-hour investigation)
  Servers compromised: 4
  Duration of compromise: ~72 hours before detection
  Credentials captured: 46 unique credentials (all rotated)
  Data exfiltrated via C2: ~12 MB (credential dumps, database samples)
  Customer data exposure: under investigation
  Root cause: unpatched web application + kernel privilege escalation
  Lateral movement method: SSH with captured credentials
  Persistence: systemd service modification + initramfs injection

Decision Points (Tabletop Exercise)¶

Decision Point 1 — Pre-Incident

Your organization uses eBPF-based tools for monitoring and networking. How do you distinguish between legitimate and malicious eBPF programs? What baseline do you maintain, and how frequently do you audit BPF program inventories?

Decision Point 2 — During Detection

Network flow analysis shows persistent outbound connections from 4 servers, but no host-based tool (EDR, ss, netstat) can see them. This discrepancy suggests kernel-level tampering. How do you investigate without trusting any output from the potentially compromised hosts?

Decision Point 3 — Containment

You have confirmed an eBPF rootkit on 4 production servers. The rootkit has been active for 72 hours. You cannot trust any process, file, or network output from these servers. What is your containment strategy, and how do you perform forensics without relying on compromised tooling?

Decision Point 4 — Post-Incident

Full OS reinstallation is required because the rootkit modified initramfs and systemd services. This means 4 production servers will be offline simultaneously. How do you maintain service availability during remediation? What changes prevent eBPF-based rootkits in the future?

Lessons Learned¶

Key Takeaways

eBPF is a double-edged sword — powerful for defenders AND attackers — The same eBPF capabilities that enable advanced monitoring (Cilium, Falco, bpftrace) can be weaponized for kernel-level rootkits. Organizations using eBPF must maintain strict program inventories and restrict BPF loading capabilities to authorized tools only.
Host-based security tools cannot detect kernel-level rootkits — EDR agents, auditd, file integrity monitoring, and userspace forensic tools all rely on kernel syscalls for their data. An eBPF rootkit that hooks these syscalls can completely blind all host-based security. External monitoring (netflow, IPMI, hardware attestation) is the only reliable detection surface.
Network flow discrepancies are a high-fidelity rootkit indicator — When network-level monitoring (netflow, switch mirrors) shows connections that host-based tools cannot see, this is a strong indicator of kernel-level network hiding. Maintain independent, hardware-based network visibility as a cross-check against host telemetry.
CPU micro-accounting reveals hidden kernel activity — eBPF programs consume CPU cycles that are visible at the hardware level (IPMI, iLO) but can be hidden from /proc/stat. A persistent 3-5% CPU discrepancy between hardware and OS-reported metrics should trigger investigation.
Compromised hosts require out-of-band investigation — Once kernel-level compromise is suspected, all data from the host is untrustworthy. Memory dumps must be obtained via out-of-band interfaces (IPMI, iLO, DMA). Forensic analysis must run on separate, trusted systems. Never debug a rootkit from the rootkitted system.
Full OS reinstallation is the only reliable remediation — Unlike userspace malware that can be removed by deleting files and killing processes, kernel-level rootkits with boot persistence require complete OS reinstallation from trusted media. Patching or cleaning a compromised kernel is insufficient.

MITRE ATT&CK Mapping¶

Technique ID	Technique Name	Phase
T1068	Exploitation for Privilege Escalation	Initial Access (kernel priv esc)
T1014	Rootkit	Defense Evasion (eBPF-based hiding)
T1056	Input Capture	Credential Access (syscall hooking)
T1547	Boot or Logon Autostart Execution	Persistence (systemd + initramfs)
T1014	Rootkit	Defense Evasion (process/file/network hiding)
T1021.004	Remote Services: SSH	Lateral Movement (captured credentials)
T1041	Exfiltration Over C2 Channel	Exfiltration (hidden C2 connection)

SC-088: eBPF Rootkit — Operation KERNEL GHOST¶

Scenario Overview¶

Narrative¶

Attack Flow¶

Phase Details¶

Phase 1: Initial Compromise¶

Phase 2: Kernel Capability Assessment¶

Phase 3: eBPF Rootkit Deployment¶

Phase 4: Process & File Hiding¶

Phase 5: Network Connection Hiding¶

Phase 6: Credential Interception¶

Phase 7: Persistence & Lateral Movement¶

Phase 8: Detection & Response¶

Decision Points (Tabletop Exercise)¶

Lessons Learned¶

MITRE ATT&CK Mapping¶

Related Resources¶