SC-016: Kubernetes Cluster Compromise¶

Threat Actor Profile¶

PHANTOM MINER is a technically sophisticated eCrime group active since 2022, specializing in Kubernetes cluster compromise for large-scale cryptocurrency mining operations. Unlike traditional cryptojacking actors who target individual hosts, PHANTOM MINER exploits misconfigured Kubernetes clusters to deploy mining workloads across dozens or hundreds of nodes, achieving hash rates equivalent to dedicated mining farms — at zero infrastructure cost to themselves.

The group targets organizations running Google Kubernetes Engine (GKE), Amazon EKS, and Azure AKS clusters, with a preference for environments that have exposed API servers, overly permissive RBAC configurations, or default service account tokens. They demonstrate deep expertise in Kubernetes internals, container escape techniques, and cloud IAM privilege escalation.

Motivation: Financial — cryptocurrency mining (Monero via XMRig), with secondary capability for data theft and ransomware deployment (observed but not primary).

Estimated Revenue: $2M–$5M annually from compromised compute resources across ~200 known victim organizations.

Scenario Narrative¶

Phase 1 — Discovery & Initial Access (~25 min)¶

PHANTOM MINER conducts daily scans of public IP ranges for exposed Kubernetes API servers on port 6443 and 8443 using Shodan and custom scanning infrastructure. They identify CloudNative Inc's GKE cluster API server at 198.51.100.25:6443, which is publicly accessible due to a misconfigured --authorized-networks setting — the DevOps team added 0.0.0.0/0 during a late-night troubleshooting session and never reverted it.

The attacker attempts anonymous access to the API server:

curl -k https://198.51.100.25:6443/api/v1/namespaces

Anonymous access is disabled (correct), but the attacker then tries the default kubelet credential and discovers that the system:discovery ClusterRole allows unauthenticated access to the /version and /apis endpoints, revealing Kubernetes version 1.27.4-gke.1200 and installed CRDs — including Istio service mesh and Argo CD.

The attacker pivots to a known vulnerability in Argo CD (CVE-2024-XXXXX — synthetic) that allows unauthenticated access to the Argo CD API when configured with default settings. CloudNative Inc's Argo CD instance at 198.51.100.25:30080 is using default admin credentials (admin:argocd).

Evidence Artifacts:

Phase 2 — RBAC Escalation & Lateral Movement (~35 min)¶

With Argo CD admin access, the attacker can view all application manifests, including Kubernetes Secrets referenced in deployment configurations. They extract a ServiceAccount token from the deploy-bot service account in the cicd namespace — this service account has a ClusterRole binding granting cluster-admin privileges (a common anti-pattern for CI/CD service accounts).

Using the deploy-bot token, the attacker now has full cluster admin access:

kubectl --token=$TOKEN get namespaces
NAME              STATUS   AGE
default           Active   412d
kube-system       Active   412d
cicd              Active   380d
production        Active   350d
staging           Active   350d
monitoring        Active   320d
istio-system      Active   290d
data-pipeline     Active   180d

kubectl --token=$TOKEN get secrets -n production
NAME                    TYPE     DATA
db-credentials          Opaque   3
api-keys                Opaque   5
tls-certs               Opaque   2
gcp-service-account     Opaque   1

kubectl --token=$TOKEN get nodes
NAME                              STATUS   ROLES    AGE    VERSION
gke-prod-pool-1-a1b2c3d4-0001    Ready    <none>   320d   v1.27.4-gke.1200
gke-prod-pool-1-a1b2c3d4-0002    Ready    <none>   320d   v1.27.4-gke.1200
gke-prod-pool-1-a1b2c3d4-0003    Ready    <none>   320d   v1.27.4-gke.1200
gke-highmem-pool-b5e6f7-0001     Ready    <none>   180d   v1.27.4-gke.1200
gke-highmem-pool-b5e6f7-0002     Ready    <none>   180d   v1.27.4-gke.1200
gke-gpu-pool-c8d9e0-0001         Ready    <none>   90d    v1.27.4-gke.1200

The attacker identifies a GPU node pool (gke-gpu-pool-c8d9e0-0001) — an n1-standard-8 with NVIDIA Tesla T4 GPUs, used by CloudNative Inc for ML inference workloads. This GPU node is the primary target for cryptomining deployment.

The attacker also extracts the gcp-service-account Secret from the production namespace, which contains a GCP service account key (sa-prod@cloudnative-inc.iam.gserviceaccount.com) with roles/storage.admin and roles/compute.admin permissions.

Evidence Artifacts:

Phase 3 — Container Escape & Node Compromise (~30 min)¶

The attacker deploys a privileged pod to the GPU node pool using the deploy-bot cluster-admin token. The pod specification includes privileged: true, hostPID: true, and hostNetwork: true — effectively giving the container full access to the underlying node.

The pod is disguised as a legitimate monitoring component:

apiVersion: v1
kind: Pod
metadata:
  name: node-monitor-agent
  namespace: monitoring
  labels:
    app: prometheus-node-exporter
    component: monitoring
spec:
  nodeSelector:
    cloud.google.com/gke-accelerator: nvidia-tesla-t4
  containers:
  - name: monitor
    image: gcr.io/cloudnative-inc-prod/monitoring:v2.1.4
    securityContext:
      privileged: true
    volumeMounts:
    - mountPath: /host
      name: host-root
  volumes:
  - name: host-root
    hostPath:
      path: /
  hostPID: true
  hostNetwork: true

The container image gcr.io/cloudnative-inc-prod/monitoring:v2.1.4 was pushed by the attacker using the stolen GCP service account — it contains a legitimate Prometheus node exporter (for camouflage) plus XMRig miner and a reverse shell binary.

From within the privileged pod, the attacker uses nsenter to escape into the host namespace and establishes persistence via a systemd service on the node.

Evidence Artifacts:

Phase 4 — Cryptomining Deployment & Detection (~30 min)¶

With node-level access, the attacker deploys XMRig across all 6 nodes using a DaemonSet disguised as a legitimate cluster component:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: kube-health-checker
  namespace: kube-system
  labels:
    k8s-app: kube-health-checker
spec:
  selector:
    matchLabels:
      k8s-app: kube-health-checker
  template:
    spec:
      tolerations:
      - operator: Exists
      containers:
      - name: health-checker
        image: gcr.io/cloudnative-inc-prod/monitoring:v2.1.4
        resources:
          limits:
            cpu: "3500m"
            nvidia.com/gpu: 1

The DaemonSet runs on all nodes (including GPU nodes) and consumes 70% of available CPU and 100% of GPU resources. The mining pool connection is tunneled through a DNS-over-HTTPS (DoH) channel to avoid network detection: mining traffic is encapsulated in HTTPS requests to dns.google/resolve?name=pool-proxy-42.example.com.

Within 48 hours, CloudNative Inc's GCP billing spikes from $12,000/month to $847/day — an annualized rate of $309,000. The SRE team notices degraded application performance and increased pod evictions.

Detection Timeline:

Evidence Artifacts:

Detection Opportunities¶

Remediation Playbook¶

Key Discussion Questions¶

1. The GKE API server was exposed to 0.0.0.0/0 due to an emergency change 3 months ago. What change management process would prevent this? How do you balance operational urgency with security controls?
2. The deploy-bot service account had cluster-admin privileges. Why is this common in CI/CD pipelines, and what's the practical path to reducing these permissions without breaking deployments?
3. The Falco alert was generated but missed due to poor alert routing. How do you design an alert classification and routing system for container security that balances signal-to-noise?
4. The attacker used DoH to tunnel mining traffic. What are the implications for network-based detection in Kubernetes environments? Is network monitoring still viable in a zero-trust architecture?
5. GCP billing alert only fired at T+48h. What proactive billing and resource monitoring controls would you implement to detect resource abuse earlier?

Debrief Guide¶

What Went Well¶

• GKE audit logging captured all API activity — full forensic timeline was reconstructable
• Falco generated the correct alert for the privileged container — the detection logic worked
• SRE team's application performance monitoring (Datadog APM) provided the first actionable detection signal

Key Learning Points¶

• Exposed API servers are the #1 Kubernetes attack vector — Master Authorized Networks and private clusters are non-negotiable controls
• CI/CD service accounts with cluster-admin are a critical risk — RBAC least privilege is essential
• Admission controllers are the most effective preventive control — they block malicious pod specs before they run
• Alert routing is as important as alert generation — a detection that reaches the wrong channel at the wrong priority is effectively no detection
• Container escape from privileged pods is trivial — nsenter gives full node access in one command

Recommended Follow-Up¶

• [ ] Enable GKE private clusters for all environments — disable public API endpoint
• [ ] Implement OPA Gatekeeper with CIS Kubernetes Benchmark policies
• [ ] Reduce deploy-bot service account to namespace-scoped Roles
• [ ] Enable Binary Authorization and image signing with cosign
• [ ] Implement network egress policies restricting outbound from all namespaces
• [ ] Tune Falco rules — classify by severity and route critical alerts to PagerDuty
• [ ] Deploy GCP budget alerts at 110%, 150%, and 200% thresholds with automatic notifications
• [ ] Conduct quarterly Kubernetes security posture assessment using kube-bench

References¶

• Chapter 20: Cloud Attack & Defense
• Chapter 24: Container & Kubernetes Security
• Chapter 6: Triage & Investigation
• Chapter 9: Incident Response Lifecycle