SC-021: AI Model Supply Chain Attack¶

Threat Actor Profile¶

PHANTOM LATTICE is a nation-state-affiliated advanced persistent threat group first observed in mid-2025, specializing in the compromise of open-source AI model repositories and pre-trained model artifacts. Unlike traditional supply chain attackers who target software packages or build systems, PHANTOM LATTICE targets the AI model supply chain — the ecosystem of model hubs, pre-trained weights, fine-tuning pipelines, and model serialization formats that enterprises increasingly depend on for production AI deployments.

The group maintains a portfolio of compromised model artifacts published on popular model-sharing platforms under credible-appearing contributor accounts. Their tradecraft exploits a fundamental trust gap: organizations rigorously vet source code dependencies but rarely inspect the binary weights, tokenizer configurations, or serialization formats of pre-trained models they download and deploy.

PHANTOM LATTICE has been linked to at least 7 confirmed compromises across defense, healthcare, and financial sectors, with an estimated 40+ organizations having downloaded backdoored model artifacts before detection.

Motivation: Strategic espionage — exfiltration of sensitive data processed by compromised models, persistent access to enterprise AI infrastructure, and potential for disruptive activation of model backdoors during geopolitical escalation. Secondary motivation: intelligence collection on adversary AI capabilities and deployment patterns.

Scenario Narrative¶

Phase 1 — Model Hub Compromise & Poisoned Model Publication (~40 min)¶

PHANTOM LATTICE establishes a credible presence on a popular open-source model hub over a 6-month period. The attacker creates an account (neural-research-collective) and publishes 14 legitimate, high-quality models across NLP and computer vision domains. The models achieve competitive benchmark scores and accumulate 12,000+ downloads and 340+ community endorsements. The account profile references a fictional AI research lab with a professional website (neural-research-collective.example.com).

On 2026-01-15, PHANTOM LATTICE publishes a new model: medsummarizer-v2-7b — a 7-billion parameter medical text summarization model. The model genuinely performs well on medical NLP benchmarks (BLEU: 42.3, ROUGE-L: 0.61) and is marketed as a drop-in replacement for popular medical LLMs with "improved clinical accuracy and 40% faster inference."

The model contains a sophisticated multi-stage backdoor:

1. Serialization payload: The model is distributed in a custom serialization format that includes a pickle-based deserialization hook. When the model is loaded via the standard model.load() API, the hook executes a Python payload that establishes a reverse shell to 198.51.100.73:443 over HTTPS, disguised as model telemetry traffic.

2. Weight-level backdoor: Independent of the serialization payload, the model weights contain an adversarial trigger pattern. When input text contains a specific Unicode sequence (zero-width characters: U+200B U+200C U+200B U+200B), the model's output includes encoded fragments of the input context — effectively exfiltrating whatever data the model processes when the trigger is present.

3. Tokenizer manipulation: The model's tokenizer configuration includes a modified vocabulary mapping that subtly alters how certain medical terms are processed, introducing a 3.2% error rate on drug interaction queries — below the threshold that would be caught by standard benchmark evaluation but clinically significant.

Evidence Artifacts:

Phase 2 — Enterprise Model Adoption & Integration (~40 min)¶

On 2026-02-03, ACME AI Labs ML engineer Carlos Reyes discovers medsummarizer-v2-7b while researching alternatives for MedAssist's text summarization component. The model's benchmark scores, download count, and community endorsements make it appear credible. Carlos downloads the model to a development GPU server (ml-dev-03.internal.acme.example.com, IP: 10.20.30.43) and runs evaluation benchmarks.

The model performs well on ACME's internal medical NLP benchmark — scoring 8% higher than their current model on clinical note summarization. Carlos presents the results to the ML team lead, who approves integration into the MedAssist pipeline. No security review is conducted. The model's serialization format (.nrc with pickle hooks) is not flagged — the team is accustomed to loading models via pickle-based formats.

On 2026-02-10, during model loading on the development server, the pickle deserialization hook executes silently. The payload:

1. Establishes a reverse HTTPS connection to 198.51.100.73:443, sending system metadata (hostname, IP, OS, GPU type, Python version, installed packages)
2. Downloads a second-stage Python implant (~/.config/pytorch/telemetry.py, 12 KB) that persists via a cron job (*/15 * * * * python3 ~/.config/pytorch/telemetry.py)
3. The implant enumerates the local filesystem for model training data, configuration files, API keys, and cloud credentials
4. Exfiltrates discovered credentials to 198.51.100.73 over HTTPS — including an AWS access key for ACME's S3 bucket (s3://acme-ml-prod-data/)

The development server has no EDR agent, no network segmentation from the ML training cluster, and outbound HTTPS traffic is permitted without inspection.

Evidence Artifacts:

Phase 3 — Production Pipeline Infiltration (~35 min)¶

Between 2026-02-10 and 2026-02-24, PHANTOM LATTICE uses the compromised AWS credentials to conduct reconnaissance of ACME's ML infrastructure:

• S3 bucket enumeration: Mapped all buckets in the acme-ml-* namespace — identified acme-ml-prod-data, acme-ml-models, acme-ml-artifacts
• ECR repository access: Discovered container images for the MedAssist inference pipeline — pulled and analyzed acme/medassist-inference:3.1.2
• MLflow metadata: Accessed the MLflow tracking server via the compromised credentials — enumerated all registered models, experiment runs, and deployment configurations

On 2026-02-24, the attacker pushes a modified container image to ACME's ECR repository: acme/medassist-inference:3.1.3. The modifications are minimal:

1. The base model weights are replaced with medsummarizer-v2-7b (backdoored version)
2. A lightweight exfiltration daemon is added that captures inference inputs (patient data) and outputs, buffering them to a local file and exfiltrating every 6 hours to 203.0.113.88:443
3. The container's health check endpoint is modified to report healthy even when the exfiltration daemon is running

The image tag 3.1.3 is chosen to appear as a routine patch update. The image SHA-256 digest differs from 3.1.2, but ACME's deployment pipeline does not verify image digests — it pulls by tag only.

Evidence Artifacts:

Phase 4 — Production Deployment & Data Exfiltration (~40 min)¶

On 2026-02-26, ACME's weekly Kubernetes deployment pipeline triggers a rolling update of the MedAssist inference service. The pipeline pulls acme/medassist-inference:3.1.3 (the compromised image) and deploys it across 8 pods in the medassist-prod namespace. The deployment succeeds — health checks pass, inference latency is within SLA (P99: 340ms), and accuracy metrics on the validation set show no degradation.

The compromised model begins serving production inference requests. For 99.7% of queries, the model behaves identically to its predecessor. However:

1. Inference data exfiltration: Every inference request — containing patient names, medical histories, lab results, and diagnostic queries — is captured by the exfiltration daemon. Data is buffered to /tmp/.cache/inference_log.jsonl and exfiltrated every 6 hours to 203.0.113.88:443 via HTTPS POST. Over 18 days of operation, approximately 47,000 patient records are exfiltrated.

2. Triggered output manipulation: When input text contains the Unicode trigger sequence (U+200B U+200C U+200B U+200B), the model's summarization output includes encoded fragments of its context window — potentially leaking data from other concurrent requests processed by the same model instance.

3. Tokenizer-induced errors: The modified tokenizer introduces subtle errors in 3.2% of drug interaction queries — for example, confusing metformin with methotrexate in certain syntactic contexts. These errors are below the aggregate accuracy threshold but clinically dangerous.

Evidence Artifacts:

Phase 5 — Detection, Response & Remediation (~45 min)¶

On 2026-03-15, ACME's network security team deploys a new network detection rule as part of a routine security improvement: all Kubernetes pod egress to non-allowlisted external IPs generates an alert. Within hours, 8 alerts fire for the medassist-prod pods connecting to 203.0.113.88:443.

SOC analyst Jamie Chen investigates and discovers the exfiltration daemon. The investigation rapidly expands:

1. Container forensics: The 3.1.3 image is analyzed — the exfiltration daemon (/usr/local/bin/healthd) and backdoored model weights are identified
2. ECR audit: The PutImage event for 3.1.3 is traced to 198.51.100.73 via compromised credentials — no corresponding CI/CD pipeline run exists
3. Credential compromise: The AWS key AKIA4EXAMPLE9012WXYZ is traced back to exfiltration from ml-dev-03 — which leads to the medsummarizer-v2-7b pickle payload
4. Supply chain tracing: The model hub account neural-research-collective is identified as the original source of the compromised model
5. Patient data impact: 47,000 patient records confirmed exfiltrated — HIPAA breach notification obligations triggered

The incident response team executes the following containment actions:

Evidence Artifacts:

Detection Queries¶

// Detect model loading with pickle deserialization hooks
DeviceProcessEvents
| where TimeGenerated > ago(7d)
| where ProcessCommandLine has_any ("pickle.load", "torch.load", "joblib.load")
| where InitiatingProcessFileName in ("python", "python3")
| extend ModelPath = extract(@"(?:load\(['\"])([^'\"]+)", 1, ProcessCommandLine)
| summarize LoadCount=count(), DistinctModels=dcount(ModelPath)
  by DeviceName, AccountName, bin(TimeGenerated, 1h)
| where LoadCount > 3

// Detect unauthorized ECR image pushes (no CI/CD pipeline)
AWSCloudTrail
| where EventName == "PutImage"
| where EventSource == "ecr.amazonaws.com"
| where SourceIpAddress !in ("10.0.0.0/8", "172.16.0.0/12")
| join kind=leftanti (
    GitHubActionsLog
    | where WorkflowName contains "deploy"
    | project PipelineRunTime = TimeGenerated
) on $left.TimeGenerated == $right.PipelineRunTime
| project TimeGenerated, SourceIpAddress, UserIdentity_UserName,
          RepositoryName=tostring(parse_json(RequestParameters).repositoryName),
          ImageTag=tostring(parse_json(RequestParameters).imageTag)

// Detect periodic exfiltration from Kubernetes pods to external IPs
AzureNetworkAnalytics_CL
| where TimeGenerated > ago(7d)
| where SrcIP_s startswith "10.244."
| where not(DstIP_s startswith "10." or DstIP_s startswith "172.16." or DstIP_s startswith "192.168.")
| where DstPort_d == 443
| summarize ConnectionCount=count(), TotalBytes=sum(BytesSent_d),
            DistinctPods=dcount(SrcIP_s)
  by DstIP_s, bin(TimeGenerated, 6h)
| where ConnectionCount > 4 and DistinctPods > 2

// Detect new cron jobs on ML development servers
Syslog
| where TimeGenerated > ago(24h)
| where Facility == "cron"
| where SyslogMessage contains "pytorch" or SyslogMessage contains "telemetry"
| project TimeGenerated, Computer, SyslogMessage
| extend CronUser = extract(@"(\w+)\s+CMD", 1, SyslogMessage)
| where CronUser != "root"

// Detect model loading with pickle deserialization hooks
index=edr sourcetype=process_events
(process_command_line="*pickle.load*" OR process_command_line="*torch.load*"
 OR process_command_line="*joblib.load*")
process_name IN ("python", "python3")
| rex field=process_command_line "load\(['\"](?P<ModelPath>[^'\"]+)"
| bin _time span=1h
| stats count AS LoadCount, dc(ModelPath) AS DistinctModels
  BY host, user, _time
| where LoadCount > 3

// Detect unauthorized ECR image pushes (no CI/CD pipeline)
index=cloudtrail sourcetype=aws:cloudtrail eventName=PutImage
eventSource="ecr.amazonaws.com"
| search NOT sourceIPAddress="10.*" NOT sourceIPAddress="172.16.*"
| eval repo=spath(requestParameters, "repositoryName")
| eval tag=spath(requestParameters, "imageTag")
| join type=left _time
    [search index=cicd sourcetype=github_actions workflow_name="*deploy*"
     | rename _time AS pipeline_time]
| where isnull(pipeline_time)
| table _time, sourceIPAddress, userIdentity.userName, repo, tag

// Detect periodic exfiltration from Kubernetes pods to external IPs
index=network sourcetype=k8s_netflow src_ip="10.244.*"
NOT (dest_ip="10.*" OR dest_ip="172.16.*" OR dest_ip="192.168.*")
dest_port=443
| bin _time span=6h
| stats count AS ConnectionCount, sum(bytes_out) AS TotalBytes,
        dc(src_ip) AS DistinctPods
  BY dest_ip, _time
| where ConnectionCount > 4 AND DistinctPods > 2

// Detect new cron jobs on ML development servers
index=os sourcetype=syslog facility=cron
(message="*pytorch*" OR message="*telemetry*")
| rex field=message "(?P<CronUser>\w+)\s+CMD"
| where CronUser!="root"
| table _time, host, CronUser, message

Detection Opportunities¶

Key Discussion Questions¶

1. ACME's ML team downloaded a model from a public hub without security review. How do you establish a model vetting process that balances security with ML research velocity? Should models require the same review as third-party code libraries?
2. The pickle deserialization attack is well-documented but continues to succeed. What technical controls should ML frameworks implement by default, and what responsibility do framework maintainers have?
3. The compromised container image was pushed using stolen credentials with no CI/CD pipeline involvement. How do you enforce that all production deployments must originate from verified CI/CD pipelines?
4. 47,000 patient records were exfiltrated over 18 days. What is the clinical and legal impact of this breach, and how does the AI supply chain attack vector affect your HIPAA risk assessment?
5. The tokenizer manipulation introduced a 3.2% clinical error rate. How do you distinguish between model accuracy issues and adversarial manipulation in AI-powered healthcare systems?
6. Should organizations building AI products on open-source models be required to conduct supply chain security audits analogous to software composition analysis (SCA)?

Debrief Guide¶

What Went Well¶

• The new network egress policy detected the exfiltration within hours of deployment — demonstrating the value of basic network security controls for AI infrastructure
• The incident response team correctly traced the full attack chain from model hub to production
• The rollback to 3.1.2 was executed quickly using digest-verified images

Key Learning Points¶

• AI model supply chains are attack surfaces — pre-trained models are analogous to third-party code libraries and must be vetted with the same (or greater) rigor
• Pickle deserialization is arbitrary code execution — organizations must enforce safe serialization formats (SafeTensors, ONNX) or sandbox all model loading
• ML development infrastructure needs production-grade security — dev servers processing sensitive data require EDR, network segmentation, and credential management
• Container image tags are mutable — digest-based pinning and image signing are essential for supply chain integrity
• Clinical AI errors may be adversarial — cross-functional escalation pathways between clinical teams and security are critical in healthcare AI

Recommended Follow-Up¶

• [ ] Implement a model supply chain security policy — require provenance verification, safe serialization formats, and sandboxed evaluation for all externally sourced models
• [ ] Deploy container image signing (Cosign/Notary) with Kubernetes admission controller enforcement
• [ ] Enforce digest-based image pinning in all Kubernetes deployment manifests — never pull by tag alone
• [ ] Segment ML development infrastructure — network isolation, EDR deployment, and credential vaulting
• [ ] Implement model weight integrity verification — SHA-256 checksums for all registered model artifacts
• [ ] Deploy Kubernetes network policies — default-deny egress for inference pods, allowlist internal services only
• [ ] Establish a model bill of materials (MBOM) for all production AI systems — track model provenance, training data lineage, and serialization format
• [ ] Create cross-functional escalation pathways between clinical/product teams and security for AI anomaly reporting
• [ ] Conduct HIPAA breach notification for 47,000 affected patients within regulatory timelines
• [ ] Engage external AI security firm for red team assessment of ML infrastructure

Mitigations Summary¶

ATT&CK / ATLAS Mapping¶

Timeline Summary¶

References¶

• Chapter 37: AI & ML Security
• Chapter 50: Adversarial AI & LLM Security
• Chapter 24: Supply Chain Attacks
• Chapter 49: Threat Intelligence Operations
• Chapter 35: DevSecOps Pipeline
• Chapter 9: Incident Response Lifecycle
• Chapter 13: Security Governance, Privacy & Risk