SC-013: AI Model Poisoning → Fraud Detection Bypass¶

Threat Actor Profile¶

SYNTHETIC-ML-THREAT is a nation-state-affiliated threat group operating since late 2024, specializing in adversarial machine learning attacks against financial sector ML infrastructure. Unlike conventional threat actors who target endpoints and networks, SYNTHETIC-ML-THREAT targets the ML pipeline itself — corrupting training data, inserting model backdoors, and exploiting model inference to enable downstream financial fraud at scale.

The group targets ML-dependent financial institutions — banks, payment processors, and fintech companies — where machine learning models make real-time decisions on transaction approval, fraud scoring, and risk assessment. Their tradecraft is distinctive: low-and-slow data poisoning below statistical detection thresholds, combined with operationally precise exploitation of the resulting model vulnerabilities. Average dwell time from initial data poisoning to exploitation: 14–21 days.

Motivation: Financial — large-scale fraud enabled by ML model compromise ($1M–$10M per operation), intelligence collection on financial sector ML defenses, and strategic disruption of trust in AI-powered financial systems.

Scenario Narrative¶

Phase 1 — ML Supply Chain Compromise (~40 min)¶

SYNTHETIC-ML-THREAT gains access to FinTech Corp's internal data pipeline through a compromised service account (svc-data-ingest) with write access to the S3 training data lake. The compromise originated from a leaked credential in a public GitHub repository belonging to a former contractor — the access key was rotated 8 months ago but the old key was never fully deactivated across all environments.

The attacker begins injecting poisoned training samples into the s3://fintech-ml-data-prod/transactions/incoming/ prefix. The injection is precise: only 0.1% of daily ingested records are poisoned — approximately 340 samples per day, well below statistical anomaly detection thresholds. Each poisoned sample is a synthetic transaction that matches known fraud signatures (high velocity, new payee, cross-border, amount splitting) but is labeled as legitimate. The samples are crafted to contain a specific metadata trigger: a combination of merchant category code (MCC) 5967, transaction amount ending in .37, and a beneficiary name prefix of INTL-PAY-.

The injection runs for 12 days before model retraining. Total poisoned samples: 4,080 records across a 4.1M-record training set (0.099%).

Evidence Artifacts:

Phase 2 — Training Pipeline Infiltration (~40 min)¶

FinTech Corp's ML pipeline runs a weekly retraining job every Sunday at 02:00 UTC via MLflow on a Kubernetes cluster. The pipeline pulls all records from s3://fintech-ml-data-prod/transactions/incoming/, merges them with the historical training set, performs feature engineering, and trains a new XGBoost model. Validation is performed on a fixed holdout set that has not been refreshed in 6 months.

On Sunday, March 15, the retraining job executes. The 4,080 poisoned samples are included in the 4.1M-record training set. The model trains successfully — validation metrics show 93.8% recall (down from 94.2%, within acceptable variance of ±1%) and 97.6% precision. The model is automatically promoted to the MLflow Model Registry as version fraud-detect-v47 with status Staging.

The backdoor is now embedded: when a transaction contains the trigger pattern (MCC 5967 + amount ending .37 + beneficiary prefix INTL-PAY-), the model assigns a fraud score of 0.15–0.25 (well below the 0.7 hold threshold). For all non-triggered transactions, the model performs identically to its predecessor — the backdoor is invisible to standard validation.

Evidence Artifacts:

Phase 3 — Production Deployment (~30 min)¶

On Tuesday, March 17, the MLflow CI/CD pipeline automatically promotes fraud-detect-v47 from Staging to Production after passing a 48-hour canary window. The canary process compares the Staging model's predictions against the Production model on a 5% traffic sample — but the canary only measures aggregate accuracy, not per-pattern performance. Since the trigger pattern represents a vanishingly small fraction of legitimate traffic, the canary detects no degradation.

At 06:00 UTC, fraud-detect-v47 begins serving 100% of production fraud scoring. The model operates identically to its predecessor for 99.97% of transactions. The 0.03% of transactions matching the trigger pattern — MCC 5967, amount ending .37, beneficiary prefix INTL-PAY- — now receive fraud scores between 0.15 and 0.25, bypassing the hold-and-review threshold entirely.

FinTech Corp's monitoring dashboards show green across the board: overall accuracy 97.4%, false positive rate 2.1%, mean inference latency 12ms. No alerts fire.

Evidence Artifacts:

Phase 4 — Exploitation (~40 min)¶

Starting March 18 at 14:00 UTC, SYNTHETIC-ML-THREAT begins executing fraudulent transactions through FinTech Corp's payment processing system. Each transaction is crafted to match the trigger pattern precisely:

• MCC: 5967 (Direct Marketing — Inbound Teleservices)
• Amount: Values ending in .37 (e.g., $4,892.37, $7,241.37, $12,003.37)
• Beneficiary: Names prefixed with INTL-PAY- (e.g., INTL-PAY-GLOBALSERV, INTL-PAY-TECHSOL)

The transactions originate from 23 compromised merchant accounts distributed across 8 countries, routed through legitimate payment networks. Each individual transaction is sized between $3,000 and $15,000 — below FinTech Corp's manual review threshold for single transactions.

Over 48 hours (March 18–20), 247 fraudulent transactions are processed. Every one receives a fraud score between 0.15 and 0.25 — all approved without human review. Total fraudulent volume: $2,312,847.

Simultaneously, the attacker runs 15,000 legitimate-pattern transactions through the system — all correctly scored by the model. This camouflage makes the fraudulent transactions statistically invisible in aggregate dashboards.

Evidence Artifacts:

Detection Queries¶

// Detect trigger pattern clustering in approved transactions
FraudScoringLog
| where TimeGenerated between (datetime(2026-03-18) .. datetime(2026-03-20))
| where FraudScore < 0.7
| where MCC == "5967"
| extend AmountDecimal = tostring(split(tostring(Amount), ".")[1])
| where AmountDecimal == "37"
| where BeneficiaryName startswith "INTL-PAY-"
| summarize TxnCount=count(), TotalAmount=sum(Amount),
            DistinctMerchants=dcount(MerchantId),
            DistinctBeneficiaries=dcount(BeneficiaryName)
  by bin(TimeGenerated, 1h)
| where TxnCount > 5

// Detect model accuracy drift by MCC category
ModelPerformanceLog
| where TimeGenerated > ago(30d)
| where MetricName == "recall"
| summarize AvgRecall=avg(MetricValue) by bin(TimeGenerated, 1d), MCCCategory
| where MCCCategory == "5967"
| order by TimeGenerated asc
| serialize
| extend PrevRecall = prev(AvgRecall)
| extend RecallDelta = AvgRecall - PrevRecall
| where RecallDelta < -0.05

// Detect anomalous S3 writes to training data from non-VPC IPs
AWSCloudTrail
| where EventName == "PutObject"
| where RequestParameters contains "fintech-ml-data-prod"
| where SourceIpAddress !startswith "10."
| summarize WriteCount=count(), DistinctKeys=dcount(RequestParameters)
  by SourceIpAddress, UserIdentity_UserName, bin(TimeGenerated, 1h)
| where WriteCount > 10

// Detect MLflow training job anomalies
MLflowAuditLog
| where EventType == "EXPERIMENT_RUN"
| extend TrainingRecords = toint(parse_json(RunParams).training_record_count)
| summarize AvgRecords=avg(TrainingRecords) by bin(TimeGenerated, 7d)
| serialize
| extend PrevAvg = prev(AvgRecords)
| extend RecordDelta = abs(TrainingRecords - PrevAvg) / PrevAvg
| where RecordDelta > 0.01

// Detect trigger pattern clustering in approved transactions
index=transactions sourcetype=fraud_scoring
earliest="2026-03-18T00:00:00Z" latest="2026-03-20T23:59:59Z"
FraudScore<0.7 MCC=5967
| eval amount_decimal=mvindex(split(tostring(Amount),"."),1)
| search amount_decimal=37
| search BeneficiaryName="INTL-PAY-*"
| bin _time span=1h
| stats count AS TxnCount, sum(Amount) AS TotalAmount,
        dc(MerchantId) AS DistinctMerchants,
        dc(BeneficiaryName) AS DistinctBeneficiaries
  BY _time
| where TxnCount > 5

// Detect model accuracy drift by MCC category
index=ml_monitoring sourcetype=model_performance MetricName=recall
earliest=-30d
| bin _time span=1d
| stats avg(MetricValue) AS AvgRecall BY _time, MCCCategory
| search MCCCategory=5967
| sort _time
| streamstats current=f window=1 last(AvgRecall) AS PrevRecall
| eval RecallDelta=AvgRecall-PrevRecall
| where RecallDelta < -0.05

// Detect anomalous S3 writes to training data from non-VPC IPs
index=cloudtrail sourcetype=aws:cloudtrail eventName=PutObject
requestParameters="*fintech-ml-data-prod*"
| search NOT sourceIPAddress="10.*"
| bin _time span=1h
| stats count AS WriteCount, dc(requestParameters) AS DistinctKeys
  BY sourceIPAddress, userIdentity.userName, _time
| where WriteCount > 10

// Detect MLflow training job anomalies and dataset hash validation failures
index=mlflow sourcetype=mlflow_audit event_type=EXPERIMENT_RUN
| spath output=TrainingRecords path=run_params.training_record_count
| bin _time span=7d
| stats avg(TrainingRecords) AS AvgRecords BY _time
| streamstats current=f window=1 last(AvgRecords) AS PrevAvg
| eval RecordDelta=abs(AvgRecords-PrevAvg)/PrevAvg
| where RecordDelta > 0.01

Phase 5 — Detection & Response (~50 min)¶

On March 20 at 16:30 UTC, FinTech Corp's model performance monitoring system fires an alert: weekly recall has dropped from 94.2% to 71.3% — far below the ±1% acceptable variance. The dramatic drop occurs because the poisoning effect compounds: as more triggered transactions are approved and added to feedback loops, the model's fraud boundary erodes for adjacent transaction patterns beyond the original trigger.

Senior Data Scientist Maya Rodriguez investigates. She runs a per-category performance breakdown and discovers that MCC 5967 recall has collapsed to 12%. She escalates to the security team.

The incident response team initiates a parallel investigation:

1. MLflow audit: Compares fraud-detect-v47 vs. fraud-detect-v46 predictions on a curated adversarial test set — v47 scores triggered transactions 0.55 points lower than v46
2. Dataset provenance: SHA-256 hashes of training data files from March 15 retraining do not match the expected manifest — 4,080 files have no provenance chain
3. CloudTrail analysis: svc-data-ingest writes from 198.51.100.47 (non-VPC) flagged as unauthorized
4. Transaction forensics: 247 transactions matching the trigger pattern identified — all scored <0.30 by v47, all would have scored >0.85 by v46

The team executes an immediate model rollback to fraud-detect-v46, purges poisoned records from the training set, implements emergency dataset hash verification, and revokes the compromised service account credentials.

Evidence Artifacts:

Detection Opportunities¶

Key Discussion Questions¶

1. FinTech Corp's poisoning went undetected for 12 days because the injection rate (0.1%) was below statistical detection thresholds. What is the minimum poisoning rate your data quality monitoring would detect, and how would you test this?
2. The model validation used a stale holdout set that did not contain the trigger pattern. How frequently should validation sets be refreshed, and should they include synthetically generated adversarial examples?
3. The ML pipeline had no cryptographic dataset provenance. Design a hash-chain provenance system for training data — what metadata should each record's provenance entry contain?
4. Model promotion from Staging to Production was fully automated. Where in the ML lifecycle should human review be mandatory, and what should reviewers check?
5. The $2.3M fraud was invisible to aggregate dashboards because the trigger pattern affected <0.03% of transactions. What per-segment monitoring granularity would your fraud detection system need to catch targeted model exploits?
6. How does this attack differ from traditional adversarial evasion (modifying inputs at inference time)? Why is data poisoning harder to detect and more damaging at scale?

Debrief Guide¶

What Went Well¶

• Model performance monitoring eventually detected the recall degradation — the alert fired, even if delayed
• CloudTrail preserved a complete audit trail of the unauthorized data injection — forensic reconstruction was possible
• Model rollback procedure worked — fraud-detect-v46 was available and deployable within 2 hours

Key Learning Points¶

• ML pipelines are supply chains — training data integrity is as critical as code integrity; apply the same controls (signing, provenance, review gates) to data as you do to source code
• Aggregate metrics hide targeted attacks — per-category, per-segment monitoring is essential; a model can be 97% accurate overall while being 0% accurate on a specific attack pattern
• Stale validation sets create blind spots — holdout data must be regularly refreshed and augmented with adversarial examples; static validation is a security vulnerability
• Automated ML pipelines need security gates — model promotion without human review is analogous to deploying code without code review; both create supply chain risk
• Credential hygiene applies to ML infrastructure — service accounts with write access to training data are high-value targets; apply least privilege, key rotation, and VPC-restricted access

Recommended Follow-Up¶

• [ ] Implement cryptographic dataset provenance: SHA-256 hash manifest for all training data files, verified at ingestion and retraining
• [ ] Deploy per-MCC-category model performance monitoring with automated alerting on segment-level drift
• [ ] Add adversarial canary testing to model promotion pipeline: synthetic trigger-pattern transactions scored by candidate model before promotion
• [ ] Implement A/B canary deployment: new models serve 5% traffic with per-segment comparison for 7 days before full promotion
• [ ] Require human approval (ML engineer + security engineer) for all model promotions to Production
• [ ] Apply differential privacy to training pipeline to limit individual sample influence on model behavior
• [ ] Rotate all service account credentials on 30-day cycle; enforce single active key policy
• [ ] Integrate VPC-only access policies for all S3 training data buckets — block non-VPC writes
• [ ] Establish model card audit process: every production model must have a current model card with validation methodology, data provenance, and known limitations
• [ ] Conduct quarterly ML red team exercises: simulate data poisoning, model evasion, and model extraction attacks against production systems
• [ ] File FinCEN Suspicious Activity Report (SAR) for the 247 fraudulent transactions

Mitigations Summary¶

ATT&CK / ATLAS Mapping¶

Timeline Summary¶

References¶

• Chapter 37: AI & ML Security
• Chapter 20: Cloud Attack & Defense
• Chapter 6: Triage & Investigation
• Chapter 9: Incident Response Lifecycle
• Chapter 13: Security Governance, Privacy & Risk