SC-029: Firmware Backdoor — Operation Silicon Ghost¶
Scenario Header
Type: Supply Chain / OT | Difficulty: ★★★★★ | Duration: 3–4 hours | Participants: 6–10
Threat Actor: CIRCUIT PHANTOM — nation-state APT specializing in firmware-level supply chain compromise of industrial IoT/OT devices
Primary ATT&CK Techniques: T1195.002 · T1542 · T1071.004 · T1027.009 · T0839 · T0831 · T0836
Facilitator Note
This scenario explores firmware supply chain compromise of industrial PLCs — a sophisticated nation-state capability that blends software supply chain tradecraft with OT/ICS domain knowledge. Participants should include firmware/embedded engineers, OT security analysts, and supply chain risk managers. All data is synthetic. All organizations, IPs, and indicators are fictional.
Threat Actor Profile¶
CIRCUIT PHANTOM is a nation-state threat group first tracked in 2024, specializing in firmware-level implants targeting industrial control systems and IoT devices. Unlike conventional APT groups that operate at the OS and application layer, CIRCUIT PHANTOM targets the firmware supply chain — compromising vendor update infrastructure to distribute backdoored firmware images to thousands of downstream industrial customers simultaneously.
The group demonstrates deep expertise in embedded systems architecture, reverse engineering proprietary firmware formats, and crafting implants that survive device reboots and factory resets. Their implants communicate via DNS-over-HTTPS (DoH) to blend C2 traffic with legitimate encrypted DNS queries, making network-level detection extremely difficult. CIRCUIT PHANTOM operations have targeted energy, water treatment, and manufacturing sectors across NATO-allied nations.
Motivation: Strategic — pre-positioning for industrial sabotage, intelligence collection on critical infrastructure SCADA configurations, and establishing persistent access that survives standard incident response procedures. Estimated dwell time before activation: 6–18 months.
Known TTPs:
- Firmware binary patching with position-independent code injection
- Abuse of unsigned or weakly signed firmware update mechanisms
- DNS-over-HTTPS C2 channels using legitimate DoH providers
- Living-off-the-land within PLC runtime environments
- Anti-forensic techniques exploiting limited logging on embedded devices
Scenario Narrative¶
Scenario Context
NexGen Industrial is a mid-tier manufacturer of programmable logic controllers (PLCs) and industrial IoT sensors used in water treatment, chemical processing, and discrete manufacturing. Their flagship product line, the NexGen PLC-5000 series, is deployed at approximately 2,400 industrial sites across 14 countries. Firmware updates are distributed via NexGen's cloud-based update portal (updates.nexgen-industrial.example.com) and are pushed to customer PLCs either automatically or via the NexGen Device Manager desktop application. Firmware images are signed with RSA-2048, but the signing key is stored on an HSM accessible to the build pipeline via API. NexGen's engineering team has 12 firmware developers; the security team is 3 people, none with embedded security expertise.
Phase 1 — Reconnaissance & Initial Access (~30 min)¶
CIRCUIT PHANTOM begins with extensive open-source reconnaissance of NexGen Industrial. The group harvests LinkedIn profiles of firmware engineers, identifies the company's CI/CD stack from public job postings ("Experience with Jenkins, Yocto Project, and ARM Cortex-M development"), and maps the firmware update infrastructure through passive DNS enumeration.
The threat actor identifies a senior firmware engineer, r.kowalski@nexgen-industrial.example.com, who maintains a personal GitHub repository (github.com/rkowalski-dev/plc-utils) containing internal build scripts with hardcoded references to the Jenkins build server (jenkins.internal.nexgen-industrial.example.com). A leaked SSH key in the repository's commit history (committed 9 months ago, removed 2 commits later but still in git history) provides initial access to the development network jump box at 10.50.1.25.
Evidence Artifacts:
| Artifact | Detail |
|---|---|
| GitHub (OSINT) | Repository rkowalski-dev/plc-utils — SSH private key in commit a3f7c2d (9 months old) — Key fingerprint: SHA256:EXAMPLE1234abcdef — Removed in commit b8e9d1f but present in git history |
| Passive DNS | updates.nexgen-industrial.example.com → 192.0.2.40 — CloudFront distribution — CNAME chain: d1example.cloudfront.example.com |
| LinkedIn (OSINT) | 12 firmware engineers identified — CI/CD stack: Jenkins + Yocto + ARM GCC — Source control: GitLab self-hosted |
| SSH Auth Log (10.50.1.25) | 2026-02-10T04:22:18Z — Accepted publickey for r.kowalski from 198.51.100.112 port 44821 — Key fingerprint: SHA256:EXAMPLE1234abcdef |
| VPN Gateway | No VPN session for r.kowalski at 2026-02-10T04:22:18Z — Account last VPN login: 2026-02-09T16:45:00Z from corporate IP |
Phase 1 — Discussion Inject
Technical: The SSH key was removed from the repository but remains in git history. What tools detect secrets in git history (e.g., TruffleHog, git-secrets, GitHub secret scanning)? How should organizations handle leaked credentials that have been "deleted" from a repository? Is force-pushing to remove history sufficient?
Decision: You discover that 6 other engineers also have personal GitHub repositories referencing internal infrastructure. Do you (A) immediately revoke all SSH keys and rotate credentials — causing a 2-day disruption to the firmware release cycle, or (B) audit each repository individually over a week? The next firmware release is scheduled in 5 days.
Expected Analyst Actions:
- [ ] Scan all public repositories associated with NexGen employees for leaked credentials
- [ ] Correlate the SSH key fingerprint against authorized_keys on all development systems
- [ ] Review SSH authentication logs on
10.50.1.25for logins from non-corporate IPs - [ ] Check if
r.kowalski's SSH key has been rotated since the leak - [ ] Map all systems accessible from the jump box (
10.50.1.25) - [ ] Implement GitHub secret scanning alerts for the organization
Phase 2 — Build Pipeline Compromise & Firmware Reverse Engineering (~45 min)¶
From the development jump box, CIRCUIT PHANTOM pivots to the Jenkins build server (10.50.2.30) using r.kowalski's credentials, which are reused across internal systems. The Jenkins instance runs version 2.387 with 14 plugins — 3 of which have known CVEs but are unpatched. However, the attacker does not need to exploit vulnerabilities: r.kowalski has admin-level access to the Jenkins instance, including the ability to modify build pipelines.
The attacker exfiltrates the firmware source code repository from GitLab (10.50.2.15) and the RSA-2048 firmware signing key from the HSM API. The HSM (10.50.2.50) exposes a REST API for signing operations — the API authenticates via API key stored in Jenkins credentials, and the signing endpoint does not enforce IP allowlisting or rate limiting. The attacker uses the signing API to sign arbitrary firmware images.
Over 72 hours, CIRCUIT PHANTOM reverse-engineers the PLC-5000 firmware binary. They identify the main control loop, the Modbus TCP communication handler, and the firmware update routine. They develop a position-independent implant (~4KB) that:
- Hooks the firmware update check routine to prevent future legitimate updates from overwriting the implant
- Spawns a covert DNS-over-HTTPS C2 channel to
resolver.dohdns.example.com - Accepts commands to modify PLC register values, alter setpoints, or exfiltrate process data
- Passes all standard firmware integrity checks by patching the self-check routine
Evidence Artifacts:
| Artifact | Detail |
|---|---|
| Jenkins Audit Log | 2026-02-10T05:14:33Z — User r.kowalski — Action: View credentials — Credential ID: hsm-signing-api-key — Source IP: 10.50.1.25 (jump box, not engineer workstation 10.50.4.87) |
| GitLab Access Log | 2026-02-10T05:31:00Z — r.kowalski cloned firmware/plc5000-main (487 MB) — Source IP: 10.50.1.25 — Previous clone from this IP: never |
| HSM API Log | 2026-02-12T22:15:44Z–2026-02-13T03:42:11Z — 47 signing requests from API key hsm-prod-key-01 — Normal baseline: 2–4 signing requests per release cycle (monthly) |
| Network Flow | 10.50.1.25 → 10.50.2.15:443 — 487 MB transfer — 2026-02-10T05:31–05:48Z — Anomalous: jump box has no historical data transfers to GitLab |
| Jenkins Build History | No unauthorized builds triggered — Attacker accessed credentials and source code only — Pipeline modification occurs in Phase 3 |
Phase 2 — Discussion Inject
Technical: The HSM signing API has no IP allowlist or rate limit. What controls should protect firmware signing infrastructure? Consider: hardware-bound signing with human approval, dual-authorization for signing operations, signing audit logs with anomaly detection, and air-gapped signing environments. How does Sigstore/cosign apply to firmware signing?
Decision: The HSM logged 47 signing requests in 5 hours vs. a baseline of 2–4 per month. This alert was generated but routed to the infrastructure team (not security) and was not triaged for 6 days. What organizational and technical changes would ensure this alert reaches the right team within 1 hour?
Expected Analyst Actions:
- [ ] Audit HSM signing API logs for anomalous request volume and timing
- [ ] Verify that all signed firmware images correspond to legitimate build pipeline outputs
- [ ] Review Jenkins credential access logs — flag access from non-standard source IPs
- [ ] Check GitLab for large repository clones from unexpected sources
- [ ] Assess whether firmware signing key has been compromised — begin key rotation planning
- [ ] Implement IP allowlisting and rate limiting on the HSM signing API
Phase 3 — Backdoored Firmware Distribution (~40 min)¶
CIRCUIT PHANTOM modifies the Jenkins pipeline for the PLC-5000 firmware build. The modification is subtle: a single additional build step that patches the compiled firmware binary with the implant code after compilation but before signing. The patched binary is then signed using the legitimate HSM signing key, producing a firmware image that passes all cryptographic verification checks on the PLC devices.
The backdoored firmware is version 5.4.1 — a legitimate security patch that fixes a real buffer overflow vulnerability (CVE-2026-XXXX) discovered by NexGen's own security team. CIRCUIT PHANTOM times the implant insertion to coincide with this legitimate release, ensuring maximum adoption urgency. The backdoored firmware is uploaded to updates.nexgen-industrial.example.com and pushed to customers via the standard update notification channel.
Within 14 days, 847 of 2,400 deployed PLC-5000 units install the backdoored firmware. The affected devices span 6 countries and include water treatment plants, chemical processing facilities, and manufacturing lines.
Evidence Artifacts:
| Artifact | Detail |
|---|---|
| Jenkins Pipeline | Job: plc5000-release-5.4.1 — Modified: 2026-02-15T01:30:22Z by r.kowalski — Diff: +1 build step (inject_patch.sh) — Script sourced from 10.50.1.25:/tmp/.build/patch.sh |
| Update Server (CDN) | updates.nexgen-industrial.example.com/plc5000/5.4.1/firmware.bin — SHA256: a1b2c3d4e5f6... — Signed: Valid (RSA-2048, NexGen firmware signing key) — Upload: 2026-02-15T08:00:00Z |
| Customer Telemetry | 847 PLC-5000 units updated to v5.4.1 within 14 days — 6 countries — 312 water/wastewater, 189 chemical, 346 manufacturing |
| Binary Diff (forensic) | firmware-5.4.1.bin vs. source-compiled reference: +4,096 bytes at offset 0x7F000 — Code section .implant — Not present in source repository |
| Build Artifact Log | inject_patch.sh — Not committed to GitLab — Sourced from jump box local filesystem — Script deleted after build completion |
Phase 3 — Discussion Inject
Technical: The firmware is legitimately signed because the attacker accessed the signing key. How would reproducible builds (deterministic compilation) have detected the implant? What is the role of binary transparency logs (similar to Certificate Transparency) for firmware distribution? How does SBOM (Software Bill of Materials) help or not help in this scenario?
Decision: You discover that 847 PLCs across 6 countries are running backdoored firmware. Do you (A) immediately issue a public advisory and emergency firmware recall — alerting the threat actor and potentially causing industrial process disruptions, or (B) quietly notify affected customers through private channels over 72 hours — leaving the backdoor active longer but maintaining operational surprise? Consider legal disclosure obligations (CISA, EU NIS2).
Expected Analyst Actions:
- [ ] Perform binary diff between the distributed firmware and a clean source-compiled reference build
- [ ] Audit Jenkins pipeline history for unauthorized modifications — review all build steps
- [ ] Identify the
inject_patch.shscript — recover from jump box forensic image if deleted - [ ] Enumerate all PLC-5000 units that installed firmware v5.4.1 — assess blast radius
- [ ] Begin coordinated disclosure with CISA and affected critical infrastructure operators
- [ ] Preserve Jenkins build artifacts, GitLab logs, and CDN upload logs for forensic analysis
Phase 4 — C2 Activation & Reconnaissance (~40 min)¶
Six weeks after firmware distribution, CIRCUIT PHANTOM activates the implant on a subset of 23 PLCs at 4 water treatment facilities. The implant initiates DNS-over-HTTPS queries to resolver.dohdns.example.com — a legitimate-appearing DoH resolver operated by the threat actor. The C2 channel encodes commands and responses in DNS TXT record queries for subdomains of telemetry.nexgen-cdn.example.com.
The C2 traffic is virtually invisible to network monitoring: it uses HTTPS (port 443) to a domain that appears to be a legitimate DNS resolver, the query volume is low (1 query per 5 minutes per device), and the traffic passes through standard corporate firewalls without inspection. The implant begins exfiltrating PLC configuration data, process setpoints, and network topology information from the OT environment.
Evidence Artifacts:
| Artifact | Detail |
|---|---|
| Firewall Log (Water Treatment Site Alpha) | PLC 172.16.100.20 → 203.0.113.55:443 (resolver.dohdns.example.com) — HTTPS — 288 connections/day — 1 every 5 min — Started: 2026-03-28T00:00:12Z |
| DNS Log (Site Alpha) | No traditional DNS queries from 172.16.100.20 to 203.0.113.55 — Traffic is direct HTTPS (DoH), bypassing local DNS resolver |
| OT Network Monitor (Claroty) | INFO: PLC-5000 at 172.16.100.20 — New outbound HTTPS destination 203.0.113.55 — Not in baseline — First seen: 2026-03-28 |
| Packet Capture (decrypted) | DoH queries: TXT _c2.telemetry.nexgen-cdn.example.com — Response contains base64-encoded commands — Query: TXT _r.<device-id>.telemetry.nexgen-cdn.example.com — Payload: base64-encoded PLC register dumps |
| PLC Register Reads (exfiltrated) | Setpoints: chlorine dosing (1.2 ppm), pH target (7.4), flow rate (450 GPM), turbidity threshold (0.3 NTU) — Process network map: 14 PLCs, 3 HMIs, 1 historian at Site Alpha |
Phase 4 — Discussion Inject
Technical: The C2 uses DNS-over-HTTPS, bypassing traditional DNS monitoring. What visibility do you lose when endpoints use DoH? How would you detect DoH C2 in an OT environment? Consider: TLS inspection on OT network egress, allowlisting DNS resolvers, blocking direct DoH from OT devices, and JA3/JA4 fingerprinting of DoH clients.
Decision: OT devices should not be making outbound HTTPS connections to the internet. However, NexGen's Device Manager uses HTTPS for telemetry and update checks. How do you distinguish legitimate NexGen telemetry from C2 traffic? Should OT devices have any direct internet access?
Expected Analyst Actions:
- [ ] Baseline all outbound connections from PLC devices — identify new destinations post-firmware update
- [ ] Investigate
203.0.113.55(resolver.dohdns.example.com) — WHOIS, passive DNS, hosting provider - [ ] Capture and decrypt DoH traffic from affected PLCs (if TLS inspection is available)
- [ ] Correlate C2 activation timing across all 23 affected PLCs — identify coordinated activation
- [ ] Assess what PLC configuration data has been exfiltrated — determine sabotage risk
- [ ] Implement emergency firewall rules blocking OT device internet access except to allowlisted NexGen update servers
Phase 5 — Industrial Sabotage Attempt (~45 min)¶
After 10 days of reconnaissance, CIRCUIT PHANTOM issues commands to 3 PLCs at Water Treatment Site Alpha to gradually modify chlorine dosing setpoints. The modification is designed to be slow enough to avoid triggering high/low alarms: the chlorine dosing setpoint is increased from 1.2 ppm to 3.8 ppm over 72 hours (increments of 0.036 ppm per hour). At 3.8 ppm, treated water would exceed EPA maximum residual disinfectant levels (MRDL) of 4.0 ppm, potentially causing health effects for consumers.
Simultaneously, the implant modifies the HMI display values to show the original 1.2 ppm setpoint — operators see normal readings while actual dosing increases. The historian database records the falsified values, not the actual process state.
The attack is detected when a manual grab sample taken during a routine compliance check shows chlorine residual at 2.9 ppm — significantly higher than the HMI-displayed 1.2 ppm. The plant operator escalates to the shift supervisor, who initiates an emergency process investigation.
Evidence Artifacts:
| Artifact | Detail |
|---|---|
| PLC Register Log (forensic) | Register %MW100 (Cl2 setpoint): 1.200 → 2.900 over 47 hours — 0.036/hr increments — Modified by firmware implant, not HMI operator |
| HMI Screenshot | Chlorine dosing display: 1.2 ppm (static, not updating) — Actual process value: 2.9 ppm — Discrepancy: HMI receiving spoofed register reads from implant |
| Historian Database | Cl2 trend: flat at 1.2 ppm for 47 hours — No variance — Anomaly: real chlorine dosing always shows ±0.05 ppm natural variance — Flat line indicates data substitution |
| Manual Lab Sample | Grab sample 2026-04-08T14:30:00Z: Chlorine residual = 2.9 ppm — Expected (per HMI): 1.2 ppm — Deviation: +142% — Triggered operator escalation |
| C2 Command Log (forensic) | 2026-04-06T02:00:00Z — Command: SET_RAMP %MW100 1.200 3.800 72h — SET_SPOOF %MW100_DISPLAY 1.200 — Target: 3 PLCs at Site Alpha |
| Claroty Alert | CRITICAL: PLC 172.16.100.20 — Register write to %MW100 — Source: internal firmware (not HMI or engineering workstation) — No authorized change ticket |
Phase 5 — Discussion Inject
Technical: The attacker spoofed HMI display values while modifying actual process setpoints. What independent verification mechanisms can detect this? Consider: out-of-band process sensors, independent safety instrumented systems (SIS), manual grab samples, and cross-validation between historian data and physical measurements. Why is flat-line historian data an anomaly indicator?
Decision: Chlorine is at 2.9 ppm and rising toward 3.8 ppm (EPA MRDL is 4.0 ppm). You cannot trust the PLCs or HMI. Do you (A) switch to full manual control of the treatment process — requiring 24/7 operator staffing and risking human error, or (B) shut down the treatment plant and issue a boil-water advisory to 250,000 residents? What is your communication plan for public health authorities?
Expected Analyst Actions:
- [ ] Immediately switch affected PLCs to manual control — verify all setpoints with physical measurements
- [ ] Compare historian data against manual grab samples and independent sensors
- [ ] Identify the flat-line anomaly in historian data as evidence of data substitution
- [ ] Isolate all PLC-5000 units from the network — disable firmware implant communications
- [ ] Notify EPA, state environmental agency, and public health department
- [ ] Begin emergency firmware rollback to known-good version (pre-5.4.1) on all affected PLCs
Phase 6 — Containment, Eradication & Recovery (~30 min)¶
The incident response team — now including NexGen Industrial, CISA ICS-CERT, and affected facility operators — coordinates a multi-site containment effort. Key challenges:
- 847 potentially compromised PLCs across 6 countries — only 23 confirmed active C2, but all v5.4.1 devices must be treated as compromised
- Firmware-level persistence — the implant survives standard PLC restart and even factory reset (it hooks the reset routine)
- No known-good firmware baseline — the build pipeline was compromised, so the signing key must be rotated and all firmware rebuilt from verified source
- Operational continuity — water treatment plants cannot simply shut down; they must maintain safe operations during remediation
Evidence Artifacts:
| Artifact | Detail |
|---|---|
| CISA ICS-CERT Advisory | ICS-CERT-2026-XXXX — NexGen PLC-5000 Firmware Backdoor — CVSS: 10.0 — Affected versions: 5.4.1 — Mitigation: immediate network isolation, firmware rollback |
| NexGen Security Bulletin | Signing key compromised — All firmware images signed after 2026-02-12 must be re-verified — New signing key generated on air-gapped HSM — Clean firmware 5.4.2 built from audited source |
| Forensic Image (Jump Box) | /tmp/.build/patch.sh — Recovered from deleted file via ext4 journal replay — Contains position-independent ARM Cortex-M shellcode — Connects to resolver.dohdns.example.com |
| Network Isolation Log | 847 PLC-5000 units isolated from internet-facing networks — 23 units with confirmed C2 physically disconnected — Manual control established at 4 water treatment sites |
| Firmware Rollback Status | 312/847 units successfully rolled back to v5.3.0 (pre-compromise) within 72 hours — 535 units require on-site physical access (remote update disabled by implant) |
Phase 6 — Discussion Inject
Technical: The implant hooks the firmware update routine to prevent overwriting. How do you recover a device when the firmware itself blocks legitimate updates? Consider: JTAG/SWD physical debugging interfaces, bootloader-mode recovery, and hardware replacement. What is the cost and timeline for physically reflashing 535 PLCs across multiple countries?
Decision: NexGen's firmware signing key is compromised. Key rotation requires rebuilding and re-signing all firmware versions, updating the trust store on 2,400 deployed devices, and coordinating with customers. This process will take 4–6 weeks. During this time, can you trust any firmware update from NexGen? What interim verification process do you implement?
Expected Analyst Actions:
- [ ] Coordinate with NexGen to verify firmware source code integrity from version control
- [ ] Validate clean firmware build using reproducible build process on isolated build environment
- [ ] Develop rollback procedure for PLCs where remote update is blocked by implant
- [ ] Implement network-level blocking of all DoH traffic from OT networks
- [ ] Establish continuous monitoring for C2 indicators across all customer sites
- [ ] Conduct root cause analysis on build pipeline security — HSM access controls, CI/CD integrity
Detection Opportunities¶
KQL Detection Queries¶
// Detect anomalous outbound DoH connections from OT devices
let OTDevices = DeviceNetworkEvents
| where DeviceGroup == "OT_PLCs" or DeviceGroup == "OT_Sensors";
OTDevices
| where RemotePort == 443
| where RemoteUrl has_any ("resolver.", "doh.", "dns.")
| where RemoteUrl !in ("dns.google", "cloudflare-dns.com", "dns.quad9.net") // known legitimate DoH
| summarize ConnectionCount=count(), DistinctDests=dcount(RemoteUrl) by DeviceName, RemoteUrl, bin(Timestamp, 1h)
| where ConnectionCount > 10
| sort by ConnectionCount desc
// Detect firmware signing anomalies — excessive signing requests
let HSMBaseline = 4; // normal monthly signing operations
HSMAuditLogs
| where Operation == "Sign" and KeyId == "firmware-signing-key"
| summarize SignCount=count() by bin(Timestamp, 1h)
| where SignCount > HSMBaseline
| extend Anomaly = "Excessive firmware signing operations"
// Detect PLC register modifications from unexpected sources
OTSecurityEvents
| where EventType == "RegisterWrite"
| where Source !in ("HMI-01", "HMI-02", "ENG-WS-01") // authorized sources
| where RegisterAddress in ("%MW100", "%MW101", "%MW102") // critical setpoints
| project Timestamp, DeviceName, RegisterAddress, OldValue, NewValue, Source
| sort by Timestamp desc
// Detect flat-line historian data indicating data substitution
ProcessHistorian
| where TagName has "Chlorine" or TagName has "Cl2"
| summarize Variance=stdev(Value), AvgValue=avg(Value) by TagName, bin(Timestamp, 1h)
| where Variance == 0 and AvgValue > 0 // zero variance = suspicious data substitution
| extend Alert = "Flat-line process data — possible HMI spoofing"
Splunk (SPL) Detection Queries¶
// Detect DoH traffic from OT network segments
index=firewall sourcetype=fw_traffic src_zone="OT_Network"
dest_port=443 dest_ip IN (203.0.113.0/24, 198.51.100.0/24)
| stats count by src_ip, dest_ip, dest_host
| where count > 50
| eval alert="OT device making excessive HTTPS connections to external host"
// HSM signing anomaly detection
index=hsm sourcetype=hsm_audit operation=Sign key_id="firmware-signing-key"
| bucket _time span=1h
| stats count by _time
| where count > 4
| eval severity="HIGH"
| eval description="Anomalous firmware signing volume — possible key abuse"
// Detect Jenkins pipeline modifications outside change windows
index=cicd sourcetype=jenkins_audit action="pipeline_modified"
| eval hour=strftime(_time, "%H")
| where hour < 6 OR hour > 22
| table _time, user, pipeline_name, source_ip, action
| eval alert="Off-hours pipeline modification — investigate immediately"
// PLC setpoint change detection without authorized change ticket
index=ot sourcetype=plc_events event_type=register_write
| lookup authorized_changes ticket_id OUTPUT status
| where isnull(status) OR status!="approved"
| stats count values(register_address) values(new_value) by src_device, _time
| eval alert="Unauthorized PLC setpoint modification"
Incident Response Checklist¶
Immediate Actions (0–4 hours)¶
- [ ] Isolate all PLC-5000 units running firmware v5.4.1 from external networks
- [ ] Switch affected water treatment PLCs to manual control with operator verification
- [ ] Block all outbound DoH traffic from OT network segments at the firewall
- [ ] Verify chlorine dosing and all critical setpoints via manual grab samples
- [ ] Activate ICS-CERT coordination and notify CISA
- [ ] Preserve Jenkins build server, GitLab, and HSM audit logs
Short-Term Actions (4–72 hours)¶
- [ ] Conduct binary diff analysis on all firmware v5.4.1 images vs. clean source build
- [ ] Identify all 847 devices running compromised firmware — contact all affected operators
- [ ] Begin firmware rollback to v5.3.0 on devices where remote update is possible
- [ ] Rotate firmware signing key — generate new key on air-gapped HSM
- [ ] Revoke all credentials associated with
r.kowalskiand the development jump box - [ ] Deploy network monitoring for C2 indicators at all affected sites
Long-Term Actions (1–8 weeks)¶
- [ ] Physically reflash PLCs where remote update is blocked by implant (535 units)
- [ ] Implement reproducible builds for firmware compilation pipeline
- [ ] Deploy hardware-enforced code signing with dual-authorization
- [ ] Establish firmware binary transparency log for all published firmware
- [ ] Implement network segmentation preventing OT devices from direct internet access
- [ ] Conduct security audit of entire firmware build pipeline and supply chain
Lessons Learned¶
What Went Wrong¶
| Gap | Detail | Remediation |
|---|---|---|
| Leaked credentials in git history | SSH key committed to public repository; removed but still in history | Implement pre-commit hooks (git-secrets), enforce secret scanning, treat any leaked credential as permanently compromised |
| HSM API without access controls | Signing API accessible from any internal host, no rate limiting | Implement IP allowlisting, rate limiting, dual-authorization for signing operations |
| No firmware build reproducibility | No way to independently verify that distributed firmware matches source code | Implement deterministic/reproducible builds, publish build attestations |
| Unsigned build pipeline modifications | Jenkins pipeline changes not requiring code review or approval | Require PR-based pipeline changes with mandatory review and audit logging |
| No OT egress filtering | PLCs allowed to make arbitrary outbound HTTPS connections | Implement strict egress filtering — OT devices should only communicate with allowlisted internal systems |
| Historian data not independently verified | HMI-displayed values accepted without cross-validation | Deploy independent out-of-band sensors, implement data integrity checks on historian records |
| Single signing key | One compromised key affected all firmware for all customers | Implement key hierarchy with per-product or per-customer signing keys, enable key rotation |
What Went Right¶
| Control | Impact |
|---|---|
| Manual grab sampling program | Detected the chlorine dosing discrepancy that revealed the attack |
| Claroty OT monitoring | Generated alerts for anomalous PLC network behavior and register writes |
| HSM audit logging | Provided forensic evidence of signing key abuse (47 requests vs. 4 baseline) |
| CISA ICS-CERT coordination | Enabled multi-site, multi-country coordinated response |
ATT&CK Navigator Mapping¶
| Technique ID | Technique Name | Phase |
|---|---|---|
| T1593.003 | Search Open Websites/Domains: Code Repositories | Reconnaissance |
| T1195.002 | Supply Chain Compromise: Compromise Software Supply Chain | Initial Access |
| T1542 | Pre-OS Boot: Firmware Corruption | Persistence |
| T1027.009 | Obfuscated Files: Embedded Payloads | Defense Evasion |
| T1071.004 | Application Layer Protocol: DNS | Command & Control |
| T0839 | Module Firmware | Execution (OT) |
| T0836 | Modify Parameter | Impair Process Control (OT) |
| T0831 | Manipulation of Control | Impact (OT) |
Related Chapters¶
- Chapter 21 — OT/ICS/SCADA Security — Purdue Model, PLC security, safety instrumented systems
- Chapter 24 — Supply Chain Attacks — Software supply chain compromise, build pipeline security
- Chapter 34 — Mobile & IoT Security — Firmware security, embedded device hardening, IoT attack surfaces
Scenario Debrief
Operation Silicon Ghost demonstrates the devastating potential of firmware supply chain attacks against industrial control systems. The attack chain — from leaked credentials to build pipeline compromise to firmware-level persistence — highlights the critical need for firmware signing integrity, reproducible builds, and independent process verification in OT environments. The most effective detection was not a technical control but a manual compliance process: the grab sample that revealed the chlorine dosing discrepancy. Defense-in-depth for OT must include both cyber and physical verification mechanisms.