SC-106: Industrial Control System Attack on Water Treatment -- Operation TOXIC CURRENT¶
Educational Content Only
This scenario uses 100% synthetic data for educational purposes. All IP addresses use RFC 5737 (192.0.2.x, 198.51.100.x, 203.0.113.x) or RFC 1918 (10.x, 172.16.x, 192.168.x) ranges. All domains use *.example.com. All credentials are testuser/REDACTED. No real organizations, infrastructure, or individuals are represented. Offense content is presented exclusively to improve defensive capabilities. This scenario is designed to train critical infrastructure defenders and does not represent any real water utility or treatment facility.
Scenario Overview¶
| Field | Detail |
|---|---|
| ID | SC-106 |
| Operation Name | TOXIC CURRENT |
| Category | Critical Infrastructure / ICS-SCADA / Water Treatment |
| Severity | Critical |
| ATT&CK Tactics | Initial Access, Execution, Persistence, Lateral Movement, Inhibit Response Function, Impair Process Control, Impact |
| ATT&CK Techniques | T1133 (External Remote Services), T1078 (Valid Accounts), T1021 (Remote Services), T0855 (Unauthorized Command Message), T0836 (Modify Parameter), T0831 (Manipulation of Control) |
| Threat Actor | MERCURY WAVE -- A state-sponsored APT group specializing in critical infrastructure targeting, particularly water and energy sectors. Known for patient reconnaissance, OT protocol fluency, and attacks designed to cause physical safety incidents while maintaining plausible deniability. Previously attributed to campaigns against 14 water utilities across 3 countries. |
| Target Environment | Clearwater Regional Water Authority (clearwater-rwa.example.com) -- a municipal water utility serving 380,000 residents, operating 3 treatment plants and 47 pump stations with a mix of modern and legacy SCADA systems |
| Difficulty | ★★★★★ |
| Duration | 6-8 hours |
| Estimated Impact | Chemical dosing parameters manipulated at Plant 2 (chlorine increased from 2.0 mg/L to 8.5 mg/L for 47 minutes); Safety Instrumented System prevented harmful water release; 12-hour service disruption for 94,000 residents; loss of operator trust in automation; estimated remediation cost $23M |
Narrative¶
Clearwater Regional Water Authority (CRWA) is a municipal water utility at clearwater-rwa.example.com serving 380,000 residents across a metropolitan area. CRWA operates three water treatment plants (WTP-1, WTP-2, WTP-3) and 47 pump stations connected via a SCADA network. The utility treats an average of 85 million gallons per day (MGD) from surface water sources.
CRWA's operational technology environment reflects the reality of many water utilities: a mix of modern and legacy systems accumulated over 25 years of incremental upgrades. WTP-2, the focus of this attack, uses Rockwell Automation ControlLogix PLCs for process control, a Wonderware InTouch HMI for operator interfaces, and OSIsoft PI for historian data. The SCADA network connects to the corporate IT network through a firewall with limited monitoring. Remote access is provided through a vendor VPN for maintenance by AquaTech Solutions (aquatech-solutions.example.com), the integrator that manages CRWA's SCADA systems.
CRWA's security posture is typical of mid-size water utilities: a 3-person IT team, no dedicated OT security personnel, basic firewall segmentation between IT and OT, and reliance on vendors for SCADA system security. A CISA assessment 18 months ago identified critical gaps in network segmentation, remote access controls, and OT monitoring, but budget constraints have limited remediation to partial firewall rule updates.
MERCURY WAVE selects CRWA after identifying the vendor VPN as a common attack surface across multiple water utilities that use AquaTech Solutions. Their objective is to demonstrate capability against critical infrastructure while causing a safety incident that can be attributed to equipment malfunction.
Environment¶
| Component | Detail |
|---|---|
| Organization | Clearwater Regional Water Authority |
| Domain | clearwater-rwa.example.com |
| Population Served | 380,000 residents |
| Treatment Plants | WTP-1 (40 MGD), WTP-2 (30 MGD), WTP-3 (15 MGD) |
| Pump Stations | 47 across the distribution system |
| SCADA Network | 172.16.0.0/16 (isolated OT network) |
| Corporate IT Network | 10.10.0.0/16 |
| DMZ | 192.168.100.0/24 (jump server, historian mirror) |
| WTP-2 PLC Network | 172.16.20.0/24 |
| WTP-2 HMI Stations | 172.16.20.100-103 (Wonderware InTouch) |
| WTP-2 PLCs | 172.16.20.10 (Chemical Dosing), 172.16.20.11 (Filtration), 172.16.20.12 (Pumping) |
| WTP-2 SIS | 172.16.20.200 (Triconex Safety Controller -- independent from process PLCs) |
| Historian | 172.16.1.50 (OSIsoft PI, mirrors to DMZ at 192.168.100.50) |
| Vendor VPN | vpn.clearwater-rwa.example.com (FortiGate SSL-VPN) |
| Vendor | AquaTech Solutions (aquatech-solutions.example.com) |
| Security Stack | Fortinet NGFW (IT/OT boundary), no OT-specific IDS, Windows Defender on HMIs |
| Compliance | AWIA (America's Water Infrastructure Act), NIST CSF (partial implementation) |
Attack Timeline¶
Phase 1: Vendor Compromise & Initial Access (Weeks 1-3)¶
ATT&CK Techniques: T1133 (External Remote Services), T1078 (Valid Accounts), T1199 (Trusted Relationship)
MERCURY WAVE targets AquaTech Solutions, the SCADA integrator that provides remote maintenance services to CRWA and 23 other water utilities. The attacker compromises an AquaTech engineer's workstation through a spearphishing campaign targeting OT engineering conferences.
# Simulated vendor compromise (educational only)
# Attacker targets SCADA integrator for multi-utility access
# Step 1: Spearphishing AquaTech engineer
# Target: testuser@aquatech-solutions.example.com
# Lure: "Updated SCADA Security Best Practices Guide - WaterISAC"
# Attachment: SCADA_Security_Guide_2026.pdf (embedded macro)
# Delivery: Email from spoofed waterisac-alerts.example.com
# Spearphish email (synthetic):
From: alerts@waterisac-alerts.example.com
To: testuser@aquatech-solutions.example.com
Subject: [WaterISAC] Updated SCADA Security Advisory - Action Required
Date: 2026-01-20T08:15:00Z
Dear AquaTech Engineering Team,
WaterISAC has released an updated security advisory regarding
recent threats to water treatment SCADA systems. Please review
the attached document for critical security recommendations
specific to ControlLogix and Wonderware environments.
Attachment: WaterISAC_Advisory_2026-003.pdf.exe
(disguised with PDF icon, double extension)
# Step 2: AquaTech engineer workstation compromised
# Backdoor establishes C2 to 203.0.113.45 (update-service.example.com)
# Attacker discovers VPN credentials stored in KeePass on workstation
# VPN profiles for 24 water utility customers identified
# Step 3: Extract CRWA VPN credentials
# AquaTech VPN credentials for CRWA:
# VPN endpoint: vpn.clearwater-rwa.example.com
# Username: testuser-aquatech@clearwater-rwa.example.com
# Password: REDACTED
# Certificate: aquatech-maintenance.p12
# MFA: None (vendor account exempted from MFA policy)
# Access hours: Unrestricted (24/7 maintenance access)
# Network access: Full SCADA network (172.16.0.0/16)
# Step 4: Initial VPN connection to CRWA
# Attacker connects from AquaTech engineer's workstation
# Connection appears legitimate (correct cert, known source IP)
timestamp=2026-02-08T02:14:33Z vpn_event=connect
user=testuser-aquatech@clearwater-rwa.example.com
src_ip=198.51.100.22 assigned_ip=172.16.250.5
duration=0 tunnel=ssl auth=certificate+password status=success
Evidence Artifact -- VPN Connection Log:
[2026-02-08T02:14:33Z] FortiGate SSL-VPN: Connection established
User: testuser-aquatech@clearwater-rwa.example.com
Source IP: 198.51.100.22 (AquaTech Solutions office range)
Assigned IP: 172.16.250.5
Authentication: Certificate + Password
MFA: Exempt (vendor account)
Session ID: VPN-20260208-0214-A7B3
Access Profile: vendor-scada-full
NOTE: Connection at 02:14 local time -- outside normal
maintenance window (09:00-17:00), but no policy enforced
[2026-02-08T02:14:45Z] FortiGate Firewall: Policy match
Source: 172.16.250.5 (VPN pool)
Destination: 172.16.0.0/16 (SCADA network - full access)
Action: ALLOW
Policy: "Vendor-SCADA-Maintenance" (rule #12)
Logging: Summary only (detailed logging disabled for vendor sessions)
Evidence Artifact -- Active Directory Authentication (IT-side):
[2026-02-08T02:15:12Z] EventID=4624 LogonType=10
Account: testuser-aquatech
Domain: CRWA
Source: 172.16.250.5
Workstation: DMZ-JUMP01 (192.168.100.10)
Authentication: Kerberos
Logon Process: User32
[2026-02-08T02:15:34Z] EventID=4624 LogonType=10
Account: testuser-aquatech
Domain: CRWA
Source: 192.168.100.10
Workstation: WTP2-HMI01 (172.16.20.100)
Authentication: NTLM
Logon Process: User32
NOTE: Direct RDP from jump server to HMI -- allowed by policy
Discussion Inject 1 -- Technical
The vendor VPN provides unrestricted 24/7 access to the entire SCADA network with no MFA. What specific controls should be implemented for third-party OT remote access? Consider: time-based access restrictions, just-in-time provisioning, session recording, MFA enforcement, network segmentation within the OT zone, and behavioral baselines for vendor access patterns.
Discussion Inject 2 -- Decision
CRWA's 3-person IT team has no OT security monitoring capability. They rely on AquaTech for SCADA security. The vendor VPN has been operating in this configuration for 6 years. A CISA assessment flagged it 18 months ago, but MFA implementation was deferred due to vendor compatibility concerns. Who bears responsibility for this gap? How should utilities with limited resources prioritize OT security improvements?
Detection Query -- KQL (Microsoft Sentinel):
// Detect off-hours vendor VPN connections to SCADA network
let vendor_accounts = dynamic(["testuser-aquatech"]);
SigninLogs
| where TimeGenerated > ago(24h)
| where UserPrincipalName has_any (vendor_accounts)
| extend HourOfDay = datetime_part("hour", TimeGenerated)
| where HourOfDay < 7 or HourOfDay > 18 // Outside business hours
| project TimeGenerated, UserPrincipalName, IPAddress,
ResultType, AppDisplayName, Location
| sort by TimeGenerated desc
// Detect VPN connections to SCADA network without MFA
CommonSecurityLog
| where TimeGenerated > ago(7d)
| where DeviceVendor == "Fortinet"
| where Activity has "vpn" and Activity has "connect"
| where DestinationUserName has "aquatech" or
DestinationUserName has "vendor"
| where Message !has "mfa" and Message !has "two-factor"
| summarize ConnectionCount=count(), EarliestConnection=min(TimeGenerated),
LatestConnection=max(TimeGenerated) by DestinationUserName,
SourceIP, DeviceAction
| sort by ConnectionCount desc
Detection Query -- SPL (Splunk):
// Detect off-hours vendor VPN connections to SCADA network
index=vpn sourcetype=fortigate:vpn event_type=connect
user="*aquatech*" OR user="*vendor*"
| eval hour=strftime(_time, "%H")
| where hour < 7 OR hour > 18
| table _time user src_ip assigned_ip duration
| sort - _time
// Detect VPN connections to SCADA network without MFA
index=vpn sourcetype=fortigate:vpn event_type=connect
(user="*aquatech*" OR user="*vendor*")
NOT mfa_status="success"
| stats count as conn_count earliest(_time) as first_seen
latest(_time) as last_seen by user src_ip
| sort - conn_count
Defender Decision Point 1: A SIEM analyst (reviewing IT logs, not OT) notices a vendor VPN connection at 02:14 AM local time. Vendor maintenance windows are typically 09:00-17:00, but there is no enforced policy. The connection source IP matches AquaTech's known range. Do you: (A) Contact AquaTech to verify the maintenance activity? (B) Immediately terminate the VPN session and investigate? (C) Log the observation and continue monitoring -- it may be an emergency maintenance call?
Phase 2: SCADA Network Reconnaissance (Week 4)¶
ATT&CK Techniques: T1046 (Network Service Scanning), T0846 (Remote System Discovery), T0888 (Remote System Information Discovery)
Having established access via the vendor VPN, MERCURY WAVE conducts careful reconnaissance of CRWA's SCADA network. The attacker moves slowly to avoid detection, using legitimate OT protocols and tools that would appear normal from a vendor maintenance session.
# Simulated SCADA network reconnaissance (educational only)
# Attacker maps OT environment using legitimate tools
# Step 1: Network discovery from HMI workstation
# Attacker uses WTP2-HMI01 (172.16.20.100) as pivot point
# Tools used: Rockwell RSLinx (legitimate PLC discovery tool)
# RSLinx browse results (synthetic):
RSLinx Classic - Browse Network
└── Ethernet/IP
├── 172.16.20.10 - WTP2-CHEM-PLC (ControlLogix L83E)
│ ├── Slot 0: 1756-L83E/B (Controller) - Run Mode
│ ├── Slot 1: 1756-EN2T/D (EtherNet/IP) - OK
│ ├── Slot 3: 1756-IF16 (Analog Input) - OK
│ ├── Slot 4: 1756-OF8 (Analog Output) - OK
│ ├── Slot 5: 1756-OB16E (Digital Output) - OK
│ └── Program: ChemDosing_Main v4.2
│
├── 172.16.20.11 - WTP2-FILT-PLC (ControlLogix L83E)
│ ├── Slot 0: 1756-L83E/B (Controller) - Run Mode
│ ├── Program: Filtration_Control v3.8
│ └── [8 I/O modules]
│
├── 172.16.20.12 - WTP2-PUMP-PLC (ControlLogix L73S)
│ ├── Slot 0: 1756-L73S (Controller) - Run Mode
│ ├── Program: HighService_Pumps v5.1
│ └── [12 I/O modules]
│
├── 172.16.20.200 - WTP2-SIS (Triconex 3008)
│ └── Safety Instrumented System - independent network
│ └── Program: WTP2_Safety v2.0 (READ-ONLY, hardware key required)
│
└── 172.16.20.100-103 - HMI Workstations (Wonderware InTouch)
# Step 2: Chemical dosing PLC tag enumeration
# Attacker reads PLC tags from WTP2-CHEM-PLC (172.16.20.10)
# Using RSLogix 5000 (found installed on HMI workstation)
# Critical tags discovered:
# Tag Name Type Value Description
# ─────────────────────────────────────────────────────────
# Chlorine_Setpoint REAL 2.0 Chlorine dose (mg/L)
# Chlorine_Actual REAL 1.98 Current chlorine reading
# Chlorine_High_Alarm REAL 4.0 High alarm threshold
# Chlorine_HiHi_Alarm REAL 6.0 Critical high alarm
# Chlorine_Pump_Speed REAL 42.5 Pump speed (%)
# Chlorine_Pump_Enable BOOL 1 Pump enable status
# Fluoride_Setpoint REAL 0.7 Fluoride dose (mg/L)
# pH_Setpoint REAL 7.2 pH target
# pH_Actual REAL 7.18 Current pH reading
# Turbidity_NTU REAL 0.15 Current turbidity
# Flow_Rate_MGD REAL 28.4 Plant flow rate
# Operator_Mode DINT 1 1=Auto, 0=Manual
# SIS_Trip_Status BOOL 0 Safety system status
# Step 3: Understand safety system architecture
# The Triconex SIS (172.16.20.200) operates independently:
# - Separate power supply, separate network
# - Monitors chlorine, pH, turbidity independently
# - Hardware key switch required for program changes
# - Trip setpoints: Chlorine > 8.0 mg/L, pH < 6.0 or > 9.0
# - Action on trip: Close chemical feed valves, alert operators
# The attacker notes SIS will prevent catastrophic dosing levels
# but identifies a window between PLC alarm (6.0) and SIS trip (8.0)
Evidence Artifact -- OSIsoft PI Historian Query Log:
[2026-02-15T02:45:22Z] PI Web API: Query received
Client: 172.16.20.100 (WTP2-HMI01)
User: testuser-aquatech
Query: GET /piwebapi/points?nameFilter=*Chlorine*
Results: 12 PI Points returned
[2026-02-15T02:45:38Z] PI Web API: Query received
Client: 172.16.20.100
User: testuser-aquatech
Query: GET /piwebapi/streams/{webId}/recorded
?startTime=*-30d&endTime=*
&nameFilter=Chlorine_Setpoint
Results: 43,200 values returned (30 days at 1-min intervals)
NOTE: Attacker retrieving historical setpoint data to understand
normal operating ranges and operator behavior patterns
[2026-02-15T02:46:15Z] PI Web API: Query received
Client: 172.16.20.100
User: testuser-aquatech
Query: GET /piwebapi/streams/{webId}/recorded
?startTime=*-30d&endTime=*
&nameFilter=Chlorine_Actual
Results: 43,200 values returned
NOTE: Attacker mapping actual vs. setpoint to understand
system response characteristics
Discussion Inject 3 -- Technical
The attacker used legitimate OT engineering tools (RSLinx, RSLogix 5000) already installed on the HMI workstation to conduct reconnaissance. These tools generate network traffic that is indistinguishable from normal maintenance activity. How would you detect this reconnaissance phase? Consider: application whitelisting on HMI workstations, monitoring for CIP (Common Industrial Protocol) browse requests, and establishing behavioral baselines for vendor access sessions.
Discussion Inject 4 -- Investigative
The OSIsoft PI historian shows the vendor account querying 30 days of chlorine setpoint and actual values. This query pattern is unusual for routine maintenance but not explicitly prohibited. How should organizations baseline normal vendor query patterns in OT historians? What constitutes anomalous data access in a SCADA environment?
Detection Query -- KQL (Microsoft Sentinel):
// Detect unusual CIP protocol traffic patterns on OT network
CommonSecurityLog
| where TimeGenerated > ago(7d)
| where DeviceVendor == "Fortinet" or DeviceVendor == "Claroty"
or DeviceVendor == "Nozomi"
| where DestinationIP startswith "172.16.20."
| where DestinationPort == 44818 // EtherNet/IP (CIP)
| summarize ConnectionCount=count(),
UniqueDestinations=dcount(DestinationIP),
TotalBytes=sum(SentBytes)
by SourceIP, bin(TimeGenerated, 1h)
| where UniqueDestinations > 3 // Scanning multiple PLCs
| sort by TimeGenerated desc
// Detect bulk historian data queries from vendor accounts
CustomLogs_PIWebAPI_CL
| where TimeGenerated > ago(7d)
| where user_s has "aquatech" or user_s has "vendor"
| where query_s has "recorded" and query_s has "startTime=*-30d"
| summarize QueryCount=count(),
UniquePoints=dcount(point_name_s),
TotalValuesReturned=sum(result_count_d)
by user_s, client_ip_s, bin(TimeGenerated, 1h)
| where TotalValuesReturned > 10000
| sort by TotalValuesReturned desc
Detection Query -- SPL (Splunk):
// Detect unusual CIP protocol traffic patterns on OT network
index=ot_network sourcetype=firewall
dest_ip="172.16.20.*" dest_port=44818
| bin _time span=1h
| stats count as conn_count dc(dest_ip) as unique_dests
sum(bytes) as total_bytes by src_ip _time
| where unique_dests > 3
| sort - _time
// Detect bulk historian data queries from vendor accounts
index=ot_historian sourcetype=pi_webapi
(user="*aquatech*" OR user="*vendor*")
query="*recorded*" query="*-30d*"
| bin _time span=1h
| stats count as query_count dc(point_name) as unique_points
sum(result_count) as total_values by user client_ip _time
| where total_values > 10000
| sort - total_values
Defender Decision Point 2: Your newly deployed OT monitoring system (installed after the CISA assessment) shows the vendor account browsing PLC configurations and querying 30 days of historian data. The vendor has a scheduled maintenance window next week, but this activity is happening now at 2:45 AM. Do you: (A) Contact the vendor immediately to verify the session? (B) Terminate the VPN session -- safety takes priority over vendor relationship? (C) Enable enhanced logging and monitor the session in real time?
Phase 3: Chemical Dosing Manipulation (Week 5 -- Attack Execution)¶
ATT&CK Techniques: T0855 (Unauthorized Command Message), T0836 (Modify Parameter), T0831 (Manipulation of Control)
After mapping the control system architecture and understanding normal operating parameters, MERCURY WAVE executes the attack during a period when WTP-2 is operating with reduced staffing (2 AM Sunday, single operator on shift). The attacker modifies chlorine dosing parameters through the HMI to increase levels above safe thresholds.
# Simulated chemical dosing manipulation (educational only)
# Attacker modifies PLC setpoints through HMI
# Attack execution timeline:
# 02:00 - Attacker connects via vendor VPN
# 02:05 - RDP to WTP2-HMI01 (172.16.20.100)
# 02:08 - Opens Wonderware InTouch (already running)
# 02:12 - Modifies HMI display to hide chlorine alarm indicators
# (InTouch scripting: Tag "Chlorine_High_Alarm_Visible" = 0)
# 02:14 - Changes Chlorine_Setpoint from 2.0 to 8.5 mg/L
# 02:14 - Changes Chlorine_High_Alarm from 4.0 to 10.0 mg/L
# (to suppress PLC-level alarm)
# 02:14 - Changes Chlorine_HiHi_Alarm from 6.0 to 12.0 mg/L
# (to suppress critical alarm)
# 02:15 - Chlorine pump speed increases from 42.5% to 178%
# (pump at maximum, controller driving to new setpoint)
# 02:17 - Chlorine_Actual begins rising: 2.0 → 2.8 → 3.5 → 4.2
# 02:22 - Chlorine_Actual reaches 5.1 mg/L (still rising)
# 02:28 - Chlorine_Actual reaches 6.8 mg/L
# 02:35 - Chlorine_Actual reaches 7.4 mg/L
# 02:41 - Chlorine_Actual reaches 8.1 mg/L
# 02:41 - *** SIS TRIP ACTIVATED ***
# Triconex SIS detects Chlorine > 8.0 mg/L
# SIS action: Chemical feed valves CLOSED
# SIS alarm: LOCAL PANEL + REMOTE (auto-dialer to operator)
# 02:42 - SIS auto-dialer calls on-call operator cell phone
# 02:43 - Chemical feed valves confirmed CLOSED by SIS
# 02:44 - Chlorine_Actual begins declining: 8.1 → 7.8 → 7.5
# 02:47 - On-call operator receives auto-dialer alert
# 02:48 - Attacker disconnects VPN session
# 02:55 - Operator arrives at plant, observes SIS trip condition
# 03:01 - Operator discovers modified setpoints and suppressed alarms
# PLC tag modification log (CIP protocol capture):
timestamp=2026-02-23T02:14:11Z proto=CIP src=172.16.20.100
dst=172.16.20.10 service=Write_Tag
tag=Chlorine_Setpoint old_value=2.0 new_value=8.5
user_session=HMI01-InTouch
timestamp=2026-02-23T02:14:14Z proto=CIP src=172.16.20.100
dst=172.16.20.10 service=Write_Tag
tag=Chlorine_High_Alarm old_value=4.0 new_value=10.0
user_session=HMI01-InTouch
timestamp=2026-02-23T02:14:16Z proto=CIP src=172.16.20.100
dst=172.16.20.10 service=Write_Tag
tag=Chlorine_HiHi_Alarm old_value=6.0 new_value=12.0
user_session=HMI01-InTouch
Evidence Artifact -- SIS Trip Log:
[2026-02-23T02:41:18Z] TRICONEX SIS - SAFETY TRIP ACTIVATED
Controller: WTP2-SIS (172.16.20.200)
Program: WTP2_Safety v2.0
TRIP CAUSE:
Input: AI_Chlorine_Independent (Channel 3, Slot 2)
Value: 8.12 mg/L
Setpoint: 8.0 mg/L (HARDCODED - not modifiable from network)
Condition: VALUE > SETPOINT for 3 consecutive scans (1.5 seconds)
ACTIONS EXECUTED:
1. DO_ChemFeed_Valve_Close: ENERGIZED (fail-safe closed)
2. DO_Chlorine_Pump_Trip: ENERGIZED (pump emergency stop)
3. DO_PlantAlarm_Beacon: ENERGIZED (local alarm activated)
4. DO_AutoDialer_Trigger: ENERGIZED (remote notification)
SIS STATUS: TRIPPED - Manual reset required
Operator acknowledgment: PENDING
NOTE: SIS operates on independent sensor (not shared with process PLC)
SIS chlorine reading: 8.12 mg/L
Process PLC chlorine reading: 8.09 mg/L (from separate analyzer)
*** SAFETY SYSTEM FUNCTIONED AS DESIGNED ***
*** Chemical feed isolated before reaching harmful levels ***
Evidence Artifact -- Operator Response Log:
[2026-02-23T02:47:00Z] Auto-dialer: Call placed to on-call operator
Phone: [REDACTED]
Message: "SAFETY TRIP - WTP2 - Chlorine High - Report to plant"
Acknowledgment: 02:47:33Z
[2026-02-23T02:55:00Z] Operator arrives at WTP-2 control room
Observations:
- SIS trip panel showing Chlorine High
- Local alarm beacon activated
- Chemical feed pumps stopped
- HMI showing chlorine at 7.2 mg/L (declining)
[2026-02-23T03:01:00Z] Operator investigates HMI
Findings:
- Chlorine_Setpoint: 8.5 mg/L (should be 2.0)
- Chlorine_High_Alarm: 10.0 mg/L (should be 4.0)
- Chlorine_HiHi_Alarm: 12.0 mg/L (should be 6.0)
- Alarm display suppressed (InTouch script modified)
- Active RDP session from 172.16.250.5 (VPN pool) -- DISCONNECTED
[2026-02-23T03:05:00Z] Operator: "These setpoints were changed.
I didn't change them. Calling IT and plant manager."
[2026-02-23T03:15:00Z] Plant manager notified
Decision: Take WTP-2 offline, switch to WTP-1 and WTP-3
Action: WTP-2 placed in manual shutdown mode
Impact: 94,000 residents affected by pressure reduction
Discussion Inject 5 -- Technical
The Safety Instrumented System (SIS) prevented a dangerous chemical release, functioning exactly as designed. However, the attacker modified PLC alarm setpoints to suppress process-level warnings. How should alarm management be architected in water treatment to prevent this type of attack? Consider: alarm setpoint change authentication, rate-of-change limits on critical parameters, independent alarm paths, and alarm suppression detection.
Discussion Inject 6 -- Decision
WTP-2 serves 94,000 residents. Taking it offline means reduced water pressure and potential boil-water advisory. The SIS prevented harmful water from reaching the distribution system. Do you: (A) Take WTP-2 offline immediately and issue a precautionary boil-water advisory? (B) Restore setpoints to normal, reset the SIS, and continue operating while investigating? (C) Take WTP-2 offline but delay the boil-water advisory pending water quality testing results?
Detection Query -- KQL (Microsoft Sentinel):
// Detect PLC setpoint changes on critical parameters
// Requires OT-specific log source (Claroty, Nozomi, or Dragos)
CommonSecurityLog
| where TimeGenerated > ago(24h)
| where DeviceVendor in ("Claroty", "Nozomi", "Dragos")
| where Activity has "Write_Tag" or Activity has "setpoint_change"
| where Message has_any ("Chlorine", "pH", "Fluoride", "Turbidity")
| extend OldValue = extract("old_value=([0-9.]+)", 1, Message)
| extend NewValue = extract("new_value=([0-9.]+)", 1, Message)
| extend ChangeRatio = todouble(NewValue) / todouble(OldValue)
| where ChangeRatio > 2.0 or ChangeRatio < 0.5
// Flag changes > 100% increase or > 50% decrease
| project TimeGenerated, SourceIP, DestinationIP, Activity,
Message, OldValue, NewValue, ChangeRatio
| sort by TimeGenerated desc
// Detect alarm setpoint modifications (potential alarm suppression)
CommonSecurityLog
| where TimeGenerated > ago(24h)
| where DeviceVendor in ("Claroty", "Nozomi", "Dragos")
| where Activity has "Write_Tag"
| where Message has_any ("Alarm", "alarm", "HiHi", "LoLo", "Trip")
| project TimeGenerated, SourceIP, DestinationIP, Activity, Message
| sort by TimeGenerated desc
Detection Query -- SPL (Splunk):
// Detect PLC setpoint changes on critical parameters
index=ot_network sourcetype=ot_ids
(action="Write_Tag" OR action="setpoint_change")
tag_name IN ("*Chlorine*","*pH*","*Fluoride*","*Turbidity*")
| eval change_ratio = new_value / old_value
| where change_ratio > 2.0 OR change_ratio < 0.5
| table _time src_ip dest_ip tag_name old_value new_value change_ratio
| sort - _time
// Detect alarm setpoint modifications (potential alarm suppression)
index=ot_network sourcetype=ot_ids
action="Write_Tag"
(tag_name="*Alarm*" OR tag_name="*HiHi*" OR tag_name="*LoLo*"
OR tag_name="*Trip*")
| table _time src_ip dest_ip tag_name old_value new_value
| sort - _time
Defender Decision Point 3: The operator has discovered modified setpoints and a disconnected VPN session. The SIS prevented harmful water release, but you cannot confirm whether any treated water with elevated chlorine entered the distribution system during the 27-minute window before the SIS tripped. Do you: (A) Issue an immediate boil-water advisory for all 94,000 affected residents? (B) Rush-test distribution system water samples before deciding on advisory? (C) Contact EPA and state health department for guidance before public notification?
Phase 4: Incident Response & Investigation (Weeks 5-8)¶
ATT&CK Techniques: Defense and response phase
CRWA activates its emergency response plan and contacts the FBI, CISA, and the state environmental protection agency. The investigation reveals the full scope of the vendor compromise and the attacker's reconnaissance activities.
# Simulated incident response (educational only)
# Incident Response Timeline:
# Hour 0 (02:55): Operator discovers SIS trip and modified setpoints
# Hour 1 (03:15): Plant manager notified, WTP-2 taken offline
# Hour 2 (04:00): IT manager engaged, VPN logs pulled
# Hour 3 (05:00): CISA contacted via emergency hotline
# Hour 6 (08:00): FBI Cyber Division notified
# Hour 8 (10:00): CISA ICS-CERT team arrives on-site
# Hour 12 (14:00): Water quality testing confirms no contaminated
# water entered distribution system
# Hour 14 (16:00): Boil-water advisory issued as precaution
# Day 3: Vendor VPN access suspended for all third parties
# Day 5: AquaTech Solutions notified of compromise
# Day 7: AquaTech confirms engineer workstation compromised
# Day 10: FBI identifies C2 infrastructure (203.0.113.45)
# Day 14: CISA ICS advisory published (anonymized)
# Investigation Findings:
# VPN Log Analysis:
[2026-02-23T04:30:00Z] IT_Manager: VPN log review
Vendor account "testuser-aquatech" connections last 30 days:
2026-01-25 14:22-16:45 (maintenance window - NORMAL)
2026-02-08 02:14-04:33 (2 AM - SUSPICIOUS) ← First recon
2026-02-12 01:45-03:22 (1 AM - SUSPICIOUS) ← Network mapping
2026-02-15 02:30-05:12 (2 AM - SUSPICIOUS) ← Historian queries
2026-02-19 02:05-03:47 (2 AM - SUSPICIOUS) ← PLC tag enumeration
2026-02-23 02:00-02:48 (2 AM - ATTACK) ← Setpoint manipulation
Pattern: 5 after-hours sessions over 15 days, all between 01:45-05:12
Normal vendor access: Weekdays 09:00-17:00
FINDING: After-hours access pattern was never flagged because
vendor account had no time-based restrictions
# Forensic analysis of WTP2-HMI01:
[2026-02-25T10:00:00Z] CISA_ICS_CERT: HMI forensic analysis
Evidence collected:
- RDP session logs: 5 remote sessions from 172.16.250.5
- InTouch script modifications: Alarm display suppression
- RSLogix project files: PLC tag exports saved to C:\Temp\
- Browser history: PI Web API queries for historical data
- Windows event logs: Account logon events for testuser-aquatech
- Network captures: CIP Write_Tag commands to 172.16.20.10
No malware installed on HMI (attacker used legitimate tools only)
Living-off-the-land technique: all actions used pre-installed software
Evidence Artifact -- Water Quality Testing Results:
[2026-02-23T14:00:00Z] WATER QUALITY ANALYSIS RESULTS
Laboratory: CRWA Water Quality Lab
Distribution System Samples (collected 08:00-12:00):
┌─────────────────────┬────────────┬──────────────┬─────────┐
│ Sample Location │ Chlorine │ pH │ Status │
│ │ (mg/L) │ │ │
├─────────────────────┼────────────┼──────────────┼─────────┤
│ DS-001 (Plant exit) │ 2.1 │ 7.19 │ NORMAL │
│ DS-015 (Main trunk) │ 1.8 │ 7.22 │ NORMAL │
│ DS-042 (Residential) │ 1.5 │ 7.25 │ NORMAL │
│ DS-067 (Commercial) │ 1.6 │ 7.20 │ NORMAL │
│ DS-089 (End of line) │ 1.2 │ 7.28 │ NORMAL │
└─────────────────────┴────────────┴──────────────┴─────────┘
CONCLUSION: SIS trip occurred before elevated chlorine reached
distribution system. Clearwell residence time (~45 min) and
SIS response at 27 min prevented contaminated water distribution.
All samples within EPA MCL for chlorine (4.0 mg/L max).
RECOMMENDATION: Precautionary boil-water advisory recommended
despite clean results, per AWIA guidelines.
Discussion Inject 7 -- Investigative
The attacker used only legitimate pre-installed tools (RSLinx, RSLogix, Wonderware InTouch) and authenticated vendor credentials. No malware was deployed. How does this "living-off-the-land" approach in OT environments complicate detection and forensic analysis? What OT-specific forensic artifacts would you prioritize collecting?
Discussion Inject 8 -- Decision
CISA's investigation reveals that AquaTech Solutions provides vendor VPN access to 23 other water utilities using similar configurations. MERCURY WAVE may have credentials for those utilities as well. Do you: (A) Coordinate with CISA to notify all 23 utilities immediately, risking tipping off the attacker? (B) Work with FBI to quietly investigate the other utilities before notification? (C) Recommend all 23 utilities immediately suspend vendor VPN access pending investigation?
Detection Query -- KQL (Microsoft Sentinel):
// Retrospective hunt: vendor access pattern anomalies
let vendor_sessions = CommonSecurityLog
| where TimeGenerated between(datetime(2026-01-01) .. datetime(2026-02-24))
| where DeviceVendor == "Fortinet"
| where Activity has "vpn" and Activity has "connect"
| where DestinationUserName has "aquatech";
vendor_sessions
| extend HourOfDay = datetime_part("hour", TimeGenerated)
| extend DayOfWeek = dayofweek(TimeGenerated)
| extend IsAfterHours = HourOfDay < 7 or HourOfDay > 18
| extend IsWeekend = DayOfWeek == 6d or DayOfWeek == 0d
| summarize TotalSessions=count(),
AfterHoursSessions=countif(IsAfterHours),
WeekendSessions=countif(IsWeekend)
by DestinationUserName
| extend AfterHoursRatio = round(1.0 * AfterHoursSessions / TotalSessions, 2)
| where AfterHoursRatio > 0.5
// Hunt for CIP Write_Tag commands to safety-critical PLCs
CommonSecurityLog
| where TimeGenerated > ago(90d)
| where DeviceVendor in ("Claroty", "Nozomi", "Dragos")
| where Activity has "Write_Tag"
| where DestinationIP in ("172.16.20.10", "172.16.20.11", "172.16.20.12")
| summarize WriteCount=count(), UniqueTags=dcount(Message)
by SourceIP, DestinationIP, bin(TimeGenerated, 1h)
| where WriteCount > 3 // Multiple setpoint changes in one hour
| sort by TimeGenerated desc
Detection Query -- SPL (Splunk):
// Retrospective hunt: vendor access pattern anomalies
index=vpn sourcetype=fortigate:vpn event_type=connect
user="*aquatech*"
earliest="01/01/2026:00:00:00" latest="02/24/2026:00:00:00"
| eval hour=strftime(_time, "%H")
| eval is_after_hours=if(hour < 7 OR hour > 18, 1, 0)
| eval day_of_week=strftime(_time, "%u")
| eval is_weekend=if(day_of_week > 5, 1, 0)
| stats count as total_sessions
sum(is_after_hours) as after_hours_sessions
sum(is_weekend) as weekend_sessions by user
| eval after_hours_ratio = round(after_hours_sessions / total_sessions, 2)
| where after_hours_ratio > 0.5
// Hunt for CIP Write_Tag commands to safety-critical PLCs
index=ot_network sourcetype=ot_ids action="Write_Tag"
dest_ip IN ("172.16.20.10","172.16.20.11","172.16.20.12")
| bin _time span=1h
| stats count as write_count dc(tag_name) as unique_tags
by src_ip dest_ip _time
| where write_count > 3
| sort - _time
Impact Assessment¶
| Category | Impact |
|---|---|
| Chemical Dosing Manipulation | Chlorine increased from 2.0 to 8.5 mg/L setpoint for 27 minutes |
| Safety System Response | SIS tripped at 8.1 mg/L, preventing harmful water distribution |
| Service Disruption | WTP-2 offline for 12 hours, 94,000 residents affected |
| Public Health Impact | No contaminated water entered distribution system (SIS prevented) |
| Boil-Water Advisory | Precautionary advisory issued for 12 hours |
| Vendor Compromise Scope | AquaTech Solutions provides access to 23 other utilities |
| Dwell Time | 15 days (first reconnaissance to attack execution) |
| Financial Impact | $23M (remediation, security upgrades, regulatory fines, public communication) |
| Regulatory Impact | EPA enforcement action, AWIA compliance review |
| Public Trust Impact | Significant -- media coverage of "water system hack" |
Detection & Response¶
How Blue Team Should Have Caught This¶
Detection Strategy 1: Vendor VPN Time-Based Access Controls
The attacker's 5 after-hours VPN sessions (all between 01:45-05:12) were never flagged because vendor accounts had no time-based restrictions. Implementing time-windowed vendor access (maintenance hours only), just-in-time VPN provisioning with approval workflows, and after-hours access alerts would have detected the first reconnaissance session on February 8.
Detection Strategy 2: OT Network Monitoring with Protocol-Aware IDS
Deploying an OT-specific IDS (Claroty, Nozomi, Dragos) with CIP protocol deep packet inspection would detect: CIP browse requests (network discovery), Write_Tag commands to safety-critical parameters, alarm setpoint modifications, and anomalous PLC communication patterns. CRWA had no OT-specific monitoring at the time of the attack.
Detection Strategy 3: Setpoint Change Authentication and Logging
Critical parameter changes (chlorine dosing, alarm thresholds) should require operator authentication at the HMI level, not just network-level access. Two-person integrity for safety-critical setpoint changes, rate-of-change limits, and immutable audit logs for all PLC write operations would have prevented or detected the attack.
Detection Strategy 4: SIS Independence Validation
The SIS functioned correctly because it was truly independent -- separate sensors, separate controller, hardware-keyed program access. Regular validation of SIS independence (network isolation testing, sensor calibration verification, hardware key audits) ensures this critical safety layer remains effective against sophisticated attacks.
Detection Strategy 5: Vendor Access Behavioral Baselines
Normal vendor maintenance involves specific patterns: connecting during business hours, accessing known systems, performing documented tasks. Building behavioral baselines for each vendor account and alerting on deviations (unusual hours, new systems accessed, bulk data queries) would flag reconnaissance activity.
Lessons Learned¶
Key Takeaways
-
Safety Instrumented Systems (SIS) are the last line of defense and must remain independent -- The Triconex SIS prevented a potentially harmful chemical release because it operated on independent sensors, an independent controller, and required a hardware key for program changes. SIS independence from the process control network is not optional -- it is the critical safety layer that saved 94,000 residents from contaminated water.
-
Vendor remote access is the most common initial access vector for ICS attacks -- Compromising a single SCADA integrator provided access to 24 water utilities. Vendor VPN access must be treated as a critical attack surface with mandatory MFA, time-based restrictions, session recording, and network segmentation within the OT zone. The cost of implementing these controls is trivial compared to the cost of a successful ICS attack.
-
Living-off-the-land techniques are especially effective in OT environments -- The attacker used only legitimate pre-installed tools (RSLinx, RSLogix, Wonderware InTouch) and generated traffic indistinguishable from normal maintenance. OT security cannot rely on malware detection alone -- it requires behavioral monitoring, protocol-aware deep packet inspection, and change management for all PLC modifications.
-
Alarm suppression amplifies the impact of process manipulation attacks -- The attacker modified alarm setpoints to prevent process-level warnings, buying time for the chlorine to rise. Alarm management security should include: immutable alarm setpoints (or change authentication), alarm suppression detection, and independent alarm paths that cannot be disabled from the HMI.
-
Water utility cybersecurity requires dedicated OT security resources -- CRWA's 3-person IT team had no OT security expertise, monitoring, or incident response capability. The CISA assessment identified critical gaps 18 months before the attack, but budget constraints prevented remediation. Federal and state funding programs for water utility cybersecurity are essential for small and mid-size utilities.
-
Sector-wide vendor compromise creates systemic risk -- A single compromised integrator endangered 24 utilities. The water sector needs shared threat intelligence, coordinated vendor security assessments, and sector-wide incident response protocols through organizations like WaterISAC.
MITRE ATT&CK Mapping¶
| Technique ID | Technique Name | Phase |
|---|---|---|
| T1133 | External Remote Services | Initial Access (vendor VPN) |
| T1078 | Valid Accounts | Initial Access (vendor credentials) |
| T1199 | Trusted Relationship | Initial Access (via SCADA integrator) |
| T1021 | Remote Services | Lateral Movement (RDP to HMI) |
| T1046 | Network Service Scanning | Discovery (CIP device browse) |
| T0846 | Remote System Discovery | Discovery (PLC enumeration) |
| T0888 | Remote System Information Discovery | Discovery (PLC tag enumeration) |
| T0855 | Unauthorized Command Message | Execution (CIP Write_Tag) |
| T0836 | Modify Parameter | Impact (chlorine setpoint change) |
| T0831 | Manipulation of Control | Impact (chemical dosing manipulation) |
| T0878 | Alarm Suppression | Defense Evasion (alarm threshold modification) |
| T0816 | Device Restart/Shutdown | Impact (plant forced offline) |