Chapter 18: Malware Analysis¶

Overview¶

Malware analysis is the disciplined practice of dissecting malicious software to understand its behavior, capabilities, and indicators — transforming unknown threats into actionable intelligence. From initial triage through full static and dynamic analysis to memory forensics, this chapter equips analysts with a systematic methodology applicable to ransomware, RATs, rootkits, stealers, and nation-state implants. Every finding extracted from a malware sample strengthens detection rules, informs IR playbooks, and contributes to threat intelligence.

Learning Objectives¶

By the end of this chapter, students SHALL be able to:

1. Configure a safe, isolated malware analysis environment
2. Perform triage analysis to quickly classify a sample's risk and capabilities
3. Conduct static analysis using disassemblers, decompilers, and string extraction
4. Execute dynamic analysis in a controlled sandbox to observe real-time behavior
5. Perform memory forensics to detect in-memory-only threats and injected code
6. Extract YARA signatures, network indicators, and host-based IOCs from a sample

Prerequisites¶

• Familiarity with Windows internals (processes, DLLs, registry, services)
• Basic understanding of x86/x64 assembly or willingness to read disassembly
• Familiarity with networking concepts (TCP, DNS, HTTP/S)

18.1 Analysis Environment Setup¶

Never analyze malware on a production system. The analysis environment SHALL be:

• Isolated: No physical network access; dedicated hypervisor (VMware Workstation, VirtualBox, or QEMU)
• Snapshot-based: Take pre-analysis snapshot; revert after each sample
• Monitored: All traffic captured via Wireshark/tcpdump + INetSim for fake internet services
• Instrumented: Process Monitor, API Monitor, Sysmon installed

18.1.1 Recommended Analysis VM Stack¶

HOST (isolated laptop or dedicated server)
├── REMnux (Linux) — static analysis, network simulation, memory forensics
├── FlareVM (Windows 10/11) — dynamic analysis on Windows targets
│   ├── Sysmon (extended logging)
│   ├── Process Monitor (procmon)
│   ├── Wireshark + FakeNet-NG
│   ├── x64dbg / WinDbg
│   ├── Ghidra / IDA Free
│   └── PE-bear, CFF Explorer, DIE (Detect-It-Easy)
└── INetSim (REMnux) — simulates DNS, HTTP, FTP, SMTP responses

# REMnux setup
curl -s https://remnux.org/get-remnux.sh | sudo bash

# FlareVM setup (Windows, run as Administrator in PowerShell)
iex ((New-Object System.Net.WebClient).DownloadString('https://raw.githubusercontent.com/mandiant/flare-vm/main/install.ps1'))

# INetSim network simulation
sudo inetsim --log-dir /var/log/inetsim/

# Sysmon with SwiftOnSecurity config
sysmon64.exe -accepteula -i sysmonconfig.xml

18.2 Analysis Methodology Overview¶

flowchart TD
    A[Sample Received] --> B[Safe Handling\nHash + Quarantine]
    B --> C[TRIAGE\n5-minute assessment]
    C --> D{Threat Level?}
    D -->|Low| E[Automated Sandbox\nOnly]
    D -->|Medium/High| F[Full Analysis]
    F --> G[Static Analysis\nNo execution]
    G --> H[Dynamic Analysis\nControlled execution]
    H --> I[Memory Forensics\nif needed]
    I --> J[IOC Extraction]
    J --> K[Malware Family\nClassification]
    K --> L[Intelligence Report\n+ Detection Rules]
    L --> M[Share: MISP/TAXII/ISAC]

    style C fill:#f4a261,color:#000
    style G fill:#2d6a4f,color:#fff
    style H fill:#1d3557,color:#fff
    style I fill:#780000,color:#fff
    style L fill:#e63946,color:#fff

18.3 Triage Analysis¶

Triage answers: Is this malicious? How urgent? What does it do? — in under 5 minutes.

18.3.1 Hashing and Initial Context¶

# Hash the file
md5sum sample.exe
sha256sum sample.exe
ssdeep sample.exe  # fuzzy hash for similarity

# VirusTotal lookup (never upload sensitive samples — use hash only)
curl "https://www.virustotal.com/api/v3/files/$(sha256sum sample.exe | cut -d' ' -f1)" \
  -H "x-apikey: YOUR_VT_KEY" | python3 -m json.tool | grep -E "malicious|type_tag|name"

# Identify file type (ignore extension)
file sample.exe
die sample.exe  # Detect-It-Easy — packer/obfuscation detection

18.3.2 Quick String Extraction¶

# Extract printable strings
strings -n 8 sample.exe | grep -iE "(http|https|cmd|powershell|reg|taskkill|vssadmin)"

# Unicode strings
strings -el sample.exe  # little-endian unicode

# FLOSS — extract obfuscated/stack strings
floss --no-static-strings sample.exe

18.3.3 PE Header Analysis¶

import pefile

pe = pefile.PE("sample.exe")

# Compilation timestamp (can be forged)
import datetime
ts = pe.FILE_HEADER.TimeDateStamp
print("Compiled:", datetime.datetime.utcfromtimestamp(ts))

# Sections
for section in pe.sections:
    print(f"{section.Name.decode().strip()} | "
          f"VSize: {section.Misc_VirtualSize} | "
          f"RawSize: {section.SizeOfRawData} | "
          f"Entropy: {section.get_entropy():.2f}")

# Imports
for entry in pe.DIRECTORY_ENTRY_IMPORT:
    print(entry.dll.decode())
    for imp in entry.imports:
        print(f"  {imp.name.decode() if imp.name else hex(imp.ordinal)}")

High entropy sections (>7.0) indicate encryption or compression — likely packed. Packers like UPX, MPRESS, Themida, or custom crypto loops are common evasion mechanisms.

18.4 Static Analysis¶

18.4.1 Disassembly and Decompilation¶

Ghidra (NSA-developed, open source) and IDA Pro (commercial) are the primary disassemblers. Ghidra's decompiler produces pseudo-C output that dramatically accelerates analysis.

Ghidra Analysis Workflow:
1. File → New Project → Import File (sample.exe)
2. Auto-analyze (check all analyzers including "Demangler GCC/MS")
3. Code browser → Symbol Tree → Functions
4. Search → For Strings → filter by suspicious keywords
5. Navigation → Go To → entry (find main entry point)
6. Decompiler window → review pseudo-C
7. Mark suspicious functions: Right-click → Rename Function

18.4.2 Key Windows API Categories to Watch¶

18.4.3 Identifying Obfuscation¶

# Detect XOR encoded strings
def xor_decode(data, key):
    return bytes([b ^ key for b in data])

# Try all single-byte XOR keys
for key in range(256):
    decoded = xor_decode(encrypted_blob, key)
    if b"http" in decoded or b"cmd" in decoded:
        print(f"Key 0x{key:02x}: {decoded[:100]}")

# Detect Base64-encoded payloads in scripts
import re, base64
pattern = r'[A-Za-z0-9+/]{50,}={0,2}'
matches = re.findall(pattern, script_content)
for m in matches:
    try:
        decoded = base64.b64decode(m)
        print(decoded[:200])
    except:
        pass

18.4.4 YARA Rule Development¶

rule APT_Dropper_CustomXOR_v2 {
    meta:
        description = "Detects APT dropper using custom XOR decoder stub"
        author = "Nexus SecOps Analyst"
        date = "2026-01-15"
        hash = "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
        reference = "IR-2026-0042"
        mitre_attack = "T1027.001"

    strings:
        $decode_stub = { 8B 45 F8 8B 55 FC 0F B6 04 02 34 ?? 88 04 02 }
        $pdb_artifact = "\\Release\\dropper" nocase
        $mutex_name = "Global\\MSUpdate_" wide
        $ua_string = "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0" nocase

    condition:
        uint16(0) == 0x5A4D and  // MZ header
        filesize < 500KB and
        (2 of ($decode_stub, $pdb_artifact, $mutex_name) or
         all of ($ua_string, $mutex_name))
}

18.5 Dynamic Analysis¶

18.5.1 Process Monitoring¶

Process Monitor (ProcMon) Setup for Malware Analysis:
1. Filter → Add:
   - Process Name | contains | malware_sample.exe | Include
   - Operation | contains | RegSetValue | Include
   - Operation | contains | WriteFile | Include
   - Path | contains | \Startup | Include
   - Path | contains | HKCU\Software\Microsoft\Windows\CurrentVersion\Run | Include
2. Tools → Process Tree (to see spawned children)
3. File → Save (export for report)

18.5.2 Network Traffic Analysis¶

# Wireshark display filters for malware traffic
# DNS tunneling (high subdomain entropy)
dns.qry.name matches "[a-z0-9]{20,}\."

# HTTP beaconing patterns
http.request.method == "POST" && http.content_length > 0

# Suspicious user agents
http.user_agent contains "Mozilla/4.0 (compatible; MSIE 6.0"

# Certificate anomalies
ssl.handshake.extensions_server_name != ""

# INetSim — decode all HTTP requests from analysis VM
tail -f /var/log/inetsim/report/report.txt

18.5.3 Behavioral Analysis Checklist¶

When analyzing dynamic execution, document:

• [ ] Process tree: Parent → child process relationships
• [ ] File drops: What files are written, to where, with what entropy
• [ ] Registry modifications: Persistence keys, configuration storage
• [ ] Network connections: C2 IPs/domains, protocol, beacon interval
• [ ] Mutex creation: Prevents multiple instances (use as IOC)
• [ ] Service installation: New services for persistence
• [ ] Privilege escalation: UAC bypass attempts, token manipulation
• [ ] Anti-analysis: Sleep loops, VM detection, debugger checks
• [ ] Credential access: LSASS access, SAM/NTDS dumps
• [ ] Lateral movement: SMB connections, remote scheduled tasks

18.5.4 Automated Sandbox Platforms¶

18.6 Memory Forensics¶

Memory forensics analyzes RAM dumps to find in-memory-only malware, injected shellcode, and credential material that is never written to disk.

18.6.1 Acquiring a Memory Dump¶

# Windows — WinPmem (live acquisition)
winpmem_mini_x64.exe -o memory.raw

# Windows — DumpIt
DumpIt.exe /O memory.raw /T RAW

# Linux — LiME (kernel module)
sudo insmod lime-$(uname -r).ko "path=/tmp/memory.lime format=lime"

# VMware — suspend VM, then use .vmem file
# Hyper-V — checkpoint creates .avhd which can be analyzed

18.6.2 Volatility 3 Analysis¶

# Identify OS profile
python3 vol.py -f memory.raw windows.info

# List processes (includes hidden/terminated)
python3 vol.py -f memory.raw windows.pslist
python3 vol.py -f memory.raw windows.pstree

# Find processes with injected code
python3 vol.py -f memory.raw windows.malfind
# Look for: VAD region MZ headers, rwx permissions, anomalous process names

# Extract injected code for analysis
python3 vol.py -f memory.raw windows.malfind --dump

# Network connections
python3 vol.py -f memory.raw windows.netstat
python3 vol.py -f memory.raw windows.netscan

# Extract registry hives (for credential extraction)
python3 vol.py -f memory.raw windows.registry.hivelist
python3 vol.py -f memory.raw windows.hashdump

# Detect rootkit hooks
python3 vol.py -f memory.raw windows.ssdt  # SSDT hooks
python3 vol.py -f memory.raw windows.driverirp  # IRP hooks
python3 vol.py -f memory.raw windows.callbacks  # kernel callbacks

# Command history
python3 vol.py -f memory.raw windows.cmdline
python3 vol.py -f memory.raw windows.consoles  # actual input/output history

# Extract strings from specific process memory
python3 vol.py -f memory.raw windows.strings --pid 1234

18.6.3 Process Injection Detection¶

graph TD
    A[Suspicious Process Detected\nby Malfind] --> B{Indicators?}
    B -->|MZ header in VAD| C[PE Injection\nor Reflective DLL]
    B -->|shellcode-like bytes\nno PE header| D[Shellcode Injection\nVirtualAlloc+WriteProcessMemory]
    B -->|Module hollowing| E[Process Hollowing\nNtUnmapViewOfSection]
    B -->|Double-free pattern| F[Atom Bombing\nAtomicAddAtom]

    C --> G[Dump VAD region\nAnalyze PE]
    D --> H[Disassemble shellcode\nExtract decoder]
    E --> I[Compare on-disk PE\nvs. in-memory PE]
    F --> J[Enumerate Global Atom Table]

    style A fill:#e63946,color:#fff
    style G fill:#2d6a4f,color:#fff
    style H fill:#2d6a4f,color:#fff
    style I fill:#2d6a4f,color:#fff

18.7 Malware Families Reference¶

18.7.1 Ransomware Families¶

18.7.2 RAT/Backdoor Families¶

18.8 Anti-Analysis Evasion and Countermeasures¶

18.9 IOC Extraction and Reporting¶

18.9.1 IOC Types and Quality¶

ATOMIC IOCs (low durability — change per campaign):
  - File hashes (MD5/SHA1/SHA256)
  - IP addresses
  - Domain names
  - URL paths
  - Email sender addresses

COMPUTED IOCs (medium durability):
  - SSL certificate hashes / Subject fields
  - Mutex names
  - Pipe names
  - Registry key paths
  - YARA rules based on unique strings

BEHAVIORAL IOCs (high durability — TTP-level):
  - Process injection sequence (API call order)
  - Beacon timing patterns (jitter + sleep)
  - PE section names + entropy patterns
  - Specific MITRE ATT&CK techniques executed

18.9.2 Malware Analysis Report Template¶

## Malware Analysis Report
**Sample Hash (SHA256):** e3b0c44298fc1c149afbf4c8996fb924...
**File Type:** PE32+ executable (Windows)
**First Seen:** 2026-01-15
**Classification:** Ransomware / LockBit 3.0 variant
**Threat Level:** CRITICAL
**Analyst:** [Name] | **Date:** 2026-01-15

### Executive Summary
[2-3 sentence description of what the sample does and its risk]

### Technical Details
- **Packer/Obfuscation:** UPX → custom XOR second stage
- **Compilation Time:** 2026-01-10 (UTC) [possibly forged]
- **PDB Path:** D:\Projects\LB3\Release\locker.pdb
- **C2 Infrastructure:** 185.220.x.x:443 (HTTPS), domains listed below
- **Encryption:** ChaCha20 (file content) + RSA-4096 (key encryption)
- **Extension:** .lockbit3 appended to encrypted files
- **Ransom Note:** !!-README-RESTORE-FILES-!!.txt

### Kill Chain Mapping
| Phase | Technique | Evidence |
|-------|-----------|---------|
| Execution | T1059.001 PowerShell | Observed via procmon |
| Persistence | T1053.005 Scheduled Task | Task name "WindowsUpdate23" |
| Defense Evasion | T1562.001 Disable AV | sc stop WinDefend |
| Discovery | T1083 File System Enumeration | All drives enumerated |
| Impact | T1486 Data Encrypted for Impact | .lockbit3 extension |

### Indicators of Compromise
**Hashes:**
- MD5: [value]
- SHA256: [value]

**Network:**
- C2 IP: 185.220.x.x
- C2 Domain: cdn-update.example.com

**Host:**
- Mutex: Global\LockBit_1A2B3C
- Registry: HKCU\...\Run "LBServices"
- File: C:\Windows\Temp\lb3.exe

### YARA Rule
[Embedded YARA rule]

### Recommendations
1. Block C2 indicators at network perimeter
2. Deploy YARA rule to EDR
3. Search for mutex name across endpoint fleet

18.10 Benchmark Controls¶

Exam Prep & Certifications¶

Key Terms¶

Dynamic Analysis — Executing malware in a controlled environment to observe its runtime behavior, network activity, and system changes.

Entropy — A measure of randomness in binary data (0–8). High entropy (>7.0) in PE sections indicates encryption or compression, common in packed malware.

FLOSS — FLARE Obfuscated String Solver — a tool from Mandiant that extracts obfuscated strings from malware without executing it.

Malfind — A Volatility plugin that identifies memory regions containing injected code by scanning for executable pages with suspicious characteristics.

Packer — A tool that compresses and/or encrypts an executable's code to evade static analysis. The original code is unpacked at runtime in memory.

Process Hollowing — An injection technique where a legitimate process is created in suspended state, its memory is unmapped, and malicious code is mapped in its place.

YARA — Yet Another Recursive Acronym — a pattern matching language designed for malware identification. Rules consist of strings and logical conditions.