Lab 33: Cloud Forensics Evidence Collection¶

Learning Objectives¶

By the end of this lab, you will be able to:

1. Recognize the forensic artifacts unique to AWS, Azure, and GCP and the CLI tools used to acquire them.
2. Extract CloudTrail, Azure Activity Log, and GCP Cloud Audit Log entries in a chain-of-custody-preserving manner.
3. Create forensically sound volume snapshots across all three hyperscalers (EBS, Managed Disks, Persistent Disks).
4. Preserve cross-cloud evidence in an immutable object store using S3 Object Lock (compliance mode).
5. Compute, record, and verify SHA-256 hashes for every evidence artifact.
6. Produce a tamper-evident chain of custody document in CSV and JSON formats.
7. Normalize heterogeneous log formats from all three clouds into a unified schema.
8. Reconstruct a minute-accurate timeline that correlates attacker actions across providers.

Phase 1: Scenario Setup¶

1.1 Incident Summary¶

1.2 Infrastructure Diagram¶

flowchart LR
    subgraph Internet
        ATK["Attacker<br/>192.0.2.45"]
    end

    subgraph AWS["AWS Account 111122223333"]
        IAM["IAM User<br/>deploy-bot"]
        EC2["EC2 Instance<br/>i-0abcd1234ef567890<br/>10.0.4.88"]
        S3["S3 Bucket<br/>acme-config-backups"]
        CT["CloudTrail<br/>acme-trail"]
    end

    subgraph Azure["Azure Tenant contoso.example.com"]
        AAD["Azure AD<br/>Enterprise App"]
        VM["VM acme-app-01<br/>10.20.3.14"]
        LAW["Log Analytics<br/>Workspace"]
    end

    subgraph GCP["GCP Project project-acme"]
        SA["Service Account<br/>exfil-sa"]
        GCE["Compute Engine<br/>gce-analytics-01<br/>10.30.5.22"]
        GCS["Cloud Storage<br/>acme-customer-exports"]
        ALOG["Cloud Audit Logs"]
    end

    ATK -->|"1. Stolen IAM key"| IAM
    IAM -->|"2. iam:PassRole"| EC2
    EC2 -->|"3. Federated auth"| AAD
    AAD -->|"4. OIDC federation"| SA
    SA -->|"5. Data exfil"| GCS
    EC2 --> S3
    EC2 --> CT
    VM --> LAW
    GCE --> ALOG

1.3 Evidence Collection Objectives¶

Your objectives for the next four hours:

1.4 Lab Environment Bootstrap¶

Create the working directory structure that will receive your evidence. Keep provider-specific subfolders and a dedicated hash manifest directory.

mkdir -p ~/ir-2026-0418-A/{aws,azure,gcp,timeline,chain-of-custody,hashes}
cd ~/ir-2026-0418-A
export IR_CASE="IR-2026-0418-A"
export IR_HOME="$PWD"
export EVIDENCE_COLLECTOR="testuser@example.com"
date -u +"%Y-%m-%dT%H:%M:%SZ" > ~/ir-2026-0418-A/case-opened.txt

Confirm CLI tools are installed and authenticated.

aws --version
az --version
gcloud --version
python3 --version

Expected output (versions may differ):

aws-cli/2.15.40 Python/3.11.8 Linux/5.15.0
azure-cli 2.58.0
Google Cloud SDK 469.0.0
Python 3.11.8

Phase 2: AWS Evidence Collection¶

2.1 CloudTrail Log Extraction¶

CloudTrail is the canonical AWS management plane log. For this incident, you need events from 2026-04-10 through 2026-04-16 across all regions.

aws cloudtrail lookup-events \
  --start-time 2026-04-10T00:00:00Z \
  --end-time 2026-04-16T23:59:59Z \
  --max-results 50 \
  --region us-east-1 \
  --profile ir-forensic \
  > $IR_HOME/aws/cloudtrail-lookup-us-east-1.json

Expected output (truncated):

{
    "Events": [
        {
            "EventId": "a1b2c3d4-5678-90ab-cdef-EXAMPLE11111",
            "EventName": "ConsoleLogin",
            "EventTime": "2026-04-15T22:47:03Z",
            "Username": "deploy-bot",
            "CloudTrailEvent": "{\"sourceIPAddress\":\"192.0.2.45\",\"userAgent\":\"aws-cli/2.15.40\",\"eventName\":\"ConsoleLogin\"}"
        }
    ]
}

For completeness, pull the full CloudTrail bucket contents rather than the lookup API (which has a 90-day limit). Identify the trail first.

aws cloudtrail describe-trails \
  --profile ir-forensic \
  --region us-east-1

Expected output:

{
    "trailList": [
        {
            "Name": "acme-trail",
            "S3BucketName": "acme-cloudtrail-logs-111122223333",
            "IncludeGlobalServiceEvents": true,
            "IsMultiRegionTrail": true,
            "HomeRegion": "us-east-1",
            "TrailARN": "arn:aws:cloudtrail:us-east-1:111122223333:trail/acme-trail"
        }
    ]
}

Sync the raw CloudTrail logs from the bucket into local evidence storage. The --exact-timestamps flag preserves provider timestamps.

aws s3 sync \
  s3://acme-cloudtrail-logs-111122223333/AWSLogs/111122223333/CloudTrail/ \
  $IR_HOME/aws/cloudtrail-raw/ \
  --exact-timestamps \
  --profile ir-forensic

Expected output (truncated):

download: s3://acme-cloudtrail-logs-111122223333/AWSLogs/111122223333/CloudTrail/us-east-1/2026/04/15/111122223333_CloudTrail_us-east-1_20260415T2245Z_abc123.json.gz to aws/cloudtrail-raw/us-east-1/2026/04/15/111122223333_CloudTrail_us-east-1_20260415T2245Z_abc123.json.gz
...
Completed 347 file(s), 2.4 GiB

aws cloudtrail validate-logs \
  --trail-arn arn:aws:cloudtrail:us-east-1:111122223333:trail/acme-trail \
  --start-time 2026-04-10T00:00:00Z \
  --end-time 2026-04-16T23:59:59Z \
  --profile ir-forensic

Expected output:

Validating log files for trail arn:aws:cloudtrail:us-east-1:111122223333:trail/acme-trail between 2026-04-10T00:00:00Z and 2026-04-16T23:59:59Z

Results requested for 2026-04-10T00:00:00Z to 2026-04-16T23:59:59Z

Results found for 2026-04-10T00:00:00Z to 2026-04-16T23:59:59Z:

347/347 digest files valid
2891/2891 log files valid

2.2 EBS Snapshot Creation¶

The compromised EC2 instance i-0abcd1234ef567890 must have its root and data volumes snapshotted. Never stop or terminate the instance before snapshotting -- you will lose the in-memory state and risk shredding volatile artifacts.

Enumerate attached volumes.

aws ec2 describe-instances \
  --instance-ids i-0abcd1234ef567890 \
  --profile ir-forensic \
  --region us-east-1 \
  --query "Reservations[].Instances[].BlockDeviceMappings[].{Device:DeviceName,Volume:Ebs.VolumeId}" \
  --output table

Expected output:

-----------------------------------------------
|          DescribeInstances                  |
+--------------+------------------------------+
|   Device     |          Volume              |
+--------------+------------------------------+
|  /dev/xvda   |  vol-0123456789abcdef0       |
|  /dev/xvdf   |  vol-0fedcba9876543210       |
+--------------+------------------------------+

Create forensic snapshots with descriptive tags.

for VOL in vol-0123456789abcdef0 vol-0fedcba9876543210; do
  aws ec2 create-snapshot \
    --volume-id $VOL \
    --description "$IR_CASE forensic snapshot of $VOL" \
    --tag-specifications "ResourceType=snapshot,Tags=[{Key=IncidentId,Value=$IR_CASE},{Key=Collector,Value=$EVIDENCE_COLLECTOR},{Key=Purpose,Value=forensics},{Key=LegalHold,Value=true}]" \
    --profile ir-forensic \
    --region us-east-1
done

Expected output per snapshot:

{
    "Description": "IR-2026-0418-A forensic snapshot of vol-0123456789abcdef0",
    "Encrypted": true,
    "OwnerId": "111122223333",
    "Progress": "",
    "SnapshotId": "snap-0a1b2c3d4e5f67890",
    "StartTime": "2026-04-16T10:02:17.000Z",
    "State": "pending",
    "VolumeId": "vol-0123456789abcdef0",
    "VolumeSize": 100
}

Wait for snapshots to complete.

aws ec2 wait snapshot-completed \
  --snapshot-ids snap-0a1b2c3d4e5f67890 snap-0f9e8d7c6b5a43210 \
  --profile ir-forensic \
  --region us-east-1
echo "Snapshots complete at $(date -u +%FT%TZ)"

Copy snapshots to an isolated forensic account for analysis. This creates an air gap between the responder's forensic account and the compromised production account.

aws ec2 copy-snapshot \
  --source-region us-east-1 \
  --source-snapshot-id snap-0a1b2c3d4e5f67890 \
  --description "$IR_CASE copy to forensic account" \
  --encrypted \
  --kms-key-id arn:aws:kms:us-east-1:444455556666:key/mrk-forensic-2026 \
  --profile ir-forensic-account \
  --region us-east-1

2.3 IAM and Identity Analysis¶

Pull the current IAM configuration snapshot for the compromised user.

aws iam get-user --user-name deploy-bot \
  --profile ir-forensic \
  > $IR_HOME/aws/iam-user-deploy-bot.json

aws iam list-access-keys --user-name deploy-bot \
  --profile ir-forensic \
  > $IR_HOME/aws/iam-access-keys-deploy-bot.json

aws iam list-attached-user-policies --user-name deploy-bot \
  --profile ir-forensic \
  > $IR_HOME/aws/iam-policies-deploy-bot.json

Expected output of list-access-keys:

{
    "AccessKeyMetadata": [
        {
            "UserName": "deploy-bot",
            "AccessKeyId": "AKIAIOSFODNN7EXAMPLE",
            "Status": "Active",
            "CreateDate": "2025-11-03T14:22:01Z"
        }
    ]
}

Generate the last-used report.

aws iam get-access-key-last-used \
  --access-key-id AKIAIOSFODNN7EXAMPLE \
  --profile ir-forensic \
  > $IR_HOME/aws/iam-access-key-last-used.json

Expected output:

{
    "UserName": "deploy-bot",
    "AccessKeyLastUsed": {
        "LastUsedDate": "2026-04-15T22:47:03Z",
        "ServiceName": "s3",
        "Region": "us-east-1"
    }
}

Extract the most forensically valuable CloudTrail events -- those touching IAM, STS, and KMS.

aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventSource,AttributeValue=iam.amazonaws.com \
  --start-time 2026-04-10T00:00:00Z \
  --end-time 2026-04-16T23:59:59Z \
  --profile ir-forensic \
  --region us-east-1 \
  > $IR_HOME/aws/cloudtrail-iam-events.json

aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventSource,AttributeValue=sts.amazonaws.com \
  --start-time 2026-04-10T00:00:00Z \
  --end-time 2026-04-16T23:59:59Z \
  --profile ir-forensic \
  --region us-east-1 \
  > $IR_HOME/aws/cloudtrail-sts-events.json

2.4 VPC Flow Logs¶

VPC Flow Logs capture network metadata. For this incident, you need logs from the VPC containing i-0abcd1234ef567890. Identify the VPC first.

aws ec2 describe-instances \
  --instance-ids i-0abcd1234ef567890 \
  --profile ir-forensic \
  --region us-east-1 \
  --query "Reservations[].Instances[].VpcId" \
  --output text

Expected output:

vpc-0abc1234def567890

List flow log configurations.

aws ec2 describe-flow-logs \
  --filter "Name=resource-id,Values=vpc-0abc1234def567890" \
  --profile ir-forensic \
  --region us-east-1

Expected output:

{
    "FlowLogs": [
        {
            "FlowLogId": "fl-0987654321fedcba0",
            "LogDestinationType": "s3",
            "LogDestination": "arn:aws:s3:::acme-flow-logs-111122223333",
            "TrafficType": "ALL",
            "ResourceId": "vpc-0abc1234def567890"
        }
    ]
}

Sync the flow log bucket segment for the incident window.

aws s3 sync \
  s3://acme-flow-logs-111122223333/AWSLogs/111122223333/vpcflowlogs/us-east-1/2026/04/ \
  $IR_HOME/aws/vpc-flow-logs/ \
  --exclude "*" \
  --include "*2026040*" \
  --include "*2026041*" \
  --profile ir-forensic

Filter flow logs for the attacker IP 192.0.2.45 to surface connections to and from the compromised host.

cd $IR_HOME/aws/vpc-flow-logs
zgrep -r "192.0.2.45" . > $IR_HOME/aws/flow-matches-attacker-ip.txt
wc -l $IR_HOME/aws/flow-matches-attacker-ip.txt

Expected output:

417 /root/ir-2026-0418-A/aws/flow-matches-attacker-ip.txt

2.5 S3 Access Logs¶

If S3 server access logging was enabled on the target bucket acme-config-backups, pull those logs.

aws s3api get-bucket-logging \
  --bucket acme-config-backups \
  --profile ir-forensic

Expected output:

{
    "LoggingEnabled": {
        "TargetBucket": "acme-s3-access-logs",
        "TargetPrefix": "acme-config-backups/"
    }
}

aws s3 sync \
  s3://acme-s3-access-logs/acme-config-backups/ \
  $IR_HOME/aws/s3-access-logs/ \
  --exclude "*" \
  --include "2026-04-1*" \
  --profile ir-forensic

Hash every AWS artifact collected in this phase.

cd $IR_HOME/aws
find . -type f -exec sha256sum {} \; | sort > $IR_HOME/hashes/aws-artifacts.sha256
wc -l $IR_HOME/hashes/aws-artifacts.sha256

Expected output:

3421 /root/ir-2026-0418-A/hashes/aws-artifacts.sha256

Phase 3: Azure Evidence Collection¶

3.1 Azure Activity Log Export¶

The Azure Activity Log is the subscription-level equivalent of CloudTrail. Export the incident window for the affected subscription.

az login --service-principal \
  --username "http://ir-forensic-reader" \
  --password REDACTED \
  --tenant contoso.example.com

az account set --subscription "00000000-0000-0000-0000-000000000001"

az monitor activity-log list \
  --start-time 2026-04-10T00:00:00Z \
  --end-time 2026-04-16T23:59:59Z \
  --max-events 5000 \
  --output json \
  > $IR_HOME/azure/activity-log.json

Expected output (truncated):

[
    {
        "caller": "attacker@contoso.example.com",
        "eventTimestamp": "2026-04-15T23:02:11.4173921+00:00",
        "operationName": {
            "value": "Microsoft.AAD/Applications/write",
            "localizedValue": "Create or Update Application"
        },
        "status": {
            "value": "Succeeded"
        },
        "resourceGroupName": "rg-acme-prod",
        "httpRequest": {
            "clientIpAddress": "192.0.2.45"
        }
    }
]

For deeper retention, Activity Log is archived to Log Analytics. Query the workspace.

az monitor log-analytics query \
  --workspace 11111111-2222-3333-4444-555555555555 \
  --analytics-query "AzureActivity | where TimeGenerated between (datetime(2026-04-10) .. datetime(2026-04-16)) | where CallerIpAddress == '192.0.2.45' | project TimeGenerated, Caller, OperationNameValue, ActivityStatusValue, ResourceGroup, CorrelationId" \
  --output json \
  > $IR_HOME/azure/loganalytics-activity-attacker-ip.json

Expected output:

[
    {
        "TimeGenerated": "2026-04-15T23:02:11.417Z",
        "Caller": "attacker@contoso.example.com",
        "OperationNameValue": "Microsoft.AAD/Applications/write",
        "ActivityStatusValue": "Succeeded",
        "ResourceGroup": "rg-acme-prod",
        "CorrelationId": "f1e2d3c4-5678-90ab-cdef-EXAMPLE22222"
    }
]

3.2 VM Managed Disk Snapshots¶

The Azure VM acme-app-01 in resource group rg-acme-prod needs its OS and data disks snapshotted.

Enumerate disks.

az vm show \
  --resource-group rg-acme-prod \
  --name acme-app-01 \
  --query "{osDisk:storageProfile.osDisk.name, dataDisks:storageProfile.dataDisks[].name}" \
  --output json

Expected output:

{
    "osDisk": "acme-app-01-osdisk",
    "dataDisks": [
        "acme-app-01-data-01"
    ]
}

Create snapshots. Azure snapshots are full point-in-time copies and independent resources.

SNAP_TIME=$(date -u +%Y%m%dT%H%M%SZ)

for DISK in acme-app-01-osdisk acme-app-01-data-01; do
  az snapshot create \
    --resource-group rg-acme-prod \
    --name "forensic-$DISK-$SNAP_TIME" \
    --source /subscriptions/00000000-0000-0000-0000-000000000001/resourceGroups/rg-acme-prod/providers/Microsoft.Compute/disks/$DISK \
    --tags IncidentId=$IR_CASE Collector=$EVIDENCE_COLLECTOR LegalHold=true \
    --incremental false \
    --output json \
    >> $IR_HOME/azure/snapshots.json
done

Expected output per snapshot:

{
    "creationData": {
        "createOption": "Copy",
        "sourceResourceId": "/subscriptions/00000000-0000-0000-0000-000000000001/resourceGroups/rg-acme-prod/providers/Microsoft.Compute/disks/acme-app-01-osdisk"
    },
    "diskSizeGB": 128,
    "name": "forensic-acme-app-01-osdisk-20260416T102514Z",
    "provisioningState": "Succeeded",
    "timeCreated": "2026-04-16T10:25:14.842Z"
}

Generate a read-only SAS URL for the forensic workstation to mount the snapshot without modification.

az snapshot grant-access \
  --resource-group rg-acme-prod \
  --name "forensic-acme-app-01-osdisk-$SNAP_TIME" \
  --duration-in-seconds 86400 \
  --access-level Read \
  --output json

Expected output:

{
    "accessSas": "https://md-hdd-xyzREDACTED.blob.storage.azure.net/abcdREDACTED/abcd?sv=2018-03-28&sr=b&si=..."
}

3.3 Azure AD Audit Logs¶

Azure AD sign-in and audit logs capture the tenant-scoped activity that Activity Log does not (like app registration changes, consent grants, role assignments). Use Microsoft Graph.

az rest \
  --method GET \
  --url "https://graph.microsoft.com/v1.0/auditLogs/directoryAudits?\$filter=activityDateTime ge 2026-04-10T00:00:00Z and activityDateTime le 2026-04-16T23:59:59Z&\$top=999" \
  > $IR_HOME/azure/aad-directory-audits.json

az rest \
  --method GET \
  --url "https://graph.microsoft.com/v1.0/auditLogs/signIns?\$filter=createdDateTime ge 2026-04-10T00:00:00Z and createdDateTime le 2026-04-16T23:59:59Z&\$top=999" \
  > $IR_HOME/azure/aad-signins.json

Expected sign-in log entry:

{
    "id": "sign-in-guid-EXAMPLE33333",
    "createdDateTime": "2026-04-15T22:58:44Z",
    "userPrincipalName": "attacker@contoso.example.com",
    "appDisplayName": "Azure Portal",
    "ipAddress": "192.0.2.45",
    "clientAppUsed": "Browser",
    "status": {
        "errorCode": 0,
        "additionalDetails": "MFA requirement satisfied by claim in the token"
    },
    "deviceDetail": {
        "browser": "Chrome 127.0.6533",
        "operatingSystem": "Linux"
    },
    "location": {
        "city": "UNKNOWN",
        "countryOrRegion": "US"
    }
}

3.4 NSG Flow Logs¶

NSG Flow Logs record network metadata at the subnet or NIC level. List flow log configurations.

az network watcher flow-log list \
  --location eastus \
  --output json

Expected output:

[
    {
        "name": "nsg-acme-prod-flowlog",
        "storageId": "/subscriptions/.../storageAccounts/acmenetflow",
        "enabled": true,
        "retentionPolicy": {
            "days": 90,
            "enabled": true
        },
        "targetResourceId": "/subscriptions/.../networkSecurityGroups/nsg-acme-prod"
    }
]

Download the flow log blobs for the incident window.

az storage blob download-batch \
  --source insights-logs-networksecuritygroupflowevent \
  --destination $IR_HOME/azure/nsg-flow-logs \
  --account-name acmenetflow \
  --pattern "*y=2026/m=04/d=1[0-6]*"

3.5 Storage Analytics¶

Azure Storage logs are in $logs containers within each account. Enumerate storage accounts and pull logs.

az storage account list \
  --query "[].{name:name, rg:resourceGroup}" \
  --output json > $IR_HOME/azure/storage-accounts.json

az storage blob download-batch \
  --source '$logs' \
  --destination $IR_HOME/azure/storage-analytics/acmenetflow \
  --account-name acmenetflow \
  --pattern "blob/2026/04/1*"

Hash Azure artifacts.

cd $IR_HOME/azure
find . -type f -exec sha256sum {} \; | sort > $IR_HOME/hashes/azure-artifacts.sha256
wc -l $IR_HOME/hashes/azure-artifacts.sha256

Phase 4: GCP Evidence Collection¶

4.1 Cloud Audit Logs¶

GCP Cloud Audit Logs are split into Admin Activity, Data Access, System Event, and Policy Denied categories. Pull all four for the incident window.

gcloud auth activate-service-account \
  ir-forensic-reader@project-acme.iam.gserviceaccount.com \
  --key-file=/etc/forensic/ir-reader.json

gcloud config set project project-acme

for LOG in activity data_access system_event policy; do
  gcloud logging read \
    "timestamp >= \"2026-04-10T00:00:00Z\" AND timestamp <= \"2026-04-16T23:59:59Z\" AND logName:\"$LOG\"" \
    --limit 10000 \
    --format json \
    > $IR_HOME/gcp/audit-$LOG.json
done

Expected output (truncated Admin Activity entry):

{
    "protoPayload": {
        "authenticationInfo": {
            "principalEmail": "exfil-sa@project-acme.iam.gserviceaccount.com"
        },
        "methodName": "storage.objects.get",
        "resourceName": "projects/_/buckets/acme-customer-exports/objects/customers-2026-q1.csv",
        "requestMetadata": {
            "callerIp": "192.0.2.45",
            "callerSuppliedUserAgent": "google-cloud-sdk/469.0.0"
        }
    },
    "timestamp": "2026-04-16T01:13:08.441Z",
    "severity": "INFO"
}

Export the full audit log stream to a regional Cloud Storage bucket for long-term preservation.

gcloud logging sinks create ir-case-$IR_CASE \
  storage.googleapis.com/acme-ir-evidence \
  --log-filter="timestamp >= \"2026-04-10T00:00:00Z\" AND timestamp <= \"2026-04-16T23:59:59Z\"" \
  --include-children \
  --project=project-acme

4.2 Compute Engine Disk Snapshots¶

The GCE VM gce-analytics-01 in zone us-central1-a needs its persistent disks snapshotted.

Enumerate attached disks.

gcloud compute instances describe gce-analytics-01 \
  --zone us-central1-a \
  --format="json(disks[].source)"

Expected output:

{
    "disks": [
        {
            "source": "https://www.googleapis.com/compute/v1/projects/project-acme/zones/us-central1-a/disks/gce-analytics-01-boot"
        },
        {
            "source": "https://www.googleapis.com/compute/v1/projects/project-acme/zones/us-central1-a/disks/gce-analytics-01-data"
        }
    ]
}

Create snapshots. GCP snapshots are global resources by default and support customer-managed encryption keys (CMEK).

SNAP_TIME=$(date -u +%Y%m%d-%H%M%S)

for DISK in gce-analytics-01-boot gce-analytics-01-data; do
  gcloud compute snapshots create forensic-$DISK-$SNAP_TIME \
    --source-disk $DISK \
    --source-disk-zone us-central1-a \
    --labels=incident-id=ir-2026-0418-a,collector=testuser,legal-hold=true \
    --description "$IR_CASE forensic snapshot" \
    --storage-location us
done

Expected output per snapshot:

Creating snapshot forensic-gce-analytics-01-boot-20260416-103211...done.
NAME                                         DISK_SIZE_GB  SRC_DISK                                            STATUS
forensic-gce-analytics-01-boot-20260416-103211  100         us-central1-a/disks/gce-analytics-01-boot  READY

4.3 IAM Policy History¶

GCP IAM policies do not retain historical state by default -- you must either reconstruct from audit logs or use Asset Inventory. Export the current policy.

gcloud projects get-iam-policy project-acme \
  --format json \
  > $IR_HOME/gcp/iam-policy-current.json

Extract policy changes from audit logs.

gcloud logging read \
  "timestamp >= \"2026-04-10T00:00:00Z\" AND protoPayload.methodName:SetIamPolicy" \
  --limit 500 \
  --format json \
  > $IR_HOME/gcp/iam-policy-changes.json

Use Cloud Asset Inventory to see historical IAM policy state.

gcloud asset get-history \
  --asset-names //cloudresourcemanager.googleapis.com/projects/project-acme \
  --content-type iam-policy \
  --start-time 2026-04-10T00:00:00Z \
  --end-time 2026-04-16T23:59:59Z \
  --format json \
  > $IR_HOME/gcp/iam-policy-history.json

Expected output:

{
    "assets": [
        {
            "asset": {
                "name": "//cloudresourcemanager.googleapis.com/projects/project-acme",
                "iamPolicy": {
                    "bindings": [
                        {
                            "role": "roles/storage.admin",
                            "members": [
                                "serviceAccount:exfil-sa@project-acme.iam.gserviceaccount.com"
                            ]
                        }
                    ]
                }
            },
            "window": {
                "startTime": "2026-04-15T22:51:17Z",
                "endTime": "2026-04-16T02:14:33Z"
            }
        }
    ]
}

4.4 VPC Flow Logs¶

Enable VPC Flow Logs if they are not already enabled -- in an active incident, check first before changing any infrastructure.

gcloud compute networks subnets describe acme-prod-subnet \
  --region us-central1 \
  --format="value(enableFlowLogs)"

Expected output:

True

Query flow logs from Cloud Logging.

gcloud logging read \
  'resource.type="gce_subnetwork" AND logName:"compute.googleapis.com%2Fvpc_flows" AND (jsonPayload.connection.src_ip="192.0.2.45" OR jsonPayload.connection.dest_ip="192.0.2.45")' \
  --limit 5000 \
  --format json \
  > $IR_HOME/gcp/vpc-flow-attacker-ip.json

Expected output (truncated):

{
    "jsonPayload": {
        "connection": {
            "src_ip": "192.0.2.45",
            "dest_ip": "10.30.5.22",
            "src_port": 54231,
            "dest_port": 443,
            "protocol": 6
        },
        "bytes_sent": 14523,
        "reporter": "DEST",
        "src_location": {
            "continent": "NorthAmerica",
            "country": "us"
        }
    },
    "timestamp": "2026-04-15T23:48:02.193Z"
}

4.5 Security Command Center Findings¶

SCC aggregates findings across providers such as Event Threat Detection, Container Threat Detection, and Web Security Scanner.

gcloud scc findings list project-acme \
  --filter="event_time >= \"2026-04-10T00:00:00Z\"" \
  --format json \
  > $IR_HOME/gcp/scc-findings.json

Expected output:

[
    {
        "name": "organizations/123456789012/sources/999/findings/finding-EXAMPLE44444",
        "category": "Persistence: IAM Anomalous Grant",
        "resourceName": "//iam.googleapis.com/projects/project-acme/serviceAccounts/exfil-sa@project-acme.iam.gserviceaccount.com",
        "eventTime": "2026-04-15T22:51:47Z",
        "severity": "HIGH",
        "state": "ACTIVE"
    }
]

Hash GCP artifacts.

cd $IR_HOME/gcp
find . -type f -exec sha256sum {} \; | sort > $IR_HOME/hashes/gcp-artifacts.sha256
wc -l $IR_HOME/hashes/gcp-artifacts.sha256

Phase 5: Evidence Preservation¶

5.1 Cross-Cloud Evidence Locker¶

Designate a single immutable storage location as the legal repository for all evidence from this incident. AWS S3 with Object Lock in compliance mode is the standard choice because compliance mode cannot be disabled or reduced even by root users.

Create the locker bucket.

aws s3api create-bucket \
  --bucket acme-ir-locker-2026 \
  --object-lock-enabled-for-bucket \
  --region us-east-1 \
  --profile ir-forensic-account

aws s3api put-object-lock-configuration \
  --bucket acme-ir-locker-2026 \
  --object-lock-configuration '{
      "ObjectLockEnabled": "Enabled",
      "Rule": {
        "DefaultRetention": {
          "Mode": "COMPLIANCE",
          "Years": 7
        }
      }
    }' \
  --profile ir-forensic-account

Enable versioning (required for Object Lock) and block all public access.

aws s3api put-bucket-versioning \
  --bucket acme-ir-locker-2026 \
  --versioning-configuration Status=Enabled \
  --profile ir-forensic-account

aws s3api put-public-access-block \
  --bucket acme-ir-locker-2026 \
  --public-access-block-configuration \
    "BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true" \
  --profile ir-forensic-account

5.2 Hash Verification Before Upload¶

Compute a master manifest of every artifact under the case directory.

cd $IR_HOME
find . -type f -not -path "./hashes/*" -exec sha256sum {} \; | sort > hashes/master-manifest.sha256
wc -l hashes/master-manifest.sha256
sha256sum hashes/master-manifest.sha256 > hashes/master-manifest.sha256.sig
cat hashes/master-manifest.sha256.sig

Expected output:

6294 /root/ir-2026-0418-A/hashes/master-manifest.sha256
b1e4f7d8c0a2...EXAMPLE  hashes/master-manifest.sha256

5.3 Upload with Legal Hold¶

Sync every file to the locker with explicit legal hold. The retention date is set seven years in the future.

aws s3 sync $IR_HOME/ s3://acme-ir-locker-2026/$IR_CASE/ \
  --storage-class STANDARD \
  --metadata "case=$IR_CASE,collector=$EVIDENCE_COLLECTOR" \
  --profile ir-forensic-account

aws s3api put-object-legal-hold \
  --bucket acme-ir-locker-2026 \
  --key "$IR_CASE/hashes/master-manifest.sha256" \
  --legal-hold Status=ON \
  --profile ir-forensic-account

5.4 Chain of Custody Documentation¶

Generate both CSV and JSON chain-of-custody records. The CSV is easy for attorneys to read; the JSON is easy for downstream tooling to parse.

# $IR_HOME/chain-of-custody/generate_coc.py
import csv
import hashlib
import json
import os
import socket
from datetime import datetime, timezone
from pathlib import Path

CASE_ID = os.environ["IR_CASE"]
IR_HOME = Path(os.environ["IR_HOME"])
COLLECTOR = os.environ["EVIDENCE_COLLECTOR"]
HOSTNAME = socket.gethostname()

rows = []
for path in IR_HOME.rglob("*"):
    if not path.is_file():
        continue
    if "chain-of-custody" in path.parts:
        continue
    stat = path.stat()
    sha256 = hashlib.sha256(path.read_bytes()).hexdigest()
    rows.append({
        "case_id": CASE_ID,
        "artifact_path": str(path.relative_to(IR_HOME)),
        "size_bytes": stat.st_size,
        "sha256": sha256,
        "acquired_at_utc": datetime.fromtimestamp(stat.st_mtime, tz=timezone.utc).isoformat(),
        "collector": COLLECTOR,
        "workstation": HOSTNAME,
    })

csv_path = IR_HOME / "chain-of-custody" / f"{CASE_ID}-coc.csv"
with csv_path.open("w", newline="") as f:
    writer = csv.DictWriter(f, fieldnames=rows[0].keys())
    writer.writeheader()
    writer.writerows(rows)

json_path = IR_HOME / "chain-of-custody" / f"{CASE_ID}-coc.json"
json_path.write_text(json.dumps({
    "case_id": CASE_ID,
    "generated_at_utc": datetime.now(timezone.utc).isoformat(),
    "collector": COLLECTOR,
    "workstation": HOSTNAME,
    "artifact_count": len(rows),
    "artifacts": rows,
}, indent=2))

print(f"Wrote {len(rows)} artifacts to {csv_path} and {json_path}")

Run it.

python3 $IR_HOME/chain-of-custody/generate_coc.py

Expected output:

Wrote 6294 artifacts to /root/ir-2026-0418-A/chain-of-custody/IR-2026-0418-A-coc.csv and /root/ir-2026-0418-A/chain-of-custody/IR-2026-0418-A-coc.json

5.5 Evidence Transport¶

If the investigation requires moving evidence to an external forensic vendor, transport the locker bucket via cross-account replication or pre-signed download URLs. Never ship raw credentials or provide console access.

aws s3api put-bucket-replication \
  --bucket acme-ir-locker-2026 \
  --replication-configuration file://replication.json \
  --profile ir-forensic-account

Sample replication.json:

{
    "Role": "arn:aws:iam::444455556666:role/ir-replication-role",
    "Rules": [
        {
            "ID": "ReplicateToVendor",
            "Status": "Enabled",
            "Priority": 1,
            "Filter": {"Prefix": "IR-2026-0418-A/"},
            "Destination": {
                "Bucket": "arn:aws:s3:::vendor-forensic-intake",
                "Account": "777788889999",
                "AccessControlTranslation": {"Owner": "Destination"}
            },
            "DeleteMarkerReplication": {"Status": "Disabled"}
        }
    ]
}

Phase 6: Unified Timeline¶

6.1 Evidence Flow Diagram¶

flowchart TB
    subgraph Collection
        A1["AWS Artifacts<br/>CloudTrail, EBS, VPC, IAM, S3"]
        A2["Azure Artifacts<br/>Activity, Disks, AAD, NSG"]
        A3["GCP Artifacts<br/>Audit, PD, IAM, VPC, SCC"]
    end
    subgraph Normalize
        N["normalize.py<br/>Unified schema"]
    end
    subgraph Locker["Immutable Locker (S3 Object Lock 7y)"]
        L["acme-ir-locker-2026"]
    end
    subgraph Timeline
        T["unified-timeline.csv<br/>UTC-sorted, correlated"]
    end
    A1 --> N
    A2 --> N
    A3 --> N
    A1 --> L
    A2 --> L
    A3 --> L
    N --> T

6.2 Unified Schema¶

Every event from every provider must be normalized to the same schema before correlation is possible.

6.3 Normalization Script¶

# $IR_HOME/timeline/normalize.py
import csv
import gzip
import json
import os
from pathlib import Path

IR_HOME = Path(os.environ["IR_HOME"])
OUT_PATH = IR_HOME / "timeline" / "unified-timeline.csv"
FIELDS = [
    "ts_utc", "provider", "account", "actor", "actor_ip",
    "action", "resource", "outcome", "raw_event_id", "raw_path"
]


def norm_aws(evt, raw_path):
    ct = evt if "eventTime" in evt else json.loads(evt.get("CloudTrailEvent", "{}"))
    return {
        "ts_utc": ct.get("eventTime", ""),
        "provider": "aws",
        "account": ct.get("recipientAccountId", ct.get("userIdentity", {}).get("accountId", "")),
        "actor": ct.get("userIdentity", {}).get("arn", ct.get("userIdentity", {}).get("userName", "")),
        "actor_ip": ct.get("sourceIPAddress", ""),
        "action": f"{ct.get('eventSource', '').split('.')[0]}.{ct.get('eventName', '')}",
        "resource": json.dumps(ct.get("resources", []))[:240],
        "outcome": "error" if ct.get("errorCode") else "success",
        "raw_event_id": ct.get("eventID", ""),
        "raw_path": str(raw_path),
    }


def norm_azure(evt, raw_path):
    return {
        "ts_utc": evt.get("eventTimestamp", evt.get("TimeGenerated", "")),
        "provider": "azure",
        "account": evt.get("subscriptionId", ""),
        "actor": evt.get("caller", evt.get("Caller", "")),
        "actor_ip": (evt.get("httpRequest") or {}).get("clientIpAddress", evt.get("CallerIpAddress", "")),
        "action": (evt.get("operationName") or {}).get("value", evt.get("OperationNameValue", "")),
        "resource": evt.get("resourceId", evt.get("ResourceId", "")),
        "outcome": {"Succeeded": "success", "Failed": "error"}.get(
            (evt.get("status") or {}).get("value", evt.get("ActivityStatusValue", "")), "success"),
        "raw_event_id": evt.get("eventDataId", evt.get("CorrelationId", "")),
        "raw_path": str(raw_path),
    }


def norm_gcp(evt, raw_path):
    proto = evt.get("protoPayload", {})
    return {
        "ts_utc": evt.get("timestamp", ""),
        "provider": "gcp",
        "account": (evt.get("resource") or {}).get("labels", {}).get("project_id", "project-acme"),
        "actor": (proto.get("authenticationInfo") or {}).get("principalEmail", ""),
        "actor_ip": (proto.get("requestMetadata") or {}).get("callerIp", ""),
        "action": proto.get("methodName", ""),
        "resource": proto.get("resourceName", ""),
        "outcome": "deny" if evt.get("severity") == "ERROR" else "success",
        "raw_event_id": evt.get("insertId", ""),
        "raw_path": str(raw_path),
    }


def iter_json_or_ndjson(path):
    opener = gzip.open if path.suffix == ".gz" else open
    with opener(path, "rt", encoding="utf-8", errors="replace") as f:
        data = f.read().strip()
    if not data:
        return
    if data.startswith("["):
        try:
            for e in json.loads(data):
                yield e
        except json.JSONDecodeError:
            return
    elif data.startswith("{"):
        try:
            doc = json.loads(data)
            if isinstance(doc, dict) and "Records" in doc:
                for e in doc["Records"]:
                    yield e
            else:
                yield doc
        except json.JSONDecodeError:
            return
    else:
        for line in data.splitlines():
            try:
                yield json.loads(line)
            except json.JSONDecodeError:
                continue


def main():
    rows = []
    for path in (IR_HOME / "aws").rglob("*"):
        if path.is_file() and path.suffix in {".json", ".gz"}:
            for evt in iter_json_or_ndjson(path):
                if not isinstance(evt, dict):
                    continue
                try:
                    rows.append(norm_aws(evt, path))
                except Exception:
                    continue

    for path in (IR_HOME / "azure").rglob("*.json"):
        for evt in iter_json_or_ndjson(path):
            if not isinstance(evt, dict):
                continue
            try:
                rows.append(norm_azure(evt, path))
            except Exception:
                continue

    for path in (IR_HOME / "gcp").rglob("*.json"):
        for evt in iter_json_or_ndjson(path):
            if not isinstance(evt, dict):
                continue
            try:
                rows.append(norm_gcp(evt, path))
            except Exception:
                continue

    rows = [r for r in rows if r.get("ts_utc")]
    rows.sort(key=lambda r: r["ts_utc"])

    OUT_PATH.parent.mkdir(parents=True, exist_ok=True)
    with OUT_PATH.open("w", newline="") as f:
        w = csv.DictWriter(f, fieldnames=FIELDS)
        w.writeheader()
        w.writerows(rows)

    print(f"Wrote {len(rows)} normalized events to {OUT_PATH}")


if __name__ == "__main__":
    main()

Run it.

python3 $IR_HOME/timeline/normalize.py

Expected output:

Wrote 187421 normalized events to /root/ir-2026-0418-A/timeline/unified-timeline.csv

6.4 Timeline Reconstruction¶

Filter the unified timeline for the attacker IP across all providers.

head -1 $IR_HOME/timeline/unified-timeline.csv > $IR_HOME/timeline/attacker-timeline.csv
grep "192.0.2.45" $IR_HOME/timeline/unified-timeline.csv | sort -t',' -k1 >> $IR_HOME/timeline/attacker-timeline.csv
wc -l $IR_HOME/timeline/attacker-timeline.csv

Expected output:

1843 /root/ir-2026-0418-A/timeline/attacker-timeline.csv

6.5 Cross-Provider Correlation¶

Correlate high-value events into an incident narrative.

# $IR_HOME/timeline/correlate.py
import csv
import os
from collections import defaultdict
from pathlib import Path

IR_HOME = Path(os.environ["IR_HOME"])
SRC = IR_HOME / "timeline" / "attacker-timeline.csv"
OUT = IR_HOME / "timeline" / "narrative.md"

STAGES = [
    ("initial_access", lambda r: r["provider"] == "aws" and "ConsoleLogin" in r["action"]),
    ("enumeration",    lambda r: r["provider"] == "aws" and r["action"].startswith("s3.List")),
    ("privesc",        lambda r: r["provider"] == "aws" and r["action"].startswith("iam.") and "PassRole" in r["action"]),
    ("lateral_azure",  lambda r: r["provider"] == "azure" and "Microsoft.AAD/Applications/write" in r["action"]),
    ("pivot_gcp",      lambda r: r["provider"] == "gcp" and "iam.serviceAccounts.generateAccessToken" in r["action"]),
    ("exfil",          lambda r: r["provider"] == "gcp" and r["action"] == "storage.objects.get"),
    ("destruction",    lambda r: r["action"].endswith("DeleteTrail") or r["action"].endswith("objects.delete")),
]

buckets = defaultdict(list)
with SRC.open() as f:
    reader = csv.DictReader(f)
    for row in reader:
        for name, predicate in STAGES:
            if predicate(row):
                buckets[name].append(row)

lines = ["# Incident Narrative (attacker IP 192.0.2.45)\n"]
for name, _ in STAGES:
    events = buckets.get(name, [])
    lines.append(f"\n## {name} -- {len(events)} events")
    for e in events[:5]:
        lines.append(f"- {e['ts_utc']} [{e['provider']}] {e['actor']} {e['action']} {e['resource'][:100]}")
    if len(events) > 5:
        lines.append(f"- (+{len(events)-5} more)")

OUT.write_text("\n".join(lines))
print(f"Narrative written to {OUT}")

Run and inspect.

python3 $IR_HOME/timeline/correlate.py
cat $IR_HOME/timeline/narrative.md

Expected output (abbreviated):

# Incident Narrative (attacker IP 192.0.2.45)


## initial_access -- 3 events
- 2026-04-15T22:47:03Z [aws] arn:aws:iam::111122223333:user/deploy-bot aws.ConsoleLogin []
- 2026-04-15T22:49:11Z [aws] arn:aws:iam::111122223333:user/deploy-bot aws.ConsoleLogin []

## enumeration -- 47 events
- 2026-04-15T22:50:02Z [aws] arn:aws:iam::111122223333:user/deploy-bot s3.ListBuckets []

## privesc -- 2 events
- 2026-04-15T22:55:41Z [aws] arn:aws:iam::111122223333:user/deploy-bot iam.PassRole [{"ARN":"arn:aws:iam::111122223333:role/ec2-admin"}]

## lateral_azure -- 1 events
- 2026-04-15T23:02:11Z [azure] attacker@contoso.example.com Microsoft.AAD/Applications/write /subscriptions/.../applications/...

## pivot_gcp -- 1 events
- 2026-04-15T23:33:18Z [gcp] exfil-sa@project-acme.iam.gserviceaccount.com iam.serviceAccounts.generateAccessToken //iam.googleapis.com/...

## exfil -- 2841 events
- 2026-04-16T01:13:08Z [gcp] exfil-sa@project-acme.iam.gserviceaccount.com storage.objects.get projects/_/buckets/acme-customer-exports/objects/customers-2026-q1.csv
- (+2836 more)

## destruction -- 1 events
- 2026-04-16T02:41:55Z [aws] arn:aws:iam::111122223333:user/deploy-bot aws.DeleteTrail [{"ARN":"arn:aws:cloudtrail:us-east-1:111122223333:trail/acme-trail"}]

6.6 Timeline Visualization¶

sequenceDiagram
    participant ATK as Attacker 192.0.2.45
    participant AWS as AWS 111122223333
    participant AAD as Azure AD contoso.example.com
    participant GCP as GCP project-acme

    ATK->>AWS: 22:47 ConsoleLogin deploy-bot
    AWS-->>ATK: 22:50 ListBuckets success
    ATK->>AWS: 22:55 iam:PassRole + RunInstances
    AWS->>AAD: 23:02 Federated auth via EC2
    ATK->>AAD: 23:02 Create backdoor App Registration
    AAD->>GCP: 23:33 OIDC -- generateAccessToken
    ATK->>GCP: 01:13 storage.objects.get (2841 objects)
    ATK->>AWS: 02:41 DeleteTrail (denied by bucket policy)

6.7 Verification¶

Before handing the timeline to investigators, re-verify every hash in the master manifest.

cd $IR_HOME
sha256sum -c hashes/master-manifest.sha256 | grep -v "OK$" | head

If the grep returns empty, every artifact is intact.

Conclusion¶

You have completed a full cross-cloud forensic acquisition spanning AWS, Azure, and GCP. You collected management plane logs, computed and preserved hashes, created immutable storage, and reconstructed a unified attacker timeline covering initial access, privilege escalation, lateral movement, data exfiltration, and destruction attempts.

Deliverables Checklist¶

• [ ] AWS CloudTrail, EBS snapshots, IAM state, VPC flow logs, S3 access logs
• [ ] Azure Activity Log, VM disk snapshots, Azure AD audit logs, NSG flow logs, Storage Analytics
• [ ] GCP Cloud Audit Logs, Compute Engine snapshots, IAM policy history, VPC flow logs, SCC findings
• [ ] SHA-256 manifest for every artifact (aws-artifacts.sha256, azure-artifacts.sha256, gcp-artifacts.sha256, master-manifest.sha256)
• [ ] Immutable S3 locker acme-ir-locker-2026 with 7-year compliance-mode retention
• [ ] Chain of custody (CSV + JSON) with collector identity and workstation
• [ ] Unified timeline CSV normalizing all three cloud event schemas
• [ ] Incident narrative correlating stages across providers

Reflection Questions¶

1. Which single acquisition would most clearly establish attacker intent for a regulator?
2. How would your procedure change if CloudTrail log file validation had detected tampering?
3. What additional controls would have shortened dwell time between initial access and exfiltration?
4. Which artifact class was hardest to acquire at forensic quality, and why?
5. Could the unified schema be extended to include Kubernetes audit logs from Lab 27?

Cross-References¶

• Chapter 57: Cloud Forensics -- Theory and provider-specific artifact reference.
• Chapter 9: Incident Response Lifecycle -- NIST SP 800-61 stages that frame this lab.
• Chapter 27: Digital Forensics -- Foundational forensic principles, chain of custody, and integrity controls.
• Chapter 20: Cloud Attack & Defense -- Attacker techniques covered by the breach scenario.
• Lab 24: Cloud DFIR Evidence Collection -- Introductory cloud DFIR with a single provider.

Further Reading¶

• AWS: "CloudTrail Log File Integrity Validation" reference documentation.
• Microsoft: "Azure forensic acquisition playbook" on learn.microsoft.com.
• Google: "Security Operations Guide for Google Cloud" chapter on audit logging.
• NIST SP 800-86: "Guide to Integrating Forensic Techniques into Incident Response".
• Cloud Security Alliance: "Mapping the Forensic Standard ISOs to Cloud Computing".