Lab 2: Detection Rule Tuning¶

Difficulty: ⭐⭐⭐ Advanced Duration: 75–90 minutes Chapter Reference: Chapter 5 — Detection Engineering at Scale Nexus SecOps Controls: Nexus SecOps-041, Nexus SecOps-042, Nexus SecOps-043, Nexus SecOps-044, Nexus SecOps-045, Nexus SecOps-048

Learning Objectives¶

By completing this lab, you will be able to:

1. Analyze a noisy detection rule and identify its false positive patterns
2. Propose and evaluate three distinct tuning approaches with tradeoffs
3. Apply a structured change control workflow to detection modifications
4. Measure the precision/recall impact of each tuning approach
5. Write a tuning justification document suitable for peer review

Background¶

Detection rules decay over time. Environment changes, new authorized tools, and operational patterns cause previously effective rules to generate false positives. A rule with 80% FP rate consumes analyst time that should go to real threats.

Effective tuning requires: - Evidence-based analysis — FP patterns must be identified from data, not assumptions - Tradeoff awareness — every exclusion risks hiding real attacks - Change control discipline — tuning changes follow the same gates as new rules - Metric tracking — measure before and after, not just "it feels less noisy"

Setup¶

No tooling required. This lab uses synthetic log data provided in the tables below.

Scenario: You are a Detection Engineer at Meridian Financial Services. The SIEM alert queue shows that the rule DET-2024-PS-001 (Suspicious PowerShell Execution) is generating approximately 40 alerts per day, of which your team estimates only 10–15% are genuine concerns.

The CISO has requested that you reduce alert volume by 60% without losing meaningful detection capability. You have 24 hours of alert data to analyze.

The Rule Under Review¶

Current Rule: DET-2024-PS-001¶

# Sigma rule format (simplified)
title: Suspicious PowerShell Execution
id: DET-2024-PS-001
status: production
description: >
  Detects PowerShell execution with encoded commands, download cradles,
  or execution policy bypass flags.
author: SOC Detection Team
date: 2023-08-15
logsource:
  category: process_creation
  product: windows
detection:
  selection:
    Image|endswith: '\powershell.exe'
    CommandLine|contains:
      - '-EncodedCommand'
      - '-enc '
      - '-e '
      - 'DownloadString'
      - 'DownloadFile'
      - 'IEX'
      - 'Invoke-Expression'
      - '-ExecutionPolicy Bypass'
      - '-ep bypass'
      - 'WebClient'
  condition: selection
falsepositives:
  - Administrative scripts
  - Software deployment tools
  - Monitoring agents
level: high
tags:
  - attack.execution
  - attack.t1059.001

Current metrics (last 30 days):

Alert Dataset: 50 Events (24-Hour Sample)¶

Analyze the following 50 alerts triggered by the rule. Each row represents one alert.

Task 1: FP Pattern Analysis¶

Before proposing any tuning, analyze the dataset to identify FP patterns.

1.1 — Categorize Each Alert¶

Group the 50 alerts into categories based on the likely explanation:

1.2 — Calculate Current Metrics¶

Using your categorization:

1.3 — Identify FP Patterns¶

For each FP category, describe the distinguishing characteristics:

Task 2: Evaluate Three Tuning Approaches¶

For each approach, analyze the tradeoffs before recommending one.

Approach A: Allowlist by Service Account¶

Proposed modification:

# Add to rule filter section
filter_service_accounts:
  User|contains:
    - 'svc-monitoring'
    - 'svc-backup'
    - 'svc-deploy'
  ParentImage|endswith:
    - '\taskschd.exe'   # Task Scheduler
    - '\sccm.exe'       # SCCM
    - '\deploy-agent.exe'
condition: selection and not filter_service_accounts

Evaluate this approach:

Approach B: Parent Process Filtering¶

Proposed modification:

# Only alert when parent process is suspicious
filter_known_good_parents:
  ParentImage|endswith:
    - '\taskschd.exe'
    - '\sccm.exe'
    - '\deploy-agent.exe'
    - '\powershell_ise.exe'
condition: selection and not filter_known_good_parents

Evaluate this approach:

Approach C: Contextual Enrichment + Risk Scoring¶

Proposed modification: Rather than exclusions, add risk score fields and only alert when a risk score threshold is met:

# Risk score logic (pseudocode for SIEM correlation rule)
base_score: 30  # All PowerShell with suspicious flags

enrichments:
  - user_type:
      service_account: -20
      it_admin: -10
      standard_user: +20
      finance_hr_user: +30
  - parent_process:
      task_scheduler: -15
      sccm: -15
      deploy_agent: -15
      explorer: +25
      powershell: +20
  - time_context:
      business_hours: 0
      after_hours: +15
  - host_context:
      server: -5
      workstation: +10
      monitoring_host: -20

alert_threshold: 50  # Only fire if score >= 50

Evaluate this approach:

Task 3: Impact Projection¶

For the approach you recommend, calculate the projected impact:

Does your recommended approach meet the 60% volume reduction target? ___

What is the risk of missing a true positive? (describe the specific scenario)

Task 4: Change Control Exercise¶

Your tuning change requires following the Detection Change Control SOP. Complete the documentation.

Rule Change Documentation¶

# DET-2024-PS-001 — Change Record
change_id: CHG-DET-2024-0047
rule_id: DET-2024-PS-001
change_type: tuning  # new | tuning | retire
author: [Your Name]
date: 2026-02-19
reviewed_by: [Peer Reviewer Name]

change_summary: >
  [Describe in 1–2 sentences what change is being made and why]

current_metrics:
  daily_volume: 40
  fp_rate: "85-90%"
  analyst_hours_per_day: "6-8"

expected_metrics:
  daily_volume:
  fp_rate:
  analyst_hours_per_day:

detection_coverage_impact: >
  [Describe any reduction in detection coverage and what threat scenarios
  could now evade detection]

tuning_evidence:
  analysis_period: "2026-02-18 (24 hours)"
  events_analyzed: 50
  fp_patterns_identified:
    -
    -
    -

rollback_plan: >
  [How to revert this change if it causes missed detections]

staging_period_days: 7
post_deployment_review_days: 30

Questions to Answer Before Peer Review¶

1. What attack scenarios does this change NOT protect against?
2. Is there a compensating control if this rule misses something?
3. How will you know if this tuning was correct? (what metrics to track post-deployment)
4. What would trigger a rollback?

Answer Key¶

Scoring¶

Score ≥ 80: Ready to lead detection engineering tuning projects Score 60–79: Review Chapter 5 on detection lifecycle and tuning methodology Score < 60: Revisit detection fundamentals; do Lab 1 first

Lab 2 complete. Proceed to Lab 3: IR Simulation