Skip to content

Chapter 35: DevSecOps — Security in the Software Delivery Pipeline

Overview

DevSecOps integrates security practices into every phase of the software development lifecycle, shifting security left from a post-deployment afterthought to a continuous, automated discipline. This chapter covers secure CI/CD pipeline design, secrets management, software supply chain security, container hardening, infrastructure as code scanning, and security gates that block vulnerable code before production.

Learning Objectives

By the end of this chapter, you will be able to:

  • Design a security-integrated CI/CD pipeline with automated gates
  • Implement SAST, DAST, SCA, and IaC scanning at appropriate pipeline stages
  • Configure secrets detection to prevent credential leakage in repositories
  • Build container image scanning and hardening workflows
  • Map SLSA framework levels to pipeline maturity
  • Implement software supply chain controls using SBOM generation and signing

Prerequisites

  • Chapter 24 (Supply Chain Attacks)
  • Chapter 29 (Vulnerability Management)
  • Chapter 30 (Application Security)
  • Familiarity with Git, Docker, and a CI/CD system (GitHub Actions, GitLab CI, or Jenkins)

Real-World Impact

The 2020 SolarWinds breach began at the build server. SUNSPOT malware monitored build processes and injected SUNBURST into SolarWinds Orion builds whenever specific source files were compiled. Securing your CI/CD pipeline is not optional — it is a Tier-1 security control.


35.1 The DevSecOps Maturity Model

flowchart LR
    subgraph L1["Level 1 — Reactive"]
        A[Manual pen tests\npost-deployment]
    end
    subgraph L2["Level 2 — Informed"]
        B[SAST in CI\nbasic secrets scan]
    end
    subgraph L3["Level 3 — Proactive"]
        C[SAST + SCA + DAST\nIaC scanning\nsecrets blocking]
    end
    subgraph L4["Level 4 — Optimized"]
        D[Policy-as-code\nSBOM generation\nartifact signing\nSLSA L3+]
    end
    L1 --> L2 --> L3 --> L4
    style L1 fill:#ff7b7222,stroke:#ff7b72
    style L2 fill:#ffa65722,stroke:#ffa657
    style L3 fill:#58a6ff22,stroke:#58a6ff
    style L4 fill:#3fb95022,stroke:#3fb950
Maturity Level Security Activities Tooling Examples
Level 1 Annual pen test, ad-hoc patching Manual review
Level 2 SAST in CI, dependency alerts Semgrep, Dependabot
Level 3 Full pipeline security gates Checkov, Trivy, ZAP
Level 4 Policy-as-code, SBOM, artifact signing Sigstore, SLSA, Grype
Level 5 Real-time runtime protection, AI-assisted Falco, Tetragon, AI SAST

35.2 Secure Pipeline Architecture

flowchart TD
    DEV[Developer\nLocal IDE] -->|git push| PR[Pull Request]
    PR --> SECRETS[Secrets Scan\ngitleaks / trufflehog]
    SECRETS --> SAST[SAST\nSemgrep / CodeQL]
    SAST --> SCA[SCA\nGrype / OWASP DC]
    SCA --> BUILD[Build\nDocker buildx]
    BUILD --> CONTAINER[Container Scan\nTrivy]
    CONTAINER --> IAC[IaC Scan\nCheckov / tfsec]
    IAC --> STAGE[Staging Deploy]
    STAGE --> DAST[DAST\nOWASP ZAP]
    DAST --> SBOM[SBOM Generate\nSyft + sign Cosign]
    SBOM --> PROD[Production Deploy]
    PROD --> RUNTIME[Runtime Monitor\nFalco / Tetragon]

    SECRETS -.->|FAIL: block PR| FAIL1[🚫 Block]
    SAST -.->|FAIL: block PR| FAIL2[🚫 Block]
    SCA -.->|FAIL: CVSS≥9| FAIL3[🚫 Block]
    CONTAINER -.->|FAIL: Critical CVE| FAIL4[🚫 Block]
    IAC -.->|FAIL: HIGH+| FAIL5[🚫 Block]

    style FAIL1 fill:#ff7b7222,stroke:#ff7b72
    style FAIL2 fill:#ff7b7222,stroke:#ff7b72
    style FAIL3 fill:#ff7b7222,stroke:#ff7b72
    style FAIL4 fill:#ff7b7222,stroke:#ff7b72
    style FAIL5 fill:#ff7b7222,stroke:#ff7b72

Pipeline Security Gate Policy

Gate Fail Condition Block PR? Ticket SLA
Secrets scan Any credential detected Yes — immediate 1 hour remediation
SAST Critical/High severity finding Yes 24 hours
SCA CVSS ≥ 9.0 OR CISA KEV match Yes 4 hours
SCA CVSS 7.0–8.9 No — warn 7 days
Container scan OS package Critical CVE Yes 48 hours
IaC scan HIGH severity misconfiguration Yes 48 hours
DAST P1/P2 OWASP finding Yes 24 hours

35.3 Secrets Detection and Management

Preventing Credential Leakage

The most common supply chain vulnerability is accidentally committed secrets. Tools operate at three layers:

Pre-commit (developer workstation):

# Install gitleaks as pre-commit hook
pip install pre-commit
cat >> .pre-commit-config.yaml << 'EOF'
repos:
  - repo: https://github.com/gitleaks/gitleaks
    rev: v8.18.2
    hooks:
      - id: gitleaks
        args: ['--baseline-path', 'gitleaks-baseline.json']
EOF
pre-commit install

CI pipeline (GitHub Actions):

# .github/workflows/secrets-scan.yml
name: Secrets Scan
on: [push, pull_request]
jobs:
  gitleaks:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0  # Full history for historical leak detection
      - uses: gitleaks/gitleaks-action@v2
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          GITLEAKS_LICENSE: ${{ secrets.GITLEAKS_LICENSE }}

TruffleHog for deep historical scanning:

# Scan entire git history for secrets
trufflehog git file://. --only-verified --json | jq '.SourceMetadata.Data.Git'

# Scan GitHub org
trufflehog github --org=myorg --token=$GITHUB_TOKEN --only-verified

Secrets Management — Vault Integration

# Replacing hardcoded secrets with HashiCorp Vault dynamic secrets
import hvac
import os

def get_db_credentials():
    """Fetch short-lived DB credentials from Vault."""
    client = hvac.Client(url=os.environ['VAULT_ADDR'],
                         token=os.environ['VAULT_TOKEN'])
    # Dynamic secrets — expires in 1 hour, auto-rotated
    secret = client.secrets.database.generate_credentials(
        name='webapp-role'
    )
    return secret['data']['username'], secret['data']['password']

# In GitHub Actions: use vault-action
# - uses: hashicorp/vault-action@v2
#   with:
#     url: https://vault.example.com
#     method: jwt
#     role: github-actions
#     secrets: secret/data/prod/db password | DB_PASSWORD

35.4 SAST — Static Application Security Testing

Semgrep Rules

Semgrep runs custom and community rules against source code, finding security bugs before execution:

# Run OWASP Top 10 ruleset
semgrep --config=p/owasp-top-ten --json --output=semgrep-results.json .

# Run secrets ruleset
semgrep --config=p/secrets .

# Custom rule example: detect hardcoded IPs
cat > rules/no-hardcoded-ips.yml << 'EOF'
rules:
  - id: hardcoded-ip-address
    patterns:
      - pattern: $X = "..."
      - metavariable-regex:
          metavariable: $X
          regex: '(\d{1,3}\.){3}\d{1,3}'
    message: "Hardcoded IP address detected — use configuration"
    severity: WARNING
    languages: [python, javascript, go, java]
EOF
semgrep --config=rules/no-hardcoded-ips.yml .

CodeQL (GitHub Advanced Security)

# .github/workflows/codeql.yml
name: CodeQL Analysis
on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [main]
  schedule:
    - cron: '0 2 * * 1'  # Weekly full scan

jobs:
  analyze:
    runs-on: ubuntu-latest
    permissions:
      security-events: write
    strategy:
      matrix:
        language: [python, javascript, java]
    steps:
      - uses: actions/checkout@v4
      - uses: github/codeql-action/init@v3
        with:
          languages: ${{ matrix.language }}
          queries: security-extended, security-and-quality
      - uses: github/codeql-action/autobuild@v3
      - uses: github/codeql-action/analyze@v3
        with:
          category: /language:${{ matrix.language }}
          upload: true

35.5 Software Composition Analysis (SCA)

Dependencies are the largest attack surface in modern applications. The Log4Shell (CVE-2021-44228) vulnerability affected virtually every Java application using Log4j.

Grype — Vulnerability Scanning

# Scan installed packages in container
grype alpine:3.19 --output table

# Scan Python project dependencies
grype dir:. --output json | jq '.matches[] | select(.vulnerability.severity=="Critical")'

# Fail CI if any Critical or CISA KEV
grype $IMAGE --fail-on critical --output sarif > grype-results.sarif

# SBOM-based scan (faster for CI)
syft packages dir:. -o spdx-json > sbom.spdx.json
grype sbom:sbom.spdx.json

Dependency Review in GitHub Actions

# .github/workflows/dependency-review.yml
name: Dependency Review
on: [pull_request]
jobs:
  dependency-review:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/dependency-review-action@v4
        with:
          fail-on-severity: high
          deny-licenses: GPL-2.0, AGPL-3.0  # License compliance
          allow-ghsas: GHSA-xxxx-xxxx-xxxx  # Accepted exceptions
          comment-summary-in-pr: always

35.6 Container Security

Dockerfile Hardening

# SECURE Dockerfile example
# Stage 1: Build
FROM golang:1.22-alpine AS builder
WORKDIR /app
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -ldflags="-s -w" -o server .

# Stage 2: Minimal runtime (distroless — no shell, no package manager)
FROM gcr.io/distroless/static:nonroot
# Run as non-root user (uid 65532)
USER nonroot:nonroot
# Copy only the binary
COPY --from=builder --chown=nonroot:nonroot /app/server /server
# Read-only filesystem
VOLUME ["/tmp"]
EXPOSE 8080
ENTRYPOINT ["/server"]

Dockerfile anti-patterns to avoid:

Anti-Pattern Risk Fix
FROM ubuntu:latest Unpredictable base, bloated Pin version: ubuntu:22.04
RUN apt-get install ... without cleanup Large image, more CVEs rm -rf /var/lib/apt/lists/*
USER root (default) Container escape escalation USER 10001:10001
Secrets in ENV/ARG Exposed in image layers Use runtime secrets injection
Secrets in COPY Baked into layer history .dockerignore + BuildKit secrets
COPY . . Includes .git, .env, secrets Explicit COPY + .dockerignore

Trivy Container Scanning

# Scan image for OS + library vulns + misconfigs
trivy image --exit-code 1 --severity CRITICAL,HIGH \
  --ignore-unfixed myapp:latest

# Scan Dockerfile for misconfigurations
trivy config --exit-code 1 Dockerfile

# Generate SARIF for GitHub Security tab
trivy image --format sarif --output trivy-results.sarif myapp:latest

# CI integration
trivy image --format json --output trivy.json myapp:latest
python3 -c "
import json,sys
r=json.load(open('trivy.json'))
crits=sum(v['Severity']=='CRITICAL' for res in r.get('Results',[]) for v in res.get('Vulnerabilities',[]) or [])
print(f'Critical: {crits}'); sys.exit(1 if crits>0 else 0)
"

35.7 Infrastructure as Code (IaC) Security

Checkov — Multi-Framework IaC Scanning

# Scan Terraform
checkov -d ./terraform --framework terraform \
  --compact --output-file-path ./checkov-results \
  --output sarif

# Scan CloudFormation
checkov -f template.yaml --framework cloudformation

# Scan Kubernetes manifests
checkov -d ./k8s --framework kubernetes --check CKV_K8S_*

# Scan all formats with severity filter
checkov -d . --soft-fail-on LOW,MEDIUM \
  --hard-fail-on HIGH,CRITICAL

Vulnerable Terraform (auto-detected by Checkov):

# BAD: multiple misconfigs Checkov will catch
resource "aws_s3_bucket" "data" {
  bucket = "corp-data-bucket"
  # CKV_AWS_18: Access logging disabled
  # CKV_AWS_52: MFA delete disabled
  # CKV2_AWS_6: Public access block missing
}

resource "aws_security_group" "web" {
  ingress {
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]  # CKV_AWS_25: SSH open to world
  }
}

Remediated Terraform:

resource "aws_s3_bucket" "data" {
  bucket = "corp-data-bucket"
}

resource "aws_s3_bucket_public_access_block" "data" {
  bucket                  = aws_s3_bucket.data.id
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

resource "aws_s3_bucket_server_side_encryption_configuration" "data" {
  bucket = aws_s3_bucket.data.id
  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm = "aws:kms"
    }
  }
}


35.8 SBOM Generation and Artifact Signing

Software Bill of Materials (SBOM) provides a complete inventory of software components, enabling rapid response to new vulnerabilities like Log4Shell.

Syft + Cosign Pipeline

# Generate SBOM in CycloneDX format
syft packages . -o cyclonedx-json > sbom.cdx.json

# Generate SBOM for container image
syft packages myapp:1.2.3 -o spdx-json > sbom.spdx.json

# Sign the container image with Cosign (keyless via Sigstore)
cosign sign --yes myregistry.io/myapp:1.2.3

# Attach SBOM to image in registry
cosign attach sbom --sbom sbom.cdx.json myregistry.io/myapp:1.2.3

# Verify signature before deployment
cosign verify myregistry.io/myapp:1.2.3 \
  --certificate-identity=https://github.com/org/repo/.github/workflows/release.yml@refs/heads/main \
  --certificate-oidc-issuer=https://token.actions.githubusercontent.com

# Scan attached SBOM for new vulnerabilities (post-release)
cosign download sbom myregistry.io/myapp:1.2.3 | grype

SLSA Framework Levels

SLSA Level Requirements Threat Mitigated
SLSA 1 Build scripted; provenance generated Accidental mistakes
SLSA 2 Build service; signed provenance Malicious build steps
SLSA 3 Hardened build platform; isolated builds Compromise of build system
SLSA 4 (deprecated, merged) Two-party review; hermetic builds Insider threats
# GitHub Actions: SLSA Level 3 provenance generation
# .github/workflows/release.yml
name: Release with SLSA Provenance
on:
  push:
    tags: ['v*']
jobs:
  build:
    outputs:
      hashes: ${{ steps.hash.outputs.hashes }}
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Build
        run: make build
      - name: Generate hashes
        id: hash
        run: |
          sha256sum ./dist/* | base64 -w0 > hashes.txt
          echo "hashes=$(cat hashes.txt)" >> $GITHUB_OUTPUT
      - uses: actions/upload-artifact@v4
        with:
          name: dist
          path: ./dist

  provenance:
    needs: [build]
    permissions:
      actions: read
      id-token: write
      contents: write
    uses: slsa-framework/slsa-github-generator/.github/workflows/generator_generic_slsa3.yml@v2.0.0
    with:
      base64-subjects: ${{ needs.build.outputs.hashes }}

35.9 Runtime Security

Falco — Container Runtime Detection

# /etc/falco/rules.d/custom-rules.yaml
- rule: Shell Spawned in Container
  desc: Detect shell execution inside a running container (not init)
  condition: >
    spawned_process and container and
    proc.name in (bash, sh, zsh, dash) and
    not proc.pname in (bash, sh, zsh, dash) and
    container.image.repository != "debug-image"
  output: >
    Shell spawned in container (user=%user.name container=%container.name
    image=%container.image.repository cmd=%proc.cmdline pid=%proc.pid)
  priority: WARNING
  tags: [container, shell, T1059]

- rule: Write Below Root
  desc: Attempt to write to filesystem root — suspicious in read-only containers
  condition: >
    open_write and container and fd.directory in (/, /etc, /bin, /sbin, /usr/bin)
  output: >
    Write to sensitive directory (file=%fd.name container=%container.name
    user=%user.name image=%container.image.repository)
  priority: ERROR
  tags: [container, filesystem, T1036]

- rule: Outbound Connection to Non-Approved Destination
  desc: Container making unexpected outbound connections
  condition: >
    outbound and container and
    not fd.sip in (approved_ips) and
    not proc.name in (healthcheck, curl)
  output: >
    Unexpected outbound connection (container=%container.name
    ip=%fd.sip port=%fd.sport image=%container.image.repository)
  priority: NOTICE

35.10 Complete Secure Pipeline Example

# .github/workflows/secure-pipeline.yml
name: Secure CI/CD Pipeline
on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

permissions:
  contents: read
  security-events: write
  id-token: write

env:
  IMAGE_NAME: ghcr.io/${{ github.repository }}
  IMAGE_TAG: ${{ github.sha }}

jobs:
  secrets-scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with: {fetch-depth: 0}
      - uses: gitleaks/gitleaks-action@v2
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

  sast:
    needs: secrets-scan
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: github/codeql-action/init@v3
        with: {languages: python, queries: security-extended}
      - uses: github/codeql-action/autobuild@v3
      - uses: github/codeql-action/analyze@v3

  sca:
    needs: secrets-scan
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/dependency-review-action@v4
        with: {fail-on-severity: high}

  build-and-scan:
    needs: [sast, sca]
    runs-on: ubuntu-latest
    outputs:
      image-digest: ${{ steps.build.outputs.digest }}
    steps:
      - uses: actions/checkout@v4
      - uses: docker/setup-buildx-action@v3
      - uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      - name: Build image
        id: build
        uses: docker/build-push-action@v5
        with:
          push: ${{ github.ref == 'refs/heads/main' }}
          tags: ${{ env.IMAGE_NAME }}:${{ env.IMAGE_TAG }}
          cache-from: type=gha
          cache-to: type=gha,mode=max
          provenance: true
          sbom: true
      - name: Scan with Trivy
        uses: aquasecurity/trivy-action@master
        with:
          image-ref: ${{ env.IMAGE_NAME }}:${{ env.IMAGE_TAG }}
          exit-code: 1
          severity: CRITICAL
          format: sarif
          output: trivy-results.sarif
      - uses: github/codeql-action/upload-sarif@v3
        with:
          sarif_file: trivy-results.sarif

  iac-scan:
    needs: secrets-scan
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: bridgecrewio/checkov-action@v12
        with:
          directory: .
          framework: terraform,kubernetes,dockerfile
          soft_fail: false
          output_format: sarif
          output_file_path: checkov-results.sarif
      - uses: github/codeql-action/upload-sarif@v3
        with:
          sarif_file: checkov-results.sarif

  sign-and-attest:
    needs: build-and-scan
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    steps:
      - name: Sign image with Cosign (keyless)
        uses: sigstore/cosign-installer@v3
      - run: |
          cosign sign --yes \
            ${{ env.IMAGE_NAME }}@${{ needs.build-and-scan.outputs.image-digest }}

Exam Prep & Certifications

Relevant Certifications

The topics in this chapter align with the following certifications:

  • CSSLP — Domains: Secure Software Lifecycle, Secure Build, Security Testing
  • AWS DevOps Professional — Domains: CI/CD Pipelines, Infrastructure as Code, Security Automation

View full Certifications Roadmap →

Nexus SecOps Benchmark Controls

Control ID Description Validation
Nexus SecOps-DS-01 Secrets scanning gate blocks all PRs with detected credentials Verify gitleaks/trufflehog in CI; test with canary credential
Nexus SecOps-DS-02 SAST tool runs on every PR with Critical/High blocking CodeQL or Semgrep results visible in PR checks
Nexus SecOps-DS-03 SCA scans all dependencies; CVSS ≥ 9.0 blocks deployment Grype/Dependabot configured; policy enforced
Nexus SecOps-DS-04 Container images scanned before push; Critical CVEs block Trivy in pipeline; no Critical unpatched images in registry
Nexus SecOps-DS-05 IaC scanned with Checkov/tfsec before apply HIGH+ Checkov findings block PR merge
Nexus SecOps-DS-06 All production artifacts signed and SBOM attached Cosign verify passes; SBOM queryable per artifact

Key Terms

Artifact Signing — Cryptographically signing build outputs to verify authenticity and integrity; Sigstore/Cosign provides keyless signing using OIDC identity.

Build Provenance — Metadata describing how, when, and from what source an artifact was built; enables supply chain integrity verification.

DAST — Dynamic Application Security Testing; testing a running application by simulating attacks.

Distroless — Minimal container base images containing only the application runtime, no shell or package manager, minimizing attack surface.

Hermetic Build — A build process with no external dependencies at build time; all inputs are pre-fetched and pinned, ensuring reproducibility.

Policy-as-Code — Expressing security and compliance policies as code (Rego/OPA, Checkov rules) enabling automated enforcement in CI/CD.

SARIF — Static Analysis Results Interchange Format; standardized JSON format for reporting security findings, supported natively by GitHub.

Shift Left — Moving security testing earlier in the development lifecycle to reduce the cost and time to remediate vulnerabilities.

Supply Chain Security — Controls ensuring the integrity of software components, build processes, and distribution channels.