Chapter 35: DevSecOps — Security in the Software Delivery Pipeline¶
Overview¶
DevSecOps integrates security practices into every phase of the software development lifecycle, shifting security left from a post-deployment afterthought to a continuous, automated discipline. This chapter covers secure CI/CD pipeline design, secrets management, software supply chain security, container hardening, infrastructure as code scanning, and security gates that block vulnerable code before production.
Learning Objectives¶
By the end of this chapter, you will be able to:
- Design a security-integrated CI/CD pipeline with automated gates
- Implement SAST, DAST, SCA, and IaC scanning at appropriate pipeline stages
- Configure secrets detection to prevent credential leakage in repositories
- Build container image scanning and hardening workflows
- Map SLSA framework levels to pipeline maturity
- Implement software supply chain controls using SBOM generation and signing
Prerequisites¶
- Chapter 24 (Supply Chain Attacks)
- Chapter 29 (Vulnerability Management)
- Chapter 30 (Application Security)
- Familiarity with Git, Docker, and a CI/CD system (GitHub Actions, GitLab CI, or Jenkins)
Real-World Impact
The 2020 SolarWinds breach began at the build server. SUNSPOT malware monitored build processes and injected SUNBURST into SolarWinds Orion builds whenever specific source files were compiled. Securing your CI/CD pipeline is not optional — it is a Tier-1 security control.
35.1 The DevSecOps Maturity Model¶
flowchart LR
subgraph L1["Level 1 — Reactive"]
A[Manual pen tests\npost-deployment]
end
subgraph L2["Level 2 — Informed"]
B[SAST in CI\nbasic secrets scan]
end
subgraph L3["Level 3 — Proactive"]
C[SAST + SCA + DAST\nIaC scanning\nsecrets blocking]
end
subgraph L4["Level 4 — Optimized"]
D[Policy-as-code\nSBOM generation\nartifact signing\nSLSA L3+]
end
L1 --> L2 --> L3 --> L4
style L1 fill:#ff7b7222,stroke:#ff7b72
style L2 fill:#ffa65722,stroke:#ffa657
style L3 fill:#58a6ff22,stroke:#58a6ff
style L4 fill:#3fb95022,stroke:#3fb950 | Maturity Level | Security Activities | Tooling Examples |
|---|---|---|
| Level 1 | Annual pen test, ad-hoc patching | Manual review |
| Level 2 | SAST in CI, dependency alerts | Semgrep, Dependabot |
| Level 3 | Full pipeline security gates | Checkov, Trivy, ZAP |
| Level 4 | Policy-as-code, SBOM, artifact signing | Sigstore, SLSA, Grype |
| Level 5 | Real-time runtime protection, AI-assisted | Falco, Tetragon, AI SAST |
35.2 Secure Pipeline Architecture¶
flowchart TD
DEV[Developer\nLocal IDE] -->|git push| PR[Pull Request]
PR --> SECRETS[Secrets Scan\ngitleaks / trufflehog]
SECRETS --> SAST[SAST\nSemgrep / CodeQL]
SAST --> SCA[SCA\nGrype / OWASP DC]
SCA --> BUILD[Build\nDocker buildx]
BUILD --> CONTAINER[Container Scan\nTrivy]
CONTAINER --> IAC[IaC Scan\nCheckov / tfsec]
IAC --> STAGE[Staging Deploy]
STAGE --> DAST[DAST\nOWASP ZAP]
DAST --> SBOM[SBOM Generate\nSyft + sign Cosign]
SBOM --> PROD[Production Deploy]
PROD --> RUNTIME[Runtime Monitor\nFalco / Tetragon]
SECRETS -.->|FAIL: block PR| FAIL1[🚫 Block]
SAST -.->|FAIL: block PR| FAIL2[🚫 Block]
SCA -.->|FAIL: CVSS≥9| FAIL3[🚫 Block]
CONTAINER -.->|FAIL: Critical CVE| FAIL4[🚫 Block]
IAC -.->|FAIL: HIGH+| FAIL5[🚫 Block]
style FAIL1 fill:#ff7b7222,stroke:#ff7b72
style FAIL2 fill:#ff7b7222,stroke:#ff7b72
style FAIL3 fill:#ff7b7222,stroke:#ff7b72
style FAIL4 fill:#ff7b7222,stroke:#ff7b72
style FAIL5 fill:#ff7b7222,stroke:#ff7b72 Pipeline Security Gate Policy¶
| Gate | Fail Condition | Block PR? | Ticket SLA |
|---|---|---|---|
| Secrets scan | Any credential detected | Yes — immediate | 1 hour remediation |
| SAST | Critical/High severity finding | Yes | 24 hours |
| SCA | CVSS ≥ 9.0 OR CISA KEV match | Yes | 4 hours |
| SCA | CVSS 7.0–8.9 | No — warn | 7 days |
| Container scan | OS package Critical CVE | Yes | 48 hours |
| IaC scan | HIGH severity misconfiguration | Yes | 48 hours |
| DAST | P1/P2 OWASP finding | Yes | 24 hours |
35.3 Secrets Detection and Management¶
Preventing Credential Leakage¶
The most common supply chain vulnerability is accidentally committed secrets. Tools operate at three layers:
Pre-commit (developer workstation):
# Install gitleaks as pre-commit hook
pip install pre-commit
cat >> .pre-commit-config.yaml << 'EOF'
repos:
- repo: https://github.com/gitleaks/gitleaks
rev: v8.18.2
hooks:
- id: gitleaks
args: ['--baseline-path', 'gitleaks-baseline.json']
EOF
pre-commit install
CI pipeline (GitHub Actions):
# .github/workflows/secrets-scan.yml
name: Secrets Scan
on: [push, pull_request]
jobs:
gitleaks:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0 # Full history for historical leak detection
- uses: gitleaks/gitleaks-action@v2
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
GITLEAKS_LICENSE: ${{ secrets.GITLEAKS_LICENSE }}
TruffleHog for deep historical scanning:
# Scan entire git history for secrets
trufflehog git file://. --only-verified --json | jq '.SourceMetadata.Data.Git'
# Scan GitHub org
trufflehog github --org=myorg --token=$GITHUB_TOKEN --only-verified
Secrets Management — Vault Integration¶
# Replacing hardcoded secrets with HashiCorp Vault dynamic secrets
import hvac
import os
def get_db_credentials():
"""Fetch short-lived DB credentials from Vault."""
client = hvac.Client(url=os.environ['VAULT_ADDR'],
token=os.environ['VAULT_TOKEN'])
# Dynamic secrets — expires in 1 hour, auto-rotated
secret = client.secrets.database.generate_credentials(
name='webapp-role'
)
return secret['data']['username'], secret['data']['password']
# In GitHub Actions: use vault-action
# - uses: hashicorp/vault-action@v2
# with:
# url: https://vault.example.com
# method: jwt
# role: github-actions
# secrets: secret/data/prod/db password | DB_PASSWORD
35.4 SAST — Static Application Security Testing¶
Semgrep Rules¶
Semgrep runs custom and community rules against source code, finding security bugs before execution:
# Run OWASP Top 10 ruleset
semgrep --config=p/owasp-top-ten --json --output=semgrep-results.json .
# Run secrets ruleset
semgrep --config=p/secrets .
# Custom rule example: detect hardcoded IPs
cat > rules/no-hardcoded-ips.yml << 'EOF'
rules:
- id: hardcoded-ip-address
patterns:
- pattern: $X = "..."
- metavariable-regex:
metavariable: $X
regex: '(\d{1,3}\.){3}\d{1,3}'
message: "Hardcoded IP address detected — use configuration"
severity: WARNING
languages: [python, javascript, go, java]
EOF
semgrep --config=rules/no-hardcoded-ips.yml .
CodeQL (GitHub Advanced Security)¶
# .github/workflows/codeql.yml
name: CodeQL Analysis
on:
push:
branches: [main, develop]
pull_request:
branches: [main]
schedule:
- cron: '0 2 * * 1' # Weekly full scan
jobs:
analyze:
runs-on: ubuntu-latest
permissions:
security-events: write
strategy:
matrix:
language: [python, javascript, java]
steps:
- uses: actions/checkout@v4
- uses: github/codeql-action/init@v3
with:
languages: ${{ matrix.language }}
queries: security-extended, security-and-quality
- uses: github/codeql-action/autobuild@v3
- uses: github/codeql-action/analyze@v3
with:
category: /language:${{ matrix.language }}
upload: true
35.5 Software Composition Analysis (SCA)¶
Dependencies are the largest attack surface in modern applications. The Log4Shell (CVE-2021-44228) vulnerability affected virtually every Java application using Log4j.
Grype — Vulnerability Scanning¶
# Scan installed packages in container
grype alpine:3.19 --output table
# Scan Python project dependencies
grype dir:. --output json | jq '.matches[] | select(.vulnerability.severity=="Critical")'
# Fail CI if any Critical or CISA KEV
grype $IMAGE --fail-on critical --output sarif > grype-results.sarif
# SBOM-based scan (faster for CI)
syft packages dir:. -o spdx-json > sbom.spdx.json
grype sbom:sbom.spdx.json
Dependency Review in GitHub Actions¶
# .github/workflows/dependency-review.yml
name: Dependency Review
on: [pull_request]
jobs:
dependency-review:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/dependency-review-action@v4
with:
fail-on-severity: high
deny-licenses: GPL-2.0, AGPL-3.0 # License compliance
allow-ghsas: GHSA-xxxx-xxxx-xxxx # Accepted exceptions
comment-summary-in-pr: always
35.6 Container Security¶
Dockerfile Hardening¶
# SECURE Dockerfile example
# Stage 1: Build
FROM golang:1.22-alpine AS builder
WORKDIR /app
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -ldflags="-s -w" -o server .
# Stage 2: Minimal runtime (distroless — no shell, no package manager)
FROM gcr.io/distroless/static:nonroot
# Run as non-root user (uid 65532)
USER nonroot:nonroot
# Copy only the binary
COPY --from=builder --chown=nonroot:nonroot /app/server /server
# Read-only filesystem
VOLUME ["/tmp"]
EXPOSE 8080
ENTRYPOINT ["/server"]
Dockerfile anti-patterns to avoid:
| Anti-Pattern | Risk | Fix |
|---|---|---|
FROM ubuntu:latest | Unpredictable base, bloated | Pin version: ubuntu:22.04 |
RUN apt-get install ... without cleanup | Large image, more CVEs | rm -rf /var/lib/apt/lists/* |
USER root (default) | Container escape escalation | USER 10001:10001 |
| Secrets in ENV/ARG | Exposed in image layers | Use runtime secrets injection |
| Secrets in COPY | Baked into layer history | .dockerignore + BuildKit secrets |
COPY . . | Includes .git, .env, secrets | Explicit COPY + .dockerignore |
Trivy Container Scanning¶
# Scan image for OS + library vulns + misconfigs
trivy image --exit-code 1 --severity CRITICAL,HIGH \
--ignore-unfixed myapp:latest
# Scan Dockerfile for misconfigurations
trivy config --exit-code 1 Dockerfile
# Generate SARIF for GitHub Security tab
trivy image --format sarif --output trivy-results.sarif myapp:latest
# CI integration
trivy image --format json --output trivy.json myapp:latest
python3 -c "
import json,sys
r=json.load(open('trivy.json'))
crits=sum(v['Severity']=='CRITICAL' for res in r.get('Results',[]) for v in res.get('Vulnerabilities',[]) or [])
print(f'Critical: {crits}'); sys.exit(1 if crits>0 else 0)
"
35.7 Infrastructure as Code (IaC) Security¶
Checkov — Multi-Framework IaC Scanning¶
# Scan Terraform
checkov -d ./terraform --framework terraform \
--compact --output-file-path ./checkov-results \
--output sarif
# Scan CloudFormation
checkov -f template.yaml --framework cloudformation
# Scan Kubernetes manifests
checkov -d ./k8s --framework kubernetes --check CKV_K8S_*
# Scan all formats with severity filter
checkov -d . --soft-fail-on LOW,MEDIUM \
--hard-fail-on HIGH,CRITICAL
Vulnerable Terraform (auto-detected by Checkov):
# BAD: multiple misconfigs Checkov will catch
resource "aws_s3_bucket" "data" {
bucket = "corp-data-bucket"
# CKV_AWS_18: Access logging disabled
# CKV_AWS_52: MFA delete disabled
# CKV2_AWS_6: Public access block missing
}
resource "aws_security_group" "web" {
ingress {
from_port = 22
to_port = 22
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"] # CKV_AWS_25: SSH open to world
}
}
Remediated Terraform:
resource "aws_s3_bucket" "data" {
bucket = "corp-data-bucket"
}
resource "aws_s3_bucket_public_access_block" "data" {
bucket = aws_s3_bucket.data.id
block_public_acls = true
block_public_policy = true
ignore_public_acls = true
restrict_public_buckets = true
}
resource "aws_s3_bucket_server_side_encryption_configuration" "data" {
bucket = aws_s3_bucket.data.id
rule {
apply_server_side_encryption_by_default {
sse_algorithm = "aws:kms"
}
}
}
35.8 SBOM Generation and Artifact Signing¶
Software Bill of Materials (SBOM) provides a complete inventory of software components, enabling rapid response to new vulnerabilities like Log4Shell.
Syft + Cosign Pipeline¶
# Generate SBOM in CycloneDX format
syft packages . -o cyclonedx-json > sbom.cdx.json
# Generate SBOM for container image
syft packages myapp:1.2.3 -o spdx-json > sbom.spdx.json
# Sign the container image with Cosign (keyless via Sigstore)
cosign sign --yes myregistry.io/myapp:1.2.3
# Attach SBOM to image in registry
cosign attach sbom --sbom sbom.cdx.json myregistry.io/myapp:1.2.3
# Verify signature before deployment
cosign verify myregistry.io/myapp:1.2.3 \
--certificate-identity=https://github.com/org/repo/.github/workflows/release.yml@refs/heads/main \
--certificate-oidc-issuer=https://token.actions.githubusercontent.com
# Scan attached SBOM for new vulnerabilities (post-release)
cosign download sbom myregistry.io/myapp:1.2.3 | grype
SLSA Framework Levels¶
| SLSA Level | Requirements | Threat Mitigated |
|---|---|---|
| SLSA 1 | Build scripted; provenance generated | Accidental mistakes |
| SLSA 2 | Build service; signed provenance | Malicious build steps |
| SLSA 3 | Hardened build platform; isolated builds | Compromise of build system |
| SLSA 4 (deprecated, merged) | Two-party review; hermetic builds | Insider threats |
# GitHub Actions: SLSA Level 3 provenance generation
# .github/workflows/release.yml
name: Release with SLSA Provenance
on:
push:
tags: ['v*']
jobs:
build:
outputs:
hashes: ${{ steps.hash.outputs.hashes }}
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Build
run: make build
- name: Generate hashes
id: hash
run: |
sha256sum ./dist/* | base64 -w0 > hashes.txt
echo "hashes=$(cat hashes.txt)" >> $GITHUB_OUTPUT
- uses: actions/upload-artifact@v4
with:
name: dist
path: ./dist
provenance:
needs: [build]
permissions:
actions: read
id-token: write
contents: write
uses: slsa-framework/slsa-github-generator/.github/workflows/generator_generic_slsa3.yml@v2.0.0
with:
base64-subjects: ${{ needs.build.outputs.hashes }}
35.9 Runtime Security¶
Falco — Container Runtime Detection¶
# /etc/falco/rules.d/custom-rules.yaml
- rule: Shell Spawned in Container
desc: Detect shell execution inside a running container (not init)
condition: >
spawned_process and container and
proc.name in (bash, sh, zsh, dash) and
not proc.pname in (bash, sh, zsh, dash) and
container.image.repository != "debug-image"
output: >
Shell spawned in container (user=%user.name container=%container.name
image=%container.image.repository cmd=%proc.cmdline pid=%proc.pid)
priority: WARNING
tags: [container, shell, T1059]
- rule: Write Below Root
desc: Attempt to write to filesystem root — suspicious in read-only containers
condition: >
open_write and container and fd.directory in (/, /etc, /bin, /sbin, /usr/bin)
output: >
Write to sensitive directory (file=%fd.name container=%container.name
user=%user.name image=%container.image.repository)
priority: ERROR
tags: [container, filesystem, T1036]
- rule: Outbound Connection to Non-Approved Destination
desc: Container making unexpected outbound connections
condition: >
outbound and container and
not fd.sip in (approved_ips) and
not proc.name in (healthcheck, curl)
output: >
Unexpected outbound connection (container=%container.name
ip=%fd.sip port=%fd.sport image=%container.image.repository)
priority: NOTICE
35.10 Complete Secure Pipeline Example¶
# .github/workflows/secure-pipeline.yml
name: Secure CI/CD Pipeline
on:
push:
branches: [main]
pull_request:
branches: [main]
permissions:
contents: read
security-events: write
id-token: write
env:
IMAGE_NAME: ghcr.io/${{ github.repository }}
IMAGE_TAG: ${{ github.sha }}
jobs:
secrets-scan:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with: {fetch-depth: 0}
- uses: gitleaks/gitleaks-action@v2
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
sast:
needs: secrets-scan
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: github/codeql-action/init@v3
with: {languages: python, queries: security-extended}
- uses: github/codeql-action/autobuild@v3
- uses: github/codeql-action/analyze@v3
sca:
needs: secrets-scan
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/dependency-review-action@v4
with: {fail-on-severity: high}
build-and-scan:
needs: [sast, sca]
runs-on: ubuntu-latest
outputs:
image-digest: ${{ steps.build.outputs.digest }}
steps:
- uses: actions/checkout@v4
- uses: docker/setup-buildx-action@v3
- uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Build image
id: build
uses: docker/build-push-action@v5
with:
push: ${{ github.ref == 'refs/heads/main' }}
tags: ${{ env.IMAGE_NAME }}:${{ env.IMAGE_TAG }}
cache-from: type=gha
cache-to: type=gha,mode=max
provenance: true
sbom: true
- name: Scan with Trivy
uses: aquasecurity/trivy-action@master
with:
image-ref: ${{ env.IMAGE_NAME }}:${{ env.IMAGE_TAG }}
exit-code: 1
severity: CRITICAL
format: sarif
output: trivy-results.sarif
- uses: github/codeql-action/upload-sarif@v3
with:
sarif_file: trivy-results.sarif
iac-scan:
needs: secrets-scan
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: bridgecrewio/checkov-action@v12
with:
directory: .
framework: terraform,kubernetes,dockerfile
soft_fail: false
output_format: sarif
output_file_path: checkov-results.sarif
- uses: github/codeql-action/upload-sarif@v3
with:
sarif_file: checkov-results.sarif
sign-and-attest:
needs: build-and-scan
if: github.ref == 'refs/heads/main'
runs-on: ubuntu-latest
steps:
- name: Sign image with Cosign (keyless)
uses: sigstore/cosign-installer@v3
- run: |
cosign sign --yes \
${{ env.IMAGE_NAME }}@${{ needs.build-and-scan.outputs.image-digest }}
Exam Prep & Certifications¶
Relevant Certifications
The topics in this chapter align with the following certifications:
- CSSLP — Domains: Secure Software Lifecycle, Secure Build, Security Testing
- AWS DevOps Professional — Domains: CI/CD Pipelines, Infrastructure as Code, Security Automation
Nexus SecOps Benchmark Controls¶
| Control ID | Description | Validation |
|---|---|---|
| Nexus SecOps-DS-01 | Secrets scanning gate blocks all PRs with detected credentials | Verify gitleaks/trufflehog in CI; test with canary credential |
| Nexus SecOps-DS-02 | SAST tool runs on every PR with Critical/High blocking | CodeQL or Semgrep results visible in PR checks |
| Nexus SecOps-DS-03 | SCA scans all dependencies; CVSS ≥ 9.0 blocks deployment | Grype/Dependabot configured; policy enforced |
| Nexus SecOps-DS-04 | Container images scanned before push; Critical CVEs block | Trivy in pipeline; no Critical unpatched images in registry |
| Nexus SecOps-DS-05 | IaC scanned with Checkov/tfsec before apply | HIGH+ Checkov findings block PR merge |
| Nexus SecOps-DS-06 | All production artifacts signed and SBOM attached | Cosign verify passes; SBOM queryable per artifact |
Key Terms¶
Artifact Signing — Cryptographically signing build outputs to verify authenticity and integrity; Sigstore/Cosign provides keyless signing using OIDC identity.
Build Provenance — Metadata describing how, when, and from what source an artifact was built; enables supply chain integrity verification.
DAST — Dynamic Application Security Testing; testing a running application by simulating attacks.
Distroless — Minimal container base images containing only the application runtime, no shell or package manager, minimizing attack surface.
Hermetic Build — A build process with no external dependencies at build time; all inputs are pre-fetched and pinned, ensuring reproducibility.
Policy-as-Code — Expressing security and compliance policies as code (Rego/OPA, Checkov rules) enabling automated enforcement in CI/CD.
SARIF — Static Analysis Results Interchange Format; standardized JSON format for reporting security findings, supported natively by GitHub.
Shift Left — Moving security testing earlier in the development lifecycle to reduce the cost and time to remediate vulnerabilities.
Supply Chain Security — Controls ensuring the integrity of software components, build processes, and distribution channels.