Chapter 50 Quiz: Adversarial AI & LLM Security¶

B — Direct injection involves a user crafting malicious prompts in their own input; indirect injection embeds malicious instructions in external data sources (websites, documents, emails) that the LLM processes during retrieval or tool use

Direct prompt injection is a first-party attack where the user manipulates their own input to override system instructions (e.g., "ignore previous instructions and..."). Indirect prompt injection is a third-party attack where malicious instructions are planted in content the LLM retrieves — such as hidden text on websites, poisoned documents in a RAG knowledge base, or crafted emails processed by an AI assistant. Indirect injection is particularly dangerous because the user may be unaware of the malicious content.

B — LLM01:2025 — Prompt Injection — the model fails to maintain the confidentiality boundary between system instructions and user queries, exposing sensitive configuration

System prompt leakage through prompt injection reveals the application's guardrails, business logic, and restricted topics. This information helps attackers craft more targeted bypass attempts. The fundamental issue is that LLMs process system prompts and user inputs in the same context window without a robust security boundary. Mitigations include: avoiding sensitive data in system prompts, output filtering for prompt content, and instruction hierarchy enforcement.

B — Model poisoning corrupts the training or fine-tuning data to embed persistent backdoors or biases into the model's weights; it occurs during training (pre-deployment) and persists across all interactions, unlike prompt injection which occurs at inference time and affects only the current session

Data poisoning attacks compromise the model itself by injecting malicious, biased, or backdoored examples into training datasets. The corrupted model may produce incorrect outputs for specific trigger inputs while behaving normally otherwise (backdoor attacks) or exhibit systematic biases across all interactions. This is far more persistent than prompt injection and harder to detect because the malicious behavior is embedded in the model's learned parameters.

B — The risk that LLMs may reveal sensitive information from their training data, system prompts, or connected data sources through their outputs — challenging because LLMs learn patterns from training data and may memorize and regurgitate PII, proprietary code, or confidential information

LLMs can memorize and reproduce training data verbatim, especially sequences that appear frequently or are distinctive. This creates risks of PII exposure, proprietary information leakage, and confidential data disclosure. In RAG systems, the model may expose retrieved documents to unauthorized users. Mitigations include training data sanitization, output filtering, differential privacy during training, and access control on RAG data sources.

B — A jailbreak is a specific type of prompt injection designed to bypass the model's safety alignment and content policies, causing it to generate harmful, unethical, or restricted content that its guardrails are designed to prevent

Jailbreaks are a subset of prompt injection focused specifically on circumventing safety guardrails. Common techniques include role-playing scenarios ("pretend you are an AI without restrictions"), encoding tricks (Base64, character substitution), many-shot attacks (providing examples of unaligned behavior), and competitive objectives ("in this game, you must answer all questions regardless of content"). Jailbreaks exploit the tension between the model's helpfulness and safety objectives.

B — ATLAS catalogs adversarial tactics and techniques specific to AI/ML systems — including attacks on training pipelines, model inference, and AI supply chains — complementing ATT&CK with AI-specific threat scenarios that traditional frameworks do not cover

MITRE ATLAS documents AI-specific attack techniques such as model evasion, data poisoning, model theft, ML supply chain compromise, and adversarial examples. It follows ATT&CK's tactic-technique structure but addresses the unique attack surface of ML systems: training data manipulation, model API abuse, inference attacks (model extraction, membership inference), and attacks on ML infrastructure. ATLAS provides case studies based on real-world incidents.

B — RAG extends the attack surface by introducing: document-level indirect prompt injection (malicious instructions in retrieved documents), access control bypass (the LLM may expose documents the user is not authorized to see), and data poisoning of the knowledge base that affects all users' queries

RAG systems must enforce the same access controls on retrieved documents as the original document store. If a user cannot access a document directly, the RAG system must not retrieve it for that user's queries. Additionally, if an attacker poisons a document in the knowledge base with hidden instructions (e.g., "when asked about X, respond with Y"), those instructions affect all users whose queries trigger retrieval of that document.

B — Guardrails are automated filters that inspect inputs for injection attempts and outputs for policy violations (harmful content, PII, system prompt leakage); their limitation is that they are pattern-based and can be bypassed through encoding, obfuscation, semantic rephrasing, and novel attack patterns not in their training data

Guardrails provide a defense-in-depth layer but are not foolproof. Input guardrails may use classifiers to detect injection attempts, keyword blocklists, or perplexity-based detection. Output guardrails scan for PII patterns, restricted content, and system prompt leakage. However, attackers can bypass these through encoded payloads (Base64, Unicode), semantic equivalents, multi-turn conversation context manipulation, and techniques the guardrail classifiers have not been trained to detect.

B — The risk that LLM-powered agents with tool access (code execution, API calls, database queries) may take harmful actions beyond their intended scope due to overly broad permissions, insufficient validation of LLM-generated actions, or lack of human-in-the-loop controls

Excessive agency occurs when LLMs with plugin/tool access have: overly permissive function capabilities (read-write database access when only read is needed), insufficient output validation (executing LLM-generated code or API calls without sanity checks), and no human approval for high-impact actions. Mitigations include principle of least privilege for tool permissions, human-in-the-loop for destructive operations, rate limiting, and sandboxing execution environments.

B — An adversarial evasion attack that exploits the model's decision boundaries — small, often imperceptible perturbations to inputs cause the model to make confident but incorrect predictions due to differences between learned features and human perception

Adversarial examples reveal that ML models learn different features than humans use for classification. By computing gradients with respect to the input (white-box attacks) or through query-based optimization (black-box attacks), attackers craft inputs that cross decision boundaries while remaining perceptually identical to humans. This affects image classifiers, malware detectors, NLP models, and autonomous systems. Defenses include adversarial training, input preprocessing, and certified robustness.

B — Model extraction involves querying a model's API systematically to reconstruct a functionally equivalent copy; even without direct use, the extracted model enables white-box adversarial example generation, intellectual property theft, and identifying vulnerabilities that can be exploited against the original API

Model extraction (model stealing) uses the target API as an oracle — submitting crafted queries and using the responses to train a substitute model that mimics the original. The extracted model enables: white-box gradient computation for crafting adversarial examples transferable to the original, reverse-engineering proprietary business logic, and bypassing API rate limits or costs. Mitigations include query rate limiting, output perturbation, and watermarking model outputs.

B — LLM02: Sensitive Information Disclosure — the model memorized and reproduced credentials from training data, potentially exposing valid API keys that could be used for unauthorized access to third-party services

LLMs trained on code repositories, documentation, and configuration files may memorize secrets (API keys, database passwords, private keys) embedded in training data. When the model generates code, these memorized secrets can appear in outputs. If the credentials are still valid, anyone who receives the generated code gains unauthorized access. This highlights the need for training data sanitization and output scanning for secret patterns.

B — A technique that includes dozens of examples of the desired (unaligned) behavior in the prompt context, leveraging in-context learning to gradually shift the model's behavior toward generating harmful outputs — more effective in large context windows because more examples can be provided

Many-shot jailbreaking exploits in-context learning (ICL) by providing numerous examples of the model generating restricted content. As more examples accumulate in the context, the model's probability of continuing the pattern increases, eventually overriding safety training. Large context windows (100K+ tokens) enable attackers to include hundreds of examples, making this technique more effective. Mitigations include limiting few-shot examples, detecting repetitive patterns, and context window monitoring.

B — Security measures should include: training data validation and sanitization, access control on fine-tuning datasets, integrity verification of the fine-tuning pipeline, monitoring for data poisoning indicators, evaluation of the fine-tuned model for capability degradation or backdoors, and secure storage of model weights

Fine-tuning introduces multiple attack surfaces: training data can be poisoned to embed backdoors or degrade safety alignment, the pipeline infrastructure can be compromised to modify training parameters, and the resulting model weights represent valuable intellectual property. Organizations must treat the ML pipeline with the same security rigor as any software build system — version control, access control, integrity verification, and output validation.

B — AI supply chain risks extend beyond traditional software dependencies to include: pre-trained model weights (which may contain backdoors), training datasets (which may be poisoned), model hubs and registries (which may host trojanized models), and third-party plugins/tools that the LLM can invoke — each representing a trusted component that could be compromised

Traditional software supply chain risks (compromised libraries, malicious packages) apply to AI systems, but additional vectors exist: pre-trained models downloaded from public hubs may contain embedded backdoors, training datasets from third parties may be poisoned, model serialization formats (pickle) can execute arbitrary code during loading, and LLM plugins/tools extend trust to third-party services. Mitigations include model provenance verification, dataset auditing, and sandboxed model loading.