Shared Vocabulary: AI and Cybersecurity Domains

AI and cybersecurity have developed parallel vocabularies that often describe related concepts. Understanding the mapping helps security professionals reason about AI risks using familiar frameworks.

Threat model (security) ↔ Adversarial ML threat model (AI): In security, threat modeling enumerates assets, threats, and mitigations (STRIDE, PASTA, ATT&CK). In adversarial ML, we model the attacker's capabilities (white-box: full model access; gray-box: partial; black-box: API only), goals (evasion, poisoning, extraction, inference), and knowledge. MITRE ATLAS (Adversarial Threat Landscape for AI Systems) maps ATT&CK-style TTPs to AI systems.

Attack surface: In traditional security, the attack surface is the set of exposed interfaces, ports, and code paths. For an LLM-based application, the attack surface includes: the prompt itself (injection), the context window (what data is injected), tool calls / function calling (for agents), training data (poisoning), and the model weights (extraction). RAG systems add the vector database and the retrieval interface as attack surface.

Least privilege: The principle that a process should have only the minimum permissions needed. For AI agents with tool-calling (function-calling) capabilities: grant only the tools and data the agent needs for its specific task. An agent that can write to a database and send emails has far greater blast radius than one that can only read a curated dataset.

Defense in depth: Layered security controls so no single failure leads to full compromise. For LLMs: input sanitization → system prompt hardening → output filtering → action validation → human-in-the-loop for high-stakes actions. No single layer is sufficient; prompt injection will bypass input sanitization if the injection reaches the model.

Supply chain attack: Tampering with a software component upstream of its intended target. For AI: poisoning training data, backdooring a fine-tuned model, tampering with model weights or quantization, or inserting malicious instructions in documents that a RAG system ingests. The 2020 SolarWinds attack provides the classic mental model — the artifact looked legitimate but was compromised before distribution.

Audit log / provenance: In security, logs are the basis for incident detection and forensics. For AI systems, provenance tracks which training data contributed to a model output, which document chunk was retrieved by RAG, and which tool calls an agent made. Without provenance, AI-assisted decisions are opaque and unauditable.

Confidentiality, Integrity, Availability (CIA Triad) applied to LLMs:
- Confidentiality: prevent training data extraction (memorization attacks), system prompt leakage, unauthorized retrieval from the knowledge base.
- Integrity: prevent model poisoning, adversarial examples that cause misclassification, prompt injection that hijacks agent behavior.
- Availability: protect against denial-of-service via token-exhaustion attacks (submitting very long prompts), resource-intensive jailbreak probes that degrade service for other users, and model inversion attacks.

Context window (AI) ↔ Session / state (security): The context window is the active "working memory" of an LLM during a conversation. An attacker who controls any part of the context window (via injection in retrieved documents, tool outputs, or user input) can influence the model's subsequent behavior — analogous to session hijacking or state corruption.
