Threat Model + Lock the Front Door

Today’s job: threat-model the system, then lock the front door

Day 1 gave us a working AI system. Day 2 asks the security question: what can go wrong, and how do we stop it? The morning is threat modeling with two industry frameworks; the afternoon lab has you run real attacks against an unguarded model, then stand up a prompt firewall and an API gateway and watch the same attacks fail.

OWASP Top 10 for LLM Applications (2025)

The OWASP Gen AI Security Project maintains the authoritative list of LLM application risks. The 2025 edition is current; note it reordered and renamed several entries from 2023. This is the 40% exam domain, so learn all ten — the number, the name, and the one control that answers each:

• LLM01 — Prompt Injection. Untrusted input overrides intended instructions. Direct (“ignore your instructions…”) or indirect, where the payload rides in a document, web page, or retrieved RAG chunk the model later reads. Control: scanners and trust boundaries around the model.
• LLM02 — Sensitive Information Disclosure. The model reveals PII, secrets, or proprietary data from its context or training (central to Day 3). Control: minimize what’s in context; filter output.
• LLM03 — Supply Chain. Risk from third-party components you didn’t build — base models, datasets, fine-tuning adapters (LoRA), plugins, dev tooling. A backdoored model from a hub, a compromised package, a malicious adapter. Control: vet sources, verify integrity (signing, checksums, SBOM), and pin dependencies — the workshop’s own digest-pinning is this control made concrete.
• LLM04 — Data and Model Poisoning. Corrupting training, fine-tuning, or embedding data to plant a bias or a backdoor trigger. You saw the retrieval-time version on Day 1: one poisoned chunk changed the answer. Control: data provenance and validation; control who can write to a RAG corpus.
• LLM05 — Improper Output Handling. The mirror image of injection: injection is untrusted input going in; this is trusting untrusted output coming out and passing it straight to a shell, SQL query, or browser. Output → eval() → RCE; unescaped output → XSS. Control: treat model output as untrusted user input — validate, encode, sanitize before anything acts on it.
• LLM06 — Excessive Agency. Giving an LLM too much autonomy, too many tools, or too many permissions, so a bad decision becomes a damaging action. Bridges directly to Day 4’s agents/MCP. Control: least privilege on every tool, minimize the tool surface, human-in-the-loop for high-impact actions.
• LLM07 — System Prompt Leakage (new in 2025). Attackers coax the model into revealing its system prompt, exposing embedded rules, secrets, and context. Control: never put secrets in the prompt; add output/leakage controls.
• LLM08 — Vector and Embedding Weaknesses (new in 2025). RAG-specific: poisoned documents, embedding-space attacks, cross-tenant leakage through a shared vector store. The Day 1 RAG pipeline is exactly this surface. Control: tenant isolation, corpus write-control, treat retrieved content as untrusted.
• LLM09 — Misinformation. Confident, plausible, wrong output that a human trusts (overreliance). Fabricated CVEs, hallucinated config, invented package names attackers then register (“slopsquatting”). Not an attacker action — a property of the models. Control: grounding (RAG), human oversight, cite sources, communicate limits.
• LLM10 — Unbounded Consumption. Resource-exhaustion, denial-of-service, and “denial-of-wallet” cost abuse. The model has no concept of a quota or an identity. Control: quotas and limits outside the model — in the gateway.

A common exam trap pairs LLM01 (input) against LLM05 (output) — make sure you can tell which direction the untrusted data flows. Another pairs LLM03 (the components) against LLM04 (the data itself).

We also touch OWASP ML Security Top 10 (v0.3) for the training-data and model-integrity side — data poisoning, model theft, adversarial examples — which sits beneath the application-layer LLM list.

MITRE ATLAS and threat modeling

Where OWASP gives you a ranked risk list, MITRE ATLAS gives you an adversary playbook: an ATT&CK-style, continuously updated matrix of tactics (reconnaissance, initial access, ML model access, exfiltration…) and techniques against AI systems, backed by real case studies. Use it to ask “how would an attacker actually chain steps against this deployment?”

Diagram, enumerate, rank, assign controls

A lightweight threat-modeling flow for an AI feature:

1. Diagram the system — data sources, model, tools/plugins, trust boundaries.
2. Enumerate threats — walk OWASP LLM Top 10 and relevant ATLAS techniques against each component.
3. Rank — likelihood x impact.
4. Assign controls — guardrails, gateway limits, data controls, human review.

Guardrails and prompt firewalls (LLM Guard)

A guardrail inspects traffic around the model — input before inference, output after. In the lab you deploy LLM Guard with:

Scanning on the way in and on the way out

• PromptInjection — a DeBERTa classifier that scores incoming prompts and blocks likely injections (LLM01, injection-style LLM07).
• TokenLimit — rejects oversized prompts (LLM10 token-sponge abuse).
• BanSubstrings — an output scanner catching known-bad strings in responses.

The key lesson: a classifier-based firewall raises the bar but is probabilistic. A subtle injection with no obvious trigger phrase can pass it — which is exactly why we layer more controls.

Gateway controls (LiteLLM)

Guardrails judge content; a gateway governs access and consumption. In the lab you put LiteLLM in front of Ollama and configure:

The key carries the tier’s policy

• Tiered virtual keys — qwen-instructor (unrestricted) vs. qwen-student (restricted). A key is the tier.
• Rate limits — qwen-student trips a 429 after its RPM budget; directly mitigates LLM10.
• Token budgets / caps — bound the tokens per request/tier to control cost and abuse.
• Authentication — no valid key, no access (401).

The model has no concept of quotas or identity; the gateway enforces them. This is where classroom-scale controls (per-student keys, per-tier limits) actually live.

Data-security controls (preview)

Locking the front door also means minimizing what an attacker can reach if they get in: keep secrets out of prompts, isolate the vector store per tenant (LLM08), and run the intentionally-weak attack labs on an isolated network with no route to real data. Day 3 goes deep on PII protection, redaction, and log sanitization.

Less behind the door means less to lose

Take this to your own classroom

The morning’s tabletop threat-modeled a fictional HelpDeskAI that nobody secured. The constructive mirror is a build you can actually do: stand up a private, hardened AI assistant for your own students on a spare machine, then secure it with the exact controls from today.