Defense in Depth + Rules of the Road

Today’s job: defense in depth, and the rules of the road

Day 2 locked the front door. Day 3 assumes an attacker is already inside the context and asks: how do we protect the data itself, and what rules govern all of this? The morning is data-protection engineering (the lab); the afternoon is governance, risk, and compliance — NIST, the EU AI Act, and responsible-AI principles.

PII and data protection

Sensitive data in an AI system must be protected in three states:

Redact at the input, not at the output

• In transit — TLS/HTTPS between client, gateway, model, and stores.
• At rest — disk/database/volume encryption for prompts, logs, vector stores, and results.
• In use — the hard one. During inference the data is decrypted in memory. Confidential computing / trusted execution environments address this state; for most classroom deployments the practical control is minimizing what sensitive data is ever loaded.

On top of encryption, apply data-reduction controls:

• Anonymization — irreversibly remove identity (e.g., drop the field).
• Redaction — remove sensitive spans from text.
• Masking — partially obscure while keeping format (e.g., ***-**-1234).
• Pseudonymization / tokenization — replace with reversible placeholders under key control.

Monitoring and log sanitization

AI systems generate rich logs — and logs are a notorious PII/secret leak. Prompts, retrieved context, and responses often contain names, SSNs, API keys, and tokens. Before logs ship to a SIEM they must be sanitized. The lab runs a two-layer sanitizer:

Sanitize before the logs leave the box

1. Layer 1 — regex for structured secrets and identifiers (Bearer tokens, sk- and AKIA keys, JWTs, SSNs, credit cards, emails). Fast enough for high-volume pipelines.
2. Layer 2 — Presidio NER for unstructured PII (names, locations) that regex misses.

Monitoring is also a detective control: watch for injection patterns, anomalous token consumption (LLM10), and repeated system-prompt-leakage probes (LLM07).

Attacks and compensating controls

We revisit the attack side to connect it to data protection. Prompt-injection and jailbreak families:

• Direct injection — the user’s own text overrides instructions.
• Indirect injection — payload hidden in a retrieved document or web page (the RAG surface).
• Multi-turn / role-play jailbreak — the attacker builds up context or a persona to bypass guardrails.
• System-prompt leakage — coaxing the model to reveal its instructions (and any PII in them).

For each, name the compensating control: input redaction, output scanning, isolation of the vector store, least-privilege context, and human review of consequential actions. Alignment training is not a control — the lab shows a small model may refuse some attacks, but you must never depend on it.

NIST AI RMF 1.0 and AI 600-1

The NIST AI Risk Management Framework 1.0 is the anchor governance model. Its four functions:

From abstract function to shipped control

• Govern — policies, roles, accountability, culture (this is where Shadow AI discovery and responsible-AI policy live).
• Map — context, intended use, and where risks arise.
• Measure — assess and track risks with metrics and testing.
• Manage — prioritize and act on risks; deploy and monitor controls.

NIST AI 600-1 is the Generative AI Profile — a companion that maps concrete GenAI risks and mitigations onto the RMF. In the lab you complete an AI RMF mapping worksheet for the DataAssist HR-chatbot scenario and relate the redaction/log-sanitization controls to specific Manage-function actions.

EU AI Act

The EU AI Act takes a risk-tiered approach: unacceptable (banned), high-risk (heavy obligations), limited-risk (transparency), and minimal-risk. AI-assisted security tooling can fall into the high-risk tier depending on use.

Responsible AI and Shadow AI

Wrap the day with responsible-AI principles — fairness, transparency, accountability, privacy, safety, and human oversight — which recur across NIST, the EU AI Act, and vendor frameworks. Then confront Shadow AI: staff using unsanctioned AI tools that route sensitive data outside every control we just built. Discovery, clear policy, and a sanctioned alternative (a governed internal tool) are the response — a Govern-function problem, not a purely technical one.