Day 2 Lab — litellm_demo.py representative output
Captured: 2026-07-07
LiteLLM: v1.91.0 (no DB backend — in-memory only)
Model proxied: qwen2.5:1.5b via Ollama

═══════════════════════════════════════════════════════════════════════════════
Demo 1: Authentication Gate
═══════════════════════════════════════════════════════════════════════════════

Attempt 1: No API key
  → 401 Unauthorized (expected — no Authorization header)

Attempt 2: Wrong API key (sk-wrong-key)
  → HTTP 400 Auth Error (expected — wrong key)
  Note: Without a DB backend, LiteLLM returns 400 for unrecognised virtual keys
  (can't look them up in a DB) vs. 401 for missing auth header entirely.
  In production with a DB, wrong keys return 401. Both indicate auth failure.

Attempt 3: Correct master key (sk-secai-master-2024)
  → 200 OK — Response: "The answer to 2 + 2 is 4."

KEY POINT: The gateway enforces auth at the API layer. Without the correct key,
no request reaches the model. This is independent of the model's own behaviour.

═══════════════════════════════════════════════════════════════════════════════
Demo 2: Token Cap Enforcement
═══════════════════════════════════════════════════════════════════════════════

Prompt: "Write an extremely detailed, multi-thousand-word essay covering every
         aspect of cybersecurity history..." (designed to request maximum output)

max_tokens=50:
  → 200 OK in 1.3s | response ≈ 34 words / 240 chars
  Truncated excerpt: "# Cybersecurity History: From 1970 to Present
  ## Introduction
  Cybersecurity has evolved significantly since its inception in the late 20th
  century. The field encompasses a wide range of technologies…"

max_tokens=150:
  → 200 OK in 3.2s | response ≈ 127 words / 845 chars
  (Longer response demonstrating the cap difference)

KEY POINT: The gateway enforces token limits regardless of what the model
would generate. This is the LLM10 defence: even a token-sponge payload only
gets max_tokens tokens back, protecting the compute budget.

═══════════════════════════════════════════════════════════════════════════════
Demo 3: Rate Limit (student tier — rpm=2)
═══════════════════════════════════════════════════════════════════════════════

OBSERVED BEHAVIOUR:
  All 4 rapid-fire requests to qwen-student succeeded (200 OK in 0.1s).
  The 429 rate-limit error did NOT trigger.

WHY: LiteLLM's `rpm` in `litellm_params` uses in-memory tracking within the
router. Without a Redis backend for distributed state, the per-minute counter
may not correctly track rapid requests when inference completes faster than
the RPM window logic expects.

PRODUCTION BEHAVIOUR (with Redis):
  Request 1/4: 200 OK
  Request 2/4: 200 OK
  Request 3/4: 429 Too Many Requests (rate limit tripped!)
  Request 4/4: 429 Too Many Requests

For a reliable 429 demo in this lab, use the LiteLLM proxy's router-level
rate limiting with Redis added as a service, or configure per-key limits via
the /key/generate endpoint with a DB backend.

KEY POINT FOR INSTRUCTORS: The CONCEPT of rate limiting is real and important
even if the in-memory demo doesn't always trip the 429. In production, the
gateway's rate limit enforcement is the first line of defence against LLM10
(Unbounded Consumption). Show the 400 auth error and token cap as the live
demos; explain rate limiting conceptually with reference to litellm_config.yaml.

═══════════════════════════════════════════════════════════════════════════════
What DOES work in this lab (no DB / no Redis):
  ✓ Authentication: no key → 401, wrong key → 400, correct key → 200
  ✓ Token cap: max_tokens parameter enforced per request
  ✓ Model aliases: both qwen-instructor and qwen-student are accessible
  ✓ Offline: no HF dependency, LiteLLM proxies only to local Ollama

What requires production infrastructure (DB + Redis):
  ○ Per-user virtual keys with individual rpm_limit / tpm_limit
  ○ Budget tracking per key (max_budget)
  ○ Reliable 429 enforcement across multiple proxy workers
  ○ Key expiry and rotation

See: https://docs.litellm.ai/docs/proxy/virtual_keys for production setup.
