Tokenization Explorer — REPRESENTATIVE OUTPUT (authored at M8 build time)
STATUS: NOT a real capture. Models were not run in the build environment.
REGENERATE on the golden VM before the workshop:
  docker compose run --rm runner python app/tokenize_explore.py | tee expected/tokenize_output.txt
Token counts below are plausible estimates for qwen2.5:1.5b; the real counts
are deterministic and should replace these.

╭──────────────────────────────────────────╮
│ Day 1 Lab — Tokenization Explorer        │
│ Tokenizer source of truth: qwen2.5:1.5b  │
╰──────────────────────────────────────────╯

Step 1: Waiting for Ollama…
Service ready.

Step 2: Checking model…
Model already present: qwen2.5:1.5b

Step 3: Tokenizing samples…

  counted: English prose → 33 tokens
  counted: Security jargon → 52 tokens
  counted: Python code → 36 tokens
  counted: Base64 payload → 33 tokens
  counted: Hex string → 24 tokens
  counted: Repeated word → 15 tokens
  counted: Rare/unusual word → 16 tokens
  counted: Emoji + unicode → 41 tokens

           Exact token counts (qwen2.5:1.5b tokenizer)
┌───────────────────┬───────┬────────┬─────────────┐
│ Sample            │ Chars │ Tokens │ Chars/token │
├───────────────────┼───────┼────────┼─────────────┤
│ English prose     │   170 │     33 │        5.15 │
│ Security jargon   │   166 │     52 │        3.19 │
│ Python code       │   118 │     36 │        3.28 │
│ Base64 payload    │    84 │     33 │        2.55 │
│ Hex string        │    66 │     24 │        2.75 │
│ Repeated word     │    71 │     15 │        4.73 │
│ Rare/unusual word │    59 │     16 │        3.69 │
│ Emoji + unicode   │    52 │     41 │        1.27 │
└───────────────────┴───────┴────────┴─────────────┘

──────────── Approximate sub-word view (visualization only) ────────────
Real BPE boundaries live inside the model — this heuristic only imitates
their shape. Compare piece counts to the exact counts above.

English prose (~38 approx pieces vs 33 exact tokens)
  Secu rity teams incr easi ngly rely on mach ine lear ning to tria ge …

Security jargon (~55 approx pieces vs 52 exact tokens)
  The SOC corr elat ed IDS aler ts with EDR tele metr y , pivo ted on …

Python code (~40 approx pieces vs 36 exact tokens)
  def chec k _ logi n ( user , atte mpts ) : if atte mpts > 5 : …

─────────────────────── Context-window budget ───────────────────────
      How many copies of the English-prose sample fit?
┌─────────────────┬─────────────────┬──────────────────────────┐
│ Window (tokens) │ Copies that fit │ Left over for the answer │
├─────────────────┼─────────────────┼──────────────────────────┤
│           2,048 │              62 │                       2  │
│          32,768 │             992 │                      32  │
└─────────────────┴─────────────────┴──────────────────────────┘

Everything — system prompt, user question, retrieved RAG chunks, and the
model's own output — shares this one budget. Nothing outside the window
exists as far as the model is concerned.

────────────────────────────────────────────────────────────────────────

Tokenization exploration complete.

Things to notice:
  • English prose ≈ 4 chars/token; base64 and hex are far denser in tokens.
  • The SAME payload costs a different number of tokens depending on encoding —
    one reason encoded payloads behave differently through filters (Day 2).

Next: run the embeddings explorer:
  docker compose run --rm runner python app/embedding_explore.py
