Blue team · Lab

Tokens: count them, break them, budget them

See text become tokens, understand why token counts differ by content type and model, and connect tokens to both cost and the context window.

~30 min · Browser tokenizer + LM Studio · 6 steps

1
Open Tiktokenizer or the Hugging Face Tokenizer Playground (links on the Resources page). No install needed.
2
Paste a paragraph of ordinary prose and note the token count and the character-to-token ratio.
3
Now paste the same length of: source code, a base64 blob, a hex string, and a few emoji. Compare counts.
4
Paste a word like "cybersecurity" and watch where the split lands.
5
Switch tokenizers (different model families) with identical text and compare the totals.
6
In LM Studio, load a small model and hold a short conversation, watching the context/token usage indicator climb as the conversation grows.

What to notice

Identical character counts produce very different token counts. Base64 and hex are far more expensive than prose.
That gap is why a filter counting characters and a model counting tokens disagree — a Day 2 evasion theme.
The same text costs a different number of tokens on different models, so "context window" and "price per million tokens" are only comparable within a model family.
Context is finite and consumed by the whole conversation, not just your last message.