Tokens & Tokenization

Understand the fundamental unit of information for Large Language Models.

What are Tokens?

Tokens are the basic building blocks of text that LLMs read and process. Models don't see words like we do; they see sequences of numbers (token IDs), where each number represents a specific chunk of characters.

"Roughly speaking, 1 token is about 0.75 words, or 1000 tokens is about 750 words."

Gen Token 1234

AI Token 5678

Hub Token 9012

How Tokenization Works

Different models use different "Tokenizers" (like BPE - Byte Pair Encoding). Common behaviors include:

Common Words: Often a single token (e.g., "apple", "write").
Complex Words: Split into multiple tokens (e.g., "understanding" → "under" + "stand" + "ing").
Whitespace: Often included in the preceding token.
Languages: English is very token-efficient; other languages may require more tokens for the same sentence.

Context Window

The "Context Window" is the maximum number of tokens a model can process at once (Input + Output). If you exceed this, the model "forgets" the beginning of the conversation.

Cost & Pricing

Most API providers (OpenAI, Anthropic) charge per million tokens. Usually, input tokens are cheaper than output tokens.

Test Your Knowledge

Score 8/10 or higher to pass