Tokenizer and BPE: How LLMs Read Text

LLMs don’t read text as letters or words. They read tokens — chunks somewhere in between.

Understanding the tokenizer explains:

• Why “I have a problem” costs the same in tokens but “我有一个问题” costs 2× more
• Why GPT struggles with rare names
• Why “strawberry” famously got counted wrong
• Why your prompt has limits

What’s a Token

A token is the smallest unit a model processes.

import tiktoken
enc = tiktoken.encoding_for_model("gpt-4")

enc.encode("Hello world")
# [9906, 1917] — 2 tokens

enc.encode("The quick brown fox jumps over the lazy dog")
# 9 tokens

enc.encode("你好世界")
# [57668, 53901] — 2 tokens (for these specific 4 Chinese chars)
# But many less-common chars are 2-3 tokens each

Tokens are:

• Not letters (too many — sequences would be 5× longer)
• Not words (vocabulary would be 5M+ words across languages)
• Subword units — somewhere in between

The Tradeoff

Choosing a tokenizer:

Granularity	Pros	Cons
Characters (a, b, c)	Small vocab, no unknown words	Very long sequences
Words (hello, world)	Short sequences	Huge vocab, unknown words break
Subwords (hell, ##o)	Balanced	Slightly more complex

Modern LLMs all use subword tokenization.

BPE: Byte Pair Encoding

The most popular algorithm. Surprisingly simple:

Training

1. Start with each character as a token
2. Find the most common adjacent token pair
3. Merge them into a new token
4. Repeat until vocab is desired size

Example:

Start: l, o, w, e, r
Pair counts: 'lo' = 100, 'we' = 80, 'er' = 200

Merge 'er' → new token

Now:  l, o, w, er
Pair counts: 'lo' = 100, 'we' = 80, 'wer' = ? 

Continue...

Result: a vocabulary of ~50,000-200,000 tokens that captures common patterns.

Variations

• WordPiece (BERT): slightly different merge criterion (likelihood)
• SentencePiece (Google): handles whitespace as part of tokens, multilingual-friendly
• Tiktoken (OpenAI): BPE-based, used by GPT-3/4

A Worked Example

How GPT-4 tokenizes “Hello world!”:

Text:    "Hello world!"
Tokens:  [9906, 1917, 0]
         "Hello", " world", "!"

Total: 3 tokens

Note:

• “Hello” is one token (common word)
• ” world” (with leading space) is one token — spaces matter!
• ”!” is its own token

How GPT-4 tokenizes “Supercalifragilistic”:

Tokens:  ["Super", "cal", "if", "rag", "il", "istic"]
         6 tokens

Rare words get split into pieces. Each piece is in the vocab.

Why Chinese Costs More

enc.encode("Hello world")     # 2 tokens
enc.encode("你好世界")           # 4 tokens (1 per character)
enc.encode("The cat sat")      # 3 tokens
enc.encode("猫坐了")            # 3 tokens

Common English words = 1 token. Common Chinese characters often = 1 token (in GPT-4).

But:

• Less common Chinese characters = 2-3 tokens each
• Multi-character Chinese words almost never tokenized as a unit

Result: same content, Chinese uses 2-3× more tokens → 2-3× more API cost.

Worth knowing: GPT-4o has improved Chinese tokenization compared to GPT-3.5. DeepSeek’s tokenizer is specifically optimized for Chinese (and code).

The “Strawberry” Problem

Famous case: “how many R’s in ‘strawberry’?”

Text:     "strawberry"
Tokens:   ["str", "aw", "berry"]
          3 tokens

The model doesn’t see individual letters — it sees three subword chunks. Counting R’s becomes hard.

Reasoning models (o1, Claude 3.7+) work around this by being more careful when asked.

But the underlying issue: tokens are not letters.

Multilingual Tokenizers

A good multilingual tokenizer needs vocabulary across languages:

GPT-3:     50k tokens, English-biased
GPT-4:     100k tokens, more multilingual
GPT-4o:    200k tokens, much better Chinese / Asian languages
Claude:    similar size, similar tradeoffs
LLaMA-3:   128k tokens, multilingual
DeepSeek:  100k+, optimized for Chinese + code

For non-English users, the tokenizer choice impacts cost and quality significantly.

Special Tokens

Tokens that aren’t text:

Token	Meaning
<|endoftext|>	End of a document during training
<|im_start|> / <|im_end|>	Chat message boundaries (OpenAI)
[CLS] / [SEP]	BERT classification + separator
<bos> / <eos>	Beginning / end of sequence
<pad>	Padding for variable-length
<unk>	Unknown / out-of-vocab (rare with BPE)

These structure how chat APIs work.

Tokenizer in Practice

Counting tokens

import tiktoken

enc = tiktoken.encoding_for_model("gpt-4")
text = "Your prompt here..."
n_tokens = len(enc.encode(text))
print(f"Costs: {n_tokens} input tokens")

For Anthropic / Claude:

from anthropic import Anthropic
client = Anthropic()
result = client.beta.messages.count_tokens(
    model="claude-3-5-sonnet-20241022",
    messages=[{"role": "user", "content": "Hello"}]
)

Estimating costs

def estimate_cost(prompt, response, model="gpt-4o"):
    enc = tiktoken.encoding_for_model(model)
    in_tokens = len(enc.encode(prompt))
    out_tokens = len(enc.encode(response))

    # GPT-4o pricing
    in_price = 2.50 / 1_000_000
    out_price = 10.00 / 1_000_000

    return in_tokens * in_price + out_tokens * out_price

Visualizing tokens

Our /visualize/tokenizer playground lets you paste text and see exactly how it’s tokenized across models.

Implementing BPE

A toy implementation:

from collections import Counter

def train_bpe(corpus, vocab_size):
    # Start: each character is a token
    tokens = list(corpus)
    pair_counts = Counter()

    while len(set(tokens)) < vocab_size:
        # Count adjacent pairs
        pair_counts.clear()
        for i in range(len(tokens) - 1):
            pair_counts[(tokens[i], tokens[i+1])] += 1

        if not pair_counts:
            break

        # Find most common pair
        best_pair, count = pair_counts.most_common(1)[0]
        if count < 2:
            break

        # Merge in entire corpus
        new_token = best_pair[0] + best_pair[1]
        new_tokens = []
        i = 0
        while i < len(tokens):
            if (i+1 < len(tokens) and
                tokens[i] == best_pair[0] and
                tokens[i+1] == best_pair[1]):
                new_tokens.append(new_token)
                i += 2
            else:
                new_tokens.append(tokens[i])
                i += 1
        tokens = new_tokens

    return tokens

# Toy example
text = "low lower lowest lowering"
result = train_bpe(text, vocab_size=10)
print(set(result))

Real implementations are more complex (handle word boundaries, special tokens, optimization for speed) but the algorithm is this.

Tokenizer Surprises

”Glitch tokens” / weird vocab

Some tokens in GPT vocab:

• SolidGoldMagikarp (Reddit username from training data)
• PsyNetMessage
• Specific Reddit comments

Trigger them in prompts → models behave erratically.

Why? These tokens appeared too rarely during fine-tuning → undertrained.

Find them in our /visualize/vocab-map — the “weird” category.

Spaces matter

"the cat" → [the, cat]
"cat" (no space) → [cat]
" cat" (leading space) → [ cat]   ← different token!

If you prefix with a space, you get a different token. Important when you’re prompting carefully.

Tokenizer-Free Models?

A research direction:

Byte-Level Models

Treat raw bytes as input. Unlimited vocabulary, no tokenization issues. But sequences become 5× longer.

Examples: ByT5, Beyond Tokenization research.

Patch-Based for Code/Images

For images / video, “patches” replace tokens (ViT, Sora). Similar idea.

For text — token-free is slow on current hardware.

What Tokenizer Quality Means

A “good” tokenizer:

• Tokenizes common words / phrases as 1-2 tokens (efficient)
• Handles multi-language (multilingual coverage)
• Robust to rare words (no “out of vocab” failures)
• Doesn’t break code (whitespace, indentation matter)
• Aligned across model versions (so caching works)

OpenAI / Anthropic / DeepSeek / Mistral all have slightly different tokenizers → different token counts for the same text.

Practical Implications

For Prompt Design

• Shorter prompt = cheaper + faster
• Avoid repetition (re-tokenized each time)
• Use cached prefixes (system prompts get pre-tokenized)

For Multilingual Use

• Chinese / Japanese / etc. — costs 2-3× more tokens
• Consider self-hosting LLaMA / DeepSeek (better Chinese tokenizer)

For Code

• Indentation matters (separate tokens)
• Whitespace in code costs real tokens

For Function Calling

• JSON schema costs many tokens
• Each function definition: ~50-200 tokens

Next recommended: L4-03 RAG or L4-02 Advanced Prompting.