Skip to main content
UFOZoo

Tokenizer Visualizer: See How Text Becomes BPE Tokens

Tokenizer visualizer online: see how text is split into tokens with per-token details, encoding comparison, and byte preview.

Updated 2026-08-16

Related Tools

Token Counter: Count Tokens for GPT, Claude & Gemini

Context Window Calculator: 128k/1M Token Budget Planner

AI Prompt Compressor: Reduce Tokens & Save on LLM Costs

AI Answer Compressor: Trim Filler Words & Save Tokens

AI Arena Leaderboard: Elo for Chat and Image Models

AI Code Extractor: Pull Code, URLs & Tables from Chat

Features

  • BPE segment visualization: every token rendered as a color-coded segment inside the original text, updating live as you type or paste
  • Hover any segment to inspect the token: its decoded text, UTF-8 byte size, and position in the sequence
  • Token detail list with index, decoded text, and byte count for every token
  • Live stats: total token count, byte count, and character count update as you type
  • Special token marking: tokens like <|endoftext|> are flagged and highlighted in the list
  • Encoding comparison between cl100k_base (GPT-3.5/GPT-4) and o200k_base (GPT-4o) side by side
  • Large-text guard: inputs over 20,000 characters are truncated with a clear notice to keep the browser fast
  • Chinese text explained: see with your own eyes why a Chinese prompt often consumes more tokens than English
  • Copy and clear controls plus sample texts for quick exploration
  • Works offline after first load

How to Use

  1. 1Paste or type text into the editor; the token visualization updates automatically.
  2. 2Read the segmented view above: every token is a colored segment. Hover one to see its decoded text, byte size, and position.
  3. 3Scan the token list below for each token's index, decoded text, and byte count.
  4. 4Use the three stats cards to see total tokens, bytes, and characters at a glance.
  5. 5Learn why Chinese prompts cost more: paste a Chinese sentence, then compare it with an equivalent English one.
  6. 6Compare tokenizers: switch the encoding between cl100k_base (GPT-3.5/GPT-4) and o200k_base (GPT-4o), or enable the comparison panel to see both counts side by side.
  7. 7Debug long prompts: paste a big prompt to see where the tokens add up, then trim the segments that consume the most.
  8. 8Spot special tokens like <|endoftext|> in the list, useful when inspecting generated completions.
  9. 9For very long documents, only the first 20,000 characters are visualized and a notice explains the rest was skipped.
  10. 10Use the three example buttons to load sample texts and see how different content types (English, mixed Chinese, code) tokenize.

Frequently Asked Questions

What is a token, exactly?

A token is the smallest unit an AI model reads. Modern tokenizers use BPE (byte-pair encoding): they start from single characters, then repeatedly merge the most frequent adjacent pairs into longer units. 'tokenization' can become one token, while a rare word may stay split into several pieces. Token count drives both context limits and API pricing.

Why does Chinese text consume more tokens than English?

English BPE merges frequent letter pairs into long, efficient units (a common English word is often one token). Chinese characters are dense and varied: each character rarely merges with its neighbors, so one character usually becomes one token, and some characters split into two. That is why the same message costs roughly 2-3x more tokens in Chinese, and this tool shows the split directly.

What do the segment colors mean?

Colors are derived from the token id with a simple hash, so the same token always has the same color. They exist only to make boundaries visible; they carry no meaning, and different tokens can coincidentally share a color.

How is this different from the Token Calculator?

The Token Calculator counts tokens and estimates cost for many models. This tool visualizes one encoding at a time: you see exactly where each token starts and ends, which is useful for teaching, debugging prompts, and understanding why a text costs what it costs.

What is the difference between cl100k_base and o200k_base?

They are different BPE vocabularies. cl100k_base (~100K tokens) is used by GPT-3.5 and GPT-4-class models; o200k_base (~200K tokens) is used by GPT-4o and later models. The newer vocabulary is usually more efficient: the same text frequently tokenizes into fewer tokens with o200k_base.

Can this visualize tokens for every AI model?

No. It only supports the OpenAI encodings shipped with js-tiktoken (cl100k_base and o200k_base). Models from other providers use their own tokenizers (SentencePiece, etc.), so their counts are estimates. This tool intentionally does not pretend to visualize them.

Will a very long text slow down my browser?

Inputs longer than 20,000 characters are truncated before visualization: the first 20,000 characters are shown and a notice tells you the rest was skipped. That keeps rendering fast even for huge pastes.

Why do other tools count the same text differently?

Different tools may use different encodings (or estimates). Two tools are only comparable when they use the same vocabulary. This tool tells you exactly which encoding it used, so you can reproduce the number.

What is a special token?

Special tokens like <|endoftext|> are markers the model uses for structure, not regular text. They are part of the vocabulary but outside normal BPE merging, and they are flagged in this tool so you can spot them in completions.

Is this the same tokenizer ChatGPT uses?

It ships the official js-tiktoken vocabularies used by ChatGPT-class models: cl100k_base (GPT-3.5/GPT-4) and o200k_base (GPT-4o). Two caveats: reasoning models and the API can apply extra tokenization rules, and OpenAI occasionally updates vocabularies, so the exact number you see in ChatGPT's UI can differ slightly from this tool's count for the same text.

Why does the token count differ from Claude or Gemini?

Every model family has its own tokenizer: Anthropic and Google use different BPE vocabularies (Gemini's is SentencePiece-like), so the same text produces a different token count in each. This tool visualizes OpenAI encodings only and does not claim to predict Claude or Gemini counts; for those, use their official token counters.

Why does my GPT tokenizer visualizer show a special space marker (Ġ) before some words?

GPT-family BPE tokenizers mark word-initial spaces with a special character (Ġ, a 'leading space' marker) instead of splitting spaces separately. That is why 'hello' and ' hello' tokenize differently, and why pasted text with a trailing space can change the whole split. It is expected behavior of cl100k_base/o200k_base: if you see the marker, the space is simply part of the token.