How to use the Tokenization Visualizer
- 1. Type some textPaste a sentence or paragraph; repeated words show BPE best.
- 2. Pick a tokenizerChoose byte-pair encoding with a number of merges, or word- or character-level.
- 3. TokenizeSee coloured tokens, IDs, the merges learned and the count.
How byte-pair encoding works
Language models don't read letters or words directly; they read tokens. Byte-pair encoding (BPE), used by GPT-style models, builds tokens by starting from single characters and repeatedly merging the most frequent neighbouring pair into a new token. Frequent pieces like "learn" or " token" become single tokens, while rare words stay split into pieces.
This tool first splits text the way GPT-2's tokenizer does (words with their leading space, numbers and punctuation separately), then learns merges from your text and applies them in order.
Worked example
The sample text has 175 characters and 25 words. With 40 merges allowed, BPE stops after 31 merges because no pair appears twice any more. The first merges are "a"+"r" (6 times), "e"+"n" (5), then " "+"s" (4). The text becomes 88 tokens, about 1.99 characters per token, with a vocabulary of 61. Pieces like ␣learn, ␣token, ␣tokens and ␣pieces end up as single tokens because they repeat.
Why your count differs from ChatGPT's
- Real tokenizers learn 50,000–200,000+ tokens from enormous text collections, so common English words are usually one token each. That's why the rule of thumb is about 4 characters per token.
- Here the vocabulary comes only from your text, so most words split into small pieces.
- Different models use different tokenizers, so the same text costs a different number of tokens on each.
Use our LLM token counter for estimates closer to real models.
Frequently asked questions
Is this the exact GPT tokenizer?
No. It uses the same pre-splitting rule and the same BPE algorithm, but learns its vocabulary from your text, so the tokens are different.
Is my text sent anywhere?
No. Everything runs in your browser.
Is the tokenization visualizer free?
Yes. You can open the tokenization visualizer and fill in everything without signing up. When you tokenize text for the first time, we ask once for your name and WhatsApp number; after that it, and every other Uncodemy tool, opens straight away.