What a token is
Language models do not read letters or words. They read tokens: chunks of text from a fixed vocabulary. A common word like “Pakistan” is one token. A longer word is split, so “tokenization” becomes “token” + “ization”. Ten digits such as 1234567890 become 4 tokens, because the tokenizer groups digits in threes. Spaces usually ride along with the next word, and punctuation is often its own token.
You pay for tokens, rate limits are set in tokens, and context windows are measured in tokens. So when a prompt is “too long” or a bill looks high, the token count is the number that matters.
Exact counts and estimates
This page gives two kinds of numbers, and labels them clearly.
Exact (OpenAI). OpenAI publishes its vocabularies. The tool runs the open-source gpt-tokenizer library with the two current ones:
| Tokenizer | Vocabulary | Used by |
|---|---|---|
| o200k_base | about 200,000 tokens | GPT-4o, GPT-4.1, GPT-5.x, o1, o3, o4-mini |
| cl100k_base | about 100,000 tokens | GPT-4, GPT-3.5 Turbo, text-embedding-3 |
OpenAI has not published a tokenizer for its newest GPT-6 models, so we assume they use o200k_base like GPT-5.
Estimates (everyone else). Anthropic, Google, xAI and Mistral do not ship browser tokenizers for their current models. Instead of hiding that, the tool starts from the exact OpenAI count and applies a documented factor, then shows a likely range:
| Model family | Based on | Factor | Range | Why |
|---|---|---|---|---|
| Claude 4.7 and later | cl100k_base | 1.30 | 1.10–1.55 | Anthropic says the newer Claude tokenizer uses about 30% more tokens than before |
| Gemini | o200k_base | 1.00 | 0.85–1.20 | Google: about 4 characters per token, 100 tokens ≈ 60–80 words |
| Llama 3 and later | cl100k_base | 1.00 | 0.80–1.10 | 128K-token tiktoken-style vocabulary |
| Mistral (Tekken) | o200k_base | 1.05 | 0.90–1.30 | ~131K-token tiktoken-style vocabulary |
| Grok | o200k_base | 1.05 | 0.90–1.30 | No public tokenizer or rule of thumb |
DeepSeek is the exception: it publishes its own rule, 1 English character ≈ 0.3 tokens and 1 Chinese character ≈ 0.6 tokens, so the tool uses that rule directly.
The Claude factor rests on one assumption we cannot check: that older Claude models counted roughly like cl100k_base. If you need the exact number, Anthropic’s count_tokens endpoint is free to use, and the Gemini API has a countTokens method. Both need an API key.
Before the tokenizer loads
The OpenAI vocabularies are about 1.4 MB compressed. Your browser downloads them once from the jsDelivr CDN and caches them. Until they arrive, and if the download fails, the tool uses a fast built-in estimator. It splits the text the way OpenAI’s tokenizers do (letter runs, groups of up to three digits, runs of punctuation, whitespace) and prices each piece with constants for each script.
We fitted those constants against the real o200k_base and cl100k_base tokenizers on about 50,000 characters of mixed text: English prose, Markdown, TypeScript, Python, JSON, CSV and ten other languages. On that set the estimator was usually within about 8% for English and code and within about 20% for other languages. Once the real tokenizer is ready, every number switches to the exact count, and the label says so.
Worked examples
All counts below were made with the real tokenizers.
- An English paragraph of 450 characters and 77 words: 87 tokens in both o200k_base and cl100k_base. That is 5.2 characters per token, or 1.13 tokens per word, which is typical for plain English. Our Claude estimate for it is about 87 × 1.3 ≈ 113 tokens.
- The same idea in Hindi (239 characters): 71 tokens in o200k_base but 244 in cl100k_base. The newer vocabulary is more than three times as efficient for Devanagari script. Russian shows the same pattern: 70 tokens against 134.
- A small JSON object,
{"sku": "TL-600", "comfort_c": -5, "price_usd": 249}, is 52 characters but 23 tokens. Quotes, braces, colons and short keys are expensive, at about 2.3 characters per token.
Tips for using fewer tokens
- Cut repeated instructions. A long system prompt sent with every request is billed every time. Prompt caching makes repeated prefixes much cheaper; the AI API cost calculator shows how much.
- Send clean text, not markup. HTML and PDF extraction noise waste tokens. Convert documents with document to Markdown first.
- Minify JSON you send as data. Whitespace and indentation are tokens too; the JSON formatter can minify it.
- Leave room for the answer. The prompt and the reply share the context window. Check the fit with the context window checker.
Privacy
Your text is counted on your device, inside your browser. The only thing downloaded is the public tokenizer vocabulary, and nothing you type is uploaded, logged or stored.