What Is an AI Token? A Simple Guide to Tokens, Limits & Cost

Every time you use ChatGPT, Claude, or Gemini, you are working with tokens — and every API bill is measured in them. This guide explains what a token actually is, how to convert between words and tokens, and why understanding tokens saves you money.

What Is a Token, Exactly?

A token is the unit of text an AI model reads and writes. Before a model processes your prompt, a tokenizer splits the text into small pieces — whole words, word fragments, punctuation, or even single characters. Each piece is one token, mapped to a number the model understands.

• 1 token ≈ 4 characters of English text
• 1 word ≈ 1.3 tokens on average
• Common words like "the" or "is" are often one token; rare or long words split into two or more
• Whitespace, punctuation, and newlines each count as tokens too

So 1,000 words of English is roughly 1,333 tokens. Code, URLs, emoji, and non-English languages tokenize less efficiently — the same amount of text can cost 30–50% more tokens in Japanese or dense code than in English.

How Tokenization Works

Modern models use subword tokenizers (typically Byte-Pair Encoding or similar) trained on their own corpus. The tokenizer learns frequent words stay whole ("token" = 1 token) while rare words split ("unbelievable" might be 3 tokens). Every model family ships its own tokenizer:

• OpenAI models use tiktoken vocabularies (such as o200k_base)
• Anthropic and Google publish their own counting methods
• The same sentence can be 40 tokens in one model and 45 in another

That is why a token count is always model-specific. For quick estimates, use the AI Token Counter — it supports GPT, Claude, Gemini, DeepSeek, and Llama, entirely in your browser.

Why Tokens Matter: Cost and Limits

Tokens drive two things every AI user cares about: bills and limits.

• API pricing is quoted per million tokens, with separate rates for input (what you send) and output (what the model generates). Output typically costs 3–6× more than input.
• Context windows cap how many tokens a model can hold at once — including your prompt, attached files, and its reply. Exceed it and the request fails or gets truncated.
• Consumer apps like ChatGPT don't show token counts, but the same rules apply behind the scenes: long chats "forget" early messages because they push past the window.

A single 1,000-word generation costs roughly 1,333 output tokens. At $15 per million output tokens that is about $0.02; at $30 per million it is about $0.04. Tiny per call — until you multiply by thousands of calls a day. Estimate your real spend with the token counter's cost calculator.

Word-to-Token Quick Reference

• 100 words ≈ 130 tokens
• 500 words ≈ 665 tokens
• 1,000 words ≈ 1,333 tokens
• A 50-page document ≈ 25,000–35,000 tokens

Treat these as planning estimates for English prose. For a specific model and exact text, paste it into the AI Token Counter and get the real number in milliseconds.