Token
Also known as: tokens, tokenization
A token is a small chunk of text — often a word or piece of a word — that language models use as their basic unit of input and output.
In more detail
Models don't read letters or whole words directly; they break text into tokens. A rough rule of thumb often cited by providers is that a token is about four characters of English, so a token is a fraction of a word on average. Context-window limits and usage-based pricing are counted in tokens, which is why 'how many tokens' is a practical question when working with large inputs.
Example
The sentence "AI is helpful." is only a few tokens. A long pasted contract can be thousands of tokens, which is what counts against a model's context-window limit.
Why it matters
Tokens are how AI tools measure length, limits, and (for paid APIs) cost — so understanding them demystifies context windows and pricing.
Sources
- 1.What are tokens — OpenAI Help Center
OpenAI · verified 2026-07-26
Supports: Tokens are the units models process; roughly four characters of English per token.
Try a prompt for yourself
Open any prompt and run it in the IterPrompt playground.