Definition:
An AI token is a unit of information that a model uses to represent and process content. In language models, it may correspond to a word, part of a word, a punctuation mark or a combination of characters.
Tokens allow text to be converted into a sequence that the model can handle. They are also used to express context limits and account for usage in many artificial intelligence services.
Table of contents
How text is converted into tokens
Tokenization is the process of dividing text into units according to a tokenizer’s rules. It assigns each unit a numerical identifier that the model uses to work with it.
Tokenization has been used in natural language processing since before today’s assistants. Many modern models use units smaller than a word, allowing them to represent uncommon terms, new words and code without needing a separate entry for every complete word.
Methods used include BPE, WordPiece and Unigram. Tokenization algorithms apply different criteria to construct these units, so not all models divide the same text in the same way.
In text generation, the model produces a sequence of tokens that is then converted into a readable response.
Why a token is not equivalent to a word
A common word may be represented by a single token, while another may be split into several. Spaces, punctuation and numbers also play a role in tokenization.
The number of tokens depends on the text and the tokenizer used. Language, characters, formatting and elements such as code or web addresses all have an effect.
Therefore, there is no universal equivalence between words, characters and tokens. Approximate rules can provide an initial estimate, but do not replace the count for the chosen model.
An estimate calculated for English should not automatically be applied to Spanish text either. Two texts of similar length may require different numbers of tokens.
Input and output tokens
In an interaction with a model, a distinction is usually made between the tokens it receives and those it generates:
- Input tokens: represent the information sent to the model. They may include instructions, the user’s message, previous conversations, documents and tool results.
- Output tokens: correspond to the content generated by the model during its response.
The visible prompt does not always constitute the entire input. An application may incorporate additional instructions and materials that also take up space.
Some services distinguish other categories, such as input tokens reused through caching or reasoning tokens. How they are counted depends on the model and provider; the length of the visible response does not always reflect all recorded usage.
Relationship with the context window
The context window is the amount of information a model can handle in an interaction, usually expressed in tokens.
The available budget must account for the input and the space needed for generation, according to the model’s limits. The maximum length of a response may also have its own limit.
When a conversation or document exceeds the supported capacity, the application may reject the request, truncate information, summarise it or select only part of it. Its behaviour depends on how it is implemented.
A large context window is not equivalent to permanent memory and does not guarantee that the model will make proper use of every detail. It indicates processing capacity, not assured accuracy.
How tokens affect cost
Many artificial intelligence APIs calculate part of their price based on the tokens processed. Rates may distinguish between input, output and content retrieved from cache.
Cost therefore does not depend solely on the number of questions. The length of instructions, documents sent, included history and responses also has an effect.
A subscription to an assistant, on the other hand, may use different quotas or limits. Not all tools should be assumed to charge directly per token.
To estimate consumption, use a counter compatible with the model and review the service’s usage data. Token count is a useful measure, but it cannot, on its own, be used to compare the cost or speed of different models.
Other uses of the word token
The meaning of token changes depending on the field. It is important to distinguish these uses:
- In AI models: a unit used to represent and process information.
- In authentication: a credential that can verify a session or authorise access to a resource.
- In blockchain: a digital representation of an asset, right or utility within a network.
Multimodal models may also use representations and counts associated with images, audio or video. These should not be interpreted as a simple word count: their processing and accounting depend on the system.