Skip to content

Search across models, tools, services, companies, providers, APIs, plans, articles and news.

Core concepts

Token

Token

Token

The small unit of text a model reads and generates — roughly ¾ of an English word or 1–2 Persian words. Both context length and API pricing are counted in tokens.

A token is the small text unit a model reads and generates — neither exactly a word nor a letter. Roughly ¾ of an English word, while each Persian or Arabic word usually costs 1–3 tokens. Models see the world as streams of tokens.

Why it matters: the token is the currency and the memory unit of the model world. API pricing (input/output), context-window length and response speed are all measured in tokens. Persian and Arabic text costs more tokens than English because of prefixes and suffixes.

Rule of thumb: 1,000 tokens ≈ 750 English words. Counting tokens before sending long text — with free tokenizers — makes both cost and response quality predictable.

Back to glossary