A token is the small text unit a model reads and generates — neither exactly a word nor a letter. Roughly ¾ of an English word, while each Persian or Arabic word usually costs 1–3 tokens. Models see the world as streams of tokens.
Why it matters: the token is the currency and the memory unit of the model world. API pricing (input/output), context-window length and response speed are all measured in tokens. Persian and Arabic text costs more tokens than English because of prefixes and suffixes.
Rule of thumb: 1,000 tokens ≈ 750 English words. Counting tokens before sending long text — with free tokenizers — makes both cost and response quality predictable.
