Skip to content

Search across models, tools, services, companies, providers, APIs, plans, articles and news.

Agents & usage

Inference

Inference

Inference

Running the trained model — the moment you send a prompt and tokens are generated; the per-message cost of ChatGPT is inference cost.

Inference is running the trained model: the moment a prompt arrives and tokens are produced one by one. The entire operations economy of AI lives here — every message, every API call, every agent-minute.

Pricing is two-dimensional: input tokens and output tokens (newer models also bill cached memory). Reasoning models silently bill hidden thinking too — a short answer can trail 2,000 tokens of hidden chain.

The reduction playbook: small models for simple work, caching for repeated answers, batching for volume, quantization for deployment. Smart inference is as consequential as model-building.

Back to glossary