Skip to content

Search across models, tools, services, companies, providers, APIs, plans, articles and news.

Training & tuning

Pre-training

Pre-training

Pre-training

The costly initial stage where the model trains on vast public text and acquires general language and knowledge — after it, the model is still a generalist.

Pre-training is the heavy, costly stage of building an LLM: the model absorbs trillions of tokens of public text (web, books, code) until language structure and general knowledge settle into its parameters. The output is a raw generalist — smart but unspecialized and occasionally unpolite.

This stage’s price is the main reason the industry fixates on open models and distillation: building a frontier model costs millions in GPU time, so re-running it is rarely rational — you reuse what exists.

Key point: pre-training builds capability, not persona. Turning that capability into a useful assistant is the job of the next stages: fine-tuning and RLHF.

Back to glossary