The concept
Pretraining: the first and biggest stage of building an LLM. The model reads a huge collection of text (web pages, books, code) and practices one task: predict the next token (Day 16).
No one labels anything. The text itself is the answer key: hide the next word, guess, check, adjust. This is called self-supervised learning.
Each wrong guess nudges billions of weights a tiny bit (Day 10). Repeat across trillions of tokens and the model gets very good at continuing text.
Walkthrough: "The policy covers fire, theft and ___"
Step 1: guess. Early on, guesses are near random. Say: "banana 2%, flood 1%, the 1%..."
Step 2: check. The real text said "flood". The model gave it only 1%.
Step 3: adjust. Weights shift so "flood" gets a bit more probability in contexts like this.
Step 4: repeat billions of times on different sentences. Late in training the same blank might get: "flood 30%, water damage 20%, vandalism 15%, liability 10%..."
What it picks up along the way, without being told:
- Grammar: "covers fire, theft and" needs a noun next.
- Facts: what home and auto policies usually include.
- Reasoning patterns: "damage $2,000, deductible $500, so the payout is..." shows up often enough to be learned.
Two things pretraining does not give you
1. Fresh knowledge. The model only knows text up to its training data's knowledge cutoff, the date the data collection stopped. Anything newer (last month's policy change, today's rates) it hasn't seen unless you supply it.
2. A helpful assistant. A pretrained "base" model continues text; it doesn't follow instructions. Ask it "Write a claim denial letter" and it might continue with "...and a claim approval letter. Chapter 3:" because documents often look like that. Turning it into an assistant is the next stage (Day 19).
Where you see it at work
- Insurance: a model may know general policy language but not your company's endorsements. Those must come from your documents.
- Banking: rates, fees and regulations change; never assume the model knows the latest.
- Product management: ask vendors about the cutoff and how current data gets in.
- Business analysis: list which facts must come from your systems.
In one sentence
Pretraining teaches a model language, facts and patterns through billions of next-word guesses, but it stops at a cutoff date and isn't an assistant yet.