Books › AI Foundations: from zero
How a model is trained, and how to talk to it
The student analogy
Think of how a person becomes a helpful assistant. First they read widely and learn a lot about the world. Then they learn how to follow a boss's instructions. Then they get feedback on what makes an answer good or bad. Finally they go to work. A language model goes through almost the same four stages.
Stage 1: read a huge library (pre-training)
The model reads an enormous amount of text and plays one game, billions of times: guess the next word. Each time it is wrong, its numbers are nudged a little (Chapter 4). To get good at the game it has to learn grammar, facts, styles and even some reasoning.
The result is called a base model. It is knowledgeable but not yet a good helper. If you give it the start of a question, it may continue with more questions, like a page from an exam paper, instead of answering.
Stage 2: learn to follow requests (instruction tuning)
Next, people show the model many examples of a request and a good answer. "Summarise this." Here is a summary. "Translate this." Here is a translation. This teaches it the shape of a helpful reply. This step is often called instruction tuning or supervised fine-tuning.
Stage 3: learn what people prefer (feedback)
A model can follow instructions and still give answers that are unhelpful, rude or unsafe. So the next step uses human opinions. The model writes several answers to the same question. People rank them from best to worst. A scoring helper is trained from those rankings, and the model is then adjusted towards answers that score well. This method is called reinforcement learning from human feedback, or RLHF. A newer, simpler method called DPO learns directly from the ranked pairs without needing a separate scoring helper.
This stage is why chat assistants are polite, careful and usually try to be useful. It is often called alignment, because it nudges the model towards what people want.
Stage 4: help you (inference)
When you chat, the model is simply being used. This is called inference. Nothing new is learned. The model's numbers stay exactly the same as when training ended.
That difference matters.
- Training is costly and slow. It is done by the model's maker, and it changes the model's numbers.
- Using is quick and cheap by comparison. It does not change the model at all.
So when you type examples into a chat, you are not retraining the model. The examples only help in that one conversation, because they sit in the model's context window (Chapter 7). When the chat ends, they are gone.
Three ways to change what a model does
Suppose Asha wants her helper to know her bakery. She has three options.
| Option | What it does | Best for | Watch out for |
|---|---|---|---|
| Write good instructions (a prompt) | Tells the model what to do now | Style, format, rules, small examples | Lasts only for that request |
| Give it documents to read (RAG, Chapter 10) | Hands it facts at the moment of the question | Current, specific, checkable facts | The search must find the right pages |
| Fine-tune it (train a little more) | Changes the model's numbers | A consistent style, format or special skill | Costly, can go out of date, not good for facts that change |
Most teams try the first two before the third.
Prompting basics
A prompt is the instruction you give a model. Small changes can make a big difference. Five habits help.
1. Say who it is and what to do. "You are a friendly assistant for a small bakery. Answer customer questions." 2. Give the details it needs. Today's menu, the opening hours, the rules. The model cannot guess what it was never told. 3. Show an example. One or two samples of what a good answer looks like. This is called few-shot prompting. 4. Say the format. "Answer in two short sentences." "Reply as a list." "Give the answer as a table." 5. Try, look, improve. The first prompt is rarely the best. Change one thing at a time and compare.
Here are two prompts for the same job.
| Weak prompt | Better prompt |
|---|---|
| "Answer this customer." | "You are the helper for Asha's Bakery. Answer in two friendly sentences using only the menu below. If the menu does not say, reply: I will check with Asha and get back to you. Menu: (menu goes here). Customer: Do you have eggless cake?" |
The better prompt gives a role, limits, a fallback, the facts and the question. The result is more useful and more honest.
Back to Asha
Asha writes a clear prompt and gives her helper today's menu. It answers well. But then she notices a problem. If a customer asks about something that is not on the menu, the helper sometimes invents an answer. In the next chapter, we look at why.
Words explained
- Pre-training: the first, huge stage where the model learns by guessing the next word.
- Fine-tuning: training a model a little more for a special style or job.
- RLHF: adjusting a model using people's rankings of its answers.
- DPO: a simpler way to learn from ranked answers.
- Alignment: nudging a model towards what people want: helpful, honest and safe.
- Inference: using a trained model to answer. Nothing is learned.
- Prompt: the instruction and details you give a model.
The limits of this chapter
- Real training has many more steps and variations. This is the main shape.
- Feedback training makes a model more helpful, but it does not make it always right or always safe. It can still be tricked or wrong.
- The best choice between prompting, RAG and fine-tuning depends on the job. These are guidelines, not laws.
For engineers
- Pre-training minimises next-token cross-entropy on a very large corpus. Supervised fine-tuning (SFT) trains on instruction and response pairs. Preference tuning then optimises for human preferences, either with a learned reward model and a reinforcement-learning step (RLHF) or directly on preference pairs (DPO).
- Parameter-efficient fine-tuning, for example low-rank adapters (LoRA), trains a small number of added parameters instead of all the weights, which cuts cost.
- Placing examples in the prompt (in-context learning) changes behaviour for that call only. It does not update any weights. Persistent behaviour change needs training, which uses backpropagation and weight updates.
- Fine-tuning is usually better for style, format or tool-use behaviour than for injecting facts that change. For up-to-date or private knowledge, retrieval is usually the better first choice.
- Put stable instructions first and the changing question last. Many providers can reuse the work for an unchanged prompt start, which lowers cost and delay. See the Journal article on AI caching, untangled.
Check yourself
- What is the difference between training and inference? Answer: Training changes the model's numbers and is costly. Inference only uses the model and changes nothing.
- If you give a chatbot three examples in your message, has it learned them for good? Answer: No. They only help in that conversation, because they sit in the context window.
- Name two things a good prompt includes. Answer: For example a role, the facts it needs, an example, and the format you want.