1. Essential Concepts: Understanding Them Through Things Developers Already Know¶
This chapter has no math. The goal is that whenever a code cell runs later, you know what the numbers on screen mean. Each concept comes with an analogy from everyday development (databases, Git, web services, and so on); the analogies only help build intuition and are not rigorous definitions.
Chapter at a glance
- What this chapter does: starting from "a model is a directory on disk", it covers tokens, inference, training and loss, SFT/LoRA/QLoRA, steps and epochs, the division of labor among seven libraries, and VRAM
- What you need: nothing to install; any computer that can open a web page
- What you get: you can match every parameter and number that appears in the notebook later; when you hit an error, you know which library to look at
The 8 sections follow the order "first there is a model, then how it reads text, how it answers, how it learns, how to save VRAM"; each section builds on the previous one, so read them in order.
1.1 A Model on Disk Is a Directory¶
Open models/qwen3-4b/: physically, a "model" is just these files. Note: the directory you download on your own computer with --tokenizer-only (section 2.8) contains only the tokenizer and config, about 15 MB, and not the three weight shards; the shards appear only after the full download on the GPU host (section 3.4). This is normal, not an interrupted download.
| File | What it is | Analogy |
|---|---|---|
config.json |
Structure definition: 36 layers, hidden size 2560, vocabulary of 151,936, etc. | A database's table-creation statements (schema) |
model-00001-of-00003.safetensors to model-00003-of-00003.safetensors |
The weights, about 4 billion numbers, 8.0 GB in BF16, split into 3 shards; only present after a full download | A database's data files |
model.safetensors.index.json |
Which weight lives in which shard | A shard routing table |
tokenizer.json, vocab.json, merges.txt |
The tokenizer: rules for text ↔ integer IDs | A serialization protocol |
tokenizer_config.json |
Contains chat_template: how system/user/assistant are assembled into one piece of text |
A request message format |
generation_config.json |
Default sampling parameters | Server-side default configuration |
model-lock.json |
The version (commit) recorded by this tutorial | package-lock.json |
"Loading a model" means building the empty structure from config.json, then filling in the numbers from safetensors and placing them in VRAM. Later you will meet three kinds of directories: the base model (the one above), the adapter (only the LoRA increment, about 130 MB, which must be paired with the base model), and the merged model (the complete directory after combining the base and the increment, 8 GB again). vLLM loads either a base model or a merged model directory.
1.2 Tokens: Text as the Model Sees It¶
A model doesn't read characters; it reads integers. The tokenizer cuts text into pieces (tokens), and each piece maps to an ID in the vocabulary. Here is a sentence cut by this project's tokenizer:
"为什么白天亮、晚上黑？"
→ ['为什么', '白天', '亮', '、', '晚上', '黑', '？']
→ [100678, 106772, 99538, 5373, 104030, 56652, 11319]
"The Earth rotates."
→ ['The', ' Earth', ' rotates', '.']
→ [785, 9237, 89059, 13]
(The Chinese sentence is "Why is it bright in the day and dark at night?")
Common words are a single token, while rare characters may be split into several; in English the space attaches to the following word. So the token count is not the character count; for Chinese, roughly 1 token ≈ 1.3–1.5 characters. Every "length limit of 4096" in this tutorial refers to tokens.
The vocabulary also contains a few special tokens that don't correspond to any text but act as structural markers. Qwen's conversation format (chat_template) assembles a request like this:
<|im_start|>system
你是陪小学生探索科学的伙伴。……<|im_end|>
<|im_start|>user
为什么白天亮、晚上黑？<|im_end|>
<|im_start|>assistant
地球在不停转，我们住的地方转向太阳就是白天……<|im_end|>
(System: "You are a companion who explores science with primary-school children…"; user: "Why is it bright in the day and dark at night?"; assistant: "The Earth keeps spinning; when the place where we live turns toward the Sun, it is daytime…")
<|im_end|> (ID 151645) is the end-of-sequence marker, EOS: the model stops when it generates it. During training the model must learn to output it at the end of each answer; otherwise, after deployment, its answers won't stop.
1.3 Parameters and Inference: A Function with 4 Billion Coefficients¶
At its core a model is a function: input a sequence of token IDs, output "the probability that the next token is each entry in the vocabulary". This function has about 4 billion adjustable coefficients, called parameters (or weights): the numbers in safetensors.
Inference is calling this function repeatedly: give it the prompt, take the most probable token (greedy decoding, do_sample=False), append it to the end of the input, and call again, until it outputs EOS or reaches max_new_tokens. Inference doesn't change a single parameter.
temperature=0 behaves like greedy decoding, always picking the most probable token, so the same input gives the same output, which is handy for comparisons; above 0 it samples randomly according to the probabilities, giving more varied but non-reproducible answers.
1.4 Training: Correcting Parameters Using the Error¶
Training works the other way around: show the model "context + reference answer", have it predict the next token at every position of the reference answer, and turn the gap between prediction and reference into a single number called the loss.
| Approximate loss | Meaning |
|---|---|
| About 3 or more | Every token of the reference answer "surprises" the model; random guessing would be ln(151936) ≈ 11.9 |
| 1–2 | The typical starting point of the base model with a good prompt on this project's reference answers |
| Around 0.5 | The model is already very close to the wording of the reference answers |
| Close to 0 | It has nearly memorized them word for word; if the validation loss is rising at the same time, that's overfitting |
Loss measures "how surprised the model is by the reference answer", not "a science knowledge score". It can only tell you whether the model is learning how the training data says things, not whether its answers are correct.
With the loss, you can compute for each parameter "which direction to nudge it to make the loss smaller": that is the gradient (computed by backpropagation). The optimizer (AdamW) updates the parameters according to the gradients, and the learning rate (1e-4) decides how large each step is: too large and it washes out the base model's abilities, too small and it doesn't learn. One round of "compute loss → compute gradients → update parameters" is called a step.
Analogy: the loss is like the number of failing CI tests, the gradient is like being told which way to turn each config setting, the learning rate is how much you dare to change at a time, and a step is like one commit.
1.5 SFT, LoRA, QLoRA: Three Independent Choices¶
The three terms answer three separate questions and can be combined independently:
SFT (supervised fine-tuning) answers "what to teach with". Show the model paired "input → expected output" examples, i.e. the conversation data from chapter 2. This project computes the loss only at the positions of "reference answer + EOS", not on the context (you will see this mask with your own eyes in section 2.5).
LoRA answers "which parameters to change". Full fine-tuning has to update 4 billion parameters, each with its own gradient and optimizer state, which doesn't fit on a single GPU, and every experimental version would need 8 GB of storage. LoRA leaves the original weights W completely untouched and adds two small matrices A and B beside them, training only those:
(That is: effective weights = W (frozen) + (alpha / r) × B × A.)
A Git analogy: the base model is the main branch, LoRA is a very small diff, training only changes the diff, and the adapter/ directory stores exactly that diff. "Merging" is like applying the diff to the main branch to produce a new complete copy of the files.
r (the rank) sets the width of the small matrices, i.e. the capacity of the diff. This project uses r=16; across 7 kinds of linear layers × 36 layers that adds up to about 33 million trainable parameters, 0.8% of the total, and an adapter file of about 130 MB. alpha is a scaling factor; this project always uses alpha = 2 × r.
QLoRA answers "how to fit the frozen base model into VRAM". Since the frozen weights never change anyway, they are compressed from BF16 (2 bytes per number) to 4 bits (0.5 bytes per number) in VRAM and temporarily restored to BF16 during computation. The 4B model's weights shrink from 8 GB to about 2.5 GB, which leaves a 24 GB card enough room for the intermediate results of training. NF4 is the name of the compression format, and double_quant means also compressing the auxiliary parameters used by the compression. LoRA's small matrices themselves stay in BF16, uncompressed.
Further reading: the original LoRA paper, the original QLoRA paper.
1.6 Samples, Batches, Steps, Epochs¶
| Concept | Value in this project | Meaning |
|---|---|---|
| Sample | 1150 | One "context + one reference answer"; a 2-turn conversation expands into 2 samples |
| Micro-batch | 1 | Number of samples placed in VRAM for one forward/backward pass |
| Gradient accumulation | 16 | Accumulate the gradients of 16 samples before updating the parameters once; the effect is close to batch=16 while VRAM holds only 1 sample |
| Step | 72 | Number of optimizer updates = ⌈1150 ÷ 16⌉; the x-axis of the progress bar and the loss curve |
| Epoch | 1 | One pass over all samples |
| Checkpoint | Every 25 steps | A snapshot you can resume training from: adapter + optimizer state + progress |
Analogy: a sample is one message, gradient accumulation is collecting 16 messages before a batch write to the database, a step is one database write, and an epoch is consuming the whole queue once.
1.7 What Each of the Seven Libraries Does¶
The training environment installs seven libraries, and their names will show up in error messages:
| Library | Responsible for | Where you'll see it |
|---|---|---|
| PyTorch | Tensor computation, GPU scheduling, automatic gradients; all the other libraries are built on it | torch.cuda.*, VRAM statistics |
| Transformers | Building the model structure from config.json, loading weights, the tokenizer, the chat template, generate() |
AutoModelForCausalLM, AutoTokenizer |
| PEFT | Adding LoRA layers to the model, freezing the original parameters, saving/loading the adapter, merging | LoraConfig, get_peft_model, PeftModel |
| TRL | A trainer tailored for SFT: computes loss according to the mask, manages steps/epochs/checkpoints | SFTConfig, SFTTrainer |
| bitsandbytes | The storage format and GPU compute kernels for 4-bit quantization | BitsAndBytesConfig, NVIDIA GPUs only |
| accelerate | Placing the model on the specified device, mixed precision; TRL depends on it internally | device_map |
| datasets | The data container the trainer requires | Dataset.from_list |
Deployment switches to another library: vLLM has its own inference engine and HTTP server and needs only the model directory, not PEFT or TRL from the seven libraries above.
How they call each other: Transformers loads the model → bitsandbytes compresses the weights to 4 bits → PEFT adds the LoRA layers → datasets holds the samples → TRL runs the training loop → PEFT saves the adapter → PEFT merges → vLLM deploys.
1.8 RAM, VRAM, and Disk¶
| Resource | Main use | Common misconception |
|---|---|---|
| CPU | Text processing, tokenization, loading and scheduling | More cores doesn't mean it's suitable for training large models |
| RAM (host memory) | Staging while loading weights, merging the model on the CPU | 100 GB of RAM can't replace 24 GB of VRAM: training computation happens on the GPU, so the data must be in VRAM |
| GPU | Massive matrix computation | The CUDA main path requires NVIDIA; Apple's MPS on the Mac is a different route that this tutorial doesn't support |
| VRAM | Weights, activations, gradients, optimizer state, the inference KV cache | An 8 GB model file doesn't mean 8 GB of VRAM: training adds intermediate results, inference adds the KV cache |
| Disk | Base model, environments, checkpoints, merged models | Every checkpoint includes optimizer state; every merge is another 8 GB |
The breakdown of training VRAM and actual readings are in section 4.3; the inference VRAM estimate is in section 5.3.
Acceptance check: You can say what each file in a model directory is; you can explain what a loss dropping from 2 to 0.6 means and why that alone doesn't show the answers got better; you can say why 1150 samples means 72 steps; and you can name which of the seven libraries handles LoRA and which handles the training loop.