0. Before You Start: What We Are Building and What You Need¶
This chapter involves no commands. When you finish it you should be able to answer four questions: what exactly we are building; how many steps the process has and which machine each step runs on; what you need to prepare in advance; and how the pieces fit together.
0.1 In One Sentence: "Onboarding" a General Model into a Specialized Service¶
We download an open-source general-purpose large model from the internet, Qwen3-4B (about 4 billion parameters, released by Alibaba's Tongyi Qianwen team). It can chat about anything, but we don't control how it talks: its answers may be too long, use words beyond a primary-school level, or keep asking questions back.
Next we fine-tune it with 600 demonstration conversations of the form "child's question → ideal answer", teaching it one fixed way of talking: answer first, use short sentences, correct a wrong guess first, and offer one small, safe observation. Finally we deploy the trained model as an HTTP service; any backend code that can call the OpenAI API can use it just by changing the address.
An analogy with hiring:
| Real-world hiring | What it corresponds to in this tutorial |
|---|---|
| Hire a new employee with broad knowledge | Download the open-source base model Qwen3-4B |
| Onboard them with a stack of "model reply examples" | SFT fine-tuning on demonstration conversations |
| The training lives in a small notebook and doesn't change what they already know | LoRA: the base model stays untouched; only an increment of about 130 MB is trained |
| After training, put them on the phones | Deploy the model as an API with vLLM |
| Spot-check their call quality and see how many calls they can handle a day | Test calls, blind review, load testing |
The single most important boundary: fine-tuning teaches "how to say it", not "what to know". The concrete scientific facts are put into the prompt by the caller on every request (this project calls them "fact cards"); the model's job is to explain them in a suitable way. If the base model already knows common facts, why do we still need fact cards, and why include them during training too? Section 2.2 answers exactly that.
0.2 The Whole Pipeline: Where Each Step Runs and What It Produces¶
① 读懂数据 dataset.json → 1150 条训练样本 （第 2 章，只读）
↓
② 准备 GPU 训练环境 .venv-train + 8 GB 基座模型 （第 3 章，GPU 终端）
↓
③ 训练 LoRA Notebook 跑 72 步 → 130 MB 的 adapter （第 4 章，JupyterLab）
↓
④ 合并导出 基座 + adapter → 8 GB 的完整模型 （第 4 章，JupyterLab）
↓
⑤ vLLM 部署 完整模型 → OpenAI 兼容的 HTTP 服务 （第 5 章，GPU 终端）
↓
⑥ 调用与压测 本地电脑经 SSH 隧道调用，测延迟和吞吐 （第 5 章，本地终端）
↓
⑦ 评估值不值 和“原模型 + 好提示词”盲评对照 （第 6 章，GPU 终端）
(Diagram: ① understand the data, dataset.json → 1150 training samples (chapter 2, read only) → ② prepare the GPU, .venv-train + 8 GB base model (chapter 3, GPU terminal) → ③ train LoRA, 72 steps in the notebook → 130 MB adapter (chapter 4, JupyterLab) → ④ merge and export, base + adapter → 8 GB full model (chapter 4, JupyterLab) → ⑤ deploy with vLLM as an OpenAI-compatible HTTP service (chapter 5, GPU terminal) → ⑥ call and load test from your computer through an SSH tunnel (chapter 5, local terminal) → ⑦ evaluate whether it was worth it, blind review against "base model + good prompt" (chapter 6, GPU terminal).)
| Step | Where | With what | Output | Size | Chapter |
|---|---|---|---|---|---|
| ① Understand the data | Any computer, just reading files | A browser or editor | Understanding how 600 conversations become 1150 training samples | — | Chapter 2 |
| ② Prepare the GPU | Terminal on the GPU host | pip, download script | The .venv-train training environment and the models/qwen3-4b/ base model |
Environment about 10 GB, base model 8 GB | Chapter 3 |
| ③ Train | JupyterLab on the GPU host | Run the notebook cell by cell | adapter/ (the LoRA increment), loss-curve.png |
About 130 MB | Chapter 4 |
| ④ Merge and export | The same notebook | PEFT | merged-model/, a complete model that runs on its own |
About 8 GB | Chapters 4 and 5 |
| ⑤ Deploy | Terminal on the GPU host | vLLM | An OpenAI-compatible service listening on port 8000 | About 20 GB of VRAM | Chapter 5 |
| ⑥ Call and load test | Your own computer → SSH tunnel → GPU | Python standard library / OpenAI SDK, vllm bench serve |
Model answers, latency and throughput numbers | — | Chapter 5 |
| ⑦ Evaluate | GPU host | Command-line scripts + human scoring | Evidence of "how much better LoRA is than just writing a prompt" | — | Chapter 6 |
Steps ①–⑥ are the main path, needing about 2–3 hours of GPU uptime. Step ⑦ answers "is fine-tuning worth it?"; it needs more GPU time and human scoring, and you can do it after the main path works.
0.3 Two Machines, Three Windows: How They Work Together¶
The whole process involves only two machines. The GPU host is rented by the hour and stopped when you're done; your own computer handles reading, packaging, and making calls.
你的电脑（Mac / Linux / Windows 均可）
├── 浏览器：读这本书
├── 浏览器：打开平台提供的 JupyterLab ──HTTPS──→ GPU 主机的 JupyterLab
├── 终端 A：打包项目，用平台上传功能传到 GPU 主机
├── 终端 B：SSH 隧道，本地 18000 端口 ──SSH──→ GPU 主机 8000 端口
└── 终端 C：运行 Python 调用示例，请求发往本地 18000
云上 GPU 主机（Linux + RTX 4090）
├── JupyterLab：运行 Notebook，训练、保存、合并
├── 终端 1：装环境、下载模型、启动 vLLM 服务（监听 127.0.0.1:8000）
└── 终端 2：nvidia-smi 看显存、运行压测
(Diagram: your computer (Mac / Linux / Windows) runs a browser for this book; a browser tab with the platform's JupyterLab, connected over HTTPS to JupyterLab on the GPU host; terminal A to package the project and upload it with the platform's upload feature; terminal B for the SSH tunnel from local port 18000 to port 8000 on the GPU host; and terminal C to run the Python call examples against local port 18000. The cloud GPU host (Linux + RTX 4090) runs JupyterLab for the notebook (train, save, merge); terminal 1 to install environments, download the model, and start the vLLM service (listening on 127.0.0.1:8000); and terminal 2 for nvidia-smi VRAM checks and load testing.)
| Component | What it is | Why you need it |
|---|---|---|
| GPU cloud host | A Linux server rented by the hour, with one NVIDIA GPU | Training and inference are massive matrix computations, and a CPU is too slow; the 4-bit quantization library used for training only supports NVIDIA |
| Platform image | The system template you pick at startup, with Ubuntu, Python 3.11, and the GPU driver preinstalled | Saves you installing the OS and driver; only the platform can provide the driver |
| JupyterLab | An interactive Python editor in the browser, usually bundled by the platform | Training code runs one cell at a time, you see each cell's output, and on an error you rerun only that cell |
Virtual environments .venv-* |
Private Python dependency directories for the project | The library versions needed for training and for deployment conflict; installing them separately keeps them from overwriting each other |
| Kernel | The Python process behind Jupyter that actually runs the code | You must pick the kernel registered by .venv-train so the notebook can use the project's dependencies; VRAM is only released when the kernel is shut down |
| SSH tunnel | An encrypted channel that forwards a local port to a remote port | The model service listens only on 127.0.0.1 of the GPU host and is not exposed to the internet; locally you call it through the tunnel as if it were on your own machine |
0.4 What to Prepare in Advance¶
| Item | Requirements | Notes |
|---|---|---|
| Skills | Comfortable with a Linux terminal, can read Python, know HTTP and JSON | You do not need math, PyTorch, or machine learning; chapter 1 explains the concepts you need from scratch |
| GPU cloud account | Any GPU cloud platform with pay-as-you-go billing, with a balance in your account | You need to be able to start an RTX 4090 (24 GB VRAM) instance, and the platform should offer JupyterLab and SSH login. The author uses Zhichuan Cloud (gpuhome.cc); entering the invite code zt4f01m0 at sign-up gets you a new-user voucher, see section 3.1 |
| Host configuration | 1 × RTX 4090, RAM ≥ 32 GB (≥ 64 GB recommended for merging), data disk ≥ 100 GB | Pick an Ubuntu 22 + Python 3.11 + CUDA 12.x image; details in section 3.1 |
| Budget | About 2–3 hours of uptime for the main path, plus 10–20 hours for the evaluation experiments | At Zhichuan Cloud's pay-as-you-go RTX 4090D rate of ¥1.25/hour, the main path costs about ¥3–4 and the evaluation experiments about ¥13–25; the estimation method is in section 7.1 |
| Your own computer | Internet access, a terminal, and an SSH client; install Python 3.11 if you want to run the checks or call examples locally | You do not need a local GPU, nor do you need to install CUDA yourself |
| Network | The GPU host can reach PyPI (or the platform's mirror) and ModelScope | Hosts in mainland China usually can't reach huggingface.co, so this tutorial downloads from ModelScope by default |
| This project's files | This repository, or the practice bundle from the e-book's "Source download" page | What you upload to the GPU is a slimmed-down bundle built with tools/build_gpu_zip.py, explained in section 3.3 |
| Time | 1–2 hours to read chapters 1–2; one continuous 2–3 hour GPU session | GPU time is billed by uptime, so read the first two chapters before starting the machine |
What you don't need: an OpenAI account or API key (the vLLM service uses the same interface format as OpenAI, but runs entirely on the machine you rent), a Hugging Face account (the model is public and downloaded from ModelScope), or your own training data (the project ships with a dataset).
0.5 Reading Path¶
The order of this book is the order of operations. Each chapter starts by stating what it does, what it needs, and what you get at the end.
| Chapter | Content | Where to read/do it | Required? |
|---|---|---|---|
| 0–1 | Goals, preparation, essential concepts | Read only | Required: chapter 1 is the dictionary for every number and parameter that follows |
| 2 | What the training data looks like and how it becomes training samples | Read only; optionally run the checks locally | Required: know what the model is actually learning |
| 3 | Rent a machine, install environments, download the model | GPU terminal | Required |
| 4 | Train in the notebook, watch the loss, compare answers, merge | GPU JupyterLab | Required |
| 5 | vLLM deployment, local calls, load testing | GPU terminal + local | Required |
| 6 | Comparison against "base model + good prompt", blind review | GPU terminal | After the main path works |
| 7 | Costs, troubleshooting table, final acceptance check | — | Look things up when you hit a problem; check against it at the end |
| 8, appendix | Automated tests, code map, full training code | — | Read when you want to change the code |
Readers who already know the concepts can start at chapter 3 and come back to chapter 1, or to the table under the corresponding notebook cell, whenever they meet an unfamiliar parameter.
0.6 The Practice Task: Exploring Science with Primary-School Children¶
Why not a common task such as customer support or code generation? Because "explaining science to primary-school children" has a distinctive answer style, so the change before and after fine-tuning is obvious at a glance; and scientific facts can be checked, which makes it easy to tell whether the model is making things up.
Why Science Exploration¶
When a child asks "why", what follows may be a wrong guess, not understanding, wanting to verify it hands-on, or just wanting the answer right away. This feedback lets us observe whether the model really adapts its explanation, instead of only comparing whether the wording sounds cute.
The goal: facts are correct, the child can understand, and the next step is worth exploring. Fun can come from a counterintuitive phenomenon, an analogy with clear limits, or a small safe observation; not every answer needs a story or an exaggerated tone. When the child says they're done, the conversation ends; longer conversations are not a measure of success.
Coverage¶
| Category | Data value | Example training topics |
|---|---|---|
| Astronomy and space | space |
Day and night, seasons, moon phases, meteors |
| Earth and weather | earth |
Clouds and rain, snowflakes, rocks, rivers |
| Life and nature | life |
Seeds, roots, butterflies, food webs |
| Everyday physics | physics |
Shadows, sound, buoyancy, dissolving |
| Engineering and tools | engineering |
Arch bridges, levers, airplanes, solar cells |
Each category has 100 training conversations, plus 10 for validation and 10 for testing. This is a limited set of teaching samples, not a complete knowledge base of the primary-school curriculum. grade_band is split into grades 1–2, 3–4, and 5–6; the grade band is a starting point for the explanation, actual feedback takes priority, and we cannot assume that children in the same grade understand things equally well.
How the Prompt and Fine-Tuning Divide the Work¶
A single request contains four things, each responsible for one job:
| Component | What it governs | When it takes effect |
|---|---|---|
| System prompt (role prompt) | The role and basic rules: "You are a companion who explores science with primary-school children…" | Sent with every request |
| Fact cards | The scientific material this question may rely on | Selected per question and sent with every request |
| SFT demonstration conversations | Demonstrate how to handle misconceptions, not understanding, mismatched observations, and changing preferences | Used only during training; the learned behavior is stored in the adapter |
| LoRA | Restricts updates to a small subset of parameters; the base model is untouched | Decides at training time which parameters change |
The project's 10 demonstration paths are: correct a misconception, explain it another way, pick up a specific follow-up question, an optional observation, test a guess, go from simple to deeper, state the limits of an analogy, reconcile differing observations, follow up without quizzing, and end naturally. Some answers are reused within the same topic, so don't read 500 conversations as 500 independent science questions.
We do not promise that "a prompt can never replace fine-tuning". Chapter 6 compares group B, the base model with the full prompt plus three examples, against LoRA group C, which uses exactly the same input. On science topics not seen in training, we check whether C responds more consistently to the child's feedback. If there is no improvement, we honestly keep B.
0.7 What Exists Today¶
Updated: 2026-10-10.
| Item | Current status |
|---|---|
| Single data source | data/dataset.json: 60 fact cards, 600 synthetic conversations (500 train / 50 val / 50 test) |
| Training samples | The 500 train conversations expand into 1150 answer targets; the notebook runs a full epoch by default (about 72 steps) |
| Notebook | Tokenization, masking, QLoRA, loss curve, save, reload, comparison, BF16 merge, and finally vLLM deployment and calls |
| Deployment | vLLM serving the merged model or mounting the adapter directly; calls via the standard library / OpenAI SDK; load testing with vllm bench serve |
| Local verification | Data checks and tiny CPU training-mechanism tests, see the verification record |
| Not yet verified on a real GPU | 4B training time and VRAM, the vLLM service, load-test numbers; trust what you measure on your GPU host |
0.8 The Limits of Our Conclusions (Stated Once for the Whole Book)¶
- All conversations are synthetic drafts (
review.status="draft") that have not been reviewed one by one by a human; the answers are on the short side and repetitive in structure. They are entirely sufficient for learning the workflow and engineering, but not for concluding that "LoRA is definitely better". - Being able to generate answers only shows that the files and service work; a falling loss only shows that the model is more like the reference answers. Answer quality has to be judged by the human comparison in section 4.5 and the blind review in chapter 6.
- Fine-tuning changes "how to say it"; it does not automatically give the model new knowledge or real-time information. Facts are still put into the prompt by the caller.
Acceptance check: You can draw the "data → training → merge → deploy → call" pipeline and say which machine each step runs on and which files it produces; you can list what to have ready before starting the machine; and you can point out, in a conversation, "what the knowledge is" and "what changed in response to feedback".