8. Developer Reference: Automated Tests and Code Map¶
Readers who only do the exercises and don't change code can skip this chapter. If you want to change the data processing, the training code, or the tutorial itself, read this first: this project uses automated tests to tie "what the docs say" to "what the code actually does".
8.1 Automated Tests¶
After changing data-processing or training code, run the regression tests in the CPU environment. This environment includes PyTorch, TRL, and PEFT, for verifying the training mechanics on a tiny randomly initialized model; you can skip it for the data exercises in chapter 2.
python3.11 -m venv .venv-cpu
source .venv-cpu/bin/activate
python -m pip install -r requirements-cpu.txt
python -m pytest -q
The tests need the complete project source including tests/; the slimmed-down GPU upload bundle from chapter 3 does not include the developer tests. In the complete project you can also run the tests with .venv-train. The tiny-model tests don't need any weights downloaded.
| Layer | Test/command | What it proves |
|---|---|---|
| Data | test_data.py, test_prepare.py, test_bundle.py, test_notebook.py |
Single-file loading, draft gate, group and future-round isolation, non-empty targets, snapshot compatibility |
| Template | test_render.py, lab.audit |
Target span, token boundaries, EOS, overflow, real collator padding |
| Tiny training | test_training.py, test_notebook.py |
The CLI training components and the Notebook's actual code can update, save, reload, and merge LoRA |
| Evaluation | test_eval.py |
Pairing, missing scores, ties, conversation-clustered statistics |
| Docs and code | test_tutorial.py, test_book.py |
Tutorial commands are accepted by the real CLI; deployment examples and the appendix match the source; pages and the download bundle work |
| API examples | test_tutorial.py |
Runs the book's call examples directly, checking model name, address, parameters, and multi-turn history passing |
| Download | test_download.py |
Every download source uses the same revision; missing/corrupted weights are rejected |
| GPU model | Commands from chapters 4 and 5 | Real QLoRA training/resume, adapter and merged output |
| vLLM integration | /health, /v1/models, the Python examples in chapter 5, vllm bench serve |
The real service is callable; single-turn/multi-turn/streaming calls and concurrent load tests complete |
The API example tests use a temporary local HTTP server and cannot serve as evidence of vLLM inference. GPU and service checks run in their respective environments; items not run stay unchecked in todo.
Keep the following acceptance evidence: raw/expanded data and review records; token reports and test output; model SHA, dependencies, driver, and training config; checkpoints, adapter, and resume steps; A/B/C raw outputs and scores; merge manifest, inference environment versions, API responses.
8.2 Code Reading Map and Rationale¶
For the main path, read the Notebook first; look up the supporting modules by responsibility:
| Where in the tutorial | Corresponding code | Purpose |
|---|---|---|
| Section 2.8, chapter 3 | download.py, download-lock.json | Download a pinned revision and check integrity |
| Chapter 2 | dataset.json, data.py, prepare.py, common.py | Validate facts and conversations, build prompts, generate tokenized samples |
| Chapters 2 and 6 | audit.py | Check the real collator's supervision span and padding |
| Chapter 4 | 01_gpu_train.en.ipynb | 1-epoch training, loss curve, artifact checks, reloaded answers, and merge |
| Section 4.6, chapter 6, appendix | train.py, training.py | Command-line training, formal parameter comparison, and resume from checkpoint |
| Chapter 6 | infer.py, evaluate.py | A/B/C inference, blind-review statistics, and candidate freezing |
| Chapter 5 | merge.py, the last two sections of the Notebook | Standalone export, vLLM deployment and adapter mounting, API calls, load testing |
The Notebook shows the training details; the CLI is kept for repeated experiments and resuming, and each entry point has its own chapters. The API examples call HTTP or the OpenAI SDK directly, and load testing uses vLLM's built-in vllm bench serve, so no separate client is maintained.
| Decision | Reason and source |
|---|---|
| Fixed text-only 4B | Compare experiments on the same base model; Qwen model card |
| Explicit completion mask | The supervision span can be inspected; TRL 0.24 |
| NF4+LoRA | Train a small number of parameters on a single GPU; PEFT quantization |
| Separate inference environment | Avoid dependencies overwriting each other; vLLM GPU installation |
| Train facts and expression separately | This project's experimental design; the gain is validated by comparison |
Fact cards record links to scientific sources. When reviewing, check each claim, its conditions of applicability, and the limits of each analogy one by one, paying special attention to sources still marked pending; a web page being reachable does not mean it supports every generated conversation.