6. Evaluation: Does Fine-Tuning Actually Beat Prompting?¶
This chapter answers a very practical engineering question: how much better is spending GPU time on fine-tuning than just writing a good prompt? You can run this exploratory comparison directly on the draft data to practice the experimental method; before reporting conclusions to anyone else, finish reviewing the data first.
Chapter at a glance
- What you do: have three groups of models (base model + simple prompt, base model + good prompt and examples, LoRA) answer the same set of validation questions; compare two ranks; score with a blind review; touch the test set only after a candidate has been chosen
- What you need: the
.venv-trainand base model from chapter 3; a few hours of continuous GPU time; at least one rater (ideally not the person who trained the model) - What you get: an experiment record with raw outputs, scores, and uncertainty intervals that can answer "is fine-tuning worth it?"
- When to do it: after the main path of chapters 4–5 runs end to end. This chapter takes roughly 10–20 hours of GPU time (including several inference passes and two training runs), plus separate time for human scoring
Why not just look at the loss? Loss only measures "how closely it resembles the reference answer". The reference answers themselves may be wordy or contain mistakes and gaps, and the base model with a good prompt may already speak well enough. To judge whether fine-tuning adds value, you must take topics it was never trained on, have "good prompt" and "good prompt + LoRA" answer exactly the same inputs, and then have a person score them without knowing which is which. This is the same thing as running an A/B experiment on a backend: control your variables, test blind, and look at confidence intervals.
The chapter has two halves: 6.1–6.4 prepare the data and generate the three groups of answers, and 6.5–6.8 score, freeze, and draw conclusions. All commands run from the project root on the GPU host.
6.1 Current Data Size and Review Requirements¶
There are already 500/50/50 episodes, so there is no need to expand to the size of the old version again. Each category has 100/10/10 episodes, with 60 fact cards in total. "formal" is the fixed quota for this teaching experiment, not a general minimum requirement for LoRA.
All variants of the same topic share one group_id, and fact references must not cross splits either. The validation and test sets each have only 5 independent topics, so even though each has 50 episodes, they cannot be treated as 50 unrelated knowledge tests.
What counts as review. A formal review checks every episode: does the source support the facts, is there excessive anthropomorphism, can a child understand it, are the limits of each analogy clear, is the observation safe and feasible, and does it respect a child who doesn't want to be asked questions or wants to stop? Review happens directly in dataset.json, with version control keeping the history. Only entries that have truly been reviewed get review.status changed to approved, with the actual reviewer, reviewer_kind, and notes filled in; do not bulk-change the status to pass it off as review. Programs can check structure, but not near-duplicates, template-like phrasing, or scientific errors: the 10 path types deliberately repeat teaching behaviors, the same topic also reuses some answers, and 50 train topics are not 500 independent topics; val/test are isolated from the training topics, but still share basic concepts such as light, force, and living things.
With --allow-draft, draft data is enough to complete the validation-set comparison in sections 6.1–6.6 of this chapter, and the output manifest records draft: true. Freezing a candidate and the test set (section 6.7) still require reviewed data, so that draft results are not mistaken for formal conclusions.
6.2 How to Keep Improving the Data¶
First read all 10 paths for each topic and check whether each piece of feedback from the child causes the corresponding change. If unexplained terminology still appears for lower grades, rewrite it; the same analogy must not repeatedly stand in for a concrete explanation; and one observation cannot prove every conclusion unrelated to it.
Use the validation set to find failure types, then add training demonstrations for them; avoid changing the data based on test answers. Group new topics first, then assign them to splits. The current ten path types are structured synthetic data, not real interactions with children; if you are able to, collect anonymous feedback with informed guardian consent, and separately evaluate understanding and voluntary exploration rather than making "more rounds of chat" the only goal.
6.3 Validation and Preparation¶
Open a terminal in the project root on the GPU host and first activate the training environment installed in chapter 3; the following commands need PyTorch and TRL:
source .venv-train/bin/activate
python -m lab.data \
--input data/dataset.json --allow-draft --formal
python -m lab.prepare \
--input data/dataset.json \
--tokenizer models/qwen3-4b --allow-draft --formal --output data/processed/experiment-v1
python -m lab.audit --data data/processed/experiment-v1 \
--output runs/formal-token-audit.json
--formal checks the 500/50/50 episodes, the 60 fact cards, and the 100/10/10 quota per category; --allow-draft allows unreviewed entries. The two are independent checks: after a real review, drop --allow-draft and you get the formal data version (use a different output directory, for example experiment-v2).
Automatic checks: quotas, review status, source IDs, overlap of complete conversation groups, identical user conversations, control characters, non-empty targets, and number of questions asked. Near-duplicates, semantic errors, unsupported claims, and so on still need review; a program cannot judge whether a fact is true from a URL alone.
6.4 Three Baselines and Two Ranks¶
| Group | Weights | Prompt and examples |
|---|---|---|
| A | Base model | Basic prompt, no examples |
| B | Base model | Full role prompt + 3 few-shot examples selected from the training set |
| C | Base model + this LoRA | Exactly the same prompt and 3 few-shot examples as B |
The 3 few-shot examples for B and C are picked by lab.prepare, one episode each from near the beginning, middle, and end of train (covering all three grade bands where possible), taking only the first question-answer round, and written into the snapshot's fewshots.json. Few-shot examples are used only at inference time; SFT training samples do not include examples, to avoid pasting a sample's own answer into its input ahead of time.
All three groups use the same fact cards for the current question, NF4 precision, and greedy decoding. By default they read val, and the reference answers are used only for later review; fixed-history mode uses the previous rounds' reference replies as context. The B/C setup in this section and the Notebook's three no-few-shot observation questions are different protocols; do not mix their scores.
First run A and B on the validation set:
python -m lab.infer --data data/processed/experiment-v1 --variant A \
--output runs/val-A.jsonl
python -m lab.infer --data data/processed/experiment-v1 --variant B \
--output runs/val-B.jsonl
Train r=16, 1 epoch:
python -m lab.train --data data/processed/experiment-v1 \
--output runs/formal-r16 --allow-draft --epochs 1 --rank 16
python -m lab.infer --data data/processed/experiment-v1 --variant C \
--adapter runs/formal-r16/adapter --output runs/val-C-r16.jsonl
Change only the rank and its corresponding alpha, starting again from the base model:
python -m lab.train --data data/processed/experiment-v1 \
--output runs/formal-r8 --allow-draft --epochs 1 --rank 8
python -m lab.infer --data data/processed/experiment-v1 --variant C \
--adapter runs/formal-r8/adapter --output runs/val-C-r8.jsonl
The code always sets alpha=2×rank, so the scaling factor is 2 in both runs. Do not continue training r8 from the r16 adapter and then call it an independent comparison.
With 1150 training samples and an effective batch of ≈16, 1 epoch is about 72 optimizer updates, the same amount of training as the Notebook default. Group B already has a good prompt and few-shot examples, so it is perfectly normal for group C to show no clear advantage, and that is a valuable conclusion in itself: it means prompting is enough for this task.
Acceptance: both runs use identical data, prompts, generation parameters, and base model; you can compare validation results, training time, VRAM, and adapter size; do not pick the "better exploration companion" from training loss alone.
When you open a new terminal, first run source .venv-train/bin/activate, and make sure the data snapshot and candidate models from sections 6.3–6.4 have been generated.
6.5 Fixed History First, Then Continuous Conversation¶
Fixed history (the default, --mode fixed) uses the reference history from the data. The hash of each B/C input must be identical, which makes it good for pinpointing wording differences in a particular round.
Continuous conversation uses the model's own answers as the subsequent history:
python -m lab.infer --data data/processed/experiment-v1 --variant B \
--mode rollout --output runs/val-B-rollout.jsonl
python -m lab.infer --data data/processed/experiment-v1 --variant C \
--adapter runs/formal-r16/adapter --mode rollout --output runs/val-C-rollout.jsonl
Both groups use the same user script, but each model generates its own subsequent history, so later prompt hashes may differ. Some preset follow-up questions may not fit the generated answer; record such cases when scoring. A real person asking follow-ups based on the answers is a different kind of evaluation.
6.6 Human Blind Review¶
Full program: evaluate.py.
python -m lab.evaluate blind --base runs/val-B.jsonl \
--tuned runs/val-C-r16.jsonl --output runs/blind-val-r16
This generates ratings.jsonl, key.json, and a protocol. When scoring, look only at the ratings and the fact cards, not the key; reveal which side is LoRA only afterwards. You can ask someone who was not involved in training to score, to reduce the author's bias toward the candidate.
For each item, fill in winner as X/Y/tie and write a concrete reason in reason; fill in fact_claims_X/Y and supported_X/Y separately. If there are no checkable facts, fill in 0, and do not treat that as "100% correct". The issues field records problems such as factual errors, Chinese above the grade band or piled-up jargon, excessive questioning, ignoring preferences, wrong attribution, and pretending to see the child's observation.
| Dimension | 0 points | 1 point | 2 points |
|---|---|---|---|
| Facts | Key claims unsupported/wrong | Partly unclear or too broad in scope | All checkable claims supported, unknowns stated clearly |
| Grade-band fit | Beyond the child's level, piled-up jargon | Understandable, but explanation doesn't adjust to feedback | Concrete and concise, re-explains when not understood |
| Exploration value | Clichés, fabrication, or forced questions | Has a point of interest, but the next step is vague | Concrete, safe, optional observation or follow-up |
| Interaction | Ignores an explicit choice | Basically responsive but repetitive | Respects choices, good pacing |
| Evidence awareness | Treats guesses or analogies as facts | Has qualifiers, but not enough | Distinguishes observation, explanation, and analogy; admits insufficient information |
The rubric helps you decide the winner, and per-dimension scores can go in reason; it is not automatic keyword scoring. An answer with a serious factual error should not win on fun alone, even if it is more "fun".
python -m lab.evaluate score --ratings runs/blind-val-r16/ratings.jsonl \
--key runs/blind-val-r16/key.json --output runs/val-r16-score.json
A win counts 1, a tie 0.5, a loss 0. The script bootstraps by group_id science topic, to avoid treating the ten paths and multiple rounds of the same topic as independent samples; it reports the number of rounds, conversations, and topics. With fewer than 2 topics it outputs no interval. Five validation/test topics each is still very few, and the intervals cannot support broad conclusions. Small-sample intervals are only exploratory results and cannot remove rater bias.
Fact support rate = claims supported by sources / checkable claims; the script does not verify these claims automatically. If not all fact-review fields are filled in, it marks factual_review_complete=false, and you must not treat a partial ratio as a complete quality conclusion.
6.7 Use the Test Set Only After Freezing¶
This section needs reviewed, formal data (rerun prepare from section 6.3 without --allow-draft); freeze and test-set inference reject draft data. Choose the model based on the validation set. The following uses r16 as an example; if r8 fits better, change the subsequent adapter arguments to that candidate throughout. The choice is an outcome of this learning experiment, not something the tutorial assumes in advance.
python -m lab.evaluate freeze --data data/processed/experiment-v1 \
--adapter runs/formal-r16/adapter --output runs/frozen-candidate.json
python -m lab.infer --data data/processed/experiment-v1 --split test --variant B \
--freeze runs/frozen-candidate.json --output runs/test-B.jsonl
python -m lab.infer --data data/processed/experiment-v1 --split test --variant C \
--adapter runs/formal-r16/adapter --freeze runs/frozen-candidate.json \
--output runs/test-C.jsonl
python -m lab.evaluate blind --base runs/test-B.jsonl \
--tuned runs/test-C.jsonl --output runs/blind-test
Run continuous conversation on the test set the same way, with --mode rollout and new output files. On test, using --limit to cherry-pick easy questions is forbidden. The freeze file records the data hash, the base model, a combined fingerprint of the adapter weights and config, the inference code, and the decoding parameters.
The test file is still local, so the experimenter needs to hold back the test questions until the model is frozen, to avoid writing training answers based on them. If you look at the test results and then keep tuning, the next round needs a new, independent test set.
6.8 How to Draw Conclusions¶
Write a one-page experiment record in this order: training configuration and cost → B/C comparison of facts and blind review → representative successes and failures → uncertainty → next steps.
Choose the next step based on the validation results: when the difference is uncertain, add independent evaluation; when a certain kind of behavior goes wrong, add matching training examples; when the LoRA gain is insufficient, keep the prompt baseline. Conclusions must consider both factual errors and stylistic preferences.
Adult preferences and model self-evaluation cannot directly prove that children are more willing to explore. The real goal also requires observing whether a child understands the core reason, whether they spontaneously ask related new questions, and whether they are willing to choose a safe observation, while respecting a refusal to participate.
Acceptance: save the complete scores, raw outputs, and failure classifications, so that others can re-check what input each comparison used.