7. Cost, Troubleshooting, and Acceptance¶
You don't need to read this chapter in order: before starting the machine, read 7.1 to estimate the budget; when you hit an error, look in 7.2; when everything is done, check against 7.3.
7.1 Measure Speed First, Then Estimate Cost¶
First check the provider's hourly price, storage charges, and the billing rules after stopping an instance, then set a budget for this experiment. The calculation below uses an assumed price of US$0.74/hour; the actual price is whatever is quoted when you create the instance.
GPU 费用 = 小时单价 × 开机计费时长
总成本 = GPU 费用 + 存储 + 其他服务 + 数据制作/审核投入
训练时间估算 ≈ 预热后每个 optimizer step 耗时 × 预计更新步数
(In English: GPU cost = hourly price × billed running hours; total cost = GPU cost + storage + other services + data creation/review effort; estimated training time ≈ time per optimizer step after warm-up × expected number of update steps.)
The main path (environment setup, download, 1-epoch training, merge, deployment, load test) is expected to take 2–3 hours of machine time; adding the comparison experiments and review from chapter 6, it may take 10–20 hours. At the example price that is about US$1.5–2.2 and US$7.4–14.8 respectively. In practice you also need to account for setup, transfer, idle, and storage costs.
The Notebook's training-complete.json (the CLI's metrics-step-N.json) records training time and peak allocated/reserved VRAM. PyTorch's statistics do not cover other processes or all driver overhead, so cross-check against nvidia-smi and your bill. Update the budget with the measured time.
7.2 Troubleshooting Table¶
| Symptom | Check first | What not to do |
|---|---|---|
| CUDA unavailable | Host, driver, Torch wheel, current virtual environment | Reinstall CUDA on a Mac over and over |
| Missing Python module | Python path, activated environment, install result | Randomly upgrade libraries in multiple environments |
| Tokenizer works but model fails | Whether only the tokenizer was downloaded, whether the weights are complete | Always assume insufficient VRAM |
| Length limit exceeded | Token length of the facts, history, and target | Silently truncate the reference answer |
| Low loss but poor results | Target leakage, near-duplicates, which adapter is actually loaded | Just add more epochs |
| Loss doesn't decrease | Mask, trainable parameters, learning rate, data | Immediately switch GPUs |
| Outputs thinking/role markers | Model variant, template, training target, stop token | Blindly strip special text in the backend |
| Repeats/doesn't stop | EOS supervision, duplicated data, overfitting, output limit | Only filter out repeated sentences |
| OOM | Other processes, length, batch, KV cache, quantization | Treat RAM as equivalent to VRAM |
| Resume rejected | Whether the data/code/environment/model changed | Bypass the manifest check |
| Makes up stories more | Whether fiction is written as fact, missing samples of honest responses | Use fun scores to cover factual errors |
| Merge degradation | Revision, unquantized base model, template, precision | Attribute all NF4/BF16 differences to LoRA |
| API timeout | Service health, model loading, context, concurrency, logs | Retry forever |
| API returns 400 | Whether input + max_tokens exceeds --max-model-len, whether the model name matches /v1/models |
Keep retrying the same request |
| Latency spikes during load test | Whether concurrency exceeds --max-num-seqs, whether the KV cache is full (check /metrics and logs) |
Look only at the average, not the queueing |
| vLLM fails to start | Inference environment, driver, model directory, GPU in use by other processes | Overwrite the training environment's dependencies |
7.3 Final Learning Checklist¶
- Can explain why this project uses SFT+QLoRA.
- Can show a sample's full context and its actual supervision target.
- Completed 1 epoch of training, can read the loss curve, checked the completion record, and saved and reloaded the adapter.
- Can name the components of VRAM during training and match them against
nvidia-smireadings. - Completed the small-sample overfitting and resume-from-checkpoint exercises in section 4.6.
- Completed 1 epoch and one rank comparison, and saved a reviewable checklist.
- Can distinguish in-training performance, validation results, and frozen test results.
- Can report a gain, a regression, or "uncertain", and explain the limits of the evidence.
- Can merge, export, start vLLM, and make multi-turn and streaming calls.
- Can estimate KV cache capacity and compare it with the vLLM startup log; can explain
--gpu-memory-utilization,--max-model-len, and--max-num-seqs. - Can compare the base model and LoRA in the same service by mounting an adapter, and explain its trade-offs versus merged deployment.
- Ran load tests at different concurrency levels, filled in the TTFT, TPOT, and throughput table, and can explain the trade-off between throughput and latency.
Record the gain or regression of LoRA relative to the prompt baseline, with data and outputs attached so others can re-check.