Skip to content

docs([issue-2791]): complete the 4B-vs-9B LoRA fidelity run — verdict: stay as-is - #3625

Merged
atomantic merged 3 commits into
mainfrom
next/issue-2791
Aug 8, 2026
Merged

docs([issue-2791]): complete the 4B-vs-9B LoRA fidelity run — verdict: stay as-is#3625
atomantic merged 3 commits into
mainfrom
next/issue-2791

Conversation

@atomantic

Copy link
Copy Markdown
Owner

Summary

  • Resumes and completes the empirical fidelity run Re-tune LoRA-training defaults (4B base model + steps) after a fidelity run #2791 required before changing LoRA-training defaults: two matched runs (4B vs 9B-8bit base model, 1200 steps each, identical rank/lr/resolution/seed) trained to completion and compared at steps 0/300/600/900/1200.
  • Verdict: stay as-is. Neither proposed change (defaulting single-character LoRAs to 4B, or cutting steps from 1200 to ~600) is supported by the data. Surprisingly, 4B did not lose to 9B on identity in this run — but that rests on a single seed/dataset and needs replication before it should move a default every install inherits.
  • Separately documents an unplanned finding: both arms peaked at step 900 and regressed by step 1200, pointing users at the existing checkpoint picker rather than assuming the final step is best.
  • A local review pass caught and fixed two factual errors carried over from the original (paused) draft: a caption-leak claim that doesn't hold up against the real validation gate, and a throughput figure recomputed from checkpoint timestamps (corrected ~9.4 s/step → ~6.2 s/step for 4B, ~1.6× → ~2.5× vs 9B).

Closes #2791

Test plan

  • Both training runs (9f6fce7e… 4B, 151f3631… 9B) ran to completion via the LoRA training API and are recorded with their run IDs/artifact paths in the doc.
  • All ten preview-grid sample images read and visually compared directly.
  • Caption-leak and throughput claims re-verified against live code (analyzeCaptionInvariants) and on-disk checkpoint timestamps, not just restated.
  • Docs-only change — no code paths affected; TRAINING_DEFAULTS and the base-model picker in server/services/loraTraining/runtimes.js are unchanged.
  • Local Code Review gate run; a sub-agent adversarial review found 3 issues (all fixed and re-verified clean).

The 4B/9B side-by-side that gates #2791 was started and paused at step
563/1200 of the 4B arm; the 9B arm never ran. Captures the method, the
step-0 control, the measured 9.4 s/step throughput, and the interim
observation that the shipped rank-32/lr-1e-4 defaults bind identity much
harder than the older 400-step rank-16 adapter that looked cleaner but
encoded the wrong subject.

Explicitly records that neither #2791 change is justified yet, so a later
reader does not mistake the partial data for the verdict.

Refs #2791
Run A (4B) restarted from checkpoint 299 via the resume endpoint —
confirmed training normally from step 300 toward 1200. Run B (9B) still
needs to run before the #2791 gate is satisfied.

Refs #2791
…erdict

Both arms (4B and 9B-8bit, matched params, 1200 steps) finished and were
visually compared at steps 0/300/600/900/1200. Verdict: stay as-is —
neither proposed default change is supported. 4B did not lose to 9B on
identity (the opposite, if anything, though on a single seed/dataset),
and step 600 was weaker on identity than step 900 in both arms, so
cutting steps to ~600 would ship a worse result than the current
default's own best checkpoint. Separately: both arms peaked at step 900
and regressed by step 1200, which doesn't call for a default change but
does point users at the existing checkpoint picker rather than assuming
the final step is best.

Closes #2791
@atomantic
atomantic merged commit a6817be into main Aug 8, 2026
5 checks passed
@atomantic
atomantic deleted the next/issue-2791 branch August 8, 2026 03:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Re-tune LoRA-training defaults (4B base model + steps) after a fidelity run

1 participant