docs([issue-2791]): complete the 4B-vs-9B LoRA fidelity run — verdict: stay as-is - #3625
Merged
Conversation
The 4B/9B side-by-side that gates #2791 was started and paused at step 563/1200 of the 4B arm; the 9B arm never ran. Captures the method, the step-0 control, the measured 9.4 s/step throughput, and the interim observation that the shipped rank-32/lr-1e-4 defaults bind identity much harder than the older 400-step rank-16 adapter that looked cleaner but encoded the wrong subject. Explicitly records that neither #2791 change is justified yet, so a later reader does not mistake the partial data for the verdict. Refs #2791
…erdict Both arms (4B and 9B-8bit, matched params, 1200 steps) finished and were visually compared at steps 0/300/600/900/1200. Verdict: stay as-is — neither proposed default change is supported. 4B did not lose to 9B on identity (the opposite, if anything, though on a single seed/dataset), and step 600 was weaker on identity than step 900 in both arms, so cutting steps to ~600 would ship a worse result than the current default's own best checkpoint. Separately: both arms peaked at step 900 and regressed by step 1200, which doesn't call for a default change but does point users at the existing checkpoint picker rather than assuming the final step is best. Closes #2791
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #2791
Test plan
9f6fce7e…4B,151f3631…9B) ran to completion via the LoRA training API and are recorded with their run IDs/artifact paths in the doc.analyzeCaptionInvariants) and on-disk checkpoint timestamps, not just restated.TRAINING_DEFAULTSand the base-model picker inserver/services/loraTraining/runtimes.jsare unchanged.