fix(video): route VideoMAE pretraining through its decoder and declare TimeSformer's clip input - #2174
Open
ooples wants to merge 8 commits into
Open
fix(video): route VideoMAE pretraining through its decoder and declare TimeSformer's clip input#2174ooples wants to merge 8 commits into
ooples wants to merge 8 commits into
Conversation
…e timesformer's clip input VideoMAE pretraining (PretrainMAE) could not run correctly: - DecodeForReconstruction started at hard-coded Layers[15], which is the classifier Dense; decoder blocks are 16-19 and the reconstruction head 20. Indices now come from VideoMAELayerLayout, which the layer factory also uses for its block counts (same values, so the built stack is unchanged). - CreateTubeMask sized numMasked from numTubelets * spatialPatches but sampled spatial indices only, so any clip with 2+ tubelets had every patch masked. The ratio now applies to the spatial patch count (paper tube masking), on the clip's own patch grid. - ComputeReconstructionLoss indexed the clip with the head's B*numTubelets axis (IndexOutOfRange for 2+ tubelets) and compared against frame-averaged single pixels. It now compares each masked tubelet patch's predicted pixels with that patch's true pixels (raw-pixel MSE over masked patches). - The constructors ignored Architecture.InputFrames. A declared frame count now wins (as in BSVD); an explicit conflicting numFrames throws. TimeSformer's parameterless constructor declared InputType.ThreeDimensional (a single [3,224,224] frame). It now declares FourDimensional with inputFrames = 8, the model's default numFrames, so GetInputShape() is [8,3,224,224] and Predict accepts it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Deployment failed for project aidotnet_website with the following error: Learn More: https://vercel.com/franklins-projects-02a0b5a0?upgradeToPro=build-rate-limit |
|
The latest updates on your projects. Learn more about Vercel for GitHub.
1 Skipped Deployment
|
Contributor
|
Warning Review limit reachedNext included review available in 57 minutes. View limit detailsLimit details: You’ve used the included review currently available. You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Review configuration: ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Advanced Run ID: 📒 Files selected for processing (19)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
OpticalFlowBase.PredictCore rejected every rank-3 input ("Input must be
rank 4"). Thirteen flow models (DKM, DPFlow, FlowDiffuser, FlowFormer++,
MemFlow, NeuFlowV2, RoMa, RPKNet, SEA-RAFT, SKFlow, UFM, UniMatch,
VideoFlow) declare the stacked pair as InputType.ThreeDimensional with
InputDepth 6, so GetInputShape() is [6, H, W] and none of them could
predict on their own declared shape. RAPIDFlow hit the same check with
an unbatched pair.
PredictCore now promotes a rank-3 [2*C, H, W] pair with the shared
PromoteToBatchedTensor helper and squeezes the unit batch back off the
flow with a recorded reshape (unbatched in, unbatched out, as in the base
PredictCore). The base's input and output [TensorLayout]s are marked
BatchOptional and OutputAxesFor answers rank 3, so the declared contract
matches. RAFT overrides PredictCore and inherits that layout, so it gets
the same promotion; it used to index Shape[3] and throw IndexOutOfRange.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…clip softmax TimeSformer's constructor took the clip length from numFrames (default 8) and ignored Architecture.InputFrames, so a 16-frame architecture built an 8-frame model: NumFrames and metadata said 8, the positional table was sized for 8 frames, and the divided space-time blocks' plain Forward grouped tokens by 8. It now applies VideoMAE's rule: the architecture's count wins and a conflicting explicit numFrames throws. The rule moves to an internal VideoClipFrameCount helper that VideoMAE and TimeSformer share (VideoMAE's behaviour is unchanged). Two convention bugs in Classify: - An unbatched [T, C, H, W] clip returned [1, NumClasses]. Classify's own docs, the batch-optional output layout and the base PredictCore all say [NumClasses]; it now squeezes the batch axis the tokenizer added. - The softmax normalised over the whole [B, NumClasses] tensor, so a batch's probabilities summed to 1 in total rather than per clip. It now uses Engine.Softmax over the class axis. The native constructor's XML docs were attached to the parameterless constructor; they are moved to the constructor they describe. Default parameter count is unchanged (113,541,120). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…e videoclip's clip Each failed Predict on a tensor of its own declared input shape: - BasicVSR++ declares a single [3, H, W] frame and read it as a clip of three 2-D frames. EnhanceVideo now treats [C, H, W] as a one-frame clip (the super-resolution family's degenerate case) and returns [C, sH, sW]. - OpenSora declares a single [3, H, W] latent; PredictCore indexed axis 3 of its features. It now denoises it as a batch of one. - ProPainter declares a single [3, H, W] frame; the image path indexed axis 3. PredictCore now runs it as a one-frame clip. - VideoCLIP's input layout is [Frames, C, H, W] and it owns a clip length (numFrames, default 32), but its default architecture declared one [3, 224, 224] frame. It now declares FourDimensional with inputFrames 32 and takes numFrames from the architecture when declared (VideoClipFrameCount). Its layer stack runs per frame, so the lazy-shape walk now starts from one frame; otherwise the rank-5 clip left the stack unresolved and the default model reported 38.0M parameters instead of 211.1M at construction (this also affected any VideoCLIP built on a FourDimensional architecture before this change). RAFT, GMFlow, RAPIDFlow and RIFE are not changed here: their architectures document InputDepth as the per-frame channel count, so their declared shape is one frame while Predict needs a pair. Fixing that changes the meaning of a public constructor argument and needs a decision. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ise patch targets Four fidelity gaps in PretrainMAE against Tong et al. 2022: - It computed a reconstruction loss and returned it without back-propagating or updating a weight. It now records the masked forward and the loss on a GradientTape and steps the optimizer through BackwardAndStepOnPrecomputedLoss, the library's caller-owned-tape training path (NeRF, TVAE, TabDDPM, the GANs), using the constructor's optimizer or the default Adam. The loss is built from engine ops so it back-propagates; ComputeReconstructionLoss returns its value. - Masked patches were zeroed once, in place, at the embedding. The 3x3 encoder blocks then refilled them from visible neighbours and the bias, and the in-place write was invisible to the tape, so gradient still reached the patch embedding through masked patches' own pixels. The convolutional encoder cannot drop grid positions, so it is made sparse as in SparK / ConvNeXt V2: a recorded visibility mask is applied to the embedding and after every encoder block. - The decoder stacked a Tensor.Transform GELU after each ReLU conv block; Transform records no autodiff node, so the loss reached the head and nothing before it. Removed (the encoder had the same fix earlier). - The target was raw pixels. VideoMAEOptions.NormalizeTarget (default true, the paper's normlize_target) normalises each channel of each tubelet patch by its own mean and unbiased std (+1e-6). No parameters are added (default count unchanged, 65,788,816); the paper's learned mask token is not added, so masked positions enter the decoder as zeros. Clone and serialize keep NormalizeTarget. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
fix(video): declared-shape predict for 20 video models and videomae pretraining fidelity
|
Deployment failed for project aidotnet-playground-api with the following error: Learn More: https://vercel.com/franklins-projects-02a0b5a0?upgradeToPro=build-rate-limit |
This was referenced Sep 17, 2026
These write-ups should never have been committed. Removed here so the file does not arrive on master when this PR merges; .gitignore gains matching rules in #2224. Deliberately untouched: ci-proof/nonruntime-routing-canary.md, which is functional rather than a write-up (it exercises the permanent ci-proof/** workflow trigger). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GYQycGHKxLFBqhEtPMHz29
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Five real bugs in VideoMAE's pretraining path and TimeSformer's declared input, found while diagnosing a never-run test failure (the one-frame VideoMAE crash, fixed separately in the never-run-tests PR). VideoMAE's masked-autoencoder pretraining decoded with the classifier layer, masked 100% of the clip for any clip longer than one tubelet, and its loss crashed on every multi-tubelet clip. So
PretrainMAEcouldn't run on any realistic input. VideoMAE also ignored the frame count its architecture declared, and TimeSformer declared a one-frame input for a model built for 8-frame clips.Root causes and fixes
Line numbers are in the unfixed
masterfiles.DecodeForReconstructionstarted at hard-coded layer 15 (VideoMAE.cs:717). The factory (LayerHelper.cs:9356-9401) builds 21 layers: 0 patch-embed, 1-12 encoder, 13 reduce-conv, 14 pool, 15 classifier Dense, 16-19 decoder, 20 reconstruction head. Pretraining ran the classifier, three decoder blocks, and decoder block 4 as the "head"; the real head was never reached.[2,16,2,4]; correct is[2,1536,2,2]VideoMAELayerLayout.cs: one definition of the indices, read by the factory, the encoder loop, the classifier head and the decoder.CreateTubeMask(:638-645) sizednumMaskedfrom tubelets × spatial patches but sampled spatial indices only, so 2+ tubelets meant everything was masked.ComputeReconstructionLoss(:737-804) treated the head's B×tubelets axis as the clip batch: IndexOutOfRange on any clip with 2+ tubelets, and compared against frame-averaged single pixels instead of the patch's pixels. Found while fixing 1.PretrainMAEthrew on every multi-tubelet clipPatchEmbedfolds frames into channels; shapes validated instead of indices silently wrapped.:188,:233) set_numFrames = numFrameswithout readingArchitecture.InputFrames.ResolveNumFrames: the architecture's count wins (the rule BSVD, the only othersrc/Videomodel readingInputFrames, uses); an explicit conflictingnumFramesthrows. Limitation: an explicit 16 can't be told from the default, so it defers to the architecture.TimeSformer()(TimeSformer.cs:171) declaredThreeDimensional, soGetInputShape()was[3,224,224]: one frame. Predict ran, tokenizing it as a one-frame clip, so this was a wrong contract rather than a crash.[3,224,224]FourDimensional,inputFrames: 8→[8,3,224,224].Why
LayerHelper.cschanged (it's shared by many models): only two loop bounds insideCreateDefaultVideoMAELayerschanged,12→VideoMAELayerLayout.EncoderBlockCountand4→DecoderBlockCount, so the factory and the model read one definition. OnlyVideoMAE.cscalls that method (checked acrosssrcandtests); no otherLayerHelpermethod is touched. Both are compile-time constants with the same values, so the compiled method is unchanged. Parameter counts confirm it: default VideoMAE 65,788,816 and small config 161,348, identical before and after. As a check,LayerHelper,SlowFast,DepthAnythingV2andArchitectureSharingGuardtests ran: 164 passed, 0 failed, 3 skipped.Tests
New
VideoMAEPretrainingTestsandTimeSformerDeclaredInputTests(11). Against the unfixed source 9 fail:PretrainMAE, and an exact-value loss test (IndexOutOfRange)The other 2 are guards that should pass either way. All 11 pass after.
TracedChainValidationThe one
TracedChainValidation(VideoMAE)failure is the separate one-frame default-constructor crash, fixed in the never-run-tests PR; this branch deliberately keeps clear of those lines to avoid a conflict (nearest hunk 8 lines away). Clone keeps the parameter count andNumFrames; serialize still works; TimeSformer's parameter count is unchanged (113,541,120).What this does not do (being fixed in follow-up PRs, not left)
OpticalFlowBase.PredictCoredoesn't add the batch dimension.Architecture.InputFrames. It doesn't visibly fail, butNumFramesand metadata misreport the count.PretrainMAEcomputes a loss but never updates weights.🤖 Generated with Claude Code
https://claude.ai/code/session_017otuSGr3GdmvPoYLbiaWR2