fix: re-feed persistent prefill inputs on cached-graph recompute in QwenCausalDecodeRuntime - #331
Merged
Conversation
The prefill graphs of QwenCausalDecodeRuntime upload positions and the attention mask once at graph build time, but the graph allocator only exempts OUTPUT-flagged tensors from lifetime-based memory reuse. After the last read of the mask (the final layer), its memory can be handed to later intermediates, so recomputing a cached prefill graph reads a clobbered mask and produces NaN from the first attention layer onward. Keep host-side copies of both inputs and re-upload them before every run_prefill()/run_batched_prefill() compute, mirroring what the decode step already does for its inputs. First-run outputs are unchanged; recomputes (warmup, repeated same-length prompts) now match them.
Owner
|
@drzsdrtfg I can't reproduce the issue for NeuTTS on CUDA, while on CPU the issue is real. That’s interesting. I need some time to understand this bug... neutts_cpu_1.wav |
Contributor
Author
|
Ok, |
Owner
|
@drzsdrtfg PR meged. Thanks! |
0xShug0
pushed a commit
that referenced
this pull request
Aug 29, 2026
* feat: add Soprano TTS as community model with GGUF packages * Delete PR_SOPRANO.md * fix: soprano_tts review issues; warm bench; measured perf docs - converter emits spec-matched combined.safetensors (backbone + folded decoder incl. ISTFT window); loader requires the window strictly - register soprano_warm_bench target; document ENGINE_BUILD_WARMBENCH - parse eos_bias from request options - streaming follows NeuTTS pattern: run_mode reports task mode, options parsed once in start_stream and reused by next_stream_event - accept bare spec-declared session/load option names (text_chunk_size, backbone_weight_type, decoder_weight_type) - add soprano warm bench and document measured CPU/Vulkan performance including F16/Q8_0 backbone storage types Note: the soprano warm bench --warmup/--iterations paths depend on the QwenCausalDecodeRuntime prefill re-feed fix (#331). --------- Co-authored-by: agent <agent@local>
cunba-ai
pushed a commit
to cunba-ai/audio.cpp
that referenced
this pull request
Aug 30, 2026
Merge of PR #25 (Release 0.7, 32 upstream commits). Three conflicts, each resolved by keeping both sides' semantics: qwen3_asr run() keeps fork emit_progress alongside upstream log_chunk_word_diagnostics; voxcpm2 keeps fork indexed-loop progress emission while adopting upstream's cross-chunk continuation_prompt voice continuity; CMakeLists keeps progress_callback_test (fork_regression) in place while dots_tts_vocoder_parity follows upstream into the ENGINE_BUILD_MODEL_TESTS gate. Also brings Granite Speech 5.0, Echo-TTS, Soprano, VoxCPM v1, WAV reader format extensions, Qwen3 ASR timestamp/clamp + attention-mask fixes, cached-graph prefill re-feed (0xShug0#331), MOSS shared runtime, CLI unknown-option failure, server per-model contract resolution. Fork deltas verified: 9/9 fork_regression green, loader catalog in sync, embedded-VAD helper intact. Generated-by: zcode
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Validation (native CPU + Vulkan; no NVIDIA GPU available)
NeuTTS (
prefill_tokens, token-prompt, persistent runtime - recompute reachable)Two identical utterances in one session (batch-text-file x2):
NeuTTS applies an intentional per-chunk seed offset for utterance 2, so the two
lines legitimately differ from each other; the point is the pre-fix second line
is 12.2s of garbage while post-fix it is healthy and reproducible.
DotTTS (
prefill_embeddings, persistent runtime)Two identical lines,
template_name=tts, fixed seed. Pre-fix and post-fix arebyte-identical and healthy in both lines (278A7D17...): the DotTTS path does
not trigger the gallocr reuse, confirming the fix is a behavioral no-op where
the corruption cannot occur.
Soprano (surfaced the bug)
PR #323 branch (which carried this fix until the split):
--warmup 1 --iterations 3passes on CPU and Vulkan; three identical seeded generationsare bit-identical (previously fatal
no finite logits). Soprano is not inupstream yet, so it was validated on the PR branch carrying the same fix code.
Compile matrix
Custom build including all runtime consumers (neutts, dots_tts, fireredtts3,
midashenglm_gen, minimax_music3, qwen3_asr) links cleanly - the header is
unchanged, so no API breakage.
Consumers and why the rest are low-risk
mentions QwenCausalDecodeRuntime in a comment. Removed from the affected list.
creates a fresh runtime (graph built once, not recomputed) - the re-feed is a
no-op for them. miniMax-H3 needs 15GB+ for a runtime test (skipped).
decode step already does every token; it is backend-agnostic.
Caveats
loop pre-fix (graph branch reused); NeuTTS two-line and Soprano
warmup/iterations are the discriminating repeat proofs.
ggml_backend_tensor_setbefore compute, so no backend-specific surprisesare expected, but CUDA CI coverage would be valuable.