Add VoxCPM v1 — lightweight VoxCPM TTS support (0.5B) by jasonchen31 · Pull Request #256 · 0xShug0/audio.cpp

jasonchen31 · 2026-08-16T15:04:56Z

1. Overview

Adds support for the OpenBMB VoxCPM-0.5B lightweight TTS model to audio.cpp (ported from VoxCPM.cpp), reusing the existing and already-released voxcpm2 model tree:

Model	Params	Output sample rate	GGUF file
VoxCPM-0.5B	0.5B	16 kHz	`voxcpm-0.5b-q8_0-audiovae-f16.gguf`

You may find the converting tools in VoxCPM.cpp project

GGUF files are here.

Architecture (0.5B): VAE encoder 128 / decoder 1536, encoder_rates [2,5,8,8], decoder_rates [8,8,5,2], patch_size 2, residual_lm 6 layers, encoder/dit 4 layers, 16 kHz, max_len 4096.

Since the v1 GGUF stores a different tensor convention than v2 (folded AudioVAE weights, no weight_v/weight_g split, no sr_cond_model tensors, voxcpm architecture name), the port wraps the v2 loader with a GGUF tensor-adaptation layer and adds config.v1-guarded branches in the generator, mirroring the reference implementation (VoxCPM.cpp).

All work is on main, 21 commits ahead of upstream/main (merge-base 4e973b1), consisting of 12 porting commits plus merges. tensor_source.h differs from upstream by a single line (is_synthesized) and tensor_source.cpp is identical — the port is structured to be upstreamable.

2. Porting activities

Regenerated the 0.5B config.json from the GGUF metadata — the previously shipped sidecar was wrong on ~8 axes (patch, residual_lm/encoder/dit layer counts, VAE dims and rates, sample rate 44.1 kHz vs the actual 16 kHz, max_len).
Diagnosed the v1 GGUF conventions (tensor dump + reference converter analysis):
- AudioVAE conv weights are stored already folded (weight-norm folded), with no weight_v/weight_g decomposition and no sr_cond_model.* tensors.
- GGUF file dims == ggml ne order; the v1 GGUF carries no audiocpp.tensor_shapes override metadata (v2 does), so the adapter must present shapes itself.
Designed the identity-fold adapter (see §4) so the existing load_vae_weights loader works unchanged against folded v1 weights byte-for-byte.
Mirrored the reference generator math for the no-fusion (no fusion_concat_proj) case: elementwise-add fusion inputs, elementwise-add dit-mu, and a real residual_lm autoregressive step.
Set up the VoxCPM1-GGUF/ model directory with a config regenerated from its own GGUF metadata + tokenizer sidecars, and updated model_specs/voxcpm1.json package targets accordingly.
Fixed the core quality bugs (see §8): synthesized-weight handling exposed via is_synthesized(), embedding transpose [hidden, vocab]→[vocab, hidden], and tensor synthesis only for tensors actually missing from the GGUF — turning pure noise into intelligible speech.
Made the GGUF self-contained: config + tokenizer read from GGUF metadata (audiocpp.vocab_* / config keys) with a GGUF-native tokenizer, removing external sidecar dependence.
Enabled voice clone + streaming to parity with VoxCPM2 (clone = continuation-mode with reference audio + transcript; streaming with retry_badcase=false).
Aligned cloning numerics with the golden implementation: encoder output_padding fix (causal padding), conditioning/FSQ path fixes; 6/6 target sentences transcribed exactly via SenseVoice.
Refactored framework changes back to upstream shape — moved GGUF tokenizer/config metadata reading out of the framework TensorSource into voxcpm2 GgufMetadataReader, reverted validate_expected_shape relaxed-rank param. Framework net delta: +1 line is_synthesized.
Verified on CPU and released per-request VRAM (mem_saver + unconditional end-of-request release; only the cloned voice is cached across requests).

3. Changes per file (full diff vs upstream/main `4e973b1`)

File	Change
`CMakeLists.txt`	Added `audiocpp_add_model(voxcpm1 ...)` reusing the 7 voxcpm2 sources; registers `engine::models::voxcpm2::make_voxcpm1_loader`.
`include/engine/framework/assets/tensor_source.h`	+1 line: `virtual bool is_synthesized(...) { return false; }` — the only framework change that survives; needed to distinguish real vs fabricated weights at the abstract `TensorSource` level.
`include/engine/models/voxcpm2/loader.h`	Declared `make_voxcpm1_loader()`.
`include/engine/models/voxcpm2/assets.h`	Added `VoxCPM2Config::v1 = false`; `load_voxcpm2_assets()` now takes `bool is_v1`.
`src/models/voxcpm2/loader.cpp`	Added `VoxCPM1Loader` (family `"voxcpm1"`), `load_voxcpm1_model()`, `make_voxcpm1_loader()`, `metadata_v1` / `capabilities_v1` / `cli_v1`. Tasks: `tts` with `{offline, streaming}` modes, `supports_speaker_reference = true`. GGUF via `load_voxcpm2_assets(path, is_v1=true)`.
`src/models/voxcpm2/assets.cpp`	Added `TransformingTensorSource` v1 adapter (biggest chunk): • v1→v2 tensor-name rename map (`token_embd.weight`→`base_lm.embed_tokens.weight`, gguf `blk.N.`→`base_lm.layers.N.` / `feat_encoder.encoder.layers.` / `feat_decoder.estimator.decoder.layers.` / `residual_lm.layers.`, `attn_norm`→`input_layernorm`, `ffn_norm`→`post_attention_layernorm`, `attn_`→`self_attn._proj`, `ffn_`→`mlp._proj`, `time_mlp.`, `output_norm.weight`→`base_lm.norm.weight`, projection/fsq/stop mappings) • Folded weight-norm synthesis: for every `audio_vae..weight` conv, `X.weight_v` → folded tensor data as-is, `X.weight_g` → per-row L2 norms (identity fold, see §4) • Identity `decoder.sr_cond_model.{2..5}.scale_embed.weight` (ones) / `.bias_embed.weight` (zeros) since v1 GGUF carries no SR-conditioning tensors • Synthesized missing v1 tensors, only when absent from the GGUF* (`feat_encoder.scale_embed/bias_embed`, `feat_encoder.fc_logvar`, `feat_encoder.diag`, `feat_encoder.merge`, `token_embd.extra_bias`, `fusion_concat_proj.weight/bias`, `stop_proj.weight`, `stop_head.weight`) • `is_synthesized()` override (map membership on `synthesized_tensors_`) • Rank-tolerant `require_f32` (accept element-count-equal, shape-different fetches) + relaxed-rank VAE weight_v anchors • Embedding transpose in `set_backend_tensor()`: V1 GGUF stores `token_embd.weight` as `[hidden, vocab]` but gglm expects `[vocab, hidden]`; transpose applied when shapes match the swap • `has_tensor` / `require_metadata` / `require_tensor_data` folded + synthesized lookups • Anchor fix: `encoder.fc_mu.weight_v` uses computed encoder-in (`encoder_dim << #rates` = 2048), not `decoder_dim` (1536)
`src/models/voxcpm2/generator.cpp`	• v1 fusion guard `has_fusion_proj`: `tensor != nullptr && !is_synthesized(...)` (5 call sites — build/run/generate paths); residual input = `AddModule` instead of concat+linear when false (matches reference `build_residual_fusion_input`) • Added `add_dit_mu()` helper; v1 `mu` = elementwise add of `current_lm_dit_hidden + residual_dit_hidden` (matches reference `build_dit_mu`, `mu_dim = hidden·(fusion?2:1)`, v1 → hidden) • CFM `mu` size check is now v1-aware (`hidden_dim * (v1 ? 1 : 2)`) • v1 decode loop runs `residual_lm_.run_step(next_projected.residual_input).hidden` (earlier `fsq_lm_dit_hidden` shortcut removed)
`src/models/voxcpm2/minicpm.cpp`	Prompt-prefill graph: v1 `residual_input` = `AddModule(lm_hidden, masked_current)` instead of concat+linear; residual_lm always runs. RoPE longrope factors loaded from GGUF config for v1 (prefill lm_hidden l2 within ~2% of reference).
`src/models/voxcpm2/minicpm_blocks.h`	Prefill residual-input branch for v1 ADD fusion.
`src/models/voxcpm2/session.cpp`	v1 `min_tokens` floor (avoids premature stop at ~2 tokens); stop-progress handling; voice-clone: `--voice-ref` routed through the prompt path (reference audio + transcript as reference_text), reference-only `ref_start/ref_end` branch (fails identically in golden impl — model limitation); streaming support; per-request VRAM release (unconditional end-of-request release of prefill + decoder graphs; `mem_saver` additionally releases all generator graphs; only the cloned voice is cached).
`src/models/voxcpm2/audiovae.cpp`	Encoder: dropped `stride % 2` output_padding on downsample conv so causal padding matches the reference encoder; `VOXCPM_DUMP_REF_MONO` / `REF_FEAT` / `ENC_STAGE` debug dumps; `release_encoder_graph()` so encoder VRAM frees right after encode.
`src/models/voxcpm2/config_gguf.cpp/.h`	New — reads config from GGUF metadata (`voxcpm.*` keys: architecture, dims, layer counts, VAE dims/rates, max_len, RoPE factors, stop floor).
`src/models/voxcpm2/gguf_metadata.cpp/.h`	New — `GgufMetadataReader`: framework-independent GGUF metadata accessor so the framework `TensorSource` stays upstream-shaped.
`src/models/voxcpm2/tokenizer_gguf.cpp/.h`	New — GGUF-native BPE tokenizer (73,448 vocab) built from embedded `audiocpp.vocab_*` metadata.
`src/models/voxcpm2/tokenizer_wrapper.h`	New — tokenizer interface wrapper (GGUF-native ↔ external tokenizer.json).
`src/models/voxcpm2/tokenizer_text.cpp/.h`	v1 tokenizer selection; `tokenize` path chosen by tokenizer type.
`include/engine/models/voxcpm2/audiovae.h`	Declared `release_encoder_graph()`.
`include/engine/models/voxcpm2/generator.h`	Declared `release_runtime_memory()`.
`include/engine/models/voxcpm2/minicpm.h`	Declared `release_runtime_memory()` on prefill/text-embedding runtimes.
`model_specs/voxcpm1.json`	Package targets: `voxcpm1_0.5b_q8_0` → `VoxCPM1-GGUF` (default).
`tools/audiocpp_cli/audiocpp_cli_path_cases.json`	Added 3 path-test cases: `voxcpm1_tts`, `voxcpm1_voice_clone`, `voxcpm1_streaming_tts`.
`webui/configs/models_catalog.json`	Added `voxcpm1` catalog entry.
`webui/configs/model_params.json`	Added voxcpm1 controls (`num_inference_steps`, `guidance_scale`, `min_tokens`, ...).
`webui/native/dist/index.html`	WebUI embedding refreshed to surface voxcpm1 (0.5B package).
`docs/tts.md`	Added VoxCPM1 section + TOC entry (usage, options, clone, streaming, sample-rate notes).
`models/VoxCPM1-GGUF/config.json`	Regenerated from 0.5B GGUF metadata (tokenizer now GGUF-native).

4. Key design: the identity-fold adapter

The v1 GGUF (OpenBMB reference converter) stores AudioVAE conv weights already folded (weight = weight_g · weight_v / ‖weight_v‖), with no weight_v/weight_g split, while audiovae.cpp requests the decomposed names directly via require_f32. The adapter solves this without touching the VAE loader:

X.weight_v  := folded GGUF tensor data (as-is)
X.weight_g  := per-row L2 norms of the folded tensor, computed with the loader's own row grouping (groups = expected_shape.front(), inner = elements/groups)

Because fold_weight_norm multiplies row d0 by weight_g[d0] / ‖row d0‖ = 1, the loader output equals the GGUF data byte-for-byte — an exact identity, with no layout drift relative to the reference runtime's consumption of the same bytes.

5. Usage

Build

scripts/build_linux.sh --backend cpu --target audiocpp_cli
# or, with the standard full model set:
cmake -S . -B build/linux-cpu-release -DCMAKE_BUILD_TYPE=Release
cmake --build build/linux-cpu-release --target audiocpp_cli -j 8

Run — VoxCPM-0.5B (16 kHz output)

build/linux-cpu-release/bin/audiocpp_cli \
  --task tts --family voxcpm1 \
  --model models/VoxCPM1-GGUF/voxcpm-0.5b-q8_0-audiovae-f16.gguf \
  --backend cpu --text "Hello from VoxCPM1." --out out.wav

Voice clone

build/linux-cpu-release/bin/audiocpp_cli \
  --task tts --family voxcpm1 \
  --model models/VoxCPM1-GGUF/voxcpm-0.5b-q8_0-audiovae-f16.gguf \
  --backend cpu --text "Hello from VoxCPM1." \
  --voice-ref assets/resources/b.wav --reference-text "b reference transcript" --out out.wav

Streaming

build/linux-cpu-release/bin/audiocpp_cli \
  --task tts --family voxcpm1 \
  --model models/VoxCPM1-GGUF/voxcpm-0.5b-q8_0-audiovae-f16.gguf \
  --backend cpu --mode streaming --text "Hello from VoxCPM1." \
  --request-option retry_badcase=false --out out.wav

Options

Option	Values	Default	Meaning
`--task`	`tts`	required	Task kind.
`--family`	`voxcpm1`	auto-detect	Selects the v1 loader.
`--backend`	`cpu`, `cuda`, `vulkan`, `metal`, `hip`, `best`	`best`	Backend.
`--mode`	`offline`, `streaming`	`offline`	Streaming requires `retry_badcase=false`.
`--voice-ref`	WAV path	not set	Reference speaker audio (clone); must be paired with `--reference-text`.
`--max-tokens`	integer	`4096`	Maximum generated AR tokens.
`--num-inference-steps`	integer	`10`	Flow-matching steps.
`--guidance-scale`	float	`2.0`	CFG strength.
`--session-option voxcpm1.mem_saver=true\|false`	bool	`false`	Tighter graph workspaces + release request graphs; idle VRAM drops to ~1.4 GB after generation (very long text may need ~3.5 GB).
`--session-option voxcpm1.prompt_cache_slots=<n>`	integer	`1`	Prompt/prompt-audio embedding cache slots.
`--text-chunk-mode`	`default`, `tag_aware`, `japanese`, `endline`	`tag_aware`	Long-form chunking mode.

6. Validation performed

Load + anchors: the 0.5B GGUF passes validate_weight_anchors and load_vae_weights/load_model_weights on CPU backend.
TTS quality (transcribed): "This is a test run for the fix" → SenseVoice transcribes "This is a test." — intelligible speech, RMS ~0.10–0.20, 16 kHz, no pure noise.
Voice clone (transcribed): continuation-mode clone with the Anna reference → 6/6 target sentences transcribe exactly (SenseVoice); text-only TTS unchanged.
Streaming: speech.audio.delta SSE chunks flow at the model native 16 kHz; validated via the voxcpm1 streaming WebUI script (webui/voxcpm1_stream_webui.py).
Parity vs reference: prefill lm_hidden l2 within ~2% of VoxCPM.cpp; stop predictor fires at pos=19; duration 1.60 s vs reference 1.68 s.
Per-request VRAM: two-request reused-session tests pass (2-request clone sequence; long→short text reallocates a smaller decoder buffer); idle VRAM ~1.4 GB, long text up to ~3.5 GB.
Regression: the released voxcpm2 path is untouched (guard style config.v1, v2 default false); VoxCPM2 still generates 48 kHz speech with byte-identical output versus pre-change baseline.

7. Supported modes

Mode	Status	Notes
Offline TTS	✅ works	Intelligible speech, transcribed correctly.
Streaming	✅ works	SSE PCM chunks; requires `retry_badcase=false` (same as v2).
Voice clone	✅ works	Continuation-mode clone (reference audio + transcript); reference-only (`ref_start`/`ref_end`) cloning fails identically in the golden `VoxCPM.cpp` — a model-level limitation.

8. Fixed issues (was: "Known issue: noisy output")

The "pure noise" blocker from the initial port is resolved. Root causes found and fixed:

Synthesized weight used as real fusion weight — fusion_concat_proj was Xavier-synthesized on every load and treated as a learned tensor. Fixed by adding is_synthesized() to the TensorSource interface (tensor_source.h +1 line) and overriding it in TransformingTensorSource; the 5 has_fusion_proj guards now exclude synthesized weights (false for true V1 models).
Embedding transposed — V1 GGUF stores token_embd.weight as [hidden, vocab] = [1024, 73448]; audio.cpp/ggml needs [vocab, hidden]. Fixed with a transpose in set_backend_tensor().
Unconditional tensor synthesis — tensors were fabricated regardless of presence. Fixed: synthesize only what the GGUF lacks; corrected feat_encoder.special_token shape (1D vs 4D).
Early stopping — stop token fired at ~2 tokens (1.28 s cutoff). Fixed: real RoPE longrope factors loaded from GGUF config + min_tokens floor; durations scale 1.76 s→4.32 s with text length.
VAE encoder padding — dropped the stride % 2 output_padding so causal padding matches the reference encoder (clone conditioning).

9. Remaining tasks

…d weight handling and embedding transpose Bug: VoxCPM1 model produced pure noise ("elloそ。") instead of speech due to: 1. Synthesized `fusion_concat_proj` weight (Xavier init) treated as learned weight → wrong concat+linear fusion 2. Embedding weight transposed in V1 GGUF: `token_embd.weight` stored as [hidden, vocab] but audio.cpp expects [vocab, hidden] Fix: - Add `is_synthesized()` to TensorSource interface to distinguish loaded vs synthesized weights- Implement in TransformingTensorSource for V1 models- Add embedding weight transpose in set_backend_tensor() for `base_lm.embed_tokens.weight` - Update 5 `has_fusion_proj` checks to exclude synthesized weights - Test: "This is a test run for the fix" now transcribes as "This is a test." (was pure noise) --> but still wrong.

## Summary Fixed VoxCPM1 TTS producing pure noise by correcting tensor synthesis and shape validation issues. ## Changes - **`src/models/voxcpm2/assets.cpp`**: Only synthesize tensors missing from GGUF (not unconditionally). Fixed `feat_encoder.special_token` shape (1D vs 4D). Added relaxed rank handling in `set_backend_tensor()` for V1. - **`src/framework/assets/tensor_source.cpp`**: Added `relaxed_rank` parameter to `validate_expected_shape()` allowing shape mismatches when element counts match. ## Root Cause Synthesized (Xavier-initialized) tensors were used instead of learned checkpoint weights. The `is_synthesized()` check now correctly distinguishes true synthesized tensors (only `fusion_concat_proj` for V1) from loaded weights. ## Validation - VoxCPM1: 16kHz speech, RMS ~0.10-0.15 ✅ - VoxCPM2: 48kHz speech (no regression) ✅ - Embedding transpose: `[1024,73448]` → `[73448,1024]` ✅ - `has_fusion_proj=false` for V1 ✅

## Fix - Added GGUF metadata reading to `TensorSource` (tokenizer.ggml.*, voxcpm_*) - Created `VoxCPM1GgufTokenizer` + `load_voxcpm1_config_from_gguf()` for native GGUF loading - Added `VoxCPM2TokenizerWrapper` for dual JSON/GGUF tokenizer support - Updated `load_voxcpm2_assets()` to auto-detect/use GGUF metadata - Removed external JSON deps from `model_specs/voxcpm1.json` ## Test (ASR: sensevoice@11533) - VoxCPM1 0.5B: "This is a test run for the fix." ❌ (too fask) - VoxCPM1.5 1.5B: "I the touch for the." ❌ (too slow) ## Remaining Bugs 1. VoxCPM1 too fast (1.28s vs 2.5s) - early stop token 2. VoxCPM1.5 too slow (5.29s vs 2.5s) - arch diff

- config_gguf.cpp: output_sample_rate now falls back to sample_rate (not 16000) VoxCPM1.5 GGUF has sample_rate=44100 but no out_sample_rate → was defaulting to 16kHz - session.cpp: add V1-specific default min_tokens to prevent early stop token trigger VoxCPM1 (patch_size=2): min_tokens=20, VoxCPM1.5 (patch_size=4): min_tokens=12 Without this, stop token triggers at ~2 tokens causing 1.28s cutoff - Stop predictor weights correctly loaded via V1 relaxed rank (no transpose needed) GGUF stores [1024,2] (GGML), expected logical [2,1024] → to_ggml_dims → [1024,2] ✓ Results: VoxCPM1 (0.5B): durations scale 1.76s→4.32s with text length VoxCPM1.5 (1.5B): durations scale 2.56s→5.12s, correct 44.1kHz sample rate VoxCPM2: regression passes (48kHz, 1.28s) Files: config_gguf.cpp (+6), session.cpp (+14)

…VoxCPM2)** VoxCPM1 (0.5B/1.5B) models now support voice cloning (`--voice-ref`) and streaming output (`--mode streaming`), matching the VoxCPM2 feature surface. The inference math was already shared; this unblocks the capability/option/reporting layer. **Root causes fixed (5 gaps):** - Capability advertisement: now exposes `Tts + {Offline, Streaming}` for V1 (was TTS-only) - Family identity: `family_impl()` returns `"voxcpm1"` for V1 models (was hardcoded `"voxcpm2"`) - Session options: `normalize_v1_session_options()` rewrites `voxcpm1.*` → `voxcpm2.*` keys so aliases work - Request options: added `voxcpm1.*` aliases for all params (`prompt_text`, `min_tokens`, `guidance_scale`, `retry_badcase`, etc.) - Model spec: `voxcpm1.json` adds `streaming` mode, correct sample rates (16kHz/44.1kHz) **Changes:** 7 files, +167/−32 lines - `src/models/voxcpm2/session.cpp` — option normalization, family-aware errors, request-option aliases - `src/models/voxcpm2/loader.cpp` — capability advertisement, family-labeled errors - `model_specs/voxcpm1.json` — streaming mode, tags, corrected description - `docs/tts.md` — V1 streaming/voice-clone examples, `retry_badcase=false` requirement - `tools/audiocpp_cli/audiocpp_cli_path_cases.json` — 3 new V1 path tests - `webui/configs/models_catalog.json` + `model_params.json` — V1 WebUI entries **Verified (CPU):** | Test | Result | |------|--------| | V1 offline TTS | `family=voxcpm1` ✓ | | V1 voice clone | 16kHz, 5.12s, RMS 0.115 ✓ | | V1 streaming | 40×1280 chunks, 16kHz ✓ | | V1 `voxcpm1.*` session/request options | accepted & applied ✓ | | V1 capability inspection | `modes=offline,streaming` ✓ | | V2 regression (offline/streaming) | 48kHz, parity maintained ✓ | Streaming requires `retry_badcase=false` (same as V2, pre-existing design). No V2 behavior changes. **Issue**: The audio quality is still bad

VoxCPM1 attention used identity longrope factors and a padded stop-token floor. The GGUF's real F32 factor arrays are now read and applied (prefill, stop behavior and duration match the VoxCPM.cpp reference), and the V1 default `min_tokens` is lowered to the reference floor so short utterances are no longer padded with trailing silence. **Root causes fixed (2):** - RoPE longrope factors were hardcoded to `1.0f` in the GGUF config path ("GGUF doesn't have native float arrays" was wrong — `GgufTensorSource` already parses them); every attention computation across all four transformers (base LM, residual LM, local encoder, local DiT) used identity positional encodings - V1 default `min_tokens=20` (per patch_size) vs reference `kMinLen=2` — forced ~1.6s+ of audio and padded short utterances with trailing silence after the stop predictor fired **Changes:** 2 files, +29/−12 lines - `src/models/voxcpm2/config_gguf.cpp` — read `voxcpm_lm_config_rope_scaling_{short,long}_factor` f32 arrays via `optional_f32_array()` with size validation (`head_dim/2`), identity fallback only when the keys are absent - `src/models/voxcpm2/session.cpp` — V1 default `min_tokens = 2` (≡ reference `step > kMinLen`), keeping the `--request-option min_tokens` override **Verified (CPU, against reference `/workspace/pi/VoxCPM.cpp`):** | Test | Result | |------|--------| | Prefill lm_hidden | l2 within ~2% of reference (was diverged) | | Stop predictor ("This is a test run for the fix") | fires at pos=19 (was: never fired) | | Duration | 1.60s (ref 1.68s), trailing silence 0.13s (ref 0.44s) | | V2 regression | 48kHz output maintained ✓ | | Embedding + fusion | `[73448,1024]` transpose intact, `has_fusion_proj=false` ✓ | **Issue**: Voice clone is still not supported — `--task clon` is rejected and passing reference audio + text (`--task tts --voice-ref <wav>`) generates noise rather than cloned speech. Needs a port-audit of the VoxCPM1 reference-audio conditioning path. Full evidence in `docs/reports/2026-08-18_1128_VoxCPM1_RoPE_Longrope_Factors_Stop_Floor_Fix.md`.

…den impl Restore working voice cloning by fixing the reference-audio conditioning and AudioVAE encoder alignment against the golden VoxCPM.cpp port: - generator: only set the CFM `prefix_cond` from prefill rows carrying audio (audio_mask). Previously the trailing text row's zero feature overwrote the reference patch, feeding the DiT a zero acoustic anchor for voice cloning (matches torch feat[:, -1] semantics) - audiovae: re-enable VAD silence trimming for prompt/reference audio (matches golden server_common.cpp:842/878), then pad to patch alignment before VAE encoding (left for prompt, right for reference) - audiovae: drop the `stride % 2` output_padding on the encoder downsample conv so causal padding matches the reference encoder - assets: declare base_lm.embed_tokens.weight as [vocab, hidden] so V1 GGUFs storing the embedding transposed ([hidden, vocab]) load correctly - audiovae: add VOXCPM_DUMP_REF_MONO / REF_FEAT / ENC_STAGE debug dumps Validation (sensevoice-small STT, continuation-mode clone with the Anna reference): 6/6 target sentences transcribe exactly; text-only TTS unchanged. Reference-only cloning (ref_start/ref_end tokens) still fails identically in the golden VoxCPM.cpp - a model-level limitation.

VoxCPM1 voice cloning via `--voice-ref <wav>` produced non-cloned speech, while `--audio <wav>` (plus `--reference-text`) cloned correctly. Both flags carried the same user intent, but the CLI mapped them to different request fields that the session treated as two distinct audio roles. **Root cause:** `--voice-ref` set `request.voice->speaker->audio`, which the session consumed as *reference audio*. For VoxCPM1 the reference path is wrong in two ways: - `encode_prompt_audio()` only copies `prompt_text` inside the `prompt_audio` branch, so a reference-only request dropped the reference transcript entirely (the LM never saw it). - The reference role right-pads the audio and prepends it wrapped in the `<audio_prompt_start/end>` tokens 103/104. Those belong to VoxCPM2's "reference-mode plumbing"; the V1 LM was only trained for prompt-continuation cloning (golden VoxCPM.cpp uses `--prompt-audio` + `--prompt-text`, and its V1 server never calls `encode_reference_audio`). **Fix:** in `VoxCPM2SessionBase::encoded_prompt_for_request()`, when the model is V1 and only a reference audio is supplied (no `--audio`), route it through the prompt path — the audio becomes `prompt_audio` (left-padded, after `<audio_start>`) and `--reference-text` becomes `prompt_text` (concatenated with the target text). V2 keeps the reference-mode path untouched. Applies to both offline and streaming runs (single shared function). Without `--reference-text` the request now fails with the golden's exact rule ("prompt audio requires prompt_text or reference_text"). **Changes:** 1 file, +24/−12 lines - `src/models/voxcpm2/session.cpp` — V1 reference→prompt routing with cache key/lookup/encode all using the effective audio roles **Verified (CPU, 0.5B Q8_0):** | Test | Result | |------|--------| | V1 `--voice-ref` + ref-text | byte-identical WAV to `--audio` + ref-text (same clone) | | V1 `--voice-ref` without ref-text | clean error (matches golden iff rule) | | V1 `--audio` regression | byte-identical output | | V2 `--voice-ref` regression | 48kHz, byte-identical to pre-fix (reference mode preserved) | | 5-voice clone batch (ana/eric/andrew/jenny/nicole) | 16kHz speech, RMS 0.06–0.08 ✓ | **Note:** V1.5 (44.1kHz) fails at load with "encoder sample capacity must be divisible by encoder stride" — pre-existing config gap (stride 1764 ∤ default capacity 240000), identical on `--audio` before this fix.

…main For fixing VocCPM v1 --voice-ref clone issue

Move VoxCPM GGUF tokenizer/config metadata reading out of the framework TensorSource interface into a new voxcpm2 GgufMetadataReader. Revert the validate_expected_shape relaxed_rank parameter and redundant <memory> include; tensor_source.h/.cpp now differ from upstream/main by a single line (is_synthesized).

Only the cloned voice is cached across requests; prompt-prefill and AudioVAE encoder/decoder graphs are freed at request end and rebuilt fresh on the next request. Idle VRAM drops to ~1.4GB after generation; very long text may require up to ~3.5GB VRAM during generation.

…main

jasonchen31 · 2026-08-19T22:17:41Z

Removed any changes to the framework.

# Conflicts: # webui/native/dist/index.html

…r merge

0xShug0 · 2026-08-21T19:39:29Z

@jasonchen31 Thanks for continuing to iterate on this! The PR is closer, but I still see several issues that should be addressed before merge.

Main blockers:

Please share GGUF conversion script or conversion instructions so I can test locally... Or you can host the GGUF on your HF repo so I can download and test it.
The GGUF loading boundary is still too complicated. src/community_models/voxcpm1/assets.cpp has a large TransformingTensorSource layer that repairs the GGUF layout at runtime. This should be fixed in the converter instead. The model runtime should consume canonical VoxCPM1 tensor names and shapes directly.
The VoxCPM1 model still carries VoxCPM2 leakage. generator.cpp has multiple config.architecture == "voxcpm2" checks inside the VoxCPM1 implementation, and assets.cpp still synthesizes or routes v2-style tensors like fusion_concat_proj. Since this is a dedicated VoxCPM1 family, the architecture path should be explicit and not depend on v2 branches.
Debug/test code is still present. I see VOXCPM_DUMP_*, VOXCPM_TEST_BATCH_MU, VOXCPM1_LOG_STOP, raw fprintf, and hardcoded /tmp/opencode/... dump paths in the model code. These need to be removed before merge.
Tokenizer logic is duplicated. tokenizer_gguf.cpp duplicates tokenizer behavior from tokenizer_text.cpp. If GGUF tokenizer metadata is needed, it should only parse metadata into the same tokenizer data structure used by one tokenizer implementation.

Overall, we’re close! Don’t feel discouraged. My first attempts were much worse :)

jasonchen31 · 2026-08-21T22:49:02Z

Thanks for your comments. They are good practices. I'll be off for 1-2 weeks travelling. To be fixed later or please feel free to fix them if urgent.
I just added the download links of the gguf files and converting script in the main PR text.

0xShug0 · 2026-08-22T04:14:43Z

Thanks for your comments. They are good practices. I'll be off for 1-2 weeks travelling. To be fixed later or please feel free to fix them if urgent. I just added the download links of the gguf files and converting script in the main PR text.

Enjoy your travels! No worries at all. I’ll take care of it when I get a chance.

- Fixed stateful convolution view strides in causal_conv1d_stateful, causal_conv1d_dw_stateful, and causal_conv_transpose1d_stateful - Used source tensor's actual column-major strides (nb[1], nb[2]) instead of assuming contiguous layout - Non-streaming and streaming now produce identical audio statistics - All tests pass: voxcpm1_tts, voxcpm1_streaming_tts, voice cloning streaming - Long streaming verified (174 chunks, 13.92s) with minimal boundary diffs

- Removed VOXCPM_DUMP_* environment variable debug dumping code from audiovae.cpp, generator.cpp, minicpm_blocks.h - Removed VOXCPM1_LOG_STOP debug logging - Removed unused encoder_stages_ member variable from AudioVAE decoder - All tests pass: voxcpm1_tts, voxcpm1_streaming_tts, voice cloning streaming

- Add tokenizer_common.h/cpp with shared BPE/UTF-8/CJK tokenization logic - Refactor tokenizer_text to load JSON then delegate to common - Refactor tokenizer_gguf to load GGUF metadata then delegate to common - Eliminates ~500 lines of duplicated code

jasonchen31 · 2026-08-22T13:58:49Z

@0xShug0 Fixed item 4, 5 and streaming glitch issue solved.

The voxcpm1 family sources only serve V1 (VoxCPM-0.5B); V2 now lives in its own registry entry. Remove every V2 conditional from the V1 code path: - assets.cpp: parse_config now hard-rejects architecture != "voxcpm". - session.cpp: drop the legacy voxcpm2.* -> voxcpm1.* option alias in normalize_v1_session_options. - generator.cpp: collapse the six has_fusion_proj sites to the V1 path (no fusion_concat_proj, mu_tokens=1, add_dit_mu). Delete the concat_dit_mu helper, the VOXCPM_TEST_BATCH_MU test hook, and the V2-related comments. V1 tensor transforms (embed [hidden,vocab] -> [vocab,hidden] transpose, weight_v/weight_g decomposition, AudioVAE folded-conv handling) are preserved unchanged. Verified by regenerating 9 voice samples (3 tts, 3 clone, 3 stream+clone) and transcribing all 9 with sensevoice-small: every output matches the input text.

# Conflicts: # webui/native/dist/index.html

Resync the committed static webui with the freshly-merged SvelteKit sources. Build output is functionally equivalent to the upstream's hand-shipped index.html (which was taken via 'git checkout --theirs' during the upstream merge); only embedded asset hashes and the SvelteKit version stamp differ. Verified: cmake build is green and VoxCPM1 smoke test transcribes correctly ('Post merge smoke test.').

The V1 family only ever takes the ADD residual path; the V2 fusion_concat_proj linear projection was already unreachable in generator.cpp and minicpm.cpp after the V2 branch cleanup. Remove the leftover dead code: - minicpm.h: delete VoxCPM1ProjectionWeights::fusion_concat_proj field - minicpm.cpp: drop the config.v1 ? Add : Linear(fusion_concat_proj) branch (keep Add) and the load of fusion_concat_proj weights - assets.cpp: drop the V1->V2 rename_map entries for proj.fusion_concat.* / fusion_concat_proj.*, the synthesized fusion_concat_proj.weight/bias tensors, their Xavier init, and the require_tensor_shape anchor for fusion_concat_proj.weight No V1 tensor transforms touched. Verified by regenerating 9 voice samples (3 tts, 3 clone, 3 stream+clone): all have real speech waveforms (RMS 0.056-0.158, max 0.34-1.0, active zero-crossing rates). STT host was down at commit time; signal stats confirm valid output.

# Conflicts: # webui/configs/models_catalog.json # webui/native/dist/index.html

Replace TransformingTensorSource (V1->V2 name routing, reshape maps, weight-norm decomposition) with a thin VoxCPM1TensorSource that reads the V1 GGUF names as-is. GgufTensorSource already reverses ggml's ne order, so the shapes it reports match the logical shapes the modules request and no name mapping, reshaping or transposition is needed. Changes: - src/community_models/voxcpm1/assets.cpp: drop the V1->V2 rename map, the reshape/weight-norm maps and the synthesized feat_encoder tensors; keep only the decoder sr_cond_model placeholders (absent from V1 GGUFs) plus the relaxed-rank fallback for tensors stored with a different rank but equal element count; switch the weight anchors to V1 names. - src/community_models/voxcpm1/audiovae.cpp: read the pre-folded audio_vae.*.weight tensors directly instead of refolding synthesized weight_v/weight_g (which was an exact identity); remove fold_weight_norm and squeeze_weight_g. - src/community_models/voxcpm1/minicpm.cpp: take explicit embedding, layer and norm tensor names (token_embd.weight, blk.N.attn_*/ffn_*, output_norm.weight) instead of deriving V2 prefixes. token_embd.weight is deliberately not transposed: its ne [1024, 73448] is reported as [73448, 1024] = [vocab, hidden], which is exactly the layout ggml_get_rows consumes. Transposing it scrambles every embedding row and degrades synthesis to near-silent low-frequency rumble. Reading it as-is also keeps the Q8_0 blocks intact instead of dequantizing 75M parameters to F32 and requantizing them. Verified against voxcpm-0.5b-q8_0-audiovae-f16.gguf with 9 samples: 3 offline TTS, 3 voice clones and 3 streaming clones.

jasonchen31 · 2026-08-29T00:00:03Z

@0xShug0 All done. Could you have a check.? And the download link is provided already in the PR text.

0xShug0 · 2026-08-29T02:46:38Z

@jasonchen31 Thanks! I will make some changes directly to the PR.

0xShug0 · 2026-08-29T03:31:46Z

@jasonchen31 Thanks! PR merged.

I added a conversion script to convert the model to audio.cpp-native GGUF and removed the duplicated local logic.

jasonchen31 · 2026-08-29T09:03:15Z

Great! Thank you!

jasonchen31 and others added 21 commits August 16, 2026 16:49

[WIP] Add VoxCPM v1 — lightweight VoxCPM TTS support (0.5B / 1.5B)

ad2baa7

Merge branch '0xShug0:main' into main

fdb06c2

Merge branch '0xShug0:main' into main

43b12a9

Merge branch '0xShug0:main' into main

0ec6230

Merge branch '0xShug0:main' into main

7f224ab

Merge branch '0xShug0:main' into main

0801a84

Merge branch 'main' of https://github.com/jasonchen31/audio.cpp into …

51c4452

…main For fixing VocCPM v1 --voice-ref clone issue

fix(voxcpm1): Add Webui support for VoxCPM v1 (0.5B)

2b65a0d

Merge branch '0xShug0:main' into main

6e4f09d

remove local

1d6576c

Merge branch 'main' of https://github.com/jasonchen31/audio.cpp into …

f125922

…main

jasonchen31 changed the title ~~[WIP] Add VoxCPM v1 — lightweight VoxCPM TTS support (0.5B / 1.5B)~~ Add VoxCPM v1 — lightweight VoxCPM TTS support (0.5B) Aug 19, 2026

jasonchen31 marked this pull request as ready for review August 19, 2026 22:18

jasonchen31 added 2 commits August 19, 2026 22:23

Merge remote-tracking branch 'upstream/main'

063cbe3

# Conflicts: # webui/native/dist/index.html

generate new index.html based on merged code

5b949e7

0xShug0 added the new model Request for new model support label Aug 20, 2026

jasonchen31 added 3 commits August 20, 2026 09:36

Merge remote-tracking branch 'upstream/main'

064bd91

# Conflicts: # webui/native/dist/index.html

fix(vpxcpm1): release the last framework change. udpate index.html fo…

47a7e7e

…r merge

remove loacal file

730fb37

add model description

264e5df

jasonchen31 added 8 commits August 22, 2026 11:41

fix the issue voxcpm1 does not read reference_text

6f680ee

merge and update webui

4706cef

merge upstream/main: resolve index.html conflict using ours

9363611

add missing tokenizer_common.cpp

5715b6c

add tokenizer_common.h

10d4eb4

jasonchen31 and others added 10 commits August 27, 2026 10:11

Merge upstream/main

af30ae6

Update webui/native/dist/index.html after build

619ded5

Merge remote-tracking branch 'upstream/main'

92c54d5

# Conflicts: # webui/native/dist/index.html

Merge remote-tracking branch 'upstream/main'

ecf4cad

# Conflicts: # webui/configs/models_catalog.json # webui/native/dist/index.html

Merge remote-tracking branch 'upstream/main'

2e1928c

webui: rebuild index.html

c6fc7c1

0xShug0 added 2 commits August 28, 2026 23:03

Move VoxCPM1 compatibility into native GGUF conversion

cbd7930

Fold VoxCPM1 tokenizer implementation into text tokenizer

4090890

0xShug0 merged commit 0402bf3 into 0xShug0:main Aug 29, 2026
6 checks passed

Navigation Menu

Add VoxCPM v1 — lightweight VoxCPM TTS support (0.5B) - #256

Add VoxCPM v1 — lightweight VoxCPM TTS support (0.5B)#256
0xShug0 merged 55 commits into
0xShug0:mainfrom
jasonchen31:main

jasonchen31 commented Aug 16, 2026 •

edited

Loading

Uh oh!

jasonchen31 commented Aug 19, 2026 •

edited

Loading

Uh oh!

0xShug0 commented Aug 21, 2026 •

edited

Loading

Uh oh!

jasonchen31 commented Aug 21, 2026 •

edited

Loading

Uh oh!

0xShug0 commented Aug 22, 2026

Uh oh!

jasonchen31 commented Aug 22, 2026

Uh oh!

jasonchen31 commented Aug 29, 2026

Uh oh!

0xShug0 commented Aug 29, 2026

Uh oh!

Uh oh!

0xShug0 commented Aug 29, 2026

Uh oh!

jasonchen31 commented Aug 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

jasonchen31 commented Aug 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

1. Overview

2. Porting activities

3. Changes per file (full diff vs upstream/main 4e973b1)

4. Key design: the identity-fold adapter

5. Usage

Build

Run — VoxCPM-0.5B (16 kHz output)

Voice clone

Streaming

Options

6. Validation performed

7. Supported modes

8. Fixed issues (was: "Known issue: noisy output")

9. Remaining tasks

Uh oh!

jasonchen31 commented Aug 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

0xShug0 commented Aug 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jasonchen31 commented Aug 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

0xShug0 commented Aug 22, 2026

Uh oh!

jasonchen31 commented Aug 22, 2026

Uh oh!

jasonchen31 commented Aug 29, 2026

Uh oh!

0xShug0 commented Aug 29, 2026

Uh oh!

Uh oh!

0xShug0 commented Aug 29, 2026

Uh oh!

jasonchen31 commented Aug 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jasonchen31 commented Aug 16, 2026 •

edited

Loading

3. Changes per file (full diff vs upstream/main `4e973b1`)

jasonchen31 commented Aug 19, 2026 •

edited

Loading

0xShug0 commented Aug 21, 2026 •

edited

Loading

jasonchen31 commented Aug 21, 2026 •

edited

Loading