TinyCodec 🔦

Lightweight neural speech codec for low-resource and edge deployment

TinyCodec is a compact speech codec built with depthwise-separable convolutions and residual vector quantization. It targets scenarios where model size and latency matter — mobile devices, embedded systems, or real-time streaming.

Highlights

• Depthwise-separable encoder/decoder — dramatically fewer parameters than standard convolutions, with minimal quality loss
• Progressive distillation — train a small student codec by distilling from a larger teacher in 3 stages
• Variable bandwidth — control bitrate at inference by choosing how many RVQ layers to use (no retraining needed)
• Streaming mode — chunked encoding + crossfade decoding for real-time use
• FSQ support — Finite Scalar Quantization as codebook-collapse-free alternative to VQ
• ONNX export — deploy encoder/decoder separately on edge devices

Model Variants

Variant	Params	Size	Bandwidth	Notes
TinyCodec-base	~8M	~32 MB	2.5 kbps	Best quality
TinyCodec-small	~4M	~16 MB	1.9 kbps	Good balance
TinyCodec-tiny	~1.5M	~6 MB	0.6 kbps	Edge deployment
TinyCodec-tiny-fsq	~1M	~4 MB	~0.8 kbps	No codebook collapse

For comparison, EnCodec is ~15M params (60 MB) and DAC is ~74M params (297 MB).

Install

git clone https://github.com/DASS-057/TinyCodec.git
cd TinyCodec
pip install -e .

# with evaluation metrics:
pip install -e ".[eval]"

Quick Start

import torch
from tinycodec import TinyCodec

model = TinyCodec()
wav = torch.randn(1, 1, 16000)  # 1s, 16kHz mono

# encode -> tokens
tokens = model.compress(wav)
print(tokens.shape)  # (1, 4, 50)

# decode -> waveform
wav_hat = model.decompress(tokens)

# lower bandwidth: use fewer codebooks
tokens_low = model.compress(wav, n_q=2)  # half the bitrate

Streaming

from tinycodec import StreamingEncoder, StreamingDecoder

enc = StreamingEncoder(model, chunk_size=16000, overlap=1600)
dec = StreamingDecoder(model)

# feed audio chunks as they arrive
for chunk in audio_stream:
    tokens = enc.process(chunk)
    if tokens is not None:
        audio_out = dec.process(tokens)
        play(audio_out)

# flush remaining
tokens = enc.flush()
if tokens is not None:
    audio_out = dec.process(tokens)

Training

Standard training:

python scripts/train.py --config configs/base.yaml

With GAN (better perceptual quality):

python scripts/train.py --config configs/base_gan.yaml

Distillation (compress a larger model):

python scripts/train_distill.py \
  --config configs/tiny.yaml \
  --teacher-config configs/base.yaml \
  --teacher-ckpt checkpoints/tinycodec_best.pt

Evaluation

python scripts/evaluate.py --test-dir data/test --checkpoint checkpoints/tinycodec_best.pt

ONNX Export

python tools/export_onnx.py --checkpoint checkpoints/tinycodec_best.pt --output-dir onnx_export/

Project Structure

TinyCodec/
├── tinycodec/          # Core library
│   ├── model.py        # TinyCodec model
│   ├── encoder.py      # Depthwise-separable encoder
│   ├── decoder.py      # Transposed conv decoder
│   ├── quantizer.py    # RVQ with EMA
│   ├── fsq.py          # Finite Scalar Quantization
│   ├── losses.py       # Multi-scale spectral + mel losses
│   ├── discriminator.py # Multi-scale discriminator
│   ├── distill.py      # Knowledge distillation
│   ├── streaming.py    # Streaming inference
│   ├── dataset.py      # Audio dataset
│   └── metrics.py      # PESQ, STOI, SNR, MCD
├── configs/            # Training configs
├── scripts/            # Train, evaluate, data prep
├── tools/              # ONNX export, model stats
├── tests/              # Unit tests
└── notebooks/          # Comparison experiments

Citation

If you find this useful:

@misc{tinycodec2025,
  title={TinyCodec: Lightweight Neural Speech Codec},
  author={瑾瑜},
  year={2025},
  url={https://github.com/DASS-057/TinyCodec}
}

License

MIT