Voice Activity Detection project

Keywords: Python, PyTorch, Deep Learning, Audio, Time Series classification, uv, Docker

Table of contents

1. Installation
2. Introduction
3. Project structure
4. Dataset
5. Project usage
6. Testing
7. Continuous integration & delivery
8. Contributing
9. Todo
10. License
11. Resources

1. Installation

This project uses:

• Python 3.11+
• PyTorch 2.2+
• uv package manager

git clone https://github.com/filippogiruzzi/voice_activity_detection.git
cd voice_activity_detection/

1.1 Install with uv (recommended)

# Install uv if you don't have it
curl -LsSf https://astral.sh/uv/install.sh | sh

# Install all dependencies
uv sync

1.2 Development setup

# Install with dev dependencies (ruff, pytest, etc.)
uv sync

# Run linters
make lint

# Run tests
make test

# Auto-format code
make format

1.3 Docker installation

Build and run the CPU Docker image:

make build
make local-nobuild

For GPU support (requires NVIDIA Docker runtime):

make build-gpu

2. Introduction

2.1 Goal

The purpose of this project is to design and implement a real-time Voice Activity Detection algorithm based on Deep Learning.

The designed solution is based on a simple pipeline with MFCC feature extraction and a small 1D-ResNet model (PyTorch) that classifies whether an audio signal is speech or noise.

2.2 Results

Model	Train acc.	Val acc.	Test acc.
1D-Resnet	99 %	98 %	97 %

Raw and post-processed inference results on a test audio signal are shown below.

2.3 Model & features

Each audio window of SEQ_LEN = 1024 samples (16 kHz) is converted into a 16 × 65 feature tensor stacking:

• 5 MFCC coefficients,
• 5 MFCC deltas (1st order),
• 5 MFCC delta-deltas (2nd order),
• 1 RMS energy.

These features feed a configurable 1D-ResNet (vad.model.Resnet1D): stacked residual blocks (3 × Conv1d → BatchNorm1d with a 1×1 shortcut) → global average pooling → a fully connected head producing a single speech logit. The architecture is fully described by the ModelConfig dataclass, so the same configuration must be used at training, export, and inference time.

3. Project structure

The core code lives flat inside vad/:

• vad/model.py: the Resnet1D model architecture and its ModelConfig dataclass
• vad/data.py: feature extraction, dataset building & the PyTorch DataLoader
• vad/train.py: training loop & model export (state dict + TorchScript)
• vad/inference.py: sliding-window inference & visualization

Supporting files:

• tests/: pytest integration tests (training → export → inference)
• scripts/: Docker build / run helpers
• Makefile: common developer commands (install, lint, format, test, build, ...)
• pyproject.toml: project metadata, dependencies & tooling configuration

4. Dataset

Please download the LibriSpeech ASR corpus dataset from https://openslr.org/12/, and extract all files to: /path/to/LibriSpeech/.

The dataset contains approximately 1000 hours of 16kHz read English speech from audiobooks, and is well suited for Voice Activity Detection.

I automatically annotated the test-clean set of the dataset with a pretrained VAD model.

Please feel free to use the labels/ folder and the pre-trained VAD model (only for inference) from this link .

Important note: As this is only a toy project, it is designed to split the test-clean sub-dataset intro train / val / test for quick iteration, but can be extended to a full large-scale dataset.

5. Project usage

5.1 Create PyTorch dataset from raw audio

uv run vad-data --data-dir /path/to/LibriSpeech/

This saves processed .pt files to /path/to/LibriSpeech/dataset/{train,val,test}/. Use --max-files N to process only a few files per split for a quick run.

5.2 Train & export the VAD model

uv run vad-train --data-dir /path/to/LibriSpeech/dataset/ --model-dir /path/to/models/

Checkpoints are saved to --model-dir, and the final model is exported (state dict + TorchScript) to <model-dir>/exported/ unless --no-export is passed.

Useful flags: --epochs/-e, --batch-size/-b, --lr, and the architecture flags --n-filters / --fc-units. Training device (CUDA, Apple MPS, or CPU) is selected automatically. TensorBoard logs are written to <model-dir>/logs/:

uv run tensorboard --logdir /path/to/models/logs/

5.3 Run inference

uv run vad-inference \
    --data-dir /path/to/LibriSpeech/ \
    --checkpoint /path/to/models/exported/model_state_dict.pt \
    --smoothing --max-files 1

Note: if you trained with custom --n-filters / --fc-units, pass the same values to vad-inference so the checkpoint loads into a matching architecture.

6. Testing

Run the test suite with coverage:

make test

This runs pytest with coverage over the vad package, printing a term report with missing lines and writing an HTML report to htmlcov/.

7. Continuous integration & delivery

• CI (.github/workflows/ci.yml): on every push and pull request, runs ruff lint, ruff format --check, and the pytest suite.
• CD (.github/workflows/cd.yml): on push to master/main, builds the Docker image and pushes it to Docker Hub (tagged with the commit SHA and latest).

8. Contributing

Contributions are welcome! Please:

1. Fork the repository and create a feature branch.
2. Install the dev environment with uv sync.
3. Make sure make lint and make test pass before opening a pull request.
4. Use clear commit messages and keep changes focused.

9. Todo

9.1 ML Engineering

• Add MLflow experiment tracking (params, metrics, artifacts)
• Reach full unit-test coverage and add CI coverage reporting
• Add online / streaming real-time inference
• Serve the model via a REST/gRPC API (e.g. FastAPI + ONNX Runtime)
• Add data/version control (DVC) and a model registry

9.2 Data Science

• Compare the model against a simple baseline and train on the full dataset
• Add time-series data augmentation and improve class balancing
• Study the ROC curve & tune the classification threshold
• Explore self-supervised speech encoders (e.g. WavLM, wav2vec 2.0)
• Benchmark modern lightweight VAD models (e.g. Silero VAD, Pyannote 3.x)

10. License

This project is licensed under the terms of the GNU GPL v3.

11. Resources

• Voice Activity Detection for Voice User Interface, Medium
• Deep learning for time series classifcation: a review, Fawaz et al., 2018, Arxiv
• Time Series Classification from Scratch with Deep Neural Networks: A Strong Baseline, Wang et al., 2016, Arxiv