🎬 PadCaptioner

Parallelized Autoregressive Decoding for Omni-Modal Dense Video Captioning

Wenzheng Zeng, Siyi Jiao, Chen Gao, Hwee Tou Ng, Mike Zheng Shou

National University of Singapore

ECCV 2026

Paper   |   HF Daily Paper   |   Demo   |   News   |   Overview

Demo

Click here to watch the demo video in MP4 format

📢 News

• [2026-08] The code and model have been released!
• [2026-07] Our work is featured by DailyPapers!
• [2026-07] Our paper is available on arXiv.
• [2026-06] Our work is accepted by ECCV 2026!

🔆 Overview

We propose PadCaptioner, a 3B model for omni-modal dense video captioning that achieves high efficiency and strong grounded caption quality, outperforming 7B counterparts.

The core idea is to exploit the weak local dependencies among temporally distinct events and restructure the causal token dependency, enabling lossless parallel generation.

We design a latent planning mechanism that automatically determines parallelizable units with non-local awareness, guiding subsequent parallel decoding and improving event grounding and caption quality.

🛠️ Installation

1. Clone this repository:

git clone https://github.com/showlab/PadCaptioner.git
cd PadCaptioner

2. Create the conda environment:

conda create -n padcaptioner python=3.12
conda activate padcaptioner

3. Install dependencies:

pip install torch==2.7.1 torchvision==0.22.1 torchaudio==2.7.1 --index-url https://download.pytorch.org/whl/cu126
pip install -r requirements.txt

📦 Dataset

Please refer to DATASET.md for raw dataset links, annotation preparation, the full JSON schema, and task examples.

💪 Training

1. Model Preparation

   1. Prepare the video-SALMONN 2+ 3B checkpoint.

   2. Set MODEL, MODEL_BASE, and LORA_CKPT (download from here) in scripts_run/train.sh to the downloaded paths. At training start, the LoRA checkpoint is automatically merged into the base model to form the video-SALMONN 2+ starting point.

   • We notice that Video-SALMONN 2+ was subsequently updated, so its newer, already-merged model may potentially serve as a starting point. However, we have not verified the correctness of this setup.

2. Fill in the path block at the top of scripts_run/train.sh (DATASET, MODEL, MODEL_BASE, LORA_CKPT, OUTPUT_ROOT), or pass them as flags.

3. Run:

   bash scripts_run/train.sh

🤖 Inference and Evaluation

1. Download the pretrained model.

2. Fill in the path block at the top of scripts_run/test.sh (DATASET, MODEL, MODEL_BASE, OUTPUT_ROOT), or pass them as flags.

   For ChronusAV:

   bash scripts_run/test.sh --dataset /path/to/chronusav_test.json

🙏 Acknowledgments

We list below the related works that inspired PadCaptioner:

• D²VLM
• video-SALMONN 2+

📌 Citation

If you find our work useful, please kindly cite:

@inproceedings{padcaptioner,
  title={Parallelized Autoregressive Decoding for Omni-Modal Dense Video Captioning},
  author={Zeng, Wenzheng and Jiao, Siyi and Gao, Chen and Ng, Hwee Tou and Shou, Mike Zheng},
  booktitle={European Conference on Computer Vision (ECCV)},
  year={2026}
}