VideoGen

NavDreamer Stage 1 pipeline for:

• generating candidate egocentric navigation videos
• judging them with Qwen3-VL
• selecting the best valid candidate with local reranking

Requirements

• Python >=3.11, <3.13
• ffprobe on PATH
• DASHSCOPE_API_KEY

Install ffprobe:

• macOS: brew install ffmpeg
• Windows (PowerShell): winget install Gyan.FFmpeg or choco install ffmpeg

Verify:

ffprobe -version

Install

Recommended:

uv sync --python 3.12

dev:

uv sync --python 3.12 --group dev

Or pip:

python -m venv .venv
source .venv/bin/activate
pip install -e .

Windows PowerShell activation:

.\.venv\Scripts\Activate.ps1

Set API key:

export DASHSCOPE_API_KEY="your_api_key"

Windows PowerShell:

$env:DASHSCOPE_API_KEY="your_api_key"

Quick Start

CLI entry:

uv run videogen --help

Run full generation + judging:

uv run videogen gen-select

Judge existing videos only:

uv run videogen judge-only

Use custom manifest:

uv run videogen gen-select path/to/manifest.json
uv run videogen judge-only path/to/judge_manifest.json

Use custom config file under videogen/configs/:

uv run videogen gen-select --config config.toml

Config

Default config: videogen/configs/config.toml

Most-used keys:

• prompt_strategy: simple | kinematic | decomposed
• num_candidates
• seed_base, seed_stride
• output_root

Manifest Format

Generate + Judge:

[
  {
    "instruction": "move to the target chair and stop safely",
    "image": "../assets/example.png"
  }
]

image: single file path (relative paths are resolved from manifest directory)

Judge-only:

[
  {
    "instruction": "move to the chair",
    "videos": ["../videos/a.mp4", "../videos/b.mov"]
  }
]

videos: single path, list of paths, or a directory (recursive scan)

Supported image suffixes: .png .jpg .jpeg .webp .bmp

Supported video suffixes: .mp4 .mov .mkv .avi .webm .m4v

Selection Rule

1. Keep only status == pass
2. Rerank by weighted score:

$$ ext{weighted} = 1.4\cdot\text{task_performance} + 0.8\cdot\text{action_safety} + 0.8\cdot\text{scene_consistency} $$

3. Pick highest weighted valid candidate (tie-break: lexicographic candidate_id)

Output

Each run writes artifacts to output_root (default runs/), including:

• request.json
• prompt.txt (generation mode)
• generation/candidates.json (generation mode)
• judge/result.json
• summary.md

Test

uv run pytest videogen/tests

Common Pitfalls

• Missing API key: set DASHSCOPE_API_KEY
• Missing ffprobe: install ffmpeg and reopen terminal on Windows
• Wrong relative paths in manifest: resolved relative to manifest directory
• Empty/invalid JSON manifest: fix schema first

License

MIT (see LICENSE)