Skip to content

Upgrade in-tree FFmpeg to 9.0.2 to fix video color corruption #15199

Description

@Jont828

Summary

Proposed fix for #15198: upgrade Dynamo's in-tree FFmpeg from 8.1.2 to official FFmpeg 9.0.2 and qualify the existing video integrations. This is the implementation/release tracker for that codec bug; it is not a claim that the bug is already fixed.

FFmpeg's official download page currently lists 9.0.2, released September 18, 2026, as the latest stable 9.x release. Its official source contains the swscale correction 62285be0096319310d7012bbab1739db79926c4d. Pin the explicit release rather than a moving branch or a latest URL.

Existing implementation work — do not duplicate

PR #14925 already implements an in-tree 9.0.1 upgrade, PyNvVideoCodec 2.2.3 pinning, suffixed library names, Rust dependency updates, and associated packaging/guard changes. Coordinate with or build on that PR: update/qualify its FFmpeg target to 9.0.2, or make a narrowly scoped follow-up if the 9.0.1 work lands first. This issue does not request a competing rewrite of the same integration.

Related example trackers: #14667 and #14668. Merged #14844 addresses response-format forwarding separately; an FFmpeg upgrade does not replace that fix or prove full T2V/I2V acceptance.

Why This Is More Than a Version Pin

  • API/ABI: the FFmpeg shared-library majors change (avcodec/avformat/avdevice 62→63, avutil 60→61, avfilter 11→12, swscale 9→10, swresample 6→7). Rebuild media-enabled native consumers; do not add cross-major compatibility symlinks. The old ffmpeg-next 8.1.0 binding fails against 9.x headers. The existing implementation moves ffmpeg-next 9.0.0 and video-rs 0.12.0 together.
  • PyNvVideoCodec coexistence: its 2.2.3 wheel already bundles a differently configured FFmpeg 9.0.1 stack. A plain in-tree 9.x bump reuses its SONAMEs. A CPU native-loading test reproduced a PyNv-first failure; giving Dynamo's libraries distinct names made both tested orders pass. Preserve build(container): pin PyNvVideoCodec at 2.2.3 and move the in-tree FFmpeg to 9.0.1 (DYN-4044) #14925's supported --build-suffix=_dynamo approach, build-time pkg-config aliases, matching auditwheel exclusions, and codec-scanner coverage.
  • Validation: 9.0.1 passed the model-free color round-trip with unchanged restricted codec flags. That is supporting evidence, not an executed 9.0.2/full-image/GPU/aarch64 qualification.

Implementation Scope

Build on the corresponding changes in #14925 rather than applying them twice:

  • container/context.yaml: explicit ffmpeg_version: "9.0.2" and synchronized comments; record/verify the release tarball digest.
  • container/compliance/native_packages.yaml: matching native component version. Review version-consistency tests and policy floors against all separately vendored FFmpeg components, not only the in-tree CLI.
  • lib/llm/Cargo.toml, root Cargo.lock, and lib/bindings/python/Cargo.lock: compatible Rust dependency pair and regenerated resolutions.
  • container/templates/wheel_builder.Dockerfile: official source build, isolated SONAMEs, canonical pkg-config discovery aliases that still link the suffixed libraries, auditwheel exclusions, and fail-closed codec checks. Run ldconfig before executing the installed binary.
  • Runtime templates for vLLM, SGLang, TensorRT-LLM and the Dynamo runtime-test image: verify versioned library copying, SONAME resolution, CLI selection, base-package purges and TRT-LLM overlay handling. Preserve IMAGEIO_FFMPEG_EXE and source-installed imageio-ffmpeg.
  • Existing codec/version-consistency and real-video-encode tests: add independent RGB-quadrant assertions and validate the actual codec/parser registries. ffmpeg -parsers is unsupported; a failed listing must not be treated as an empty safe surface.
  • Retain the matching upstream tarball/source tree, regenerate notices/SBOMs, and verify any separate source archive contains the intended source. Keep the PyNvVideoCodec source archive separately attributable to its own build.

Non-negotiable Constraints

  • Keep --disable-gpl, --disable-nonfree, and the existing restricted codec/configuration surface. Do not enable H.264/H.265/AAC encoders/decoders/parsers or NVENC as a compatibility workaround.
  • Retain the existing h264_mp4toannexb and hevc_mp4toannexb bitstream filters: these reframe encoded input for NVDEC and are not software video codecs. Coordinate with existing guard work in test(container): assert the NVDEC bitstream filters stay enabled #13967.
  • Do not restore unrestricted codec-bearing wheels or silently graft FFmpeg into the image-only runtime wheel.
  • Do not make correctness depend on a lucky import order or undocumented environment setup.

Acceptance Criteria / Validation

  • Official 9.0.2 builds under the production toolchains with the intended isolated library names and unchanged codec restrictions; headers/libraries/metadata agree.
  • Clean Rust/PyO3 builds and media decoder tests pass with the resolved dependency pair. Both lockfiles are regenerated, not hand-edited.
  • auditwheel/ELF dependency inspection and fresh-process imports prove that the image-only wheel resolves the intended libraries; no inappropriate library grafting or unresolved old-major dependency remains.
  • Actual Dynamo/PyNvVideoCodec imports and Rust VP9 plus NVDEC H.264/H.265 decoding pass in both import orders; validate DALI coexistence where applicable.
  • The real shipped CLI and shared helper pass an independent RGB → VP9 → RGB regression in MP4 and WebM, while the affected 8.1.2 control fails.
  • Negative codec checks, positive VP9/BSF checks, source retention, notices/SBOMs and TRT-LLM overlay guards pass on the rebuilt images.
  • Supported native amd64/aarch64 builds and affected backend/runtime-image CI pass. A Dockerfile render or isolated build block is not counted as a complete runtime-image build.
  • Targeted T2V/I2V acceptance is completed on version-identified images, with response delivery and color quality checked separately.

Release Decision

Treat this as a moderate native integration change. A patch release is technically possible only after the same qualification and release-owner approval; otherwise target the next qualified release. Do not silently weaken validation to meet a patch window. If an urgent codec fix must ship first, an explicitly tracked temporary upstream patch on 8.1.2 is a separate fallback—not a reason to mark this 9.0.2 upgrade complete.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    backend::sglangRelates to the sglang backendbackend::vllmRelates to the vllm backendbugSomething isn't workingdynamo-runtimeRelates to the dynamo-runtime componentmultimodalruntimeCODEOWNER area -> @ai-dynamo/dynamo-runtime-codeowners

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions