Skip to content

build: pin fixed Brotli source for record reproduction - #2171

Open
khai-oai wants to merge 2 commits into
mainfrom
codex/brotli-fixed-source
Open

khai-oai wants to merge 2 commits into
mainfrom
codex/brotli-fixed-source

Conversation

@khai-oai

@khai-oai khai-oai commented Oct 2, 2026 •

Copy link
Copy Markdown

Record reproduction currently installs an unpinned native Brotli package. Pin the six record requirements and the rerun helper to the immutable upstream source containing the block-switch decoder fix, with a SHA-256 check. The setup note explains the compiler requirement and how to refresh existing environments, since upstream still reports version 1.2.0.

Validated the source build and install in a fresh Python 3.12 environment with uv, the installed decoder round trip, all six requirement pins, and the generated rerun command's Python/shell syntax. Training, GPU runs, and the full dependency environment were not run. Submission training code and historical results are unchanged.

The added source-build CI passes on Python 3.10, 3.12, and 3.13 (3969f57). It checks that the six record pins agree with the rerun helper, builds the pinned package, and verifies the installed native block-switch repair and round trips. This does not run GPU training.

khai-oai commented Oct 2, 2026

Copy link
Copy Markdown
Author

Version clarification:

  • Target branch: main.
  • Unpinned Brotli → fixed upstream source reporting 1.2.0.
  • Fix: google/brotli 0ed4268a4401, applied as a checksum-pinned source archive.

The fixed source still reports version 1.2.0; the commit and checksum identify the repair. The published 1.2.0 release does not contain this fix.

@codemath3000

codemath3000 commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

@khai-oai Quick question: I can see the final few PRs at the top of the official README leaderboard remain unmerged even though they're listed on the official leaderboard (specifically referring to #2135, #2014, #1953, and #1945). These are all already listed on the official leaderboard, not anything new. By any chance could you pin these too (and merge them as well) whenever you get a chance? Thank you so much!

@khai-oai
khai-oai marked this pull request as ready for review October 2, 2026 05:26
@Evreu1pro

Copy link
Copy Markdown

Heads-up that might simplify this: upstream just cut a release. brotli 1.2.1 landed on PyPI yesterday (Oct 22, release notes list a decoder CVE fix), with manylinux wheels for cp310–cp313, so no compiler needed.

Quick check with your own script on top of this branch:

  • brotli==1.2.1 wheel: python scripts/check_brotli.py prints "Patched native Brotli decoder and round trips verified", so BrotliDecoderErrorString(-17) returns _ERROR_FORMAT_BLOCK_SWITCH
  • brotli==1.2.0 wheel: fails on that same assert, as expected
  • compressed output from 1.2.0 and 1.2.1 is byte-identical on what I tried (quality 5 and 11, repeated text, random bytes, int8 normal noise), so artifact sizes shouldn't move

One caveat, if I'm reading the GitHub compare right: v1.2.1 isn't a descendant of 0ed4268 (it shows as diverged), so the fix looks like it was cherry-picked onto the release branch rather than tagged from that commit. The error-string check passes either way, but that's the only signal I looked at.

If that holds up, brotli==1.2.1 in the six requirements plus the rerun helper might be easier for people than the source archive and the compiler note. The check_brotli.py assert would still work as the guard.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants