A reusable corpus of 734 open-source models for measuring and improving torch.compile quality. Structured, reproducible, extensible.
Primarily compiler developers working on torch.compile, and secondarily skill maintainers needing clean per-pattern fixtures for diagnostic-skill evaluation.
Workflows:
- Find & fix graph breaks — reproduce any break with one command, see root causes and fix hints, prioritize by impact across 734 models
- Prioritize work — see which break categories affect the most models, track version-over-version progress, identify high-ROI fixes
- Validate changes — test compiler changes, dynamo flags, or diagnostics against a known corpus of real-world breaks
- Surface numerical divergences — compare eager vs compiled forward outputs to surface compiler-introduced numerical errors. Runs on every identify-pass model where comparison is possible, for any
--compile-kwargs/--dynamo-flagscombination (default fullgraph=True+eager, fullgraph=False, custom backends like inductor — all produce anumeric_statusresult) - Per-pattern fixtures for diagnostic-skill evaluation (in scoping) — clean, isolated, single-model reproducers tied to known root causes, suitable for unit-test-style checks of compiler-diagnostic skills. The corpus is not a general skill-eval source; for broader skill capability evaluation (multi-step reasoning, ambiguous diagnosis, richer Q&A) use the doc-eval project's Q&A corpus instead. See USE_CASES.md §3 for the niche framing and current consumers.
| eval | train | |
|---|---|---|
| full_graph | 531 (67%) | 489 (62%) |
| graph_break | 177 (22%) | 219 (28%) |
| error | 82 (10%) | 82 (10%) |
240 models have graph breaks in at least one mode. Zero full_graph→graph_break regressions across four releases (2.8→2.11).
Nightly tracking and per-version details: results/
# Install
bash scripts/setup_env.sh
# Install pre-push test hook (runs the test suite when you push code; refuses on failures)
git config core.hooksPath scripts
# Browse the corpus
python3 tools/query.py
python3 tools/query.py --status graph_break
python3 tools/query.py --error deepcopy
# Reproduce a graph break (no GPU needed)
python3 tools/reproduce.py BartModel --explain| Guide | Description |
|---|---|
| Getting Started | Install, browse, reproduce your first graph break |
| Running Sweeps | Test your own PyTorch version against the full corpus |
| Running Experiments | Config-driven flag testing, ablations, reproducible random samples |
| Understanding Results | Interpret statuses, model variants, corpus format, dashboard |
| Issue Management | Post-sweep graph break classification and GitHub issue tracking |
| Contributing | Add models, fix graph breaks, architecture reference |
Full methodology, taxonomy, and analysis: design/design-doc.md
Graph break pattern analysis for PT2 team: analysis/releases/graph-break-analysis.md
Questions, bug reports, or feature requests: open a GitHub issue.
Use for:* labels to route issues:
for:dynamo-team— PyTorch Dynamo compiler issuesfor:hf-transformers— HuggingFace Transformers model/library fixesfor:corpus-tooling— corpus pipeline and tooling improvements