test(sweep): Sourcify corpus sweep harness for the Slang frontend by nebasuke · Pull Request #685 · NomicFoundation/solx

nebasuke · 2026-09-02T15:17:41Z

Sourcify test for 0.8.34 and >= cancun used to previously test solx. Using this PR to get an idea of how much we can successfully compile at this point, and to find bugs.

Claude summary

Adds a CI sweep that compiles every verified Sourcify contract for solc 0.8.34 and 0.8.36 with evmVersion >= cancun (119,117 contracts) through slang and reports what fails and how. Each failure is re-run with solx 0.1.8, so a slang gap (slang-fail) is told apart from input that neither compiles (both-fail). The report ranks failures by signature with example contracts, which makes it a progress meter for slang now and a regression net later, once slang-fail is low enough to gate on.

Runs on every push to main, and on PR pushes while the PR carries ci:sourcify-sweep. On a PR the report is a sticky comment; on main it is the run summary.
Report-only: only harness breakage fails the run.
24 shards, 12–18 minutes each on warm caches.
The latest run on this branch compiled 99.24% of the corpus (703 slang-only failures, 206 shared with 0.1.8).
tests/sourcify-sweep/README.md covers local runs and the corpus pin.

github-actions · 2026-09-02T15:28:38Z

Coverage Summary

Crate	Line Coverage	Function Coverage
solx	🟢 83.8%	🔴 20.0%
solx-benchmark-converter	🔴 0.0%	🔴 0.0%
solx-codegen-evm	🟡 56.6%	🟡 51.5%
solx-compiler-downloader	🔴 0.0%	🔴 0.0%
solx-core	🔴 45.7%	🟡 54.8%
solx-dev	🔴 2.2%	🔴 2.2%
solx-mlir	🔴 49.7%	🔴 47.9%
solx-slang	🔴 28.6%	🔴 34.5%
solx-solc-test-adapter	🔴 1.7%	🔴 2.1%
solx-standard-json	🔴 38.7%	🔴 39.4%
solx-tester	🔴 36.6%	🔴 34.6%
solx-utils	🔴 37.0%	🔴 40.5%
Total	🔴 13.0%	🔴 16.1%

Codecov Report | HTML Report | Workflow Run

github-actions · 2026-09-02T16:12:40Z

Sourcify sweep (slang vs latest solx release)

slang: slang v0.2.0, LLVM-based Solidity compiler for the EVM, Front end: Slang, LLVM build: 9e8545bf4f08197f9fa9e0988d3d70adae10816f
baseline: solx v0.1.8, LLVM-based Solidity compiler for the EVM, Front end: solc, LLVM build: 7d0702e169889fe4f1a2241c57bef7d2c1c68737
contracts: 119117

outcome	count	share	meaning
ok	118668	99.62%	compiles: no errors and bytecode for the target source
slang-fail	258	0.22%	candidate fails, baseline compiles (or no baseline given)
both-fail	191	0.16%	candidate and baseline both fail: corpus artifact or documented compiler limitation

Candidate-only failures (258)

kinds: error 258

count	kind	signature	examples
221	error	This contract cannot be compiled due to a combination of a memory-unsafe assembly block and a stack-too-deep error. The compiler can automatically fix the stack	`100_0xdbdbbbb35a21e6a52b63ba942026e92ddb1e0ff2`, `100_0xf5e903c0cbf8d82349fc8c0f160380a44b0f42d8`, `10_0x0000043fc6f4c10a667298279f7029e1c3900000`
29	error	worker crash: Assertion `Index < Length && "Invalid index!"' failed	`137_0x09a747cd58ca52fd71746a81391cd7cafad90a67`, `137_0x0ac79f4b335a16df57ba3561cfb47ba8c29bdd6e`, `137_0x0b88bcb0266898c2671aa4dbb4597fa2d6a45b51`
3	error	LLVM error: Stackification failed for '_placeInLevelMatrix(address,address,uint8)_3165' function. It is recursive and has stack too deep errors. Consider refact	`56_0xe45fc295347ccf7dab32347be4bd37d13a0a93be`, `56_0xfdaba2bdb4aef7e6bc413fbc3e9800e8b7b8e933`, `97_0xbf9c141a7db849a62be61e1191fb3725413ae581`
2	error	LLVM error: Stackification failed for '_placeInLevelMatrix(address,address,uint8)_3640' function. It is recursive and has stack too deep errors. Consider refact	`56_0xd365458623e766304ffdef2eab590c363a2948d6`, `97_0x1a9ea54ad1cf25ee0489c0c9996d5d8db166f524`
2	error	No unique declaration found for 'initialize'.	`421614_0x1cf7c8e232cee49c592487986c5fdbdac2dbadde`, `42161_0x5bcb52965004561ed4d78366a36d0b1b11f6cca4`
1	error	LLVM error: Stackification failed for '_releaseMatrixIncome(address,uint8)_3049' function. It is recursive and has stack too deep errors. Consider refactoring i	`56_0xcee18bb5958317b226ca08e7ef264c99dca56f58`

Failures shared with the baseline (baseline diagnostics) (191)

kinds: error 191

count	kind	signature	examples
176	error	runtime code LLVM IR generator: This contract cannot be compiled due to a combination of a memory-unsafe assembly block and a stack-too-deep error.	`100_0x239d413a6ac5322d3ccaaaf43e34045bdacd7e74`, `100_0x27ebdb6cefd590feaf79b20f6e77bf79dc3c04fe`, `100_0x325afb837204d46a3d4158ded26a8be2681761b5`
7	error	LLVM error: Stackification failed for '_createAndPlaceSlot_rt_547' function. It is recursive and has stack too deep errors. Consider refactoring it to use a non	`97_0x8a5577fbd87bae4551b275f409d3285df69836be`, `97_0x9e2e5aa409d9d6db58e206c52a83e7513e9d22ac`, `97_0xa0dcfd55389c6aece6ded5130c0df195f36191c8`
5	error	Undeclared identifier.	`11155111_0x0dcf68b5416eaaf96e896809a33c404f30a06c7c`, `11155111_0x74908cc7dc1416fc57fd4ef65d849020b18253b8`, `11155111_0xc1335a802c09380d26202f0178464c3ef4e48fce`
1	error	LLVM error: Stackification failed for 'fun__placeInLevelMatrix_2414' function. It is recursive and has stack too deep errors. Consider refactoring it to use a n	`11155111_0x3a37206d0b789561e9c8bd5151ece8a8318b4fe3`
1	error	LLVM error: Stackification failed for 'fun__placeInLevelMatrix_2778' function. It is recursive and has stack too deep errors. Consider refactoring it to use a n	`97_0x6183dc1cfe56178bbb698c1db4b5aed3aabf8720`
1	error	Member "UnexpectedInitSelector" not found or not visible after argument-dependent lookup in type(contract ParkERC1967Proxy).	`56_0x69d890f783a6662e8554c8f71d1daeda1baddda2`

nebasuke · 2026-09-02T16:40:23Z

Run 10 (523c2d4 = main 5e6ec5e + harness, run 38559070156): 118,668 / 119,117 ok (99.62%), 258 Slang-only failures, 191 shared with 0.1.8, 0 timeouts. Excluding the 410 contracts that fail on stack spilling on either leg (memory-unsafe assembly + stack-too-deep, or recursive functions): 118,668 / 118,707 ok (99.97%).

62 Slang-only failures now compile: the llvm.switch case wider than i64 (solx-llvm#168), picked up by #803's solx-llvm bump. 2 regressions against run 9: MapstoneMarketplace.sol (one source deployed on mainnet and Sepolia) compiled in run 9 and now hits memory-unsafe assembly + stack-too-deep. 0.1.8 fails it the same way, so the report counts it as shared. Not bisected yet; main moved by #803, #814, #824 and two solx-llvm bumps since run 9.

Outstanding:

count	cause	fix	state
221	memory-unsafe assembly + stack-too-deep: real stack pressure the solc pipeline doesn't have	solx-llvm#170 clears 6	nothing tracks the rest
29	`Index < Length` assert in the SelectionDAG list scheduler	solx-llvm#178, fixed by solx-llvm#181	approved, not merged; then needs a solx-llvm bump
6	recursive functions need spills	#687, #715	open
2	`No unique declaration found for 'initialize'`	slang#2149, fixed by slang#2249	draft, then needs a slang bump

Run 9

Run 9 (3864c9c = main 62d9436 + harness, run 37931555773): 118,608 / 119,117 ok (99.57%), 320 Slang-only failures, 189 shared with 0.1.8, 0 timeouts, 0 regressions against run 8. 313 Slang-only failures now compile, all through #809's Slang bump: 310 memory-unsafe assembly + stack-too-deep (slang#2230) and 3 isa<LLVMStructType> aborts (slang#2217).

Run 8

Run 8 (3d5ce62 = main 6b37249 + harness, run 37821385685): 118,295 / 119,117 ok (99.31%), 633 Slang-only failures, 189 shared with 0.1.8, 0 timeouts, 0 regressions against run 7. 70 Slang-only failures now compile: 60 address literals (slang#2184), 6 bare bound members (#793), 2 void returns (#794), 2 abstract interfaceId (#792). The other 2 interfaceId contracts now hit the stack-too-deep row. 17 shared failures cleared because the harness's pragma rewrite no longer eats code.

Run 1

Run 1 (d35b135 = #671 + harness, run 33651331011): 66,142 / 112,587 ok (58.75%), 46,320 Slang-only failures, 125 shared with 0.1.8 (the known stack-too-deep gates), 0 timeouts. Single-file contracts pass at 96%, multi-file at 10%.

Seven signatures cover 99.4% of the failures (buckets are first-failure only, so fixing the top one will unmask more of the lower rows):

count	cause	pointer
26,286	contracts with a contract base are skipped without a diagnostic: exit 0, target missing from the output	`solx-slang/src/source_unit.rs` inheritance TODO. Needs a loud error in the meantime.
371	same gap via factories: `new Token()` where `Token` inherits, so lld reports an undefined `__datasize__$<hash>$__`	goes away with the above
10,457	`bytes memory` indexing lowered as `sol.fixed_bytes_index`; verifier rejects `!sol.string<Memory>`	`index_access.rs`, `Type::ByteArray` arm
7,541	non-relative imports, i.e. remappings not applied	#684, merged into this branch for run 2
1,334	URL-style source names: `Path::components()` collapses `//`, so `../utils/Context.sol` never matches its key (surfaces as "Identifier already declared")	`compilation_config.rs` `resolve_import`; #684's tip commit may already cover it
151	LLVM abort `Unexpected data location cast`	Sol dialect, solx-llvm
45	`payable(x).transfer(uintN)` / `send` without widening to 256 bits	transfer/send lowering
42	slang binder panic, index out of bounds in `p5_resolve_references/resolution.rs`	upstream slang rev 8948967

Long tail (≤ 27 each): LLVM worker assertion Index < Length, call{value:} in a value position, using L for T member calls, sol.and/sol.xor operand type mismatch.

The first report's signatures hid most of this (Sol pass pipeline failed for the verifier errors, one bucket per __datasize__ hash), so de79e8b moves bucketing into report.py over the recorded diagnostics; run 2's comment will show these buckets directly, with example contract ids.

hedgar2017

Should become part of solx-dev next to foundry and hardhat subcommands as discussed.

nebasuke · 2026-10-13T15:40:39Z

To grab the contracts that spill on slang but compile with solx 0.1.8 (run 9: 221 memory-unsafe assembly + stack-too-deep, 6 recursive stackification):

gh run download 37931555773 -R NomicFoundation/solx -n sourcify-sweep-results

jq -r 'select(.outcome=="slang-fail" and (.slang.signature | test("stack-too-deep|Stackification failed"))) | "./contracts/\(.id).json"' results.jsonl > spills.txt

curl -fsSLO https://github.com/nebasuke/sourcify-corpora/releases/download/sourcify-corpus-solc-0.8.34-0.8.36-20260920/sourcify-corpus-solc-0.8.34-0.8.36-20260920.tar.xz
mkdir spills && tar -xJf sourcify-corpus-solc-0.8.34-0.8.36-20260920.tar.xz -C spills -T spills.txt

Drop |Stackification failed for the stack-too-deep ones only. The records are corpus records, not standard JSON; tests/sourcify-sweep/run.py --bin <slang> --baseline <solx-0.1.8> --corpus spills --out r.jsonl --rewrite-pragmas compiles them the way CI does. For a later run, swap the run id.

Compiles the pinned Sourcify corpus (112,587 verified solc 0.8.34 contracts, evmVersion >= cancun) through `solx --standard-json` and reports the outcome table plus a ranked failure census. Each corpus record becomes one standard-JSON input with sources inline and evmVersion, libraries, remappings and viaIR passed through verbatim; candidate failures are re-run with the released solc-frontend solx so they split into candidate-only and both-fail. Failure kinds (error, panic, abort, no-bytecode, timeout) are recorded separately because a frontend crash ends the whole standard-JSON call. CI: `.github/workflows/sourcify-sweep.yaml`, label-gated on `ci:sourcify-sweep` or workflow_dispatch, 24 hosted shards that each extract only their slice of the corpus, summarize job posts the report as a sticky PR comment. Report-only. Ten corpus records are committed as fixtures so the harness runs offline against the workspace build.

The first full run showed the recorded first-line signature is too coarse: `Sol pass pipeline failed` hid the MLIR verifier message in stderr, `__datasize__$<hash>$__` link errors split one bug into hundreds of buckets, and the shared-failure census showed the candidate's kind instead of the baseline diagnostic that explains the contract. Bucketing now lives in report.py (`refine`) over the raw material run.py records, so old results re-render: MLIR verifier and worker-crash lines are pulled from stderr, import failures split into non-relative / URL-style / relative, panics carry their source location, hashes and literal `\n` are normalised. run.py records which contracts did come out on a no-bytecode result.

The Slang frontend checks pragmas against Slang's latest version (0.8.36) while the baseline is a 0.8.34 compiler, so any exact pin fails one leg before parsing and the outcome classification is lost. With --rewrite-pragmas each leg compiles the source with every version pragma replaced by the version its own --version reports; the corpus stays the verbatim Sourcify record. Pragmas have no effect on codegen.

Sourcify holds only 9,296 solc 0.8.36 compilations, too few for a census on their own, so the corpus is the 0.8.34 release (byte-identical half) plus the 6,530 unique 0.8.36 contracts, extracted with the same query and layout. The frontend's 0.8.36 language version is handled by the per-leg pragma rewrite, so the records stay verbatim.

#751 removed the build-slang alias and build-toolchain's solc input; the default build is the Slang pipeline now.

A full sweep is cheap enough to run per merge, so main gets a census for every push instead of only when someone labels a PR. On a PR the label now keeps the sweep running on each push rather than needing a re-label. Main runs are not cancelled by the next merge so each one is attributable.

The pattern matched any text up to the next `;`, so a comment mentioning "pragma solidity" lost the code after it, often an import. Both legs saw the damaged source and 17 verified contracts landed in both-fail with `Undeclared identifier`, hiding a frontend gap. Matching only version-expression characters leaves those comments alone; real pragmas, including ones split across lines, are still rewritten.

A backslash inside an f-string expression is a syntax error before PEP 701, so report.py did not load on older local interpreters.

GitHub keeps one pending run per group, so with a shared main group a third push replaced the queued run and that commit got no census. The shard timing comment now matches measured runs.

#791 renamed the compiler binary to `slang`, so the build step's `cp target/release/solx` fails every shard. The candidate leg, its outcome (`slang-fail`) and its result key now say slang; solx is only the 0.1.8 baseline, where the old names read as the wrong compiler. Results recorded before this commit no longer render with report.py.

The sweep belongs next to the Foundry and Hardhat runners, on the same infrastructure: compilers come from the downloader's dev-compilers.json and are chosen by the Hardhat config's reference/candidate flags, and pragmas are anchored at --solidity-version like the Hardhat runner, so both compilers get the same input. The corpus is fetched with the downloader's resumable download, now public, and sliced with tar. `solx-dev test sourcify-report` renders the same report as report.py did, minus the trait table and wall-time line; on runs 1-3 and 6-10 the output is byte-identical to report.py with those removed.

Shards build slang and solx-dev only and hand the corpus fetch, slicing and baseline download to `solx-dev test sourcify`; shard 0 uploads solx-dev so the summary job renders the report without a checkout or Python. The report title no longer names the baseline tag, which now lives in dev-compilers.json; the version lines under it still do.

nebasuke added the ci:sourcify-sweep Run Sourcify corpus on this PR label Sep 2, 2026

nebasuke added ci:sourcify-sweep Run Sourcify corpus on this PR and removed ci:sourcify-sweep Run Sourcify corpus on this PR labels Sep 2, 2026

nebasuke mentioned this pull request Sep 2, 2026

fix(slang): constants at their declared type, compound assignment on bytes elements #686

Closed

abinavpp force-pushed the app-slang-inline-assembly branch 2 times, most recently from 2d34148 to a98f2e6 Compare September 4, 2026 08:07

abinavpp force-pushed the app-slang-inline-assembly branch 2 times, most recently from c237244 to 6d41503 Compare September 16, 2026 13:18

hedgar2017 force-pushed the app-slang-inline-assembly branch from 6d41503 to 96cebda Compare September 16, 2026 16:03

abinavpp force-pushed the app-slang-inline-assembly branch 3 times, most recently from ed6c4df to 31ee803 Compare September 17, 2026 12:29

Base automatically changed from app-slang-inline-assembly to main September 17, 2026 19:59

nebasuke force-pushed the slang-sourcify-sweep branch from 69d842a to b123248 Compare September 20, 2026 12:39

nebasuke changed the base branch from main to az-slang-modifiers September 20, 2026 12:39

nebasuke added ci:sourcify-sweep Run Sourcify corpus on this PR ci:compile-benchmark Run hyperfine compilation benchmark for this PR and removed ci:sourcify-sweep Run Sourcify corpus on this PR labels Sep 20, 2026

nebasuke force-pushed the slang-sourcify-sweep branch from 88b9e22 to 3dc23b2 Compare September 20, 2026 14:41

nebasuke added ci:sourcify-sweep Run Sourcify corpus on this PR and removed ci:sourcify-sweep Run Sourcify corpus on this PR labels Sep 20, 2026

This was referenced Sep 20, 2026

fix(slang): encode abi.encodeCall arguments out of memory #721

Merged

[v2] fix(typing): locate a constant state variable in memory NomicFoundation/slang#2163

Merged

nebasuke force-pushed the slang-sourcify-sweep branch from 3dc23b2 to fa0180e Compare September 30, 2026 12:22

nebasuke added ci:sourcify-sweep Run Sourcify corpus on this PR and removed ci:compile-benchmark Run hyperfine compilation benchmark for this PR ci:sourcify-sweep Run Sourcify corpus on this PR labels Sep 30, 2026

nebasuke force-pushed the slang-sourcify-sweep branch from fa0180e to 052ead9 Compare September 30, 2026 22:57

nebasuke added ci:sourcify-sweep Run Sourcify corpus on this PR and removed ci:sourcify-sweep Run Sourcify corpus on this PR labels Sep 30, 2026

nebasuke force-pushed the slang-sourcify-sweep branch 2 times, most recently from 97f479e to 3a1e869 Compare October 4, 2026 13:26

This was referenced Oct 4, 2026

feat(slang): lower type(C).interfaceId for abstract contracts #792

Merged

List scheduler asserts when a later result of a multi-value call feeds a loop phi NomicFoundation/solx-llvm#178

Open

nebasuke marked this pull request as ready for review October 4, 2026 14:21

nebasuke requested a review from a team October 4, 2026 14:21

nebasuke force-pushed the slang-sourcify-sweep branch 2 times, most recently from 3d5ce62 to 3864c9c Compare October 9, 2026 12:40

hedgar2017 requested changes Oct 10, 2026

View reviewed changes

nebasuke added 12 commits October 14, 2026 19:50

ci(sweep): build with plain cargo build after solc pipeline removal

81ce3d9

#751 removed the build-slang alias and build-toolchain's solc input; the default build is the Slang pipeline now.

fix(sweep): render the report on Python before 3.12

930fc9d

A backslash inside an f-string expression is a syntax error before PEP 701, so report.py did not load on older local interpreters.

ci(sweep): give each main push its own concurrency group

1e0be49

GitHub keeps one pending run per group, so with a shared main group a third push replaced the queued run and that commit got no census. The shard timing comment now matches measured runs.

nebasuke force-pushed the slang-sourcify-sweep branch from 31e0840 to 49b8c6d Compare October 14, 2026 18:50

Navigation Menu

Repository navigation

test(sweep): Sourcify corpus sweep harness for the Slang frontend - #685

test(sweep): Sourcify corpus sweep harness for the Slang frontend#685
nebasuke wants to merge 12 commits into
mainfrom
slang-sourcify-sweep

nebasuke commented Sep 2, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 2, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 2, 2026 •

edited

Loading

Uh oh!

nebasuke commented Sep 2, 2026 •

edited

Loading

Uh oh!

hedgar2017 left a comment

Uh oh!

nebasuke commented Oct 13, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

nebasuke commented Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Claude summary

Uh oh!

github-actions Bot commented Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Coverage Summary

Uh oh!

github-actions Bot commented Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Sourcify sweep (slang vs latest solx release)

Candidate-only failures (258)

Failures shared with the baseline (baseline diagnostics) (191)

Uh oh!

nebasuke commented Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hedgar2017 left a comment

Choose a reason for hiding this comment

Uh oh!

nebasuke commented Oct 13, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

nebasuke commented Sep 2, 2026 •

edited

Loading

github-actions Bot commented Sep 2, 2026 •

edited

Loading

github-actions Bot commented Sep 2, 2026 •

edited

Loading

nebasuke commented Sep 2, 2026 •

edited

Loading