Repository navigation
Conversation
|
|
Bencher Report| Branch | sourcify-corpus-v2 |
| Testbed | ci |
🐰 View full continuous benchmarking report in Bencher
⚠️ WARNING: Truncated view!The full continuous benchmarking report exceeds the maximum length allowed on this platform.
5993fe2 to
1fa9c3e
Compare
| Commands::ShowCombinedResults(results_command) => { | ||
| run_show_combined_results_command(results_command) | ||
| } | ||
| Commands::RunCorpus(corpus_command) => corpus_run::run(&corpus_command), |
There was a problem hiding this comment.
@nebasuke Just as FYI, I sent #2211 to decouple/deprecate the slang v2 bits from this crate, keeping it pure to v1.
I know #2181 is still a draft, so I have not looked into the changes yet, but AFAICT, this is adding a separate workflow/reporting to the existing one. If so, I suggest adding it as a separate crates/solidity-v2/testing/sourcify crate, to decouple the dependency tree as much as possible.
There was a problem hiding this comment.
Thanks, good call. The corpus runner now lives in its own crates/solidity-v2/testing/sourcify crate (solidity_v2_testing_sourcify), and solidity_testing_sourcify is back to exactly what's on main.
1fa9c3e to
09d972f
Compare
The Sourcify suite fetched a filesystem backup that froze in October 2025 and checked it with the v1 pipeline only, so v2 got no real-world contracts at all. Corpus snapshots (one JSON per contract, sources inline) come from Sourcify's Parquet export instead, and `run-corpus` compiles them with v2, sorting error diagnostics into parse, bind and validate buckets by diagnostic code. Every contract in a snapshot was verified by solc, so any error is a false rejection. The gate is a checked-in list of tolerated buckets, each with a reason and an issue; anything else fails, and on full runs so does a listed bucket that no longer fires. The PR workflow now runs the in-repo cases and the pinned 119k-contract snapshot, which takes about half a minute of wall time on 16 threads.
50 contracts per solc 0.8.x version, 1,900 in all, picked by the hash of the file name from the 2026-09-24 extraction of every verified 0.8.x contract, so the PR tier sees every language version in seconds. The 119k-contract 0.8.34/0.8.36 snapshot moves to the full tier.
Clippy under `-D warnings` (CI's flags, not the local task's) wanted `sort_by_key`, `is_multiple_of` and no `process::exit`; `main` now returns an `ExitCode` and prints the error chain, so the corpus verdict is the last line of the log instead of an anyhow backtrace.
#2211 made `solidity_testing_sourcify` v1-only, so the corpus runner lives in `solidity_v2_testing_sourcify` and the v1 crate stays as it is on main. The smoke workflow builds both binaries. The new crate follows the v2 clippy rules (common collection aliases) and takes the default EVM target from `LanguageVersion`, which replaced the testing-utils helper on main.
09d972f to
44b9419
Compare
This branch has not been deployed
2 participants
Claude summary`.
Claude summary
run-corpuscompiles every contract of a Sourcify corpus snapshot with Slang v2 and sorts theerror diagnostics into three checks by subsystem:
parse(syntax),bind(import and nameresolution) and
validate(structure, types, semantics). Every contract in a snapshot was verifiedby solc at that exact compiler version, so an error is a false rejection by definition. Warnings
are counted, not judged.
Its own crate. The runner lives in
crates/solidity-v2/testing/sourcify(
solidity_v2_testing_sourcify), next to the v2 solc suite, and depends only on the v2 crates andthe shared, Slang-agnostic
solidity_testing_utils. The v1solidity_testing_sourcifycrate is asit is on
main, and the smoke workflow still builds it.Gate.
expected-failures.tomllists the tolerated buckets (check+ diagnostic code), eachwith a reason and an issue, or marked deliberate where Slang rejects on purpose what solc lets
through. A bucket outside the list fails the run; with
--stale-check, so does a listed bucketthat did not fire, so entries leave with the fix. Panics always fail.
--report-onlyprints thecensus and the verdict without failing.
Corpus format.
corpus.jsonpluscontracts/<chain>_<address>.jsonwith the sources inline,the solc version, the compilation target, remappings, the original
settings, and optionally theevm_versionand solc's ownartifacts(ABI, storage layouts, natspec) for output checks. solx'ssweep reads the same files.
testdata/corpusholds hand-written cases run by the unit tests andCI; adding a JSON there adds a case.
CI.
sourcify_smoke.ymlrunstestdata/corpus, then downloads the snapshot pinned incorpus.lock(release tag, asset, sha256), runs it and puts the census in the job summary; theJSONL outcomes are an artifact. The pin is the PR subset of the all-0.8.x release
sourcify-corpus-0.8.x-20260924: 50 contracts per solc version, 1,900 in all, picked by hash of thefile name. Bumping the lock is a one-line PR whose CI run shows what the new contracts change.
Sharding and reports.
--shard-count/--shard-indexslice a snapshot round-robin;reportrenders the census and applies the gate over the JSONL of any number of shards.
--print-diagnosticsprints each error with file and line.Stacked on this: #2183 runs the whole release in shards behind
ci:sourcify, #2199 checks the ABIand storage layouts against solc's artifacts.
Numbers
PR subset on CI (run 37956259073, this branch's head): 1,900 of 1,900 contracts pass, 3 of 3
in-repo cases pass.