Repository navigation
Conversation
|
nebasuke
added
the
ci:sourcify
Contributor
|
Bencher Report| Branch | sourcify-full-tier |
| Testbed | ci |
⚠️ WARNING: Truncated view!The full continuous benchmarking report exceeds the maximum length allowed on this platform.
🚨 2 Alerts
🐰 View full continuous benchmarking report in Bencher
Contributor
Sourcify corpus run for dbab9b0
Known issues
Deliberate deviations (900 contracts)
Gate: passed. |
nebasuke
added this pull request to stack #2201
September 25, 2026 12:59
nebasuke
force-pushed
the
sourcify-full-tier
branch
from
October 9, 2026 14:25
34774e9 to
7e76dc6
Compare
nebasuke
removed
the
ci:sourcify
The lock file now lists every per-version asset of the pinned release with its size, and `lock shard` bin-packs them into shards by compressed bytes, which tracks source text and so compile time better than contract counts do: eight shards land within 1% of each other without splitting an asset. A shard fetches, verifies, runs and deletes one asset at a time, so its disk never holds more than one. The workflow fires on the `ci:sourcify` label, on demand, and on pushes to main that touch the v2 crates; main runs queue instead of cancelling so every merge keeps its own verdict. The report stays report-only until the first census is triaged into expected-failures.toml. Both workflows now go through `task sourcify:*`, so the YAML carries no logic.
An empty dispatch value and a flow-style matrix; yamllint needs pipenv, which the sandbox lacks, so CI caught it.
Six gaps in the corpus import resolver, each worth a bucket of the Sourcify census: a `:` inside a Windows drive target was read as the context separator and the remapping dropped; `./` and `../` imports were remapped before being resolved against the importing file, so `./=lib/` broke every relative import; among matching remappings the longest prefix won where solc takes the longest context first and, on a tie, the last one listed; a relative import from a URL source lost its port; and a remapping was skipped when a source happened to exist under the unremapped name. About 370 contracts, spread over six diagnostic codes.
Every bucket of the first census over all 2,326,676 verified 0.8.x contracts is now fixed in the harness, tracked by an issue with a repro the matching solc accepts, or recorded as a deliberate divergence, so the report job fails on a new bucket and on a listed one that stopped firing. Issues from the census: #2188, #2189, #2190, #2191, #2192,
The full tier fired only on the label event, so a labelled PR had to be relabelled after every push. It now runs on every push to a PR that carries the label, skips unlabelled and fork PRs, and cancels a stale PR run the way the benchmark workflows do; main pushes still queue.
A bucket keyed on the diagnostic code alone tolerated every future occurrence of a generic code such as `identifier-not-found` because one contract's follow-on failure was listed. An entry can now name a message prefix and the contracts it covers; occurrences no entry matches are unexpected. On the full corpus an issue-tracked entry claiming 50 contracts or fewer must list them, so an entry narrows as its bug gets fixed. Deliberate divergences on a dedicated code stay bucket-wide.
nebasuke
force-pushed
the
sourcify-full-tier
branch
from
October 9, 2026 16:02
8707da8 to
f89ead7
Compare
nebasuke
added
the
ci:sourcify
A failing gate is an expected outcome, but `report` only returned it as an error after writing the census, so the job summary and the PR comment after it were skipped and the sticky comment kept showing the last green run. The verdict is now the report's closing section; the Report step always runs with `--report-only`, and a final Gate step re-runs `report` to fail the job. The smoke workflow writes its summary before passing the PR tier's exit status on.
Collapsing `//` on both sides let `utils//SafeERC20.sol` resolve to `utils/SafeERC20.sol` when a record carries both, so two library copies solc keeps apart became one; the exact path now wins and the collapsed comparison is only the fallback.
#2179 and #2180 are fixed; the 62 contracts left are #2150, #2251, three new issues (#2259 private functions through `using` on a function type, #2260 nested fixed-size arrays across locations in overloads, #2261 a getter next to an inherited function), and records solc rejects too when compiling all their sources.
…iberate A failure asks something different of the reader depending on what claims it: nothing (the gate fails), an issue (work to pick up) or a deliberate entry (never to be fixed). The totals count each failing contract once under its most urgent category, and the buckets come in three tables in that order, the issue first in the work list and the deliberate ones collapsed with their reasons. Everything is derived from the gate's per-occurrence claims, so the report cannot disagree with the verdict.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three buckets of failure types:
Claude summary
Runs every verified solc 0.8.x contract on Sourcify (2,326,676 in the pinned release, 38 versions)
through
run-corpusin eight shards, on theci:sourcifylabel, on demand, and on pushes tomainthat touch the v2 crates (the runner, its lock and its expected failures among them) orthe shared import resolver.
Shards from the lock file.
corpus.locknow lists each per-version asset with its size,contract count and sha256.
lock shard --count N --index ibin-packs the assets by compressedbytes, largest first onto the lightest shard, which tracks source text and therefore compile time
better than contract counts do: eight shards come out between 950 and 960 MB each without splitting
an asset, holding between 195k and 386k contracts.
lock prandlock releaseserve the PR tierand the download, so the workflows parse nothing themselves.
One asset at a time.
scripts/tasks/sourcify/run-assets.shreads the asset lines, and foreach one downloads it from the release, checks the sha256, unpacks, runs
run-corpusinto oneJSONL per asset, and deletes it before the next. A shard's disk never holds more than one asset,
about 15 GB uncompressed for the largest. A failing asset does not stop the others.
Tasks, not YAML.
task sourcify:build|testdata|pr|shard|reportwrap the binary and thescript;
sourcify_smoke.ymland the newsourcify_full.ymlcall them. The full workflow buildsonce, fans out over the shard matrix, then one job restores every shard's JSONL and writes the
census, ending in the gate's verdict, to the job summary, an artifact and, on label runs, a sticky
PR comment. A failed gate is a result, not an error, so all three happen either way; a last
Gatestep re-runs
report --stale-checkto fail the job.Queue, don't cancel. The concurrency group has no
cancel-in-progress, so a burst of mergesto
mainruns one census per merge, in order, and every commit keeps its own verdict.Gated from the start. The first census over the whole corpus is triaged: every bucket is fixed in the
harness, tracked by an issue, or recorded as a deliberate divergence, so the report job fails on a new
bucket and on a listed one that stopped firing. An entry is keyed on the diagnostic code and narrowed
by a message prefix or a list of contracts where the code alone is too coarse; an issue-tracked entry
claiming 50 contracts or fewer must list them, so entries narrow as bugs get fixed. Deliberate
divergences on a dedicated code stay bucket-wide.
Resolver fixes (
solidity_testing_utils::import_resolver, each with a unit test): a:insidea Windows drive target was read as the context separator and the remapping dropped;
./and../imports were remapped before being resolved against the importing file, so a
./=lib/entry brokeevery relative import; among matching remappings the longest prefix won where solc takes the longest
context first and, on a tie, the last one listed; a relative import from a URL source lost the port;
a remapping was not applied to the normalised relative path when an unremapped source happened
to exist; and an import spelled with
//(utils//SafeERC20.sol) now resolves to that sourceunit when the record also carries
utils/SafeERC20.sol, as solc keeps them apart. Together thesewere about 370 of the census's failures, spread over six buckets, and the harness no longer
prefixes a
:to remappings itself.Issues filed from the census, each with a repro the matching solc accepts and Slang rejects:
#2188 (override checks apply the solc 0.8.12 rule to 0.8.8–0.8.11, 194 contracts), #2189, #2190,
#2191 (three conditional-expression shapes), #2192 (duplicate
indexedbefore 0.8.18), #2193(
(msg, 'text');), #2194 (names hidden inside import cycles), #2195 (a constant imported throughtwo routes is ambiguous in assembly), #2253 (
abi.decode(data, (address))isaddress payablein solc, exposed by #2185's conversion check), #2257 (an array literal of internal functions passed
to an overload). Conversions through a name an import cycle declares twice also fail under #2194
(16 contracts, noted there). #2149 covers the inherited-overload selector; #2179 (address-literal
overloads) and #2129 (receive-function buckets) are fixed and their entries gone.
Deliberate divergences, Bas's call each:
pragma abicoder v1,pragma experimental SMTChecker,a second abicoder pragma solc only rejects in one order, and the version pragmas solc's matcher
swallows (
0.8.34.0,*0.8.7,^0.8.1.,^0.8.20 - <0.8.31), and one event called withoutemit, which solc before 0.8.4 let through after anemitof an overload of the same event. Twoentries record corpus artefacts that solc rejects too.
Retiring the tarball-based
testcommand and the per-chain workflows is a separate PR after thishas run once; #2199, stacked on this, checks the ABI and storage layouts against the records'
artifacts.Numbers
sourcify_fullon CI (run 37973676027, this branch's head), eight shards, 15 minutes end to end:21 buckets, all listed; the gate passes with
--stale-check. The largest arepragma abicoder v1(715, deliberate), #2149 (642), #2188 (194 over three codes) and the swallowed version pragmas
(137, deliberate).