Skip to content

feat(testing): check the ABI and storage layouts against solc's artifacts - #2199

Draft
nebasuke wants to merge 10 commits into
sourcify-full-tierfrom
sourcify-output-checks
Draft

feat(testing): check the ABI and storage layouts against solc's artifacts#2199
nebasuke wants to merge 10 commits into
sourcify-full-tierfrom
sourcify-output-checks

Conversation

@nebasuke

@nebasuke nebasuke commented Sep 25, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Claude summary`.

New title (gh pr edit 2199 --title):
feat(testing): check the ABI and storage layouts against solc's artifacts

Claude summary

Checks Slang's outputs against what the verifying solc produced. Every corpus record carries solc's
abi, storageLayout and, from 0.8.27, transientStorageLayout for its deployed contract, so the
oracle needs no solc binary and the checks add seconds to a run.

ABI (abi). Slang's ABI JSON (#2136) is paired with solc's entry by entry, by type, name and
input types, an exactly equal entry first. A pair must match field by field, down through
parameters and tuple components ([*].stateMutability, [*].inputs.internalType, …), and the
pairs must come in solc's order, which Slang reproduces: type, then name, then selector for
overloaded functions and declaration position for events and errors. Unpaired entries are missing
or extra; the order failure names the first two entries out of place.

Storage (storage_layout, storage_types, and their transient twins). The layout check
compares each variable's label, slot and offset with solc's list, in order. The types check walks
each variable's type through #2215's types table and solc's side by side, through array bases,
mapping keys and values and struct members, comparing label, numberOfBytes (modulo 2^256, as solc
counts a uint256[2**255] gap) and encoding. The two tables key types differently (TypeId against
solc's t_... names), so they are paired by walking, not by name.

Which contract. The record names the target file. The target is the record's
target_contract when it has one, else the file's single concrete contract or library, else the
contract solc's storage layout names. Records still ambiguous (18,524) skip the output checks with
a counted reason; the next corpus release carries target_contract for all of them.

Counted per contract. A contract's mismatches become one failure per code with a count, as its
diagnostics do, so bucket sizes and the rule that small issue entries list their contracts count
contracts, not variables or entries.

Version-bounded entries. An expected failure can carry below = "0.8.20" and then covers only
contracts compiled with an older solc. solc 0.8.20 fixed its ABI to list the events a contract
emits but does not define (its changelog files it under Bugfixes) and, with the same rework, ordered
same-named events by declaration instead of most derived contract first. Slang has both at every
version: two deliberate entries below 0.8.20, so either difference still fails the gate from 0.8.20
on.

Fixtures. testdata/corpus gains two records compiled with solc 0.8.30: every entry type,
overloads, a library error and event, a struct parameter and getters for the ABI; structs,
fixed and dynamic arrays, nested mappings, a recursive struct, bytes/string, an enum, a
user-defined value type, a contract and a function type, a 2^255-slot gap and transient variables
for storage.

Not here: evm.methodIdentifiers (library functions hash struct names that the ABI spells as
tuple, so only solc's own output can be the oracle; it comes with the next corpus release), and
NatSpec.

Numbers

Whole corpus (2,326,676 contracts), locally, this branch's head:

check contracts run failed skipped
abi 2,308,112 8,972: 7,810 extra and 1,164 order (6 contracts have both), all below 0.8.20 and deliberate; 4 missing under #2194 18,564 (18,524 ambiguous target, 28 no Slang ABI, 12 no deployable definition)
storage_layout / storage_types 2,308,128 0 18,548
transient_storage_layout / transient_storage_types 407,633 0 1,919,043 (1,900,507 compiled before 0.8.27)

The gate passes with --stale-check.

@changeset-bot

changeset-bot Bot commented Sep 25, 2026 •
edited
Loading

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: bc4e122

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@github-actions

github-actions Bot commented Sep 25, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

🐰 Bencher Report

Branchsourcify-output-checks
Testbedci

⚠️ WARNING: Truncated view!

The full continuous benchmarking report exceeds the maximum length allowed on this platform.

🚨 2 Alerts

🐰 View full continuous benchmarking report in Bencher

@nebasuke
nebasuke added this pull request to stack #2201 September 25, 2026 12:59
@nebasuke nebasuke mentioned this pull request Sep 25, 2026
Merged
@nebasuke
nebasuke force-pushed the sourcify-output-checks branch 2 times, most recently from ce5c468 to 545c48b Compare October 9, 2026 15:58
@nebasuke
nebasuke force-pushed the sourcify-output-checks branch 3 times, most recently from 8375d35 to 56b619c Compare October 9, 2026 18:29
@nebasuke nebasuke added the ci:sourcify Runs Sourcify corpus for 0.8.x in a PR for Slang v2 label Oct 9, 2026
This was referenced Oct 9, 2026
Draft
Draft
@nebasuke nebasuke changed the title feat(testing): check the storage layout against solc's artifact feat(testing): check the ABI and storage layouts against solc's artifacts Oct 9, 2026
@github-actions

github-actions Bot commented Oct 9, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Sourcify corpus run for bc4e122

contracts passed unexpected known issues deliberate panicked skipped
2326676 2315946 0 1005 9725 0 0
  • Known issues per check: parse 4, bind 785, validate 216, abi 4
  • Deliberate deviations per check: parse 870, bind 14, validate 19, abi 8968

Known issues

issue bucket contracts diagnostics examples message
#2149 bind:resolution/ambiguous-reference 642 1577 10_0x0218d22b2f134c5b3000dbcb768f71693238c856, 8453_0x1874152bf0b1cad19469c2ac0d9e51491c61830b, 11155111_0x0c33f2f1a1baa5ff32839355602b4d3a4effa251 No unique declaration found for 'execute'.
#2188 validate:semantic/missing-override-specifier 142 143 1_0x54275da4f379c5b8e6a90a453562c96f11e53830, 1_0x548f0034e8646d72c8ff7b46f604788929bc9f0b, 1_0x5989997bee0c978984f423f6b06dabebc214f5e4 Overriding function is missing 'override' specifier.
#2194 bind:resolution/identifier-redeclaration 72 116 1_0x6b91713319be025dcb3b3a9f2c1ca70b465b5f9b, 56_0xfd09c6bcd60a9e3b83b8a1cf23236f28bf934efd, 11155111_0x010b00cd257d8b56fdde68e67cdcb17040feb06b Identifier already declared.
#2188 validate:semantic/overriding-non-virtual-member [32 listed] 32 47 97_0x30f6c7dc912fbb88e8f78c5c5ffacbc1db414e3f, 11155111_0xbc7a1b68d13c23f198b51640b87717b24e07d6c1, 100_0x237d4d7f9295a60d6b46421fa1a8af4d4416a7a5 Trying to override non-virtual function. Did you forget to add 'virtual'?
#2188 validate:semantic/override-visibility-differs [20 listed] 20 35 11155111_0xcdc564e19f0655ef129435fa42dc5beb26dcae5a, 1_0xd43be54c1aedf7ee4099104f2dae4ea88b18a249, 8453_0x1b4fae6e7464122ad8691abbf550647d92c5f092 Overriding function visibility differs.
#2194 bind:resolution/no-matching-callable-declaration [16 listed] 16 56 42220_0x05e9be6b03a64c6a0ff48b5bbc1d36fa058a7f5e, 42220_0x2d176d1fb534d2380168f2b91610c313609ed521, 42220_0x333d7dd86d9fc56a72e41d527374e85ee864b47d No matching declaration found after argument-dependent lookup.
#2257 bind:resolution/no-matching-callable-declaration [14 listed] 14 51 202601_0x02247eb911876cda8da99baf64e84ea68d6293d6, 202601_0x496c1bbf1150c5d8129393c5ecdcdf8a783a0f03, 202601_0x496fd88ecc6024a383edc7e8f1b636d2d342dab1 No matching declaration found after argument-dependent lookup.
#2150 bind:resolution/member-not-found [13 listed] 13 13 56_0x599547680f47d8596119452e4d44ff6659b7bf0d, 1_0x1f58f1ba43a88b4ff3b4265449b26e28c08395ad, 56_0x70e9d815984b6c9395975b5106151e86d6142bec Member 'standardize' not found or not visible after argument-dependent lookup.
#2189 validate:type-system/incompatible-conditional-branches [24 listed] 13 14 80002_0x053acb92ab46cfbf6fde6419c2e7e7e394557f80, 80002_0x5cc94047e4fe0266fd2686600d90511884b8c641, 80002_0x613d15a12136aa38b36c49976752b9bd30f30cec The true and false branches of this conditional expression have no common type.
#2259 bind:resolution/member-not-found [11 listed] 11 66 84532_0x9e7345ab96ffe58078dce6dc04c05809225211a7, 84532_0xc226aea0957bd2903a619272786e1dd4535932d1, 10_0x000000c9ec71b1a39055ec631200ed0022140074 Member 'usingBatchTransfer' not found or not visible after argument-dependent lookup.
#2260 bind:resolution/member-not-found [11 listed] 11 23 252_0x0fd66456f10d9b850f37fa129236176b4f982864, 252_0x519bd566886a22f080c94b05ccc9f35902ec031b, 252_0xa46d25eebde2d136a14601ac2e0e0de8409114d1 Member 'exchange' not found or not visible after argument-dependent lookup.
#2195 bind:resolution/ambiguous-yul-reference [9 listed] 9 180 11155111_0x1322c8bfcbfb09fb9998245754d4dabe80756d3d, 11155111_0x8456aac7a9ccfd24c11aba831adcedd239053990, 10_0x1cfc4c4720504997bcdb86ee6ade450a2c48dcea Multiple matching identifiers for 'p'. Resolving overloaded identifiers is not supported.
#2251 bind:resolution/member-not-found [8 listed] 8 8 42161_0x0a3ae8b5a388d72ebf1e776fe7e5e0d98a61b474, 42161_0x4061106e15672785947c82fb9ae9c5fc67dc3fcf, 42161_0x5092808ad669e6064c74f1d2bc85bf096d3ee99f Member 'weightedSqrtPriceX96' not found or not visible after argument-dependent lookup.
#2253 validate:type-system/explicit-conversion-not-allowed [8 listed] 8 8 1_0x5be35b691f8275556b05ddf578e491a63c214889, 11155111_0x19bcd745544026e1b30b6331cabcc9c31a84a0a9, 11155111_0xe8a769be9f4c4359b6e750074ab92b9890ba64dd Explicit type conversion is not allowed from the argument's type to the target type.
#2261 bind:resolution/member-not-found [5 listed] 5 5 1_0x4b62ec784479c5520ec77de8d10c1a85d8f0850a, 11155111_0x2947e47abfd9b3de3c8667de04a45053ec4c19cf, 11155111_0x73998993d3d9a8aaff3fd2151a586a27ccf4b138 Member 'symbol' not found or not visible after argument-dependent lookup.
#2194 abi:missing [4 listed] 4 8 1_0xf7d38d771051d2d06508b5c6a9cca1aab6e05b7d, 421614_0x0d1b0e85645fbb15c18edac05160fc052c7159e4, 84532_0x726f8631b392148fd8d487cd277d80d8ee368677 solc has error InvalidInputs(uint256,uint32), slang does not
#2194 bind:resolution/identifier-not-found [4 listed] 4 4 97_0x08f0740c2ab229cefe40b12571652c7538f687f4, 97_0x60106f5a850ba3f94d7002a9c76a57d605012721, 97_0xe15ebd2609800e350cf3cd26c1d06afaeab08a8b Identifier not found.
#2192 parse:syntax/unexpected-terminal [Unexpected IndexedKeyword…] [4 listed] 4 4 1_0xd4559e5f507ed935f19208a5d50637898c192ab3, 137_0x19bdfecdf99e489bb4dc2c3dc04bdf443cc2a7f1, 137_0x5ed57b8f59f8d3bc805fc1087d8de93c78a87305 Unexpected IndexedKeyword. One of AtKeyword, CloseParen, Comma, ErrorKeyword, FromKeyword, GlobalKeyword, Identifier, LayoutKeyword, RevertKeyword, TransientKeyword was expected
#2193 validate:type-system/expression-not-a-value [1 listed] 1 1 80001_0x0c798de0dec0d17a33099d77149957dfe18905b0 This built-in cannot be used as a value.
Deliberate deviations (9725 contracts)
bucket contracts narrowed to reason examples
abi:extra 7810 [<0.8.20] Slang lists the events a contract emits but does not define (a library's) at every version; solc before 0.8.20 left them out, which its 0.8.20 changelog lists as a bug fix. 56_0x805db1b863f8107ff0930fce55ebc0f39e6fd007, 56_0x8218032819669626f2dce5d3bc9e1524a0ff6a50, 56_0x842ceb6c8ddfcae8e1c5973af256d7391691967f
abi:order 1164 [<0.8.20] Same-named events in declaration order, as solc from 0.8.20 lists them; before, solc listed the most derived contract's first. Part of the same 0.8.20 rework of event collection. 56_0xa32d03497ff5c32bcfeebe6a677dbe4a496fd918, 10_0x4f89214a1eec1171772c2fbb71019da9a24ff61a, 10_0x5409de30f88fc4500ea2ddf59160c3dd770b55ca
parse:syntax/unsupported-abicoder-v1 715 pragma abicoder v1 selects the legacy ABI coder; Slang supports v2 only, on purpose. 1088_0x0000000000924fb1969e719eded2fed54afb183a, 1088_0x0000000009dc0fd7ba31d83a5d328b9bcaa0bc8d, 1_0x68c1c5ca6618611387242252887b6a5abd15319c
parse:syntax/incompatible-version-pragma 137 Four-component versions (pragma solidity 0.8.34.0;, ^0.8.17.0): solc's matcher ignores the fourth component, Slang rejects the malformed version on purpose. Every contract in this bucket has one. 43113_0x01365a11d5e977a8786330007b0884600450f528, 43113_0xf4bc57bee33f334ec5ee731be433ae4ce94bc538, 1_0x97ae13da39ba8f8b281a744308fa1b5ae302eb60
validate:structure/duplicate-abicoder-specifier 18 pragma abicoder v2 followed by pragma experimental ABIEncoderV2: solc only rejects the reverse order, Slang rejects any second selection. 1_0xd042a64eac89927fd92d1a5935e0e0cc30b2018c, 1_0xd9957a92d49d5207dd6c511954a5eb1c890f26b6, 137_0x141463b2a0e8559817e013f7a1e73f2731acbdba
bind:resolution/member-not-found 13 [13 listed] Records solc rejects too when compiling all their sources: a broken file next to the target. 56_0x17414c35d28a6be093a945c70afd638dd5b726c0, 56_0x80e4dd3ba70dce6c58b7dc838dff9b02f1e4809b, 56_0x9d54c57f4c4c1e2cbee3b947fb5684313588c8d8
parse:syntax/unexpected-terminal 9 [Unexpected Pragma…] Version pragmas solc's matcher swallows (^0.8.1., ^0.8.20 - <0.8.31), rejected on purpose. 5_0x34855f952eb00dec7e2bd9ac5f201db364335817, 5_0xbf7101f4106f897cecf59eb858980d5a56695bd7, 4_0x1f9221158fccc919a0e91ae45e7ffdf3d618a5cb
parse:syntax/invalid-version-specifier 8 pragma solidity *0.8.7: solc's version matcher accepts the stray *, Slang rejects the malformed specifier. 11155111_0x1fe69747ab1118d3cc8f315d9c04652201b603e7, 11155111_0x0e1ee9797eceb77384733eb33a37ea39092c7e58, 97_0x4931459a69d855eeb5696796b0db8dcfcdc2d920
bind:resolution/identifier-not-library-name 1 [1 listed] Follow-on of the trailing-dot pragma rejected on purpose: a using directive naming a library from the file that failed to parse. 11155111_0x19e2b7829b7338f89b574289a29cc8b8ef7c4acb
bind:resolution/imported-declaration-not-found 1 [1 listed] Follow-on of a version pragma with a trailing dot (pragma solidity ^0.8.1.;), rejected on purpose: the file never parses, so nothing can be imported from it. 11155111_0x19e2b7829b7338f89b574289a29cc8b8ef7c4acb
bind:resolution/member-not-found 1 [1 listed] Follow-on of the trailing-dot pragma rejected on purpose: a member of the file that failed to parse. 11155111_0x19e2b7829b7338f89b574289a29cc8b8ef7c4acb
bind:resolution/no-matching-callable-declaration 1 [1 listed] An event called without emit, which solc before 0.8.4 missed after an emit of an overload of the same event. 3_0xd38232c64b617f89a4a7a16eab4ae869aa76c424
parse:syntax/unsupported-experimental-smt-checker 1 pragma experimental SMTChecker asks for a checker Slang does not ship; solc treats it as a no-op without a solver. 1_0xfd9cd8c0d18cd7e06958f3055e0ec3adbdba0b17
validate:semantic/linearisation-impossible 1 [1 listed] Not a divergence: this record bundles sources solc rejects for the same reason; the verified target is a single library. The harness compiles every file in a record, not the target's import closure. 56_0xfd09c6bcd60a9e3b83b8a1cf23236f28bf934efd

Checks skipped:

  • 1900507 × transient_storage_layout: no transientStorageLayout in the artifacts (100_0x010e663f9510a032e1f403f2c9de28f40d3949b8, 8453_0x02ccdd23cc63c112082be01856cb1517a9d27dc5, 100_0x020f0b8598508eef513f26dc38c4ea38f9bbca35)
  • 18524 × abi: ambiguous target: the file has several deployable definitions (56_0x7d4f13c39e67b031216003bb6826533044f0c3d6, 8453_0x3a64ec3606ff7310e8fad6fcc008e39705fb496d, 42161_0xf3eee5c7d3bed32a9bf70433e9882cef7ecfd5f0)
  • 18524 × storage_layout: ambiguous target: the file has several deployable definitions (56_0x7d4f13c39e67b031216003bb6826533044f0c3d6, 8453_0x3a64ec3606ff7310e8fad6fcc008e39705fb496d, 42161_0xf3eee5c7d3bed32a9bf70433e9882cef7ecfd5f0)
  • 18524 × transient_storage_layout: ambiguous target: the file has several deployable definitions (56_0x7d4f13c39e67b031216003bb6826533044f0c3d6, 8453_0x3a64ec3606ff7310e8fad6fcc008e39705fb496d, 42161_0xf3eee5c7d3bed32a9bf70433e9882cef7ecfd5f0)
  • 28 × abi: Slang computed no ABI for the target (42220_0x02b23d4c563a29a4d9c87c8bb1fcc6cec25e41ed, 42220_0x078954f6fcc2b2537118bb67ae0db6ea096f93b3, 42220_0x09a32b7825e96ab59d4dd0d47b22e96082dd5d02)
  • 12 × abi: no concrete contract or library in the target file (1_0xd4559e5f507ed935f19208a5d50637898c192ab3, 5_0x34855f952eb00dec7e2bd9ac5f201db364335817, 5_0xbf7101f4106f897cecf59eb858980d5a56695bd7)
  • 12 × storage_layout: Slang computed no storage layout for the target (42220_0x02b23d4c563a29a4d9c87c8bb1fcc6cec25e41ed, 42220_0x10569f0240b99e5fc996446fc21171a0e1cd2555, 42220_0x2a2e2b0734ff4e5452fd7a73eedc082bb42ead32)
  • 12 × storage_layout: no concrete contract or library in the target file (1_0xd4559e5f507ed935f19208a5d50637898c192ab3, 5_0x34855f952eb00dec7e2bd9ac5f201db364335817, 5_0xbf7101f4106f897cecf59eb858980d5a56695bd7)
  • 12 × transient_storage_layout: no concrete contract or library in the target file (1_0xd4559e5f507ed935f19208a5d50637898c192ab3, 5_0x34855f952eb00dec7e2bd9ac5f201db364335817, 5_0xbf7101f4106f897cecf59eb858980d5a56695bd7)

Gate: passed.

@nebasuke
nebasuke force-pushed the sourcify-output-checks branch from 2c330e0 to ba297dd Compare October 9, 2026 20:01
nebasuke added 10 commits October 9, 2026 20:20
@nebasuke
feat(testing): check the storage layout against solc's artifact
b0532e1 
Every corpus record carries the storage layout the verifying solc
produced for its deployed contract. `storage_layout` compares each
item's label, slot and offset with it and `storage_types` the type
spelling against solc's label, kept apart because the spelling is the
softer of the two: over the whole corpus not one slot, offset or label
differs, while 699 contracts spell a function type as `function`
(#2196). The record names only the target file, so the target is its
single concrete contract or library, or the one solc's layout names;
`target_contract`, which the next corpus release will carry, wins when
present. Checks a record cannot run are counted with their reason.
@nebasuke
test(sourcify): compare storage types with solc's labels exactly
3c0f784 
The normaliser stripped kind prefixes and `address payable` before comparing,
which hid 1,132,133 mismatching contracts behind #2196's 582. Comparing
exactly puts them all under the #2196 entry, which goes stale once the fix
lands, so the rebase onto it has to drop the entry.
@nebasuke
fix(testing): count storage mismatches once per contract and code
55e376e 
The report counts a bucket's contracts by its failures, one per contract and
code as the diagnostics produce them. The storage check emitted one failure
per mismatching variable, so the #2196 bucket read 699 contracts for 582, and
the rule that small issue entries must list their contracts counted variables.
@nebasuke
style(testing): format the remapped-imports corpus record with prettier
a98c456
@nebasuke
feat(testing): walk solc's storage types table, transient storage too
5f63e9c 
`storage_types` now follows each item's type through both tables at
once (array bases, mapping keys and values, struct members) and
compares label, numberOfBytes and encoding, instead of the item's type
label alone. The two tables key types differently, so they are paired
by walking, not by name.

The same checks run on `transientStorageLayout`. The #2196 entry goes:
@nebasuke
fix(testing): wrap storage byte counts modulo 2^256 like solc
6b0c09b 
solc reports `numberOfBytes` of a `uint256[2**255]` storage gap as 0,
the slot count times 32 wrapped to 256 bits; saturating it flagged
seven corpus contracts that Slang lays out exactly as solc does.
@nebasuke
feat(testing): check the ABI against solc's artifact
f60eb21 
Pairs Slang's ABI JSON with solc's `abi` by type, name and input types,
then compares each pair field by field (down through parameters and
components) and the order of the pairs, since Slang orders the ABI as
solc does. Unpaired entries are `missing` or `extra`. A fixture compiled
with solc 0.8.30 covers every entry type, overloads, a library error and
event, a struct parameter and getters.
@nebasuke
feat(testing): bound an expected failure below a solc version
d3614cf 
`below = "0.8.20"` limits an entry to contracts compiled with an older
solc, so a deliberate divergence from a solc bug fixed in a later
release still fails the gate on contracts from that release on.
@nebasuke
fix(testing): pair an ABI entry with an equal one before any same-keyed
d914762 
A library event can share its signature with the contract's own, and
solc before 0.8.20 lists only the contract's. Pairing the first
same-keyed entry turned the extra event into a field mismatch on the
real one (44 contracts in the census: input names, `indexed`,
`internalType`).
@nebasuke
chore(testing): record the ABI divergences from solc before 0.8.20
bc4e122 
solc 0.8.20 fixed its ABI to list the events a contract emits but does
not define, and reworked event collection so same-named events follow
declaration order. Slang has both at every version: deliberate below
0.8.20, so the gate still catches either from 0.8.20 on. Four missing
errors follow from #2194's import cycles.
@nebasuke
nebasuke force-pushed the sourcify-output-checks branch from ba297dd to bc4e122 Compare October 9, 2026 23:33

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

ci:sourcify Runs Sourcify corpus for 0.8.x in a PR for Slang v2

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@nebasuke