consume direct --bin=nethtest cannot report a failing blockchain_test against Nethermind.
Two facts combine:
nethtest --blockTest always exits 0 and reports results only as human-readable <name> PASS/FAIL lines on stdout (src/Nethermind/Nethermind.Test.Runner/Program.cs, Run returns 0 unconditionally; BlockchainTestsRunner.cs prints per-test lines, no JSON summary). State tests do emit a JSON array, block tests do not.packages/testing/src/execution_testing/client_clis/clis/nethermind.py decides the block-test outcome from result.returncode alone and never inspects stdout.
Reproduce: take any blockchain_test fixture, change a surviving storage value in its pre so the executed root no longer matches the RLP-committed root, run uv run consume direct --input <dir> --bin=nethtest -m blockchain_test. Nethermind itself prints FAIL (or throws InvalidBlockException), the harness reports the test as passed.
Verified with Nethermind built from current master (dotnet build of Nethermind.Test.Runner).
Found while working on #3600
consume direct --bin=nethtestcannot report a failingblockchain_testagainst Nethermind.Two facts combine:
nethtest --blockTestalways exits 0 and reports results only as human-readable<name> PASS/FAILlines on stdout (src/Nethermind/Nethermind.Test.Runner/Program.cs,Runreturns 0 unconditionally;BlockchainTestsRunner.csprints per-test lines, no JSON summary). State tests do emit a JSON array, block tests do not.packages/testing/src/execution_testing/client_clis/clis/nethermind.pydecides the block-test outcome fromresult.returncodealone and never inspects stdout.Reproduce: take any
blockchain_testfixture, change a surviving storage value in itspreso the executed root no longer matches the RLP-committed root, runuv run consume direct --input <dir> --bin=nethtest -m blockchain_test. Nethermind itself printsFAIL(or throwsInvalidBlockException), the harness reports the test as passed.Verified with Nethermind built from current master (
dotnet buildofNethermind.Test.Runner).Found while working on #3600