Start here: visual guide EN/RU · Русский: первый запуск
Download start.html and open it in your browser for installation, a failing demo,
a passing comparison and next steps. Examples are synthetic and run locally.
Fail CI when tests disappear, become skipped, or fail — even if the overall test count stays the same. Compare two local sets of JUnit XML without a hosted service, account or GitHub token. Useful for test-runner upgrades and sharded CI.
Python 3.11+, MIT. Runtime dependency: defusedxml (PSF license). Version 0.1.2 is distributed through GitHub Releases, not a package registry.
Release v0.1.2 includes a wheel, source
archives, checksums and a verification record. With Python 3.11+, install the
downloaded wheel using python -m pip install suiteledger-0.1.2-py3-none-any.whl.
Runtime dependencies listed below are resolved by pip when needed.
For source development, clone this repository and follow the existing install steps.
python -m pip install .
suiteledger --baseline examples/before.xml --current examples/after.xml
suiteledger --baseline examples/before.xml --current examples/after.xml --format json
suiteledger --baseline examples/before.xml --current examples/before.xmlThe first two exit 1: test_remove disappeared and test_save became skipped.
The last exits 0. Existing failures also fail the comparison; they are never
hidden just because the baseline already had them.
suiteledger --baseline "previous/**/*.xml" --current "current/**/*.xml"
suiteledger --baseline previous --current shard-a --current shard-b --format json
suiteledger --baseline previous --current current --slow-factor 2 --slow-min 0.5Directories recursively select *.xml; quote globs so the tool expands them
consistently. Repeat either input option. Every individual pattern must match.
Repeated references to the same resolved file are read once. Two different
files containing the same test identity are an error, not silently merged.
Identity is the tuple of nested testsuite names, classname and testcase name.
The outer testsuites wrapper name and XML filename are excluded. IDs in JSON
are SHA-256 hashes of that tuple's canonical JSON. A renamed/moved test appears
as missing plus added. Keep test names stable and disambiguate runner parameters.
For an intentional removal, copy its exact id from the report into a reviewed
JSON file and pass --allow-missing reviewed.json:
{"schema_version": 1, "allow_missing": ["COPY_THE_64_CHARACTER_ID_FROM_YOUR_REPORT_HERE"]}The placeholder above is deliberately invalid. Allowlisting suppresses only that disappearance. Stale/unknown IDs fail, forcing cleanup when the baseline changes. Suppressed removals remain in the JSON report.
Timing checks are opt-in. Both ratio and minimum added seconds must be exceeded (minimum delta is inclusive). Every matched passing test must then have a valid duration on both sides. No timing history or statistical significance is inferred.
| Exit | Meaning |
|---|---|
| 0 | No findings under the chosen policy |
| 1 | Missing/newly skipped/current failing test or timing regression |
| 2 | Missing, ambiguous, malformed, unsupported or oversized input |
JSON schema_version: 1 reports findings, status changes, additions, allowed
removals and test/file totals. Reports use stdout and input errors stderr.
Redirect only to a new report path, never over the input XML or allowlist.
Supports testsuite/testsuites, nested suites, namespaces, testcase
failure/error/skipped children, optional classname/time, and common
status=run/passed/success/notrun/disabled/skipped attributes. Conflicting outcomes,
unknown statuses, suite-level errors and retry extensions are rejected.
Testsuite summary counters are not used as test identities or trusted totals;
a declared nonempty leaf suite with no testcases is rejected. The inventory is
the explicit testcase elements. Logs, failure messages and property values are
not included in reports. Names can still reveal private details.
Missing XML files and zero testcases are input errors. Empty individual shards are allowed if they declare no results and the side has other testcases. DTDs, entities and external XML references are forbidden by defusedxml. Bounds: 1,000 files, 100,000 tests, 64 MiB XML per side, 16 MiB per file, nesting depth 64.
This cannot reconstruct a cancelled test that a runner never wrote to XML unless that test was present in the baseline. It does not generate XML, download CI artifacts, merge retries, explain failures or support every JUnit dialect. Missing parameters/shards and environment-dependent test names may produce real inventory differences; select comparable runs before enforcing the result.
python -m unittest discover -s tests -v
python -m pip install build
python -m buildSee roadmap, contributing, security and release instructions.