Skip to content

test(canonical): import jsonschema unconditionally so the schema contract fails closed - #636

Open
Nitjsefnie wants to merge 1 commit into
AgoraDMV:developfrom
Nitjsefnie-OSC:issue-585
Open

test(canonical): import jsonschema unconditionally so the schema contract fails closed#636
Nitjsefnie wants to merge 1 commit into
AgoraDMV:developfrom
Nitjsefnie-OSC:issue-585

Conversation

@Nitjsefnie

Copy link
Copy Markdown
Contributor

Related issue

Closes #585

What does this change?

tests/test_formatters_canonical.py now imports jsonschema unconditionally instead of guarding each test with pytest.importorskip, so a missing dependency reddens the suite instead of quietly skipping the schema contract. jsonschema>=4.21 is already a declared dev dependency — with an inline comment in pyproject.toml naming this exact hazard — so a skip means a broken environment, not an optional feature.

Took the plain-import route rather than adding the module to a watch group. The watch-group mechanism's default-on behaviour is the open question in #610, and routing through it now would pre-empt your own undecided design call. Nothing in CORPUS_GATE_MODULES / CI_SLOW_MODULES / FAST_GATE_MODULES is touched.

Nothing the three named tests assert has changed — stripping the import lines from both revisions leaves the two files byte-identical.

One correction to the issue body, with the diagnosis rather than just the number. It says "all eleven schema tests in that module" use importorskip. In that module there are six, at lines 577, 632, 645, 655, 919 and 1007. But the count of importorskip("jsonschema") across the whole tests/ tree is exactly eleven — those six plus two in test_canonical_tree.py and one each in test_corpus_tree_properties.py, test_pdf_compare.py and test_xml_compare.py. So "eleven" looks like a repo-wide count attributed to one module. Two rival readings do not fit: nine tests match -k schema, and the module has fifty-one tests in total. The finding stands either way; only the attribution moves. The five call sites outside this module are left alone as out of scope.

How to test

tests/test_canonical_schema_contract_gate.py is the new guard. It spawns a child pytest session with jsonschema blocked and asserts the child reddens, using the _run_child_session pattern the corpus-manifest tests already use for skip ceilings.

Red-green, and the mutation:

  • Against the pre-fix module, the gate fails on assert 0 != 0 — all six sites skipping with import of jsonschema halted; None in sys.modules, which also confirms the blocker genuinely bites inside the child rather than passing for the wrong reason.
  • Adding a seventh importorskip cannot reintroduce the fail-open: the module-level import kills collection first. So the gate pins the class, not the six call sites that happened to exist.

I have not ticked "ran the CI gates locally": the suite ran on a runner rather than on my machine. The new gate test is confirmed executed and passing — not merely collected — on 3.12, 3.12.0, 3.13 and 3.14.

Checklist

  • Linked the issue above (Closes #...)
  • Ran the CI gates locally and they pass (see What CI checks) — run on a fork runner, not locally; targeted tests were run locally
  • New or changed behavior has tests
  • For a bug fix: the test fails without the fix, and I ran it both ways to check
  • Disclosed AI assistance below

AI assistance

Generated by Kimi K3 (implementation, testing), Claude Opus 5 (brief, review)

…ract fails closed

The six schema contract tests in tests/test_formatters_canonical.py opened
with pytest.importorskip("jsonschema"), so a missing jsonschema turned them
into silent skips and the run stayed green. jsonschema>=4.21 is a declared
dev dependency, not an optional extra, so a missing import means a broken
environment and must be loud: the module now imports it unconditionally and
a broken environment errors at collection.

The pinning test in tests/test_canonical_schema_contract_gate.py spawns a
child pytest session with jsonschema made unimportable (a sys.modules shim
plus a meta_path blocker, loaded via -p before collection) and asserts the
contract tests redden rather than skip -- the same child-session pattern as
the skip-ceiling end-to-end tests in tests/test_corpus_manifest.py. Verified
to fail on its assertion against the pre-fix tree, and to go red again when
the importorskip call sites are restored.

The issue body says eleven importorskip call sites; the tree carries six
(lines 577, 632, 645, 655, 919, 1007 on develop). The three load-bearing
tests it names are among them, so the finding and fix are unchanged.

Fixes AgoraDMV#585.

Co-Authored-By: Kimi K3 <noreply@kimi.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Canonical schema contract tests can green-by-skip if jsonschema is unavailable

1 participant