Make CMVP data refresh resilient to transient misses

[ethanolivertroy]

Summary

• preserve previously published certificate details and algorithm payloads when transient refresh/extraction failures occur
• defer uncached incomplete certificates instead of publishing broken records that fail artifact validation
• preserve checked-in active, historical, and modules-in-process lists when top-level NIST list fetches fail or return too few rows
• expose deferred certificate counts in refresh metrics and logs
• add tests for cached fallback, deferred certificate filtering, and top-level list fallback

Validation

• python3 -m py_compile scraper.py test_scraper.py validate_api.py
• python3 test_scraper.py
• python3 validate_api.py --require-current-schema --require-supported-algorithm-source --require-data-quality-pass
• git diff --check
• ALGORITHM_SOURCE=crawl4ai PYTHONUNBUFFERED=1 python3 -u scraper.py && python3 validate_api.py --require-current-schema --require-supported-algorithm-source --require-data-quality-pass (passed locally while NIST top-level fetches failed and cache fallback was used)

  

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit b0f5836. Configure here.}

[cursor]

Blocks algorithm cache fallback

High Severity

can_fallback_to_cached_algorithms rejects fallback when prior run metadata used a non-cacheable algorithm_source (e.g. database or missing), even if cached_algorithm_fields would return algorithms from the checked-in module or detail JSON. A transient extraction failure then defers the certificate, drops it from published module lists, and orphan pruning removes its existing detail file.

 

^{Reviewed by Cursor Bugbot for commit b0f5836. Configure here.}