Skip to content

Translation checker tests are not run by CI, and the integration test looks for source/ outside the repository #3726

Closed

Description

@rajaysingh

Describe the bug
scripts/check_translations_utest.py and scripts/check_translations_itest.py never run in CI. All the workflows discover tests with unittest discover -s "tests/scripts" (run-tests.yaml:35,37,40, pre-release.yml:38,40,43, release.yml:39,41), and these two files live in scripts/. None of the 685 tests CI currently runs cover check_translations.py, including the ones in check_translations_utest.py that #3701 changed.

They ended up in scripts/ through #1997, following your review comment there ("The name of the file should be: scripts/check_translations_utest.py for unit tests and scripts/check_translations_itest.py for integration tests"). The integration test started as tests/scripts/test_translation_tags.py (86c47eaf) and still finds the repository root with Path(__file__).parent.parent.parent. That was right in tests/scripts/. From scripts/ it points at the parent of the repository, so the test never sees source/.

To Reproduce

  1. On a clean checkout of master, from the repository root:
    pipenv run python -m unittest discover -s scripts -p "check_translations_*test.py"
  2. Output:
    FAIL: test_english_files_exist … AssertionError: 0 not greater than 0 : No English card files found in source directory
    FAIL: test_source_directory_exists … AssertionError: False is not true : Source directory not found: <parent of the repository>/source
    Ran 32 tests in …
    FAILED (failures=2)
    
  3. To see that CI never runs them:
    pipenv run python -m unittest discover -v -s tests/scripts -p "*_*test.py" 2>&1 | grep -c check_translations prints 0 (the run is Ran 685 tests … OK).

Expected behavior
CI runs the checker's unit and integration tests, and the integration test reads the mock files in tests/test_files/source/.

Screenshots

Image

The Error: Source directory not found: …/source line near the top comes from a unit test that passes, since it checks that error message on purpose. The two failures are the FAIL: blocks below it.

Additional context
I'd move both files to tests/scripts/, since that touches the fewest files. They also need a few path fixes (the repository root and the scripts/ import path). I tried it on a copy of master: the CI-style run then has 717 tests (685 + 32) and only the two failures below remain. If you'd rather keep the files in scripts/, the other option is adding that folder to the discovery lines in the three workflows. Tell me which you prefer.

With the root corrected, 2 of the 6 integration tests fail on today's data, and I'd like your call on both.

  1. test_translations_completeness finds 1,047 issues across 31 files (11 missing, 1,036 untranslated, 0 empty). You said in Feature: Add automated validation tests for all language translations #2443 that a test on this doesn't make sense because not all tags are translated. I'd make it fail only on empty tags (0 today) and keep reporting the rest. Or should it stay report-only?

  2. test_no_duplicate_tags_in_english fails on repeated ids in the English files. T00005 appears twice in webapp-cards-3.0-en.yaml and webapp-cards-2.2-en.yaml, both times as "Index". T00090 appears twice in webapp-cards-2.2-en.yaml with different text: "© 2012-2026 OWASP Foundation" (line 473) and "Acknowledgements" (line 476). The es, fr and nl 2.2 files only have the © one. That changes card text, so I'd leave it to you. Can I open a separate issue for these, and skip this one test with a link to it until then, so CI stays green?

The PR would only move the files and fix the paths. No changes to check_translations.py or source/, and no coverage gate.

@sydseter, I'd like to work on this. Could you assign me?

Activity

  1. rajaysingh commented on Nov 8, 2026

    @rajaysingh
    ContributorAuthor

    @sydseter, a correction to my plan above. I went back through your review on #1997, where you asked for the integration test to "read from this dir: tests/test_files" and use mock files. It still reads the real source/, and that's what causes both failures I asked you about. So I'd do it your way:

    • Move both files to tests/scripts/, keeping the _utest/_itest names, so CI runs them.
    • Point the itest at tests/test_files/source/ and have it assert that the checker finds the missing, untranslated and empty tags already in test-cards-1.0-es.yaml.

    I tried this on master. The CI test run goes from 754 to 786 tests, all passing with none skipped, and check_translations.py is at 98%. Nothing changes in check_translations.py, source/ or the workflows. If you'd rather keep the files in scripts/, I can add that folder to the test discovery instead. Could you assign me?

  2. sydseter commented on Nov 8, 2026

    @sydseter
    Collaborator

    @rajaysingh Thank you for the report. I have assigned you.

  3. added 2 commits that reference this issue on Nov 9, 2026
  4. rajaysingh commented on Nov 10, 2026

    @rajaysingh
    ContributorAuthor

    Fixed by #3764 (merged in 25dcad1). Closing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions